Utilizing Edge Features in Graph Neural Networks via Variational Information Maximization
Authors: Pengfei Chen
Weiwen Liu
Chang-Yu Hsieh
Guangyong Chen
Pheng Ann Heng
Venue:  (2020)
Keywords: Graph Neural Network
Edge Feature
Mutual Information

Abstract: Graph Neural Networks (GNNs) broadly follow the scheme that the representation vector of each node is updated recursively using the message from neighbor nodes, where the message of a neighbor is usually pre-processed with a parameterized transform matrix. To make better use of edge features, we propose the Edge Information maximized Graph Neural Network (EIGNN) that maximizes the Mutual Information (MI) between edge features and message passing channels. The MI is reformulated as a differentiable objective via a variational approach. We theoretically show that the newly introduced objective enables the model to preserve edge information, and empirically corroborate the enhanced performance of MI-maximized models across a broad range of learning tasks including regression on molecular graphs and relation prediction in knowledge graphs.

Under review as a conference paper at ICLR 2020
UTILIZING EDGE FEATURES IN GRAPH NEURAL NETWORKS VIA VARIATIONAL INFORMATION MAX- IMIZATION
Anonymous authors Paper under buuble-blind review
ABSTRACT
Graph Neural Networks (GNNs) broadly follow the scheme that the repre- sentation vector of each node is updated recursively using the message from neighbor nodes, where the message of a neighbor is usually pre-processed with parameterized transform matrix. To make better use of edge fea- tures, we propose the Edge Information maximized Graph Neural Network (EIGNN) that maximizes the Mutual Information (MT) between edge features and message passing channels. The MI is reformulated as a differentiable objective via a variational approach. We theoretically show that the newly introduced objective enables the model to preserve edge information, and empirically corroborate the enhanced performance of MI-maximized models across a broad range of learning tasks including regression on molecular graphs and relation prediction in knowledge graphs.
INTRODUCTION
Many real-world datasets naturally come in the form of graphs, such as citation networks (Kipf & Welling 2017), social networks Hamilton et al. (2017), knowledge graphs (Schlichtkrull lt al. 2018) molecular graphs (Scarselli et al. 2009 Duvenaud et al. 2015) etc., all of which consist of a number of nodes and edges equiped with their inherent features. Recently, impressive performance has been achieved in graph learning tasks with various forms of Graph Neural Networks (GNNs) [Zhou al. 2018). Compared to prior works, such as node2vec (Grover & Leskovee 20166, GNNs learn the state of a node by recursively aggregating messages from its neighbors: combining the graph structure with node features. Intuitively, edge features should play important role in graph learning tasks, For example, chemical bonds a molocule have a high impact on chemical properties of molccules, and edge features in knowledge graphs encode important relations between concepts, data, and entities. Our proposed method focuses on improving the usage of edge features in GNNs.
The expressive power of GNNs largely depends on how the message is passed between nodes. widely adopted scheme is multiplying neighbor node states with a parameterized transform matrix before aggregation (Gilmer al.) 2017 Xu et al. 2019). Despite tremendous success of GNNs, existing models do not exhaustively exploit the full potentials of edge features on graphs. For example, many GNNs such as GCN (Kipf & Welling 2017), Cheby (Defferrard
let al.) 2016) and GAT (Veličković et al. 2018) do not even consider categorized edge types. To utilize edge features in multi-relational graphs, RGCN (Schlichtkrull al. 2018 proposes to learn different transform matrix for each edge type, respectively. However, it does not
generalize to edge features in continuous space. MPNN (Gilmer et al.; 2017) introduces edge network that takes edge feature vectors input and outputs transform matrices, which are used to transform states of neighbor nodes. In principle, the MPNN framework can handle complex edge features. Yet. the lack of maximization of MI between edge and message channels implics that the MPNN may give an odge-independent transform matrix.
In this work, aim more efficiently exploit the full potentials of edge features from the perspective of training. We propose the Edge Information maximized Graph Neural Network (FIGNN) that maximizes the Mutual Information (MT) between edge features

---

Under review as a conference paper at ICLR 2020
and the message passing channel which is parameterized the transform matrix in the widely-accepted message passing framework (transformation and aggregation) (Gilmer et al. 2017 Xu et all, 2019 Considering the challenge of computing the MI, we adopt a variational approach to reformulate it as an differentiable objectíve, which can be easily applied as a regularization term. We theoretically show that EIGNN can reduce information loss of edge features. Apart from demonstrating the impressive performance of EIGNN on extensive benchmarks of molecular graphs and knowledge graphs, we also analyze and attribute the enhanced effectiveness of EIGNN to the exploitation of edge features instead of the regularization effects. Notably, attribution analysis on molecular graphs show that EIGNN can caplure domain knowledge without human interference.
Preliminaries Let G (V,8) be graph with node feature vectors 2. E R for node E V and edge feature vectors evw E & for the edge connecting node and w. In GNNs, the state of each node updated recursively using neighbor nodes. Let Ny be the set of neighbor nodes of e and h, R4 be the hidden state of v at 1-th layer, where dj the dimension of the hidden layer. For simplicity of notation, we use single d to denote the
dimension such that h E Rd. We also have h = Tep at the input layer.
2
RELATED WORKS
2.1 RELATIONAL MODELING IN GRAPH NEURAL NETWORKS
Single-relational modeling. Many variants such GCN
Kipf & Welling 2017
GAT Veličković et al. 2018), ChebyNet (Defferrard et al. 2016),
GraphSAGE (Hamilton
et 2017 locus on learning node states. These models can assign weight to neighbors, but they can not handle various edge features. A typical neighborhood aggregation scheme is
((+1)) =a (E WEN. Olwan (r(h ( + w
(1) where a denotes an activation function, a.g can be a normalization constant or a learned attention coefficient (Veličković et al. 2018). States of all neighbors multiplied by the same trainable transform matrix w Sometimes the self-connection also treated in the same way, s.t., w _
Multi-relational modeling. simple stralegy handle multi-relational graphs is as- signing each edge type with separate transform matrix as presented in RGCN (Schlichtkrull eet al. 2018) and adopted by GGNN (Li et al. 2016) and LNet (Liao et al, 2019). RGCN updates node states according to the following scheme
h(+1) = = <E E avw, (mm,,W(0H WD W: w((n(())
(2) where No is the collection of neighboring nodes of u with relation r E R and aew,r is a normalization constant similar as Onw in Eq. T.. Such scheme faces challenge in handling edge features of continuous space. GGNN and LNet do not focus on the improvement. of edge expressibility. GGNN introduces Gated Recurrent Unit (GRU) (Coo al. 2014 and LNet focus on handling multi-scale connections.
Complex-relational modeling. The relation in graph can be quite complex, expressed as a general feature vector e. MPNN (Gilmer et al. 2017) introduces an edge network which takes edge feature vectors input and outputs transtorm matrices. A single edge network is shared in a MPNN model. The forward propagation is formalized as
m(+1 ((+1) = (Ewex, f(eww)h w:! h.9), " h((11) GRU(H m(+
(3)
where f:e + W denotes the edge network. Recently, some research works treat a multi- relational problem as the complex-relations one by introducing continuous edge embedding vector for each edge type, so as to handle increasing number of relations (Nathani et al. 2019).
Although the MPNN architecture allows the usage of arbitrary edge features, this advantage not utilized in practice. MPNN can actually learn an edge-independent transform matrix. A GNN model that efficiently utilizes edge features is yet. to emerge.
2

---

Under review as a conference paper at ICLR 2020
2.2 READOUT FUNCTIONS
After several forward propagations, GNNs yield final states of all nodes, which are suitable for mode/edgy classification or regression. For graph classification or regression lasks, we can apply a readout function (Ying et. al. 2018 Vinyals et. al., 2015 such that
R((h GJ),
(4)
where h is the state of at the last layer, the readout function that outputs a graph-lovd representation y, e-g,, summing up the final node stales, applying hierarchical pooling Ying el. al. 2018 or using the sel model (Vinyals et al. 2015).
OUR METHOD
3.1
THE USAGE OF MUTUAL INFORMATION
In probability theory and information theory, MI is mcasure of mutual dependence between two random variables. Our method proposes to preserve edge information in GNNs, which important in many real-world graph structures such as molecules apart from node (atom) features, attributes of edges (bonds) are equally important for predicting properties of molecules. To this end, we maximize [e;; W) the MI between the edge feature vector e and the message passing channel, i.c., the transform matrix W which is used to transform neighbor node states in the forward propagation. Our method can be casily generalized to directly maximize the MI between edge features and the message itself in methods that do not explicitly have the transform matrix, e.g., the message from node w to node v can be expressed (hy Eow, )w) rather than )eoww) which is shown in Section
An general principle of maximum MI is described for unsupervised learning task by Linsker 1988) and MI inspired objective functions have long been adopted in unsupervised learn-
ing BBidle et al. 1992 Barber & Agakov| 2006 Veličković et al. 2019 Hjelm et al.
2019
scmi-supervised learning (Kranse et al.) 2010) and generative adversarial networks (Chen
ct al. 2016). Specifically, DGI (Velicković et al. 2019) also applics MI to GNNs. DGI
proposes to learn node-wise representations in an unsupervised manner by maximizing the MI between node representations and corresponding high-level summaries of graphs, using adver- sarial learning and negative sampling. The node representations may then be retrieved and used for downstream tasks, such as node classification. DGI can be used to pre-train GNNs, as demonstrated in Hu et al. (2019). Our EIGNN also proposes information maximization but targets a completely different objective and adopts quite different approach.
3.2 A VARIATIONAL APPROACH TO MAXIMIZE MUTUAL INFORMATION
Computing (e; W) itself is intractable in practice needless to say that training a model requires the derivative. Thus, we adopt variational approach (Agakov) 2004) to reformulate I(e; () as a differentiable objective. We show that our objective is an approximated lower bound of T(e; W) and notably, optimizing our objective does lead to maximizing T(e; W). Following MPNN Gilmer et al.) 2017), we use an edge network to parameterize the transform matrix W and relate it to edge features. Therefore, the prior p(Wle) is
(5) where 8(.) is the Dirac delta function. The posterior p(e[W) is intractable, so we define a variational distribution g(e[W), which can be obtained by defining neural network g W - Specifically, ()) substitutes to some distribution (such as Gaussian distribution) with parameter g(W). In this way, f and g are similar to the probabilistic encoder and decoder in the Variational Auto-Encoder (VAE) KKngmma & Welling 2013). Then we can approximate I(e; W) with differentiable objective L1(f.g: C) follows.
p(Wle)= 5(W - f())),
Theorem 1. Let e be the edge feature vector, W be the transform matriz with conditional distribution p(W)e) specified by the probabilistic encoder as shown in Eg. (5) and g(e(W) be the variational distribution specified by the probabilistic decoder g, then we have
T(e:W) 2 H(e) + Be-pie) (Fr.
where Lr log g(e(s(e)) and H()) denotes the entropy.
3

---

Under review as a conference paper at ICLR 2020
Proof. Let DEL(- || denote the KL-divergence, which should be nonnegative, then we have
[(e; W) He)) H(e(W)
(()) + Ew~r(W) E [ooe
(e) + Ew>)(W) [8 (eepp log g(elW) ) + log {(e W]]]
= He)) + Ew> (DL( |I ((elW)) + Be>p(eWW) log [{e
2 (e) + Bw ( wee q(e W)]l
(e) + We
(e) + -e- [log (e f()))]
where the equality (a) follows from Eq.
According to Theorem 1 can maximize the variational lower bound for [e; W). The bound becomes tight when the variational distribution q(elW) approaches the true posterior p(e(W). Moreover, (e) constant because the distribution of edge feature vector is fixed for given graphs, hence we can equivalently maximize (f e).. We choose the widely accepted Gaussian distribution the prior distribution for the probabilistic decoder g.
((clW) =
(7)
Then we have
Lr(J,ge)) = log ((og((()((())) = log logN(ee o21) -
(8)
where À 0 is constant determined by and the dimension of e, taken as tunable
parameter. The following Theorem 2 shows that maximizing the objective in Eq. ( does
lead to the maximization of [(e; W), hence enables the model to preserve edge information.
Theorem 2. Assume the optimal solution of mazimizing Li(f.gte) is f and g", then f* also marimizes [ee; W).
Proof. Note that (e) is constant when the graphs are given. In information theory, have
H(g(f(e))) H(f(e)) < H(e). I(e; W) is upper bounded by H(e),
W)); = [(e; = H((())) - H(S(e))e) H((()) < H(e).
Since S and g" is the optimal solution of maximizing L. (r (Je)e)) presented in Eq. 8 it is not difficult to see that = (f" (e)), Ve E. In this case, the inequalities in Eq. become equalities, i..,
(f = (e) (e).
Therefore, we have [(e; W. = He), i.e., the maximum is attained in this case.
3.3 EDGE INFORMATION MAXIMIZED GRAPH NEURAL NETWORKS
Our EIGNN is derived by implementing our MI objective in GNNs. As a concrete example, the forward propagation of our model follows the formulation in Eq. (3) where the dege network f W is expressed as a multi-layer perceptron (MLP). According to theoretical
analysis presented in Sec. 3.2
introduce another MLP W + the decoder.
For graph regression or classification tasks, the model outputs a prediction for each graph G, which has label ÿ. Without MI maximization, denote the vanilla loss Lo(ūs y; G). Common choice of Lo includes Mcan Square Error (MSE), Mean Absolute Error (MAE) and Cross Entropy (CE). For graph G (V, EIGNN maximizes Li(f.g; and minimizes Lo y; G) using the following loss function
LEIGNN(G) = Lo(g. G) (f
(10)
where E: denotes taking the mean over all edges in G (V,E) and À is the regularization parameter. When EIGNN is trained using mini-batches (ū. y: G) is averaged over all graphs in the batch while (r g;; g)e) is averaged over all edges of all graphs in the batch.
4

---

Under review as a conference paper at ICLR 2020
Similarly, for relational prediction tasks in knowledge graphs, EIGNN directly yields node- level representations hy for each node u E V and edge-level representations for each relationship. The objective function of EIGNN can be derived from the translational scoring function Bordes et al. (2013), which learns embedding such that for a given valid triple few (Mv; Cow,Au, from the valid set S, the condition de... h2 Cove hw holds. Let Lo = Be Es' max[de,,. - dtw. + 7, 0}, where 5" denotes a set of invalid triples and margin
EIGNN can be trained by minimizing the following loss,
LEIGNN Co (f
(11)
EXPERIMENTS
In this section, we first conduct experiments on a large quantum chemistry benchmark QM9, which is challenging for most baselines. Then we evaluate EIGNN on several useful molecule benchmarks and use attribution analysis to show that EIGNN increases the impact of edges and captures domain knowledge without human interference. Finally, adopt our method to large-scale knowledge graphs and evaluate the performance on challenging relation prediction tasks using wide variety of real-world datasets. All experimental results demonstrate clear and substantial improvement EIGNN over the state-of-the the-art methods.
4.1 QUANTUM CHEMISTRY
QM9 (Ramakrishnau el. al. 2014) large benchmark containing 134k molecules with 12 quantum chemistry regression properties, which have been show to be quite challenging for many GNNs (Gimmrr et al 2017). Feature engineering of nodes and edges exactly follows Gilmer et al. 2017 such that molecules preprocessed
graphs according to atom features and bond features. We compare our EIGNN with nine state-of-the-art baselines which can be categorized into three groups according to the ability of handling edge features:
i) GCN, ChebyNet, GAT and GIN (Xu et al.) (201)) which simply use binary edge features to indicate the existence of bond without any other edge features; ii) RGCN, GGNN, LNet and simplified MPNN (sMPNN) which consider bond types (no bond, single, double, triple, or aromatic); fii) MPNN and our EIGNN which use edge feature vectors to indicate both edge types and pairwise distance between atoms.
For a fair comparison, we repeat all experiments times with different random seeds while during cach run, all methods share the same random sced. We randomly choose 10k molecules for validation, 10k molecules for testing, and keep the rest for training. Each target property is normalized to zero mean and unit variance for training, Each model is trained to predict the 12 target properties simultaneously. mean square error (MSE) loss to train the models for at most 300 epochs till convergence, and the performance is measured by mean absolute error (MAE). For LNet and GGNN, implementation of the readout. function follows the original paper. While for all other models,
naively set to for EIGNN. We use
Table 1: Quantumm property regressions for 12 targets and overall performance (top two raws) on QM9. We repeat all experiments 3 times with different. random seeds and report the average performance. Full results with standard deviation are presented in Appendix A e.g.
for MPNN and EIGNN, we have Avg. mMAE 0.0398 + 0.0002 and 0.0357 + 0.0005.
Method
GCN
ChebyNet
GAT
GIN
RGCN
GGNN
LNet
<MPNN
MPNN
EIGNN
Avg. aMAE
0.135
0.121
0.137
0.100
0.102
0.099
0.099
0.089
0.040
0.036
MAE
5,306
4.303
5.470
3.480
3.817
3.861
3.653
3.161
0.693
0.633
mu
0.568
D.518
0.667
0.478
0.006
0.518
0.472
0.472
D.110
0.097
alpha
0.621
0.605
0.623
0.294
HOMO(10
5.451
4.775
5.429
4.183
4.453
4.483
3.889
3.854
2.481
2.230
LUMO(110
8.331
4.796
5.133
2.862
KAP(10 3)
8.201
T.097
8.193
6.096
6.500
6.502
5.813
5.634
3.620
3.276
R2
53.56
54.52
39.68
35.27
6.064
5.646
2PVB(1D-3)
2.533
2.527
2,271
1.744
1.477
1.2912
1.438
1.345
D.679
0.612
UO
2.042
1.984
2.200
1.422
1.059
0.697
1.806
0.791
0.416
0.357
2.290
0.607
0.357
2,042
1.954
2.290
1.422
1.059
0.697
1.796
0.791
D.416
G
2.042
1.964
2.260
1.422
1. 1.059
0.696
1.778
0.791
0.416
0.357
Ov
0.473
0.420
0.479
0.309
0.317
0.316
0.312
0.262
D.134
0.121
5

---

Under review as a conference paper at ICLR 2020
Training Error
Validation Error
EIGNN
EIGNN
MPNN
0.09
MPNN 1e-5
0.08
MPNN L2 1e-5
0.020
MPNN E 1e-4
Aug. MMAE
MPNN + L. 1e-4
MPNN concat
0.07
concat
0.015
0.010
0.05
0.005
0.04
0.000
50
160
150 Epochs
200
250
300
0.03
100
150
250
300
Epochs
Figure 1: Ablation study, Training and validation error on QM9. The shadow area indicates mean + std over 3 runs.
we use the same set 2set (Vinyals et al. 2015 readout, which has been demonstrated to work well in GGilmer et al. 2017].
In Table 1 we list regression results for all methods. We report individual MAE for each target in their original scale, averaged MAE (Avg. MAE) over 12 properties, and averaged normalized MAE (Avg. nMAE; averaged over normalized target properties since different targets have different units and ranges). Our EIGNN achieves the best performance for each metric and each target. Now we are ready to answer the following research questions. i) Are edge features important? Yes. The error has trend of decreasing with increasing edge features. The comparison between sMPNN (using edge types) and MPNN (using edge types and distance) directly verifies the importance of edge features. It is also consistent with the expert knowledge that distances between pairwise atoms are closely related to quantum properties. For example, the smaller the distance between the two atoms, the stronger the bond is, and consequently a higher bond energy is associated with this atom pair. ii) cach target,
Docs the EIGNN work? Yes. EIGNN achieves the best performance outperforming the strong baseline MPNN. Moreover, the advantage of EIGNN over MPNN consistent over 3 runs and the standard deviation on this task is quite small. Detailed results are shown in Table i of Appendix A iii) How does the EIGNN work? Our MI objective is easily implemented on top of vanilla loss function. We have shown that our objective enables preserving of edge information. Fig. [ demonstrates that regularization such as L2 weight decay can increase training error while our objective does not. Morcover, the validation performance verifies that regularization itself does not reduce the validation error. Thus, the offectiveness of EIGNN due to exploiting edge features rather than the regularization effect. We further run ablation study where concatenate edge features to node representation (i.e., MPNN +cnncal Fig. [ in message passing. Concatenation is unable to identify correlations between edges and nodes [Gilmer et al. 2017 and our results show that slightly reduces the mean validation error but increases the variance.
4.2 MORE MOLECULE BENCHMARKS WITH POTENTIAL APPLICATIONS
We further evaluate EIGNN on three molccule benchmarks: Lipophilicity (Wu al. 2018). ESOL (Delancy] 2004) and FreeSolv (Mobley & Guthriq 2014). These datascts contain tewer molecules, and have potential usages in applications such as chemistry, drug discovery, and materials science. For example, the properly lipophilicity is an important feature of drug molecules that affects both membrane permeability and solubility. The dataset Lipophilicity contains 4200 compounds. ESOL provides water solubility data for 1128 compounds. FreeSolv contains hydration free energy of 642 small molecules in water. We conduct graph regression experiments these benchmarks. All datasets are split into training, validation and test according to a proportion of 0.8/0.1 0011 MPNN and our EIGNN share the same architecture with 3 layers of message passing and steps of set2set. We repeat each experiment 3 times
Table 2: Testing RMSE on Lipophilicity, ESOL and FreeSolv.
Dataset
Lipophilicity
ESOL
FreeSolv
Method meanistt
MPNN
EIGNN 0.653-10.02
MPNN
EIGNN 0.776-10.071
MPNN
EIGNN
0.618
0.80520.061
0..08
1.273:10.177
6

---

Under review as a conference paper at ICLR 2020
[8)
Edge sttrisatio 21% Atom atribattor 79%
Toge strrisution: 35% Atom att Wittiistonn 64%
edge atiiiution 27% Alam diibbiirr 50%
zóge attroution 52% Atom sll Brator 60%
Figure 2: Attribution analysis. The color indicates the impact of an edge atom on the output, i.e., the regression result. EIGNN i) increases the edge attribution, ii) reduces the prediction error and iii) can learn domain knowledge without human interference.
with different random seeds. Results of testing root mean square error (RMSE) in Table 2 verify the effectiveness of our method. Our EIGNN outperforms MPNN on each dataset and each run. Detailed results for each run are presented in Appendix B
Attribution analysis. To understand how our EIGNN reduces the regression error, conduct attribution analysis, ie., attributing the prodiction a deep network to its input features, which usually builds up the standard gradient operator (Sundararajan et al. 2017j. For an output y, i.e., the prediction of a GNN, we define its sensitivily lo an edge with features e as S. = =1 where |-] denotes the L1 norm. Similarly, the sensitivity to atom Oy
with features x is S² = J Then S. and Sp are used the metrics of attribution in our ex- periments. As an example, we show in Fig. the attribution for two molecules in Lipophilicity: (a) Netnonel Nefmone1C(=NO Ne2ccc())))
and (b) Dele2nnc3errcccccn 14cccco44 Molecule in (a) has the potential to be used to treat. prevent and/o /oror diagnose cancer Prinz et al. (2019). Compared with MPNN, we can observe an increasing of overall edge attribution under our EIGNN and decreasing of prediction error in both cases. Interestingly, the attribution under EIGNN is similar to the expert knowledge chemists: halogen atoms such as {CI, Br, I} and their bond with the carbon atom greatly effect the lipophilicity of molecule Wilcken et al. 22033), while atoms (O, N} also have high impact on the lipophilicity but usually in a negative way (Augustijns & Brewster 2007). In Fig. (a), the attribution of the halogen bond C-CI and the pair {0, NJ under our FIGNN much higher than the
one under MPNN, which is consistent with the expert knowledge. In Fig. 2 (b). similarly, the attribution of atoms {O, N} and the bond C-O under EIGNN is much higher. More examples are presented in Appendix
4.3 PREDICTING RELATIONS IN KNOWLEDGE GRAPHS
In this subsection, we adopt EIGNN to tackle the problem of relation prediction in knowledge graphs (KGs), which entails predicting whether a given triple valid not. For example, triple (London, capilal of, United Kingdom) should be classified as valid or London should be predicted as the capital of United Kingdom. KGs represent human knowledge as directed graph, and have been widely used in practical applications, such as semantic search, dialogue generation, question answering etc. Recovering missing relations in KGs have been major task for practical usages of KGs. We evaluate our methods on three benchmark datascts, WNISRR (Dettmers et al. 2018 FB15k-23 [Toutanova et al. 2015) and NELL-995 (Xiong
et al. 2017 Without the reversible relation problem (Dettmners et al. 2018). WNI8RR includes II relations scraped from WordNet for 40, 943 synsets. FB15k 237 is a subset of Freebase, and contains 14, 541 entities associated with 237 types of edge. NELL-995 is constructed from the t9 iteration of NELL system, containing 75, 492 entities and 200
types of edge.
7

---

Under review as a conference paper at ICLR 2020
Table 3: Experimental results on WNISRR, FB15K-237 and NELI-995 test sets Hits@N
values are in percentage. The best score is in bold and second best score is underlined.
Dataset
WNISRR
PB1SK-237
NELL-995
HitoN
HitoN %
MRR
M1
WI0
MRR
121
W23
W10
MIR
W1
523
DistMult
0.444
41.2
47
50.4
0.281
19.9
30.1
44.6
0.485
10.1
52.4 4
61
ComplEx
0.449
40.9
46.9
53
0.278
19.4
20.7
45
0.482
39.9
52.8
60.6
Conv
0.456
41.9
47
53.1
0.312
22.5
34.1
49.7
0.491
40.3
53.1
61.3
TransE
44.1
53.2
44.1
50.1
ConvKR
0.265
58.2
44.5
55.8
0.289
19.8
32.4
47.1
0.43
37.0
54.5
T-GCN
0.123
8
13.7
20.7
0.164
10
18.1
30
0.12
8.2
12.6
16.8
KBGAT
0.436
35.8
48.1
57.8
0.431
36.1
45.8
56.9
0.514
42.9
55.3
67.8
EIGNN
0.438
30.7
48.8
58.1
0.451
37.4
48.2
60.5
0.523
43.8
56.1
68.3
A critical issue of applying Eq. (3 to KGs is that high-dimensional embedding vectors are required to distinguish massive amount of entities and relations, leading to a rapid growth in number of parameters in EIGNN. To address this issue, we adopt the architecture of [Nathani it al.) 2019) and learn graph attention based embeddings that target relation prediction on KGs as Tollows,
Mew 5(h e'v ), a'w = =oftmaxx(a (w h ((+1)== -o (E "ww)
(12)
Compared with Eq. [3, where myw = f(eew)hw the above equation absorbs the transform matrix f(eww) into f(h eve, (w)) and reduces the model parameters. In the following experiments, we implement I using MLP as in previous experiments, and maximizes the MI between Crw and Mew by introducing another MLP with À = 0.01. Multi-hoad attention is further introduced to stabilize the learning process and encapsulate more information about neighbors according to (Veličković et al. 2018).. After training EIGNN, CouvKB (Nguyen et al. 2018) is adopted as a regression function for a given triple by analyzing the global embedding properties across each dimension.
In the relation prediction task, the aim is to predict a triple (v, eve, w) with v or w missing, We can generate set of candidate triples for each missing entity by randomly replacing with an arbitrary onc. Scores can be calculated by ConvKB for all triples, and we find the rank of a correct triple by sorting all scores in ascending order. Thus, the performance of relation prediction task can be evaluated by mean reciprocal rank (MRR) and the proportion of correct entities in the top N ranks (Hits&N) for N 1,3, and 10 (Bordes et al. 2013). We compare our EIGNN with seven state-of the-hhherr art baselines focusing on this task: DistMult (Yang al., 2014), ComplEx (Trouillon et al. (0066). ConvE (Dettmers al.
2018 TransF (Bordes et al. 2013). ConvKB (Nguyen et al. 2018 RGCN (Schlichtkrull
ct al. 2018) and KBGAT Nathani et al. 2019 As shown in Table 3 our EIGNN achieves the best performance for each metric on FBI5K-237 and NELL-995, and achieves the best performance on WN1SRR with Hit@3 and 10 metrics. The results of KBGAT are reproduced following the official implementation
[ and the results of other methods can be found in the
previous peer-reviewed publications, i.c. Nathani et al. 2019
CONCLUSIONS
In this work, to make better use of edge features in GNNs, we proposed the edge information maximized graph neural network (EIGNN) that maximizes the mutual information between edge feature vectors and message passing channels. We reformulated the mulual information as a differentiable objective by adopting a variational approach. We have theoretically proved that our proposed objective enables EIGNN to preserve edge information and empirically evaluated EIGNN's performance on a variety of benchmarks incorporating an array of challenging molecular datasets and knowledge graphs. These results clearly manifested a substantial improvement of EIGNN over the prior state-of-tho-ars methods. Apart from demonstrating the impressive performance of EIGNN, we also showed that its effectiveness is due to exploitation of edge features instead of the regularization effect. Notably, attribution analysis on molecular graphs show that EIGNN can capture domain knowledge in an end-to-end fashion.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Barber Felix Agakov. The im algorithm: a variational approach to information maximization. NeurlPS, 2004.
Patrick Augustijus and Marcus Brewster. Solvent systems and their selection in pharma- ceuties and biopharmaceutics, volume 190. Springer, 2007.
David Barber and Felix V Agakov. Kernelized infomax clustering. In Newr/PS, 2006.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NewrIPS, pp. 2787-2795, 2013.
John S Bridle, Anthony JR Heading, and David JC MacKay. Unsupervised classifiers, mutual information and' pha'tohntom targets. In NeurIPS, 1992.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. NewrIPS, 2016.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decodes approaches. arXiv preprint arXin:140.. 1259, 2014.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst, Convolutional neural networks graphs with fast localized spectral filtering, In NeurIPS, 2016.
John S Delaney. Fsol: estimating aqueous solubility directly from molecular structure. Journal of chemical information and computer sciences, 44(3):1000- 1005, 2004.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In AAAI, 2018.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In NewrIPS, 2015.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In TCML, 2017.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 455-864, ACM, 2016.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NewrIPS, pp. 1024-1034, 2017.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. 2019.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Pre-training graph neural networks. arXiv preprint arXiv:1905.1200 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. ar preprint 1rX1::1312.6114, 2013.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Andreas Krause, Pietro Perona, and Ryan G Gomes. Discriminative clustering by regularized information maximization. In NewrIPS, 2010.

---

Under review as a conference paper at ICLR 2020
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. In ICLR, 2016.
Renjie Liao, Zhizhen Zhao, Raquel Urtasun, and Richard Zemel. Lanczosnet: Multi-scale deep graph convolutional networks. In ICLR, 2019.
Ralph Linsker. Selfoorganization in perceptual network. Compuler, (1 -17, 1988.
David L Mobley and Peter Guthrie. Freesolv: a database of experimental and calculated hydration free energies, with input files. Journal of computer-aided molecular design, 28 (7):711-720, 2014.
Christopher Morris, Martin Ritzert, Matthias Fey, William Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI, volume 33, pp. 4602 4609, 2019.
Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. Learning tentioon based embeddings for relation prediction in knowledge graphs. The 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapler of the Association for Compulational Linguistics: Humon Language Technologies, Volume (Short Papers), pp. 327 327-333, 2018.
Bianka Prinz, Jerry M Thomas, Ansgar Brock, Viviana Cremasco, Catherine Anne Sabatos- Peyton, Glenn Dranoff, Scott Chapel, Andrew Lake, Alison Paterson, Rachel W O' O'connor, et al. Antibody molecules to cd73 and uses thereof, January 31 2019. US Patent App. 16/014,744.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilicnfeld. Quantum chemistry structures and properties of 134 kilo molccules. Scientific data, 1: 140022, 2014.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfar- dini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1): 61-80, 2009.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling, Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pp. 393-607. Springer, 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, pp. 3319 -328. JMLR. org, 2017.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1499-1509, 2015.
Th6o Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In ICML, pp. 2071 2080, 2016.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. Tn ICI.R, 2018.
Petar Veličković, William Fedus, William Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR, 2019.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. Xi preprint orxiu: 1.511.0039.... 2015.
Rainer Wilcken, Markus O Zimmermann, Andreas Lange, Andreas C Joerger, and Frank M Boeckler. Principles and applications of halogen bonding in medicinal chemistry and chemical biology. Journal of medicinal chemistry, 56(4))1363-138 1388, 2013.
10

---

Under review as a conference paper at ICLR 2020
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Ancesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513-556) 2018
Wenhan Xiong, Thien Hoang, and William Yang Wang. Deeppath: A reinforcement learning method for knowledge graph reasoning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 564-53 164,-73, 2017.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. ICLR, 2014.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In NewrIPS, pp. 4800-481 2018.
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Graph neural net works: A review of methods and applications. or Xio preprint or 1iv:188 1812. 08494, 2018.
11

---

Under review as a conference paper at ICLR 2020
A FULL RESULTS ON QM9
Table 4: Full results of quantum property regressions for 12 targets and overall performance (nMAE and MAF in top two raws) on QM9. We repeat all experiments 3 times with different random seeds and report the average performance and standard deviation. This is a supplement for Table I in the main text.
Method
GCN
ChebyNet
GAT
GIN
Avg. aMAE Avg. MAE
0.1350#0.0016
0.1206+0.0000
0.1367==0.005
0.1001=000000
5.306310.1990
4.303210. 4814
5. 5.4698.10.204
3.4799110.0402
mu
0.5679+0.007
0.5180+00001 0131
0,5670=000010
0.178320.000 0.011
alpha
0.8811
0.7932:4: 0778
0 19913+0.000
0.6209 0002
HOMO
5.4510=0.0790
1.7750= 2.40
5. 5.4290==011
4. 1.1830=0.0377
LUMO ")
6. 9,4000=0,110 1230
5.6710=::00 2670
6.3310==0,200 2390
4. 1.7960==00055
gap(10
8, 0.2010::0.2150
0.6620
8. 8.19:30:10.201 2910
6. 6.0960 0,0420
R2
50.56311 1.031
41.950.1448228
54.51911 1.992
34. 647 10.2167
ZPVE(10
2.5330=001070
2.5270++035660
2.2710=+0.140
1.74
UO
2.042210.322
1.9842
2.289910. 2000
1.4215 1. 0.0857
2.0422:0.3281
1.984244..0033
2.2899+0.2000 2000
1.4215=000857
H
2.0422=0.328
1.9842+-0.200 ,0355
0.289990.2220 2000
1.4215=0.0855
2.0423+0..0 3.2281
1.9842# 2036
2 2.2899+0
1.4213=0.006 08557
Cy
0.0230
0.4199:0.0040
10.0114
0.309310.0035
Method
RGON
GGNN
LNet
sMPNN
Avg. aMAE
0.102110.0016
0.0992:1..0011
0.0992:10.0011
0.0888-10.0014
Avg. MAE
3.817520.0000
3.6608:t0.0722
3 3.6527=0.34177
3.161020.0697
mn
0.5056110.008
0.5179-10.0077
0.4717-10.006
0.4718-10.0080
alpha
10.014
0 0092
0. 0.6225-00.0m 0508
0.5278+0.0100
HOMO
1.453010.1290
4.4830.10.00 0.0060
3.8889110.1119
3.654010.0010
LUMO(1O
5. 0.
5.1530:10.0880
4. 4.1935 2205
4.5490-0.0880
gap(10 9)
6.5000=0.180 1390
6.6020::0. 1400
5.8132++0.64M0
5.6340+00.050
R2
10.10220.111 7128
39.685 8212
35.275+3.0533
33. 18948.990000 6566
ZPVE(10
1.4770=0.0000
1.292010000340
1.4376+0.0769
1.3450
UO
10589=0.0285
0.6969+
1.808 2533
0.7914#0.04M
U
1.0589+0.00 0231
0.6966 0418
1.7555+0.2199
0.791400.044
H
1.058910.0220
0.697410.004 0.0408
1.7964110.244 2428
0.791410.0446
G
1.0589-1.0023
0.6961 1:0.021
1 1.778010.2445 2458
0.791410.0446
Cv
0.3170:0.0152
0.3145::.00255
0.31244400303
0.2625+0.0040
Method
MPNN
EIGNN
Avg. aMAE
0.039810.0002
0.0357+0.0000
Avg. MAE
0.02910
0.6331.10.022
mu
1.1095=000011
0.0974+0.0000
alpha
0.331810.0022 0.0026
1993910.0003
HOMO(10 3
2.4810=0.0 0200
2 0310
LUMO(10
0..00=0000777
2.5930=
gap(10-2)
3. 0.620010.0188
3.2750: 0.0520
R2
6.0637 2.1111 2511
5.6464 0.3098
ZPVE(IO 3)
0.6790=1000100
0.6120
UO
0.4164=0.0222
0.3574+0.001 01000
U
0.4 0.4164410.022
0.3575:1 0.000
H
0.416440.00 0225
0.3574+0.0100
G
0. 016-1=:000225
0.3575:+0.010
Cy
0.10:3
0.120810.0022
We present full results of quantum property regressions on QM9 with average performance and standard deviation in Table 4 The (10-3) in the parentheses indicates that the values in the corresponding raw of the table should multiply by 10-3, This is simply for clear presentation of the values. We also present. a detailed descriptions on the target properties in Table 5 for your reference.
In our results, we directly report MAE instead of Error Ratio (MMA))(Chemine
Accu- racy) (Gilmer et al., 2017)), because it is common to report MAE in terms of chemical unit. This practice has been widely adopted not only in computational chemistry but also in the machine learning community working on molecular graphs (Wu et al.) 2018 Morris et al. 2019). Still, we add Table 6 which contains the Error Ratio for MPNN and our EIGNN. In this comparison, we follow Gilmer et al. 2017) and train models separately to predict each target. Our EIGNN consistently outperlorms MPNN.
12

---

Under review as a conference paper at ICLR 2020
Table Regression targets on QM9.
Target property
Description
Unit
mu
Dipole momentt
D
alpha
Isotropic polarizability
dg
HOMO LUMO
Highest oncupied molecular orbital energy
Fj
Lowest: unoceupied molecular orbital energy
gAp R2 ZPVE
Gap hetween HOMO and LUMO
Eb
Electrouic spatial extent
Zero point vibrational energy
Ex
Ub
Iuternal energy OK
E
Internal energy at 298. 15K
Enthalpy 298. 15K
Fj
Free energy at 298. 15K
E
Heat capavity at 298 15K
ccl
m maxX
Table 6: Error Ratio ((MAE)
(MAA))/Chemical (Chemical Accuracy) Gilmer et al. 2017 on QM9. Note that the energy values of {UO, U, Н, G} are per molecule rather than per atom. Following Gilmer et al. 2017). models are separately trained on each target.
Target
MPNN
EIGNN
Improvement (%)
alpha
2.64
2.44
7.57
HOMO
1.39
10.2
LUMO
1.31
1.28
2.13
gap
2.20
1.97
10.4
R2
1.00
0.72
28.4
ZPVE
5.31
4.43
16.5
U
56.6
23.6
58.3
77.4
35.2
54.5
46.6
34.4
26.1
Cv
1.69
1.79
-6.22
B FULL RESULTS ON LIPOPHILICITY, ESOL AND FREESOLV
In this section, we present experimental results on Lipophilicity, ESOL and FreeSolv with detailed results for each run. The results Table 7 verify that our EIGNN consistently outperforms MPNN.
C MORE EXAMPLES OF ATTRIBUTION
In this scction, we present more examples of attribution analysis on Lipophilicity. As supplement for Fig. 2 in the main text, the observation here is similar. Compared with MPNN, we can observe an increasing of overall edge altribution under our EIGNN and decreasing of prediction error in both cases. Notably, EIGNN is able to capture the ex- pert knowledge. (a) The molecule CN(C@@H[CC)C( (
F) The attribution of {O, N} and the halogen atom is higher under EIGNN. (b) The molecule is (CCC))CCCCCCEE)]CCCC NCCCN(CC2)) c3ccc) (eee(C()))))) Our EIGNN successfully captures the im- portance of two critical halogen atoms Cl and several atoms N. (c) The molecule is COclcco(cc))((= 2)CCCCC2 O. The attribution of two atoms O at the top is
Table 7: Testing RMSE on Lipophilicity, ESOL and FreeSolv. This a supplement for Table 2 in the main text.
Dataset
Lipophilicity
ESOL
FreeSolv
Seed
MPNN
EIGNN
MPNN
EIGNN
MPNN
EIGNN
0.718
0.678
0.770
0.718
1.396
1.109
0.606
0.664
0.750
0.733
1.299
1.265
0.620
0.619
0.894
0.876
1.499
1.443
meaniistd
0.618::0.042
0.653:t0.025
0.805-20.061
0,776:t0.071
1.398=000081
1.273::0.137
13

---

Under review as a conference paper at ICLR 2020
much higher under EIGNN. (d) The molecule is Cclo2NC CC C=) The attribution of atoms O is much higher under EIGNN.
(a)
Model: MPNN, Error: 84, Edge attribution: 11%, Atom attribution. 89%
Mode: BIGNN, Ertor: 0.07,
Edge attribution: 26% Atom actribution: 74%
(b)
Model: MPMN, Error:
Mode: EIGNN, Error: 67%
Edge
Edge
(c)
Edge 12%, attribution:
Edge attribution: actribution: 68%
(d)
Model: MPNN, Errar: 18,
Model: EIGNN, Enror: 0,046, Edge attribution: 31%, Atom actribution: 69%
Edge attribution: 21%, Atom attribution: 79%
Figure 3: More examples of attribution analysis. The color indicates the impact of an edge/aton the output, i.e, the regression result. EIGNN i) increases the edge attribu- tion, ii) reduces the prediction error and iii) can learn domain knowledge without human interference.
14


Reviews:
[{"review_id": "rke365VAKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a mutual information term into the training objective of message passing graph neural networks.  The additional term favors the preservation on information in a mapping from an input edge feature vector e_{i,j} to a weight matrix f(e_{i,j}) used in computing messages across the edge from node i to node j.  A variational lower bound on the mutual information is used in training.  Impressive empirical results are given for chemical property prediction and relation prediction in knowledge graphs.  I have no real complaints other than I might recommend citing the original work on infomax:\n\n\"Self Organization in a Perceptual Network\", Ralph Linsker, 1988.\n\nPostscript:  I have been swayed by the complaints of reviewer 1 and reduced my score to weak reject.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJg2znvb5H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an auxiliary loss based on mutual information for graph neural network. Such loss is to maximize the mutual information between edge representation and corresponding edge feature in GNN ‘message passing’ function. By assuming a gaussian distribution of edge feature given edge representation, the training can be done efficiently with tractable density. Experiments on molecule regression and knowledge graph completion show better performance than MPNN. \n\nOverall the paper is written in a clear way which is easy to follow. The idea of using mutual information as some kind of regularization is also interesting. However, there are some concerns I have with the paper:\n\nRegarding formulation\n\n1. The derivation up to Eq(8) looks fine to me, where the assumptions are reasonable. However from Eq(8) one can see this is reduced to an ‘auto-encoder’ type of regularization, where one can have a trivial solution for reconstruction -- the identity network, when the hidden dimension is larger than input dimension. And in this paper, dimension of W should always be larger than the dimension of e (for example, in molecules e should be low dimension vector with the bond type, distance, etc., while W should have dimension that matches the node embeddings).\n\nI think the original loss (i.e., the supervised MSE, cross entropy etc) would help a bit with such degenerated case, but it is possible that the learned f(e) contains both identity mapping (or equivalent) and the representation that contributes to original loss.  \n\n2. Actually I’m also not sure if I get the motivation here. If one needs to do this regularization for edges, why don’t we consider this auxiliary loss for node embeddings as well? As in molecules, atoms have more interesting features than bonds, which should account more if the mutual information loss is needed. \n\nRegarding experiment\n\n1. In Figure 1, the training loss of EIGNN is better than MPNN. This is a bit counterintuitive to me, as I think the auxiliary is a kind of regularization -- which might help with generalization but not necessarily the training loss. \n\n2. The original paper of MPNN reports the relative MAE. Is it possible to report the results using the same metric as previous paper? It would make the comparison more consistent -- though showing the improvement in current way is not too bad. \n\n3. I think one simple ablation study would to concat the edge feature directly inside the ‘message passing’ procedure, or have some ‘residual’ type of connection for edge features. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyg-qWu3tB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a new kind of graph neural network that can use continuous edge features. Specifically, a variational lower bound is proposed for mutual information and integrated into the GNN model so that MI between edge features and the message passing channel is maximized. Experiments on molecule graph datasets and knowledge graph datasets show the effectiveness of the proposed method compared to the state-of-the-art GNN models.\n\nUtilizing continuous edge features in GNN is an important and difficult task. The authors proposed an elegant solution. The paper is well written and extensive experiments on large scale datasets are compared with 9 competitors that are all outperformed by EIGNN on nearly all the cases.\n\nTo conclude, I think the paper will be a good addition to the conference.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BylQWd5str", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new graph neural network to utilize the edge features. In particular, it proposes the Edge Information maximized Graph Neural Network (EIGNN) that maximizes the Mutual Information (MI) between edge features and message passing channels. The MI is reformulated as a differentiable objective via a variational approach. The experimental results have some improvement over existing methods.  Overall, the idea is novel and well presented. \n\nPros:\n1. The idea of utilizing edge features looks novel.\n2. The writing is clear.\n3. Extensive experiments are done to verify the performance of the proposed method.\n\nCons:\n1. The theoretical analysis is just a regular routine. \n2. How does the hyper-parameter affect the performance? In other words, how does the component of the edge features affect the model performance?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed an auxiliary loss based on mutual information for graph neural network. Such loss is to maximize the mutual information between edge representation and corresponding edge feature in GNN ‘message passing’ function. GNN with edge features have already been proposed in the literature. Furthermore,  the reviewers think the paper needs to improve further in terms of explain more clearly the motivation and rationale behind the method. 
Decision: Reject