Composition-based Multi-Relational Graph Convolutional Networks
Authors: Shikhar Vashishth
Soumya Sanyal
Vikram Nitin
Partha Talukdar
Venue:  (2020)
Keywords: Graph Convolutional Networks
Multi-relational Graphs
Knowledge Graph Embeddings
Link Prediction

Abstract: Graph Convolutional Networks (GCNs) have recently been shown to be quite successful in modeling graph-structured data. However, the primary focus has been on handling simple undirected graphs. Multi-relational graphs are a more general and prevalent form of graphs where each edge has a label and direction associated with it. Most of the existing approaches to handle such graphs suffer from over-parameterization and are restricted to learning representations of nodes only. In this paper, we propose CompGCN, a novel Graph Convolutional framework which jointly embeds both nodes and relations in a relational graph. CompGCN leverages a variety of entity-relation composition operations from Knowledge Graph Embedding techniques and scales with the number of relations. It also generalizes several of the existing multi-relational GCN methods. We evaluate our proposed method on multiple tasks such as node classification, link prediction, and graph classification, and achieve demonstrably superior results. We make the source code of CompGCN available to foster reproducible research.

Published as a conference paper at ICLR 2020
COMPOSITION-BASED MULTI-RELATIONAL GRAPH CONVOLUTIONAL NETWORKS
Shikhar Vashishth* 1.2 Soumya Sanyal" Vikram Nitin Partha Talukdar'
Indian Institute of Science, *Carnegie Mellon University, $Columbia University evashishècs emu, edu, (shikhar, vi vikram, nit ine co Bon.kharrsoumneeennnr soumyasanyal, reiisc. ac.
ABSTRACT
Graph Convolutional Networks (GCNs) have recently been shown be quite suc- cessful in modeling graph-structured data. However, the primary focus has been on handling simple undirected graphs. Multi-relational graphs are more general and prevalent form of graphs where each edge has label and direction associ- ated with Most of the existing approaches handle such graphs suffer from over- parameterization and are restricted to learning representations of nodes only. this paper, we propose COMPGCN, novel Graph Convolutional framework which jointly embeds both nodes and relations in relational graph. COMPGCN leverages a variety of entity-relation composition operations from Knowledge Graph Embedding techniques and scales with the number of relations. It also gen- eralizes several of the existing multi-relational GCN methods. We evaluate our proposed method multiple tasks such as node classification, link prediction, and graph classification, and achieve demonstrably superior results. We make the source code of COMPGCN available foster reproducible research.
INTRODUCTION
Graphs are one of the most expressive data-structures which have been used to model a variety of problems. Traditional neural network architectures like Convolutional Neural Networks (Krizhevsky et al. 2012) and Recurrent Neural Networks (Hochreiter & Schmidhuber 1997) are constrained to handle only Euclidean data. Recently, Graph Convolutional Networks (GCNs) (Bruna et al.) 2013 Defferrard et al., 2016) have been proposed to address this shortcoming, and have been success- fully applied to several domains such social networks (Hamilton et al. 2017). knowledge graphs (Schlichtkrull et al. 2017 Shang et al. 2019), natural language processing (Marcheggiani & Titov)
2017 Vashishth al.) 2018a 6 2019). drug discovery (Ramsundar et al.
2019), crystal property
prediction (Sanyal et al. 2018), and natural sciences (Fout et al., 2017).
However, most of the existing research on GCNs (Kipf Welling, 2016 Hamilton al. 2017 Veličković et al., (0188) have focused on learning representations of nodes in simple undirected graphs. more general and pervasive class of graphs are multi-relational graph:" notable ex- ample of such graphs is knowledge graphs. Most of the existing GCN based approaches for han- dling relational graphs (Marcheggiani & Titov] 2017, Schlichtkrull et al., (017)) suffer from over- paraneterization and are limited to learning only node representations. Hence, such methods not directly applicable for tasks such as link prediction which require relation embedding vectors. Initial attempts learning representations for relations in graphs (Monti et al.) 2018 Beck et al., 2018) have shown some performance gains on tasks like node classification and neural machine translation.
There has been extensive research on embedding Knowledge Graphs (KG) Nickel et al. 2016 Wang al. (017) where representations of both nodes and relations are jointly learned. These meth- ods are restricted to learning embeddings using link prediction objective. Even though GCNs can
"Equally Contributed
Work done while at IISc, Bangalore
In this paper, multi-relational graphs refer to graphs with edges that have labels and directions.

---

Published as a conference paper at ICLR 2020
London
London
United Kingdom
United Kingdom
CEDO
CBcB
Born-in_inv
Born-n
of
Borcwin
Wo
Ciizen-a of
Christopher Nolan
Christopher Nolan
W1
Wa
S
Directed--by. Jow
Directed-byy
The Dark Knight
The Dark Knight
Relational Graph with Embeddings
CompGCN Update
Figure 1: Overview of COMPGCN. Given node and relation embeddings, COMPGCN performs composition operation <(-) over each edge in the neighborhood of central node (e.g. Christopher Nolan above). The composed embeddings are then convolved with specific filters Wo and Wr r for original and inverse relations respectively. We omit self-loop in the diagram for clarity. The message from all the neighbors are then aggregated to get an updated embedding the central node. Also, the relation embeddings transformed using separate weight matrix. Please refer to Section for details.
learn from task-specific objectives such as classification, their application has been largely restricted to non-relational graph setting. Thus, there a need for a framework which can utilize KG embed- ding techniques for learning task-specific node and relation embeddings. In this paper, propose COMPGCN, a novel GCN framework for multi-elatiinna graphs which systematically leverages entity-relation composition operations from knowledge graph embedding techniques. COMPGCN addresses the shortcomings of previously proposed GCN models by jointly learning vector repre- sentations for both nodes and relations in the graph. An overview of COMPGCN presented in Figure The contributions of our work can be summarized as follows:
We propose COMPGCN, novel framework for incorporating multi-relational information in Graph Convolutional Networks which leverages variety of composition operations from knowl- edge graph embedding techniques to jointly embed both nodes and relations a graph.
2. We demonstrate that COMPGCN framework generalizes several existing multi-relational GCN methods (Proposition 4.1 and also scales with the increase in number of relations in the graph (Section 6.3).
3. Through extensive experiments on tasks such as node classification, link prediction, and graph classification, we demonstrate the effectiveness our proposed method.
The source code of COMPGCN and datasets used in the paper have been made available at http: 7/github.
RELATED WORK
Graph Convolutional Networks: GCNs generalize Convolutional Neural Networks (CNNs) to non-Euelidean data. GCNs were first introduced by Bruna et al. (2013) and later made scalable through efficient localized filters in the spectral domain (Defferrard et al.) 2016). A first-order ap- proximation GCNs using Chebyshev polynomials has been proposed by Kipf & Welling (016)) Recently, several of its extensions have also been formulated (Hamilton et al. [0017; Veličković et al, 2018 Xu et al. 2019 Vashishth et al. 2019 Yadav et al., 2019). Most of the existing GCN methods follow Message Passing Neural Networks (MPNN) framework (Gilmer et al., 2017) for node aggregation. Our proposed method can be seen as an instantiation of the MPNN framework. However, is specialized for relational graphs.
2

---

Published as a conference paper at ICLR 2020
GCNs for Multi-Relational Graph: An extension of GCNs for relational graphs is proposed by Marcheggiami & Titov (2017). However, they only consider direction-specitie filters and ignore rela- tions due to over- parameterization, SSchlichtkrull et al. (2017) address this shortcoming by proposing basis and block-diagonal decomposition of relation specific filters. Weighted Graph Convolutional Network (Shang et al.) 2019) utilizes learnable relational specific scalar weights during GCN ag- gregation. While these methods show performance gains on node classification and link prediction, they are limited embedding only the nodes of the graph. Contemporary to our work, Ye et al. (201)) have also proposed an extension of GCNs for embedding both nodes and relations in multi- relational graphs. However, our proposed method more generic framework which can leverage any KG composition operator. We compare against their method in Section 6.1
Knowledge Graph Embedding: Knowledge graph (KG) embedding a widely studied field (Nickel et al., 2016) Wang et al., (017) with application in tasks like link prediction and ques- tion answering (Bordes et al.J 2014). Most of KG embedding approaches define a score function and train node and relation embeddings such that valid triples are assigned higher score than the invalid ones. Based on the type of score function, KG embedding method are classified as transla- tional (Bordes et al., 2013) Wang et al. 2014b), semantic matching based ( (Yang et al., 20144] Nickel jet al. 2016) and neural network based (Socher et al.] [013] Dettmers et 20188] Vashishth et al. 2019) In our work, evaluate the performance of COMPGCN link prediction with methods of all three types.
BACKGROUND
In this section, give brief overview of Graph Convolutional Networks (GCNs) for undirected graphs and its extension to directed relational graphs.
GCN on Undirected Graphs: Given a graph 9 EV, where V denotes the set of vertices, E is the set of edges, and X V represents do-dimensional input features of each node. The node representation obtained from a single GCN layer is defined as: H f fÂx'). Here, - Đ + (A+ D)D & is the normalized adjacency matrix with added self-connections and D is defined as D = (A (A+ +o++++++ D).- The model parameter is denoted by W R and f is some activation function. The GCN representation H encodes the immediate neighborhood of cach node in the graph. For capturing multi-hop dependencies in the graph, several GCN layers can be stacked, one on the top of another as follows: H =f f(AH*W' kk where ie denotes the number of layers,
W E RRy xdx is layer-specific parameter and H ===X X.
GCN on Multi-Relational Graphs:
For multi-relational graph g RV, where R denotes the set of relations, and each edge u. represents that the relation E R exist from node u to v. The GCN formulation as devised by MMacchegianii & Titov (017)) based on the assumption that information in directed edge flows along both directions. Hence, for each edge (u, v) E E, an inverse edge (v, u, r 11) is included in G. The representations obtained after layers of directed GCN is given by
H #+1 J(AH'W;))
(1)
Here, W denotes the relation specific parameters of the model. However, the above formulation leads to over-parameterizzaionn with an increase in the number of relations and hence, MMrcheggiani & Titov (2017) use direction-specific weight matrices. [Schlichtkrull et al. (2017) address over- parameterization by proposing basis and block-diagonal decomposition W
4 COMPGCN DETAILS
In this section, we provide detailed description of our proposed method, COMPGCN, The overall
architecture is shown in Figure [ as defined in Section 3 where Z E R| do denotes the initial relation features. Our model is motivated by the first-order approximation of GCNs using Chebyshev polynomials (Kipf & Welling 2016). Following Marcheggiani & Titov (2017), we also allow the information in a directed edge to flow along both directions. Hence, extend & and R with corresponding inverse edges and
We represent a multi-relational graph by G = (V, R, E, X, Z)
3

---

Published as a conference paper at ICLR 2020
Methods
Node Eimbeddings
Relation Finbeddings
Number
Directions
Relations
Parameters
KCp] Welling 2016
O(KdF) O(Kd2)
DC
Maraniggaaa Titow 2017
Weighted-GGC GCrce Shang
2019
O(Kd' KJR() О(BКa BK(R)
Relational-GCN Schlcchkrull et 2017
COMPGCN (Proposed Method)
O(Ke + Bd + B(R]
Comparison of our proposed method, COMPGCN with other Graph Convolutional meth- ods. Here, K denotes the number of layers in the model, d is the embedding dimension, B rep- resents the number bases and [R] indicates the total number of relations in the graph. Overall, COMPGCN is most comprehensive and more parameter efficient than methods which encode relation and direction information.
Table 1:
relations, i.e.,
u, u(, (u, v,r) E EEJU((U u, T) V)}, = EE
and R' R U Rine
(T), where Rinv = (r-lrr E R) denotes the inverse relations and T
indicates the self loop.
4.1 RELATION BASED COMPOSITION
Unlike most of the existing methods which embed only nodes in the graph, COMPGCN learns d-dimensional representation h, E IRd Vr E R along with node embeddings h. E R Ve E V. Representing relations as vectors alleviates the problem of over-parameetrrizaion while applying GCNs on relational graphs. Further, it allows COMPGCN to exploit any available relation fea- tures (Z) as initial representations. To incorporate relation embeddings into the GCN formulation, we leverage the entity-relation composition operations used in Knowledge Graph embedding ap- proaches (Bordes et al. 2013) Nickel et al. 2016), which are of the form
e, o(e,, e,).
Here, é Rdd Rdd TRd is a composition operator, 8, r, and denote subject, relation and object in the knowledge graph and e(.) E Rdd denotes their corresponding embeddings. In this paper, we restrict ourselves to non-parameterized operations like subtraction (Bordes et al.) 2013), multiplica- tion (Yang et al. 2014) and circula--coreeation (Nickel et al.) 2016). However, COMPGCN can be extended to parameterized operations like Neural Tensor Networks (NTN) (Socher et al., 2013) and ConvE (Dettmers et al.) 2018). We defer their analysis future work.
As we show in Section 6 the choice of composition operation is important deciding the quality of the learned embeddings. Hence, superior composition operations for Knowledge Graphs developed in future can be adopted to improve COMPGCN's performance further.
4.2 COMPGCN UPDATE EQUATION
The GCN update equation (Eq. defined Section 3 can be re-written as
h =
Σ W,h
(u.r)EN(())
where ((v) is set of immediate neighbors of for its outgoing edges. Since this formulation suffers from over-parameten pveeeearamttriization,, in COMPGCN we perform composition (d) of a neighboring node with respect its relation while being linear (O(R)) d)) in the number of feature dimensions. Moreover, for treating original, inverse, and self edges differently, define separate filters for each of them. The update equation of COMPGCN is given as:
defined above. This allows model to be relation aware
h
Σ
(2)
(M.T)EN())
4

---

dir(r) defined in Published as a conference paper at ICLR 2020
Methods
W A(r)
Kipf-GCN (Kipf & Welling 2016)
W
h
Relational- (Schlichtkrull et al.
2017)
W
h.
Directed-GCN Marcheggiani & Titov 2017
W dir())
h
Weighted-GGN (hang et al.
2019)
W
a
Table 2: Reduction of COMPGCN to several existing Graph Convolutional methods. Here, of is relation specific scalar, W denotes separate weight for each relation, and W Equation Please refer to Proposition 4.1 for more details.
where Ta. Z, denotes initial features for node u and relation respectively, h, denotes the updated representation of node v, and WA(r) E RRd xdo a relation-type specific parameter. In COMPGCN, use direction specific weights, i.c. A(r) dir(r), given as:
Wo: E R
Wdie())
Wr. r E Rine
Ws, = T (self-loop)
Further, in COMPGCN, after the node embedding update defined in Eq. 2 the relation embeddings are also transformed as follows:
h Zy.
(4)
where W Rdd Xdo is learnable transformation matrix which projects all the relations to the same embedding space as nodes and allows them to be utilized in the next COMPGCN layer. In Table D we present contrast between COMPGCN and other existing methods in terms of their features and parameter complexity.
Scaling with Increasing Number of Relations To ensure that COMPGCN scales with the increas- ing number of relations, we use variant of the basis formulations proposed in (Schlichtkrull et al. (017). Instead of independently defining embedding for each relation, they expressed linear combination of set of basis vectors. Formally, let {11 u2...., Rg} be a set of learnable basis vectors. Then, initial relation representation is given as:
a Ub.
Here, Or,, E IR is relation and basis specific learnable scalar weight.
On Comparison with Relational-GCN Note that this is different from the basis formulation in Schlichtkrull et al. (2017) where a separate set of basis matrices is defined for each GCN layer. In contrast, COMPGCN uses embedding vectors instead of matrices, and defines basis vectors only for the first layer. The later layers share the relations through transformations according to Equation This makes our model more parameter efficient than Relational-
We can extend the formulation of Equation 2 to the case where we have k-stacked COMPGCN layers. Let h denote the representation of node obtained after k layers which is defined as
h
k++
Σ
W Xtr O ht))
(5)
(.,))EN())
Similarly, let h denote the representation of relation r after k layers. Then,
h 1++1
W h rel
Here, h and h are the initial node (æ,) and relation (z,)) features respectively.
Proposition 4.1. COMPGCN generalizes the following Graph Convolutional based methods: Kipf- GCN (Kipf Welling 2016), Relational GCN (Schlichtkrull et al. 2017), Directed GCN (Marcheg. giani & Titov 2017), and Weighted GCN (Shang al. 2019)
5

---

Published as a conference paper at ICLR 2020
Proof. For Kipf-GCN, this can be trivially obtained by making weights x(r) and composition function (Φ) relation agnostic in Equation 5 i.e., Wa(r) W and @hh,; hr) = h Similar reductions can obtained for other methods as shown in Table
5 EXPERIMENTAL SETUP
5.1 EVALUATION TASKS
In our experiments, we evaluate COMPGCN on the below-mentioned tasks.
Link Prediction is the task of inferring missing facts based on the known facts in Knowledge Graphs. In our experiments, we utilize FB15k-237 (Toutanova & Chen 2015) and WNI8RR (Dettmers et al. 20188) datasets for evaluation. Following Bordes et al. (013), we use filtered setting for evaluation and report Mcan Reciprocal Rank (MRR), Mean Rank (MR) and Hits @ N.
Node Classification is the task predicting the labels nodes in a graph based on node features and their connections. Similar to Schlichtkrull et (2017), evaluate COMPGCN MUTAG (Node) and AM (Ristoski & Paulheim 2016) datasets.
Graph Classification, where, given a set of graphs and their corresponding labels, the goal is to learn representation for each graph which fed to a classifier for prediction. We evaluate on bioinformatics dataset: MUTAG (Graph) and PTC Yanardaag & Vishwanathan 2015).
A summary statisties of the datasets used is provided in Appendix A.2
5.2 BASELINES
Across all tasks, compare against the following GCN methods for relational graphs: (1) Relational- (R-GCN) (Schlichtkrull et al. 2017) which uses relation-specifi weight matri- ces that are defined as a linear combinations of of basis matrices. (2) Directed-GCN (D-GCN) (Marcheggiani & Titov 2017) has separate weight matrices for incoming edges, outgoing edges, and self-loops. It also has relation-specifie biases. (3) Weighted-GCN (W-GCN) (Shang et al.) 2019) assigns a learnable scalar weight to each relation and multiplies an incoming "message" by this weight. Apart from this, we also compare with several cask-specific baselines mentioned below.
Link prediction:
For evaluating COMPGCN, we compare against several non-neural and neural baselines: TransE [Bordes al. ),013), DistMult (Yang et al.) 2014), ComplEx (Trouillon et al, R-GCN (Schlichtkrull al. | 2017). KBGAN (Cai & Wang 2018), ConvE (Dettmers al.
2016
2018
ConvKB (Nguyen et al. 2018). SACN (Shang et al. 2019). HypER (Balažević et al. 2019)
RotatE (Sun et al. 2019), Convk (liang et al. 2019), and VR-GCN (Ye et al. 2019).
Node and Graph Classification: For node classification, following Schlichtkrul et al. 201 7 ). we compare with Feat (Paulheim & Fümkranz 2012). WL (Shervashidze et adl.|201] and RDF2Vec (Ristoski & Paulheim, 2016). Finally, for graph classification, we evaluate against PACHYSAN Niepert al., (2066), Deep Graph CNN (DGCNN) (Zhang al., 2018), and Graph Isomorphism Network (GIN) (Xu et al. 2019).
6 RESULTS
In this section, we attempt to answer the following questions.
Q1. How does COMPGCN perform on link prediction compared to existing methods? 6.1
Q2. What is the effect of using different GCN encoders and choice of the compositional opera- tor COMPGCN on link prediction performance? (6.1)
Q3. Does COMPGCN scale with the number relations the graph? 6..3
Q4. How does COMPGCN perform node and graph classification tasks? 66.4

---

Published as a conference paper at ICLR 2020
FB15k-237
WNISRR
MRR
MR
H6 10
H@3
H601
MRR
MR
H@ 10
H03
H@1
TransE
Bordes al.
3013
.294
357
.465
.226
3384
.501
DistMult
Yang et al.
2014
.241
254
.419
.263
.155
.43
5110
.49
.44
.39
ComplE'x Trouition el 2016
.247
.428
.255
5261
.51
.66
R-GCN Schicctkrull
.248
.417
.151
KBGAN
Lat & Wang
201S
.278
.458
.214
472
ConvE
Detmers et al.
2018
.325
.501
.356
.237
.43
4187
.52
A4
Conv INguyen
2018
3324
-524
.417
SACN Shang ot at. 2019
.35
.54
.39
.26
.47
.54
.48
.43
HypER (Balazevie ef al.
2019
.341
250
.520
.376
.252
.465
5798
.522
.477
.436
RotatE Sun et al. 2019
.338
177
.533
.375
.241
.476
3340
.571
.492
ConvR Uhung
2019
.350
.528
.385
.261
.475
.537
.489
.443
VR-GCN Ye al. 2019
.248
.432
.272
.159
COMPGCN (Proposed Method)
.355
197
.535
.390
.264
.479
3533
.546
.494
.443
Table Link prediction performance COMPGCN and several recent models on FB15k-237 and WN18RR datasets. The results of all the baseline methods are taken directly from the previous papers (-" indicates missing values). We find that COMPGCN outperforms all the existing methods on 4 out of 5 metrics on FB15k-237 and 3 out of 5 metrics WNISRR. Please refer Section 6. for more details.
6.1 PERFORMANCE COMPARISON ON LINK PREDICTION
In this section, we evaluate the performance COMPGCN and the baseline methods listed Sec- tion 5.2 on link prediction task. The results on FB 15k-237 and WNI8RR datasets are presented in Table The scores of baseline methods are taken directly from the previous papers (Sun al, 2019 al & Wang, 2018 [Saang et al.) 2019) Balažević et al. [019 Jiang et al.) [019 Ye et al. 2019). However, for ConvKB, we generate the results using the corrected evaluation code (Sun) al. 2019). Overall, we find that COMPGCN outperforms all the existing methods in 4 out of 5 metrics on FB15k-237 and out of 5 metrics on WN18RR dataset. We note that the best perform- ing baseline RotatE uses rotation operation in complex domain. The same operation can be utilized in a complex variant our proposed method to improve its performance further. We defer this as future work.
6.2 COMPARISON OF DIFFERENT GCN ENCODERS ON LINK PREDICTION PERFORMANCE
Next, we evaluate the effect of using different GCN methods sentative score function (shown in Figure 2 from each category: TransE (translational), DistMult (semantic-basede and ConvE (neural network-based). In results, X M (Y) denotes that method M is used for obtaining entity embeddings (and relation embeddings in the case of COMPGCN) with X as the score function as depicted in Figure Y denotes the composition operator in the case of COMPGCN. We evaluate COMPGCN on three non-parametric composition operators inspired from TransE (Bordes al. 2013), DistMult (Yang al. 2014). and HolE NNikel al. 2016) defined as
encoder along with repre-
Subtraction (Sub): ole,, e,) =e,- e,. Multiplication (Mult): d(e., er) =e, er- Circular-correlationn (Corr): o(e., =e, *
The overall results are summarized in Table 4 Similar to Schlichtkrull et al. (2017), we find that utilizing Graph Convolutional based method encoder gives substantial improvement in perfor- mance for most types of score functions. We observe that although all the baseline GCN methods lead to some degradation with TransE score function, such behavior is observed for COMPGCN. On average, COMPGCN obtains around 6%, 4% and 3% relative increase in MRR with TransE, DistMult, and ConvE objective respectively compared to the best performing baseline. The superior performance of COMPGCN can be attributed to the fact that learns both entity and relation embed- dings jointly thus providing more expressive power learned representations. Overall, we find that COMPGCN with ConvE (highlighted using is the best performing method for link prediction!
We further analyze the best performing method for different relation categories in Appendix A.1

---

Published as a conference paper at ICLR 2020
Scoring Function (=X)
TransE
DistMult
ConvE
Methods
MRR
MR
H@ 10
MRR
MR
H@
MRR
MR
H 10
0.294
357
0.465
0.241
354
0.419
0.325
244
0.501
X D-GCN
0.299
351
0.469
0.321
225
0.497
0.344
200
0.524
X + R-GCN
0.281
325
0.443
0.324
230
0.499
0.342
197
0.524
X W-GCN
0.267
1520
0.444
0.324
229
0.504
0,344
201
0.525
X COMPGCN (Sub)
0.335
194
0.514
0,336
231
0,513
0.352
199
0.530
+ COMPGCN (Mult)
0,337
233
0,315
0.338
200)
0,518
0.353
216
0.532
COMPGCN (Corr)
0.336
214
0.518
0.335
227
0.514
0.355
197
0.535
COMPGCN (B 50)
0.330
203
0.502
0.333
210
0.512
0.350
193
0.530
Performance on link prediction task evaluated on FB15k-231 dataset. method M used for obtaining entity (and relation) embeddings with X as the scoring function. In the case of COMPGCN, denotes the composition operator used. B indicates the number of relational basis vectors used, Overall, we find that COMPGCN outperforms all the existing methods across different scoring functions. ConvE: + COMPGCN (Corr) gives the best performance across all settings (highlighted using D.. Please refer Section for more details.
Table 4:
(Y) denotes that
All
Entity ngs
Number of Basis
99.4
CompGcN
TransE
Relation Emboodinngs
50
98.8
Knowledge Graph
R-GCN/ Wich
Entity Embeddlings
TransE
25
98.0
Relation Embeddings
97.2
Encoder (M)
Score Function (X)
25
50
75 100
Relative
Figure 2:
Knowledge Graph link prediction with COMPGCN and other methods. COMPGCN gener- ates both entity and relation embedding opposed just entity embeddings for other models, For more details, please refer Section
Performance of COMPGCN with dif- ferent number of relation basis vectors on link pre- diction task. We report the relative change in MRR on FB 15k-237 dataset. Overall, COMPGCN gives comparablc performance even with limited parame- ters. Refer Section 6.3 for details.
Figure 3:
Effect of composition Operator: The results link prediction with different composition oper- ators are presented in Table 4 We find that with DistMult score function, multiplication operator (Mult) gives the best performance while with ConvE, circular-correlation surpasses all other op- erators. Overall, we observe that more complex operators like circular-correlation outperform or perform comparably to simpler operators such as subtraction.
6.3 SCALABILITY OF COMPGCN
In this section, we analyze the scalability of COMPGCN with varying numbers of relations and basis vectors. For analysis with changing number of relations, we create multiple subsets of FB15k-237 dataset by retaining triples corresponding to top-m most frequent relations, where = [10. 25, 50, 237). For all the experiments, we use our best performing model (ConvE COMPGCN (Corr)).
Effect of Varying Relation Basis Vectors: Here, we analyze the performance of COMPGCN on changing the number of relation basis vectors (B) as defined in Section 4 The results are summa- rized in 3igure We find that our model performance improves with the increasing number of basis vectors. We note that with B 100, the performance of the model becomes comparable to the case where all relations have their individual embeddings. In Table 4 we report the results for the best performing model across all score function with set to 50. We note that the parameter-efficien variant also gives comparable performance and outperforms the haselines in all settings.
8

---

Published as a conference paper at ICLR 2020
0.345
All
97.2
Number of Relations
Number of Relations
0.331
100
100
99.4
0.316
50
100.0
0000
0
25
100.0
0.316
10
0.209
Comp@ON (B+5) R-GON
0.230
0.275
0.300
0,325
0.500
Relative MRR
Mean Reciprocal Rank
Figure 5: Performance of COMPGCN with differ- ent number of relations on link prediction task. We report the relative change in MRR pruned ver- sions of FB15k-237 dataset. Overall, COMPGCN gives comparable performance even with limited pa- rameters. Refer to Section 6.3 for details.
Figure 4: Comparison of COMPGCN (B = 5) with R-GCN for pruned versions of Fb15k-237 dataset containing different number of relations. COMPGCN with relation basis vectors outperforms R-GCN across all setups. For more details, please refer to Sec- tion 6.3
MUTAG (Node)
AM
MUTAG (Graph)
PTC
Feat" WL'
77.9
66.7
PACHYSAN! DGCNN
92.6 + 4.2
60.0 + 4.8
80.9
87.4
85.8
58.6
RDF2Vec
67.2
88.3
GINT
89.4 1 4.7
64,6 7.0
R-GCN"
73.2
89.3
R-GCN
82.3 9.2
67.8 13.2
SynGCN
74.8 + 5.5
86.2 + 1,9
SynGCN
79.3 10.3
11.5
WGCN
77.9 3.2
90.2 + 0.9
WGCN
78.9 = 12.0
67.3 + 12.0
COMPGCN
85.3 + 1.2
90.6 + 0.2
COMPGCN
89.0 1 11.1
71.6 1 12.0
Table Performance comparison on node classification (Left) and graph classification (Right) tasks. * and indicate that results are directly taken From[Schlichtkrie et al. (2017) and Xu et al, (2019) respectively. Overall, find that COMPGCN either outperforms or performs comparably compared the existing methods. Please refer to Section 6.4 for more details.
Effect of Number of Relations: Next, we report the relative performance of COMPGCN using relation basis vectors (B = 5) against COMPGCN, which utilizes a separate vector for each relation the dataset. The results are presented in Figure Overall, we find that across all different numbers of relations, COMPGCN, with a limited basis, gives comparable performance to the full model. The results show that a parameter-efficien variant of COMPGCN scales with the increasing number of relations.
Comparison with R-GCN: Here, we perform a comparison a parameter-efficient variant of COMPGCN (B- 5) against R-GCN on different number of relations. The results are depicted in Figure We observe that COMPGCN with limited parameters consistently outperforms R-GCN across all settings. Thus, COMPGCN is parameter-efficien and more effective encoding multi- relational graphs than R-GCN.
6.4 EVALUATION ON NODE AND GRAPH CLASSIFICATION
In this section, we evaluate COMPGCN on node and graph classification tasks on datasets as de- scribed in Section 5.1 The experimental results are presented in Table 5 For node classification task, we report accuracy on test split provided by Ristoski et al. (2016), whereas for graph classi- fication, following Yanardag & Vishwanathan (2015) and Xu et al. (2019), we report the average and standard deviation of validation accuracies across the 10 folds cross- validation. Overall, we find that COMPGCN outperforms all the baseline methods node classification and gives compara- ble performance on graph classification task. This demonstrates the effectiveness of incorporating relations using COMPGCN over the existing GCN based models. On node classification, compared

---

Published as a conference paper at ICLR 2020
to the best performing baseline, obtain average improvement of 3% across both datasets while on graph classification, we obtain an improvement 3% on PTC dataset.
7 CONCLUSION
In this paper, we proposed COMPGCN, novel Graph Convolutional based framework for multi- relational graphs which leverages a variety of composition operators from Knowledge Graph em- bedding techniques to jointly embed nodes and relations in graph. Our method generalizes several existing multi-relational GCN methods. Moreover, our method alleviates the problem of over-paren pareemmermmteiiiioonnn by sharing relation embeddings across layers and using basis decomposition. Through extensive experiments on knowledge graph link prediction, node classification, and graph classification tasks, showed the effectiveness of COMPGCN over existing GCN based methods and demonstrated its scalability with increasing number relations.
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their constructive comments. This work is supported in part by the Ministry of Human Resource Development (Govermment of India) and Google PhD Fellowship.
REFERENCES
Ivana Balažević, Carl Allen, and Timothy M Hospedales. Hypernetwork knowledge graph embed- dings. In International Conference on Artificial Neural Networks, 2019.
Daniel Beck, Gholamreza Haffari, and Trevor Cohn. Graph-to-sequence learning using gated graph neural networks. In Iryna Gurevych and Yusuke Miyao (eds.), 201. The 56th Annual Meet- ing of the Association for Computational Linguistics, pp. 273-283. Association for Computational Linguistics (ACL), 2018. ISBN 9781948059 178408887322.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Ok- sana Yakhnenko. Translating embeddings for modeling multi-rclational data. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances
Neural Information Processing Systems 26, pp. 2787-2795.
Curran Associates, Inc., 2013. URL http://paper:: n nips cc/paper/ 5071 wrrnssaaaainnne wwwpralltiiineebemdiinmmee embeddings for mode ing muwti- -eaiti--ellatee atllationn onaal data.
pdf
Antoine Bordes, Sumit Chopra, and Jason Weston. Question answering with subgraph embeddings. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 615-620, Doha, Qatar, October 2014. Association for Computational I Linguistics. doi: 10.31 1/..1155//DD114110000 1067. URL wtt.:htttt///tttmm
web. org/ aaaebbooggaanthoooygeee DD 1067
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. CoRR, abs/1312.6203, 2013. URL lttt:://axxiv.
abs/133 /111/.11111 6203
Liwei Cai and William Yang Wang. KBGAN: Adversarial learning for knowledge graph embed- dings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1470-1480, 2018. URL
https:
Victor de Boer, Jan Wielemaker, Judith van Gent, Michiel Hildebrand, Antoine Isaac, Jacco van Ossenbruggen, and Guus Schreiber. Supporting linked data production for cultural heritage in- stitutes: The amsterdam museum case study. In Proceedings of the 9th International Conference The Semantic Web: Research and Applications, ESWC' 12, pp. 733-747, Berlin, Heidelberg, doi: 1007/977 3- 6424-02284-8 -_52.. URL http: /ttp://dx..o doi wtt.//...100/911-033303 320-2-66222222222
2012. Springer- Verlag. ISBN 978-3-64223 30283-1.
10

---

Published as a conference paper at ICLR 2020
Asim Kumar Debnath, Rosa L. Lopez de Compadre, Gargi Debnath, Alan J. Shusterman, and Cor- win Hansch. Structure-activitn -activiiraaiiity relationship of mutagenic aromatic and heteroaromatic nitro com- pounds. correlation with molecular orbital energies and hydrophobicity. Journal of Medicinal Chemistry, 3462)786-797,, 1991. doi: 10. (0.1011/m00106a4444 URI httpt:ttt///......... http://ooooooooooo (021/3m001060222
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering, CoRR, abs/1606.09375, 2016. URL http:
09375
Tim Dettmers, Minervini Pasquale, Stenetorp Pontus, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedingy of the 32th AAAI Conference on Artificial Intelli- gence, pp. 181 18818, February 2018. URL htlp:::::: //axivv org/ abtp//0s//110... 01446
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop Representation Learning Graphs and Manifolds, 2019.
Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur. Protein interface prediction using graph convolutional networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6530-6539, Curran Associates, Inc., 2017. URL http: / http::://ppapp..........es . httpppppprr.i.ppp.cppppppppeee
7231
using www.ragggaaao-ooootutttoooooaaaa networks.
pdf
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 1263-1272. JMLR.org, 2017. URL http: 3305512,
7/al. acm. wttp:///taaa......iirieeeeeeeee.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth Interna tional Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL http:::::t:cceeeeeeeeehee
ngs m. wwtpps/eesggleeeeeeepppp 10a.:ttt0a.... html
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. NIPS, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 98): 1780, November 1997. ISSN 0899-7667. doi: 19w../9....99...8... URL http::/dx
doi. doi.org/10. 1 /2/ nww../eec....mmmm 1997.9997 1735)
Xiaotian Jiang, Quan Wang, and Bin Wang. Adaptive convolution for multi-relational learn- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, 2019. URL https:
7 7/www. aclweb. wwt.o/////ao.. wtt..//aaacoooooy/l111--1-11
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 12 2014.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- 02907
works. CoRR, abs///609002900 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097- 1105. Curran Associates, Inc., 2012. URL http:://pppers ni nips cips.ccc/aperr wttoolltiiooooo -ee--nnurll-nnn --e--nuurrletrrrrreeee networks.
4824-1 www.iiaangetoooooooeooee wlwssifiasiia..ea cat con-onn w with
pdf
Diego Marcheggiani and Ivan Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1506-1515. 1515. Association for Computational Linguistics, 2017. URL
http:// //ac: web. wtgg/ http://anhholooy//1111155
11

---

Published as a conference paper at ICLR 2020
George A. Miller. Wordnet: A lexical database for english. Commun. ACM. 38(1))339 41, Novem- ber 1995. ISSN 0001-0782. doi: 10.1145/19777.219944. URL http::/tp/// acm. org/
10.1144/219771 219748
Federico Monti, Oleksandr Shchur, Aleksandar Bojchevski, Or Litany, Stephan Günnemann, and Michael M. Bronstein. Dual-primal graph convolutional networks. CoRR, 0oss/8806000770 2018.
URL
org/ abs/1806.... 00770
Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoe Nguyen, and Dinh Phung. A novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Short Papers), pp. 327-333. Association for Computational Linguistics, 2018. doi: 10.18653//1/N18-20000
URL
org/ hogg/anthology/Nii 2053
M. Nickel, K. Murphy, V. Tresp, and Gabrilovich. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11-33, Jan 2016. ISSN 0018-9219. doi: 10.1109/PROC.221552433333
Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. Holographic embeddings of knowledge graphs. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'6, pp. 1955-1961. AAAI Press, 2016. URL http:://dd acm org/ warg/citann citatiin.tno... cfm.iiiion.ccccce fmdid= 3016100.. 3016172
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural net- works for graphs. In Proceedings of the 33rd International Conference on International Con- ference on Machine Learning Volume 48, ICML' 16. pp. 2014 2023. JMLR.org, 2016. URL ffm?: d=3045330 3045603)
http://dd acm. wwwg//itaaaeooooooo
Heiko Paulheim and Johannes Fümkranz. Unsupervised generation of data mining features from linked open data. Proceedings of the 2Nd International Conference on Web Intelligence, Mining and Semantics, WIMS 12, pp. 31:1-31:12, New York, NY, USA, 2012. ACM. ISBN 978-1- 4503-0915-8. doi: 10. (0.1145/225412 2254416.
URL http: acm. wwt.aac......./00////1
2254129 2254168
Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O'Reilly Media, 2019. https: wwww. amazon. com/ Deep Learning Scienceennssssss caniss-Mirrr wicrocscoo///pp/1111033333
Petar Ristoski and Heiko Paulheim. Rdf2vec: Rdf graph embeddings for data mining. In Interna- tional Semantic Web Conference, pp. 498-514. Springer, 2016.
Petar Ristoski, Gerben Klaas Dirk de Vries, and Heiko Paulheim. A collection of benchmark datasets for systematic evaluations of machine learning on the semantic web. In Paul Groth, Elena Sim- perl, Alasdair Gray, Marta Sabou, Markus Krötzsch, Freddy Lecue, Fabian Flöck, and Yolanda Gil (eds.), The Semantic Web ISWC 2016, pp. 186-194, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46547-..
Soumya Sanyal, Janakiraman Balachandran, Naganand Yadati, Abhishek Kumar, Padmini Ra- jagopalan, Suchismita Sanyal, and Partha Talukdar. Mt-egenn: Integrating crystal graph con- volutional neural network with multitask learning for material property prediction. arXiv preprint arXiv:1811.05660, 2018
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. arXiv preprint 1rX1v:1703.06103, 2017.
Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. End-to-end structure- aware convolutional networks for knowledge base completion, 2019.
Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M. Borgwardt. Weisfeiler-lehman graph kernels. J. Mach. Learn. Res., 12:2539-2561, Novem- ber 2011. ISSN 1532-4435. URL attttp::// ///dl. acm. wwt./ooottiiiooooo e 1fm21d=195300 2078187
12

---

Published as a conference paper at ICLR 2020
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26. pp. 926-934. Curran Associates, Inc., 2013. URL http: / httpp/aapaaaaaa/aa n ps. cc pcppe/ edge 5028 10-8--easonni with -etn-rrrrra---st----------ee---e new.wr---wwoeee-eeeom for-kwww htw.ror--eewwwwedeeeeeeeeeeeeeee
letion.
pdf
A. Srinivasan, R. D. King, S. H. Muggleton, and M. J. E. Sterbberg. The predictive toxicology evaluation challenge. In Proceedings of the 15th International Joint Conference on Artifical In- telligence Volume 1, IJCAI997 pp. 4-9, San Francisco, CA, USA, 1997. Morgan Kaufmann Publishers Inc. ISBN 1-555860-480-4, URL htLhhtp::/// //cl:://cl... acm. org/ citat ion. cfm?id= 1624162 1624163
Zhiqing Sun, Zhi-Hong Deng, Jian- Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representationss 2019. URL UtLhttpss:/
ptttps://operrevicce
met/foormm? wtt.orrmmm......eeeeeeee...
Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha Talukdar, and Yiming Yang. A Re- evaluation of Knowledge Graph Completion Methods. arXiv e-prints, art. arXiv:1911.03903 Nov 2019.
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionaliiy, pp. 57-66, 2015.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, fric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In Proceedings of the 33rd International Con- ference on International Conference Machine Learning Volume 48, ICML 16, pp. 2071- 2080. JMLR.org. 2016. URL http:/ /dl. acm. wwtg/citaaion.. cf 3045609
Shikhar Vashishth, Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. Dating docu- ments using graph convolution networks. In Proceedings of the 56th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pp. 1605 1615. Association for Computational Linguistics, 2018a. URLhttp: http:/acllwa aclweb wwtp//attooooy/./e.e.e1
Shikhar Vashishth, Rishabh Joshi, Sai Suman Prayaga, Chiranjib Bhattacharyya, and Partha Taluk- dar. RESIDE: Improving distantly-supervisedd neural relation extraction using side information. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1257-1266, Brussels, Belgium, October- 2018b. Association for Computational Linguistics. URL
org/ anthology/
Shikhar Vashishth, Manik Bhandari, Prateck Yadav, Piyush Rai, Chiranjib Bhattacharyya, and Partha Talukdar. Incorporating syntactic and semantic information in word embeddings using graph convolutional networks. In Proceedings the 57th Conference of the Association for Com- putational Linguistics, pp. 300033080333 3318, Florence, Italy, July 2019. Association for Computational Linguistics. URL http:: www. aawwebebb..... http///ncchoogg//11-1113333
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, Nilesh Agrawal, and Partha Talukdar. InteractE: Improving Convolution- Knowledge Graph Embeddings by Increasing Feature Interactions. arXiv e-prints, art. arXiv: 1et1111110000 1.00119, Nov 2019.
Shikhar Vashishth, Prateek Yadav, Manik Bhandari, and Partha Talukdar. Confidence-based graph convolutional networks for semi-supervised learning. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research. pp. 1792-1801. PMLR, 16-18 Apr 2019. URL http:// proceedings mlr. wwwpppessssseeeee
httpaiinssmm...pesseeeeeeee /tress//////aashsstth...... html
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018. URL RtRl.httss://opeeee ew. fttttt/fformmmiaarr.u..
accepted as poster.
13

---

Published as a conference paper at ICLR 2020
Q. Wang. Z. Mao, B. Wang, and L. Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12):2724-274) Dec 2017. ISSN 1041-4347. doi: 10.1 10.11099TKDEE.21...754444.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by trans- lating on hyperplanes, 2014a. URL https: //w:://wwwww.. aaal. wtt.a///.../////////aa/aaaooo/// http://ccsinnex...pe wwwpp/hpppp..///
AAA11//papesvview/55555
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by
translating on hyperplanes. In Proceedings of the Twenty-Eighth AAAI Conference Artificial
Intelligence. 14, pp. 1112-1119. AAAI Press, 2014b. URL http:://dlaccm.. org/
aintiion.cm??2 0ooi:2ddd22987223 2893873 2894046
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural
networks? In International Conference on Learning Representations, 2019. URL https:
wttp://wwww..eetttooooo//rearaaa A5Km
Prateek Yadav, Madhav Nimishakavi, Naganand Yadati, Shikhar Vashishth, Arun Rajkumar, and Partha Talukdar. Lovasz convolutional networks. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learn- ing Research, pp. 1978-1987. PMLR, 16-18 Apr 2019. URL http::/pprcceediinie mlr.
Exess/v89/yen yaap://ydalllaaaa
Pinar Yanardag and S. V.N. Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '15, pp. 1365-1374, New York, NY, USA, 2015. ACM. 978-1453-330 doi: 10.1145/2783258.
2783417. URL /ttp: /do::::/do.... acm. org 1141/ 1 2783417
Bishan Yang. Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Deng. Embedding entities and relations for learning and inference in knowledge bases. CoRR, abs/1411.6577 2014. URL
http:://rxx wrt.oorg//ooo....
Rui Ye, Xin Li, Yujie Fang, Hongyu Zang, and Mingzhong Wang. A vectorized relational graph con- volutional network for multi-relational network alignment. In Proceedings of the Twenty-Eighth International Joins Conference on Artificial Intelligence, IJCAI-19, pp. 4135 4141. International Joint Conferences Artificial Intelligence Organization, 7 2019. doi: (0.24963/jccai222115574 URL httlh:tts://// //to: ttg.//oooooooo//////////oo/ooo/ 2tt///000.9911/aaa. 20t.2.0195577
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. AAAI, pp. 4438 4445, 2018.
14

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 EVALUATION RELATION CATEGORY
In this section, we investigate the performance of COMPGCN on link prediction different relation categories on FB15k-237 dataset. Following Wang al. (2114)); Sun et al. (2019), based on the average number of tails per head and heads per tail, we divide the relations into four categories: one-to-one, one-to-many, many-to-one and We observe that using GCN based encoders for obtaining entity and relation embeddings helps to improve performance all types of relations. In the case of one-to-one relations, COMPGCN gives an average improvement of around 10% on MRR compared to the best performing baseline (ConvE the corresponding improvements
The results are summarized in Table 6
W-GCN). For one-to-many, many-to-one, and 10.5%, 7.5%, and 4%. These results show that COMPGCN is effective at handling both simple and complex relations.
ConvE
ConvE W-GCN
ConvE + COMPGCN (Corr)
MRR
MR
H@ 10
MRR
MR
H@ 10
MRR
MR
H@10
Head Pred
1-1
0.193
459
0.385
0.422
238
0.547
0.457
150
0.604
1-N
0.068
922
0.116
0.093
612
0.187
0.112
604
0.190
0.438
0.638
0.454
0.647
0.471
99
0.656
N-N
0,246
189
0.436
0,261
169
0.459
0.275
179
0,474
Taill Pred
1-1
0.177
402
0.391
0.406
319
0.531
0.453
193
0.589
I-N
0,756
66
0.867
0.771
43
0.875
0.779
34
0,885
N-1 0.049
783
0.09
0.068
747
0.139
0.076
792
0.151
N-N
0.369
119
0.587
0.385
107
0.607
0.395
102
0.616
Table 6: Results on link prediction by relation category on FB15k-237 dataset. Following Wang et al. 2014a). the relations are divided into four categories: one-to-one (1-1), one-to-many (1-N), many- to-one (N-1), and many-to-many (N-N). We find that COMPGCN helps to improve performance on all types of relations compared to existing methods. Please refer to Section A. for more details.
A.2 DATASET DETAILS
In this section, we provide the details of the different datasets used in the experiments. For link prediction, we use the following two datasets:
FB15k-237 (Toutanova & Chen 2015) a pruned version of FB15k (Bordes al. 2013 dataset with inverse relations removed to prevent direct inference.
WNISRR (Dettmers et al. 2018), similar FB 5K-237, is subset from WN18 Bordes al,
2013) dataset which is derived from WordNet MMiller 1995).
For node classification, similar to Schlichtkrul et al. 2017). we evaluate on the following two datasets:
MUTAG (Node) a dataset from DL-Learner toolki It contains relationship between complex molecules and the task is to identify whether molecule is carcinogenic not.
AM dataset contains relationship between different artifacts in Amsterdam Museum (de Boer et al., 2012). The goal is to predict the category of given artifact based on its links and other attributes.
Finally, for graph classification, similar to Xu et al. 2019). we evaluate on the following datasets:
MUTAG (Graph) Debnath et al. [199]) is a bioinformatics dataset 188 mutagenic aromatic and nitro compounds. The graphs need to be categorized into two classes based their mutagenic effect on bacterium.
hhtp://www.dl-emmmr.org
15

---

Published as a conference paper at ICLR 2020
Link Prediction
Node Classification
Graph Classification
FB1
WNI8RR
MUTAG (Node)
AM
MUTAG (Graph)
PTC
Graphs
1
188
344
Entities
14,541
40,943
23,644
1,666,764
17.9 (Avg)
25.5 (Avg)
Edges
310,116 237
93,003
74.227
5,988.321
39.6 (Avg)
29.5 (Avg)
Relations
11
23
133
4
Classes
2
11
2
2
Table 7: The details of the datasets used for node classification, link prediction, and graph classifi- cation tasks. Please refer to Section 5.1 for more details.
PTC [Srinivasan al. (1997) a dataset consisting of 344 chemical compounds which indicate carcinogenicity of male and female rats. The task is to label the graphs based their carcino- genicity on rodents.
summary statistics of all the datasets used is presented Table
A.3 HYPERPARAMETERS
Here, we present the implementation details for each task used for evaluation in the paper. For all the tasks, we used COMPGCN build on PyTorch geometric framework (Fey & Lenssen 2019).
Link Prediction:
For evaluation, 200-dimensional embeddings for node and relation embeddings are used. For selecting the best model we perform hyperparameter search using the validation data over the values listed Table 8 For training link prediction models, we use the standard binary cross entropy loss with label smoothing Dettmers et al. (2018).
Following (Schiichtrrll et al. (2077, we use 10% training data as validation for selecting the best model for both the datasets. We restrict the number of hidden units to 32. We use cross-entropy loss for training our model.
Node Classification:
Graph Classification: Similar to Yanardag & Vishwanathan (2015); Xu al. (2019), report the mean and standard deviation validation accuracies across the 10 folds cross- volidavaii..ion.. Cross- entropy loss is used for training the entire model. For obtaining the graph-level representation, use simple averaging of embedding of all nodes as the readout function, i.e.,
hg
Σ hv,
Chu
[V]
where h. is the learned node representation for node in the graph.
For all the experiments, training is done using Adam optimizer (Kingma & Ba 2014 and Xavier initialization (Glorot & Bengio 2010 is used for initializing parameters.
Hyperparameter
Values
Number of GCN Layer (K)
2, 3]
Learning rate Batch size Dropout
[0.001, 0.0001)
{128, 256)
(0.0, 0. 0.2, 0.3)
Table 8: Details of hyperparameters used for link prediction task. Please refer to Section A.3 for more details.
16


Reviews:
[{"review_id": "rJlDK_SAFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a graph convolutional  network based model for joint embedding of nodes and relations in a multi-relational graph. The framework comprises of node/relation embedding, nonparametric compositional operation as in knowledge graph embedding, and finally convolution operation with direction specific weight matrices. The performance is evaluated on link prediction, and node/graph classification tasks.\n\nOverall, the paper is well written and literature is sufficiently discussed. The performance of the proposed model on the diverse tasks looks good.  I have just two minor comments:\n\n1. The difference and overlap with Schlichtkrull et al. (2017) should be more elaborated on. Right now, the paper reads that the main difference is only in employing the basis representation for initial relation features in your work.\n\n2. There are some inconsistencies in Table 2. For example, H@3 for SCAN is same as COMPGCN for FB15k dataset. Also, for WN18RR dataset, MR of ConvKB is better but it's not in boldface!", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlBICeCYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces a GCN (Graph Convolutional Network) framework for multi-relational graphs. \n\nAuthors conducted a decent literature review and generalized several existing approaches to Knowledge Graph embedding into one framework regarding these model’s components:\n- entity-relation composition operator\n- relation weights matrix definition\n\nThere are two very close recent works (mentioned by authors), namely Weighted GCN (Shang et al., 2019) and Vectorized Relational GCN (Ye et. al., 2019), but this work proposes a more generic model. Therefore, although the idea is quite incremental, the overall work is well written and experiments with ablation study on different model’s components bring the value into it.\n\nMinor comments:\n- In table 2 SACN method outperforms COMPGCN based on H@10 (0.54 vs 0.535) for FB15k-237 dataset, but not highlighted with bold.\n- Authors write: “limited to embedding only the nodes of the graph” about Weighted Graph Convolutional Network (Shang et al., 2019), however that paper in fact does relation embeddings (see. e.g. figure 1 in that paper). The block at the bottom of the figure 1 called \"Relation Embedding\". I would propose to discuss this issue and comment on exact differences/similarities with the proposed approach.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkeK-XITYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors developed GCN on multi-relational graphs and proposed CompGCN. In comparison with existing multi-relational GCN, CompGCN leverages insights from knowledge graph embedding and learns representations of both nodes and relations, with the aim to alleviate the problem of over-parameterization. Moreover, to improve the scalability w.r.t. the number of relations, the initial relation representations are expressed as a linear combination of a fixed number of basis vectors. In contrast to existing works, the basis vectors are only defined for initialization but not for every GCN layer. The authors also compared the proposed CompGCN with other existing GCN variants and summarized the relationships between CompGCN and other models. In the experiments, three tasks including link prediction, node classification and graph classification were performed to evaluate the performance of the proposed method. By comparing with existing methods, the effectiveness of the proposed method was demonstrated.\n\nThere are several concerns on this paper. First, this work is somewhat incremental on R-GCN (\"Modeling relational data with graph convolutional networks\"). Although the differences were highlighted, the overall technical contribution is limited. In this paper, relation representations are learned jointly with node representations, but it is not clear on what are the benefits by doing so, or how significant the over-parameterization problem can be alleviated in this manner. Second, the experimental setup is unclear. As for the datasets, some more descriptions about the meaning of relationships and labels should be provided. As for the three different tasks, what loss functions were used for model training, how to obtain graph-level representations from node-level representations for graph classification, should be clarified. Third, in Table 2, there are some missing values of some compared methods, but it is not clear on their absence. Finally, from the results of Figure 3, it is hard to see the scalability of the proposed method w.r.t. the increasing number of relations since the variable is the number of basis. It is suggested to provide some more results to show the insights of advantages of the proposed method, such as the accuracy change w.r.t. increasing number of relations and the corresponding running time, in comparison with other methods.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes and evaluates a formulation of graph convolutional networks for multi-relation graphs. The paper was reviewed by three experts working in this area and received three Weak Accept decisions. The reviewers identified some concerns, including novelty with respect to existing work and specific details of the experimental setup and results that were not clear. The authors have addressed most of these concerns in their response, including adding a table that explicitly explains the contribution with respect to existing work and clarifying the missing details. Given the unanimous Weak Accept decision, the ACs also recommend Accept as a poster.
Decision: Accept (Poster)