Composition-based Multi-Relational Graph Convolutional Networks

1 INTRODUCTION . Graphs are one of the most expressive data-structures which have been used to model a variety of problems . Traditional neural network architectures like Convolutional Neural Networks ( Krizhevsky et al. , 2012 ) and Recurrent Neural Networks ( Hochreiter & Schmidhuber , 1997 ) are constrained to handle only Euclidean data . Recently , Graph Convolutional Networks ( GCNs ) ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ) have been proposed to address this shortcoming , and have been successfully applied to several domains such as social networks ( Hamilton et al. , 2017 ) , knowledge graphs ( Schlichtkrull et al. , 2017 ; Shang et al. , 2019 ) , natural language processing ( Marcheggiani & Titov , 2017 ; Vashishth et al. , 2018a ; b ; 2019 ) , drug discovery ( Ramsundar et al. , 2019 ) , crystal property prediction ( Sanyal et al. , 2018 ) , and natural sciences ( Fout et al. , 2017 ) . However , most of the existing research on GCNs ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) have focused on learning representations of nodes in simple undirected graphs . A more general and pervasive class of graphs are multi-relational graphs1 . A notable example of such graphs is knowledge graphs . Most of the existing GCN based approaches for handling relational graphs ( Marcheggiani & Titov , 2017 ; Schlichtkrull et al. , 2017 ) suffer from overparameterization and are limited to learning only node representations . Hence , such methods are not directly applicable for tasks such as link prediction which require relation embedding vectors . Initial attempts at learning representations for relations in graphs ( Monti et al. , 2018 ; Beck et al. , 2018 ) have shown some performance gains on tasks like node classification and neural machine translation . There has been extensive research on embedding Knowledge Graphs ( KG ) ( Nickel et al. , 2016 ; Wang et al. , 2017 ) where representations of both nodes and relations are jointly learned . These methods are restricted to learning embeddings using link prediction objective . Even though GCNs can ∗Equally Contributed †Work done while at IISc , Bangalore 1In this paper , multi-relational graphs refer to graphs with edges that have labels and directions . learn from task-specific objectives such as classification , their application has been largely restricted to non-relational graph setting . Thus , there is a need for a framework which can utilize KG embedding techniques for learning task-specific node and relation embeddings . In this paper , we propose COMPGCN , a novel GCN framework for multi-relational graphs which systematically leverages entity-relation composition operations from knowledge graph embedding techniques . COMPGCN addresses the shortcomings of previously proposed GCN models by jointly learning vector representations for both nodes and relations in the graph . An overview of COMPGCN is presented in Figure 1 . The contributions of our work can be summarized as follows : 1 . We propose COMPGCN , a novel framework for incorporating multi-relational information in Graph Convolutional Networks which leverages a variety of composition operations from knowledge graph embedding techniques to jointly embed both nodes and relations in a graph . 2 . We demonstrate that COMPGCN framework generalizes several existing multi-relational GCN methods ( Proposition 4.1 ) and also scales with the increase in number of relations in the graph ( Section 6.3 ) . 3 . Through extensive experiments on tasks such as node classification , link prediction , and graph classification , we demonstrate the effectiveness of our proposed method . The source code of COMPGCN and datasets used in the paper have been made available at http : //github.com/malllabiisc/CompGCN . 2 RELATED WORK . Graph Convolutional Networks : GCNs generalize Convolutional Neural Networks ( CNNs ) to non-Euclidean data . GCNs were first introduced by Bruna et al . ( 2013 ) and later made scalable through efficient localized filters in the spectral domain ( Defferrard et al. , 2016 ) . A first-order approximation of GCNs using Chebyshev polynomials has been proposed by Kipf & Welling ( 2016 ) . Recently , several of its extensions have also been formulated ( Hamilton et al. , 2017 ; Veličković et al. , 2018 ; Xu et al. , 2019 ; Vashishth et al. , 2019 ; Yadav et al. , 2019 ) . Most of the existing GCN methods follow Message Passing Neural Networks ( MPNN ) framework ( Gilmer et al. , 2017 ) for node aggregation . Our proposed method can be seen as an instantiation of the MPNN framework . However , it is specialized for relational graphs . GCNs for Multi-Relational Graph : An extension of GCNs for relational graphs is proposed by Marcheggiani & Titov ( 2017 ) . However , they only consider direction-specific filters and ignore relations due to over-parameterization . Schlichtkrull et al . ( 2017 ) address this shortcoming by proposing basis and block-diagonal decomposition of relation specific filters . Weighted Graph Convolutional Network ( Shang et al. , 2019 ) utilizes learnable relational specific scalar weights during GCN aggregation . While these methods show performance gains on node classification and link prediction , they are limited to embedding only the nodes of the graph . Contemporary to our work , Ye et al . ( 2019 ) have also proposed an extension of GCNs for embedding both nodes and relations in multirelational graphs . However , our proposed method is a more generic framework which can leverage any KG composition operator . We compare against their method in Section 6.1 . Knowledge Graph Embedding : Knowledge graph ( KG ) embedding is a widely studied field ( Nickel et al. , 2016 ; Wang et al. , 2017 ) with application in tasks like link prediction and question answering ( Bordes et al. , 2014 ) . Most of KG embedding approaches define a score function and train node and relation embeddings such that valid triples are assigned a higher score than the invalid ones . Based on the type of score function , KG embedding method are classified as translational ( Bordes et al. , 2013 ; Wang et al. , 2014b ) , semantic matching based ( Yang et al. , 2014 ; Nickel et al. , 2016 ) and neural network based ( Socher et al. , 2013 ; Dettmers et al. , 2018 ; Vashishth et al. , 2019 ) . In our work , we evaluate the performance of COMPGCN on link prediction with methods of all three types . 3 BACKGROUND . In this section , we give a brief overview of Graph Convolutional Networks ( GCNs ) for undirected graphs and its extension to directed relational graphs . GCN on Undirected Graphs : Given a graph G = ( V , E , X ) , where V denotes the set of vertices , E is the set of edges , and X ∈ R|V|×d0 represents d0-dimensional input features of each node . The node representation obtained from a single GCN layer is defined as : H = f ( ÂXW ) . Here , Â = D̃− 1 2 ( A+ I ) D̃− 1 2 is the normalized adjacency matrix with added self-connections and D̃ is defined as D̃ii = ∑ j ( A + I ) ij . The model parameter is denoted by W ∈ Rd0×d1 and f is some activation function . The GCN representation H encodes the immediate neighborhood of each node in the graph . For capturing multi-hop dependencies in the graph , several GCN layers can be stacked , one on the top of another as follows : Hk+1 = f ( ÂHkW k ) , where k denotes the number of layers , W k ∈ Rdk×dk+1 is layer-specific parameter and H0 = X . GCN on Multi-Relational Graphs : For a multi-relational graph G = ( V , R , E , X ) , where R denotes the set of relations , and each edge ( u , v , r ) represents that the relation r ∈ R exist from node u to v. The GCN formulation as devised by Marcheggiani & Titov ( 2017 ) is based on the assumption that information in a directed edge flows along both directions . Hence , for each edge ( u , v , r ) ∈ E , an inverse edge ( v , u , r−1 ) is included in G. The representations obtained after k layers of directed GCN is given by Hk+1 = f ( ÂHkW kr ) . ( 1 ) Here , W kr denotes the relation specific parameters of the model . However , the above formulation leads to over-parameterization with an increase in the number of relations and hence , Marcheggiani & Titov ( 2017 ) use direction-specific weight matrices . Schlichtkrull et al . ( 2017 ) address overparameterization by proposing basis and block-diagonal decomposition of W kr . 4 COMPGCN DETAILS . In this section , we provide a detailed description of our proposed method , COMPGCN . The overall architecture is shown in Figure 1 . We represent a multi-relational graph by G = ( V , R , E , X , Z ) as defined in Section 3 where Z ∈ R|R|×d0 denotes the initial relation features . Our model is motivated by the first-order approximation of GCNs using Chebyshev polynomials ( Kipf & Welling , 2016 ) . Following Marcheggiani & Titov ( 2017 ) , we also allow the information in a directed edge to flow along both directions . Hence , we extend E and R with corresponding inverse edges and relations , i.e. , E ′ = E ∪ { ( v , u , r−1 ) | ( u , v , r ) ∈ E } ∪ { ( u , u , > ) | u ∈ V ) } , and R′ = R ∪ Rinv ∪ { > } , where Rinv = { r−1 | r ∈ R } denotes the inverse relations and > indicates the self loop . 4.1 RELATION-BASED COMPOSITION . Unlike most of the existing methods which embed only nodes in the graph , COMPGCN learns a d-dimensional representation hr ∈ Rd , ∀r ∈ R along with node embeddings hv ∈ Rd , ∀v ∈ V . Representing relations as vectors alleviates the problem of over-parameterization while applying GCNs on relational graphs . Further , it allows COMPGCN to exploit any available relation features ( Z ) as initial representations . To incorporate relation embeddings into the GCN formulation , we leverage the entity-relation composition operations used in Knowledge Graph embedding approaches ( Bordes et al. , 2013 ; Nickel et al. , 2016 ) , which are of the form eo = φ ( es , er ) . Here , φ : Rd × Rd → Rd is a composition operator , s , r , and o denote subject , relation and object in the knowledge graph and e ( · ) ∈ Rd denotes their corresponding embeddings . In this paper , we restrict ourselves to non-parameterized operations like subtraction ( Bordes et al. , 2013 ) , multiplication ( Yang et al. , 2014 ) and circular-correlation ( Nickel et al. , 2016 ) . However , COMPGCN can be extended to parameterized operations like Neural Tensor Networks ( NTN ) ( Socher et al. , 2013 ) and ConvE ( Dettmers et al. , 2018 ) . We defer their analysis as future work . As we show in Section 6 , the choice of composition operation is important in deciding the quality of the learned embeddings . Hence , superior composition operations for Knowledge Graphs developed in future can be adopted to improve COMPGCN ’ s performance further .

Summary: This paper proposes a graph convolutional  network based model for joint embedding of nodes and relations in a multi-relational graph. The framework comprises of node/relation embedding, nonparametric compositional operation as in knowledge graph embedding, and finally convolution operation with direction specific weight matrices. The performance is evaluated on link prediction, and node/graph classification tasks.