Composition-based Multi-Relational Graph Convolutional Networks

1 INTRODUCTION . Graphs are one of the most expressive data-structures which have been used to model a variety of problems . Traditional neural network architectures like Convolutional Neural Networks ( Krizhevsky et al. , 2012 ) and Recurrent Neural Networks ( Hochreiter & Schmidhuber , 1997 ) are constrained to handle only Euclidean data . Recently , Graph Convolutional Networks ( GCNs ) ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ) have been proposed to address this shortcoming , and have been successfully applied to several domains such as social networks ( Hamilton et al. , 2017 ) , knowledge graphs ( Schlichtkrull et al. , 2017 ; Shang et al. , 2019 ) , natural language processing ( Marcheggiani & Titov , 2017 ; Vashishth et al. , 2018a ; b ; 2019 ) , drug discovery ( Ramsundar et al. , 2019 ) , crystal property prediction ( Sanyal et al. , 2018 ) , and natural sciences ( Fout et al. , 2017 ) . However , most of the existing research on GCNs ( Kipf & Welling , 2016 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) have focused on learning representations of nodes in simple undirected graphs . A more general and pervasive class of graphs are multi-relational graphs1 . A notable example of such graphs is knowledge graphs . Most of the existing GCN based approaches for handling relational graphs ( Marcheggiani & Titov , 2017 ; Schlichtkrull et al. , 2017 ) suffer from overparameterization and are limited to learning only node representations . Hence , such methods are not directly applicable for tasks such as link prediction which require relation embedding vectors . Initial attempts at learning representations for relations in graphs ( Monti et al. , 2018 ; Beck et al. , 2018 ) have shown some performance gains on tasks like node classification and neural machine translation . There has been extensive research on embedding Knowledge Graphs ( KG ) ( Nickel et al. , 2016 ; Wang et al. , 2017 ) where representations of both nodes and relations are jointly learned . These methods are restricted to learning embeddings using link prediction objective . Even though GCNs can ∗Equally Contributed †Work done while at IISc , Bangalore 1In this paper , multi-relational graphs refer to graphs with edges that have labels and directions . learn from task-specific objectives such as classification , their application has been largely restricted to non-relational graph setting . Thus , there is a need for a framework which can utilize KG embedding techniques for learning task-specific node and relation embeddings . In this paper , we propose COMPGCN , a novel GCN framework for multi-relational graphs which systematically leverages entity-relation composition operations from knowledge graph embedding techniques . COMPGCN addresses the shortcomings of previously proposed GCN models by jointly learning vector representations for both nodes and relations in the graph . An overview of COMPGCN is presented in Figure 1 . The contributions of our work can be summarized as follows : 1 . We propose COMPGCN , a novel framework for incorporating multi-relational information in Graph Convolutional Networks which leverages a variety of composition operations from knowledge graph embedding techniques to jointly embed both nodes and relations in a graph . 2 . We demonstrate that COMPGCN framework generalizes several existing multi-relational GCN methods ( Proposition 4.1 ) and also scales with the increase in number of relations in the graph ( Section 6.3 ) . 3 . Through extensive experiments on tasks such as node classification , link prediction , and graph classification , we demonstrate the effectiveness of our proposed method . The source code of COMPGCN and datasets used in the paper have been made available at http : //github.com/malllabiisc/CompGCN . 2 RELATED WORK . Graph Convolutional Networks : GCNs generalize Convolutional Neural Networks ( CNNs ) to non-Euclidean data . GCNs were first introduced by Bruna et al . ( 2013 ) and later made scalable through efficient localized filters in the spectral domain ( Defferrard et al. , 2016 ) . A first-order approximation of GCNs using Chebyshev polynomials has been proposed by Kipf & Welling ( 2016 ) . Recently , several of its extensions have also been formulated ( Hamilton et al. , 2017 ; Veličković et al. , 2018 ; Xu et al. , 2019 ; Vashishth et al. , 2019 ; Yadav et al. , 2019 ) . Most of the existing GCN methods follow Message Passing Neural Networks ( MPNN ) framework ( Gilmer et al. , 2017 ) for node aggregation . Our proposed method can be seen as an instantiation of the MPNN framework . However , it is specialized for relational graphs . GCNs for Multi-Relational Graph : An extension of GCNs for relational graphs is proposed by Marcheggiani & Titov ( 2017 ) . However , they only consider direction-specific filters and ignore relations due to over-parameterization . Schlichtkrull et al . ( 2017 ) address this shortcoming by proposing basis and block-diagonal decomposition of relation specific filters . Weighted Graph Convolutional Network ( Shang et al. , 2019 ) utilizes learnable relational specific scalar weights during GCN aggregation . While these methods show performance gains on node classification and link prediction , they are limited to embedding only the nodes of the graph . Contemporary to our work , Ye et al . ( 2019 ) have also proposed an extension of GCNs for embedding both nodes and relations in multirelational graphs . However , our proposed method is a more generic framework which can leverage any KG composition operator . We compare against their method in Section 6.1 . Knowledge Graph Embedding : Knowledge graph ( KG ) embedding is a widely studied field ( Nickel et al. , 2016 ; Wang et al. , 2017 ) with application in tasks like link prediction and question answering ( Bordes et al. , 2014 ) . Most of KG embedding approaches define a score function and train node and relation embeddings such that valid triples are assigned a higher score than the invalid ones . Based on the type of score function , KG embedding method are classified as translational ( Bordes et al. , 2013 ; Wang et al. , 2014b ) , semantic matching based ( Yang et al. , 2014 ; Nickel et al. , 2016 ) and neural network based ( Socher et al. , 2013 ; Dettmers et al. , 2018 ; Vashishth et al. , 2019 ) . In our work , we evaluate the performance of COMPGCN on link prediction with methods of all three types . 3 BACKGROUND . In this section , we give a brief overview of Graph Convolutional Networks ( GCNs ) for undirected graphs and its extension to directed relational graphs . GCN on Undirected Graphs : Given a graph G = ( V , E , X ) , where V denotes the set of vertices , E is the set of edges , and X ∈ R|V|×d0 represents d0-dimensional input features of each node . The node representation obtained from a single GCN layer is defined as : H = f ( ÂXW ) . Here , Â = D̃− 1 2 ( A+ I ) D̃− 1 2 is the normalized adjacency matrix with added self-connections and D̃ is defined as D̃ii = ∑ j ( A + I ) ij . The model parameter is denoted by W ∈ Rd0×d1 and f is some activation function . The GCN representation H encodes the immediate neighborhood of each node in the graph . For capturing multi-hop dependencies in the graph , several GCN layers can be stacked , one on the top of another as follows : Hk+1 = f ( ÂHkW k ) , where k denotes the number of layers , W k ∈ Rdk×dk+1 is layer-specific parameter and H0 = X . GCN on Multi-Relational Graphs : For a multi-relational graph G = ( V , R , E , X ) , where R denotes the set of relations , and each edge ( u , v , r ) represents that the relation r ∈ R exist from node u to v. The GCN formulation as devised by Marcheggiani & Titov ( 2017 ) is based on the assumption that information in a directed edge flows along both directions . Hence , for each edge ( u , v , r ) ∈ E , an inverse edge ( v , u , r−1 ) is included in G. The representations obtained after k layers of directed GCN is given by Hk+1 = f ( ÂHkW kr ) . ( 1 ) Here , W kr denotes the relation specific parameters of the model . However , the above formulation leads to over-parameterization with an increase in the number of relations and hence , Marcheggiani & Titov ( 2017 ) use direction-specific weight matrices . Schlichtkrull et al . ( 2017 ) address overparameterization by proposing basis and block-diagonal decomposition of W kr . 4 COMPGCN DETAILS . In this section , we provide a detailed description of our proposed method , COMPGCN . The overall architecture is shown in Figure 1 . We represent a multi-relational graph by G = ( V , R , E , X , Z ) as defined in Section 3 where Z ∈ R|R|×d0 denotes the initial relation features . Our model is motivated by the first-order approximation of GCNs using Chebyshev polynomials ( Kipf & Welling , 2016 ) . Following Marcheggiani & Titov ( 2017 ) , we also allow the information in a directed edge to flow along both directions . Hence , we extend E and R with corresponding inverse edges and relations , i.e. , E ′ = E ∪ { ( v , u , r−1 ) | ( u , v , r ) ∈ E } ∪ { ( u , u , > ) | u ∈ V ) } , and R′ = R ∪ Rinv ∪ { > } , where Rinv = { r−1 | r ∈ R } denotes the inverse relations and > indicates the self loop . 4.1 RELATION-BASED COMPOSITION . Unlike most of the existing methods which embed only nodes in the graph , COMPGCN learns a d-dimensional representation hr ∈ Rd , ∀r ∈ R along with node embeddings hv ∈ Rd , ∀v ∈ V . Representing relations as vectors alleviates the problem of over-parameterization while applying GCNs on relational graphs . Further , it allows COMPGCN to exploit any available relation features ( Z ) as initial representations . To incorporate relation embeddings into the GCN formulation , we leverage the entity-relation composition operations used in Knowledge Graph embedding approaches ( Bordes et al. , 2013 ; Nickel et al. , 2016 ) , which are of the form eo = φ ( es , er ) . Here , φ : Rd × Rd → Rd is a composition operator , s , r , and o denote subject , relation and object in the knowledge graph and e ( · ) ∈ Rd denotes their corresponding embeddings . In this paper , we restrict ourselves to non-parameterized operations like subtraction ( Bordes et al. , 2013 ) , multiplication ( Yang et al. , 2014 ) and circular-correlation ( Nickel et al. , 2016 ) . However , COMPGCN can be extended to parameterized operations like Neural Tensor Networks ( NTN ) ( Socher et al. , 2013 ) and ConvE ( Dettmers et al. , 2018 ) . We defer their analysis as future work . As we show in Section 6 , the choice of composition operation is important in deciding the quality of the learned embeddings . Hence , superior composition operations for Knowledge Graphs developed in future can be adopted to improve COMPGCN ’ s performance further .

Summary: In this paper, the authors developed GCN on multi-relational graphs and proposed CompGCN. In comparison with existing multi-relational GCN, CompGCN leverages insights from knowledge graph embedding and learns representations of both nodes and relations, with the aim to alleviate the problem of over-parameterization. Moreover, to improve the scalability w.r.t. the number of relations, the initial relation representations are expressed as a linear combination of a fixed number of basis vectors. In contrast to existing works, the basis vectors are only defined for initialization but not for every GCN layer. The authors also compared the proposed CompGCN with other existing GCN variants and summarized the relationships between CompGCN and other models. In the experiments, three tasks including link prediction, node classification and graph classification were performed to evaluate the performance of the proposed method. By comparing with existing methods, the effectiveness of the proposed method was demonstrated.