INDIGO: GNN-Based Inductive Knowledge Graph Completion Using Pair-Wise Encoding

1 Introduction . Knowledge graphs ( KGs ) are graph-structured knowledge bases where nodes and edges represent entities of interest and their relations [ 7 ] . KGs are commonly represented as sets of triples in a standard format such as the Resource Description Framework ( RDF ) [ 9 ] . Many prominent KGs are highly incomplete , which limits their usefulness in practice ; hence KG completion ( or link prediction ) —the problem of extending a KG with missing triples—has received significant attention [ 14 ] . Most approaches to KG completion , such as TransE [ 3 ] , DistMult [ 28 ] , and RotatE [ 20 ] , are based on graph embedding techniques , which first embed the KG into a vector space ( e.g. , by learning a feature vector for each entity ) and then generate the predicted triples by applying a predefined scoring function to the learnt vectors [ 14 ] . These models demonstrate good performance in transductive settings , where missing triples are assumed to mention only constants already occurring in the incomplete KG . A key limitation of these models , however , is that they are not applicable in inductive settings [ 5 , 24 , 21 ] , where missing triples may involve constants unseen during training ; this setting is especially relevant in practice since KGs are evolving : they may be extended with triples describing new objects or integrated with external KGs . Consider , for instance , the KG inside the frame in Fig . 1 , where an embedding-based model has been used to extend the KG with the triple ( Plato , lives , Greece ) ; thus , the model seems to have successfully learnt the pattern that students and their teachers tend to live in the same country . Assume the graph is now extended with the triples ( Aristotle , student , Plato ) , ( R.Feynman , student , J.Wheeler ) , ( J.Wheeler , lives , USA ) , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . introducing constants for Aristotle , Feynman , Wheeler , and the USA . Since the parameters learnt by embedding-based models are tied to the constants in the original KG , such a model can not generalise the learnt pattern to the constants unseen during training , and fails to complete the KG with triples ( Aristotle , lives , Greece ) and ( R.Feynman , lives , USA ) . Indeed , for the model to become applicable , it would require re-training using also the newly introduced triples . There are various approaches for inductive KG completion . Rule-based methods [ 29 , 10 , 15 ] show good results , but only when the shape of the rules is known in advance . Other methods utilise side information about unseen constants , such as attributes [ 6 ] and textual descriptions [ 25 , 18 ] , which is not always available in practice . Recent approaches [ 5 , 17 , 30 , 24 , 21 ] aim to overcome these limitations by exploiting Graph Neural Networks ( GNNs ) , which are directly applicable to graphs annotated with feature vectors [ 16 ] . GNNs are biased towards graph symmetries ( nodes with same neighbourhoods receive the same value ) and have built-in inductive capabilities : they are invariant under isomorphisms ( i.e. , insensitive to the identity of nodes ) and can capture general patterns represented as logic formulas [ 2 , 11 , 27 ] . Conceptually , we see GNN-based KG completion approaches as working in three stages . First , they encode the input KG as a graph with nodes annotated by feature vectors ; second , this graph is fed to a GNN ; third , the predicted triples are decoded from the output vectors of the GNN . Approaches differ in their encoding of the KG , the kind of GNN they use , and the chosen function for decoding . In the R-GCN approach [ 17 ] , the encoder generates a node with a randomly initialised vector for each entity in the KG and triples are encoded as directed coloured edges ( colours correspond to binary relations ) ; their GNN model aggregates separately for each colour , and a colour-specific scoring function is applied to the outermost GNN layer for decoding the prediction . Hamaguchi et al . [ 5 ] and Wang et al . [ 24 ] developed similar approaches with decoding also based on a scoring function . A drawback of these approaches is that their prediction for a triple depends on the standalone neighbourhoods of its constants in the input KG , but does not take into account what is the common part of these neighbourhoods . This drawback has been recently addressed in the GraIL system by Teru et al . [ 21 ] . To predict whether a given triple should be added to the KG , GraIL identifies a sub-graph of the KG for that triple and encodes it in a way similar to R-GCN . Then , GraIL applies a GNN and makes a prediction for this triple using a scoring function applied globally to the output vectors of all nodes in its dedicated subgraph . To make predictions for many triples , the approach requires generating a subgraph to each of them , which is a significant bottleneck in practice . In this paper , we propose a novel approach where the KG is encoded into the input graph to a graph convolutional network [ 8 ] ) in a transparent and direct way , so that the inductive capabilities of the GNN are fully exploited . In contrast to existing approaches , our encoding establishes a one-to-one correspondence between elements of the feature vectors in the innermost and outermost layers of the GNN and triples over the KG ’ s signature , and hence the predicted triples can be read out directly from the outermost layer without the need for an external scoring function . As a result , our approach has neither the aforementioned drawbacks of scoring-based approaches , nor the bottleneck of GraIL , being able to process the entire graph at once ; moreover , our approach allows to make predictions for several triples in one run , which may provide significant speed up in practice . We have implemented our approach in a system called INDIGO and compared it with R-GCN , GraIL and the system in [ 5 ] on the inductive benchmarks developed in [ 21 ] and [ 5 ] , as well as on a new benchmark we have developed based on Freebase . Our results show that INDIGO not only outperforms the baselines on these benchmarks , but can also be trained and applied more efficiently . We have finally studied the ability of our system to learn in practice common inference patterns represented using logical rules ; our experiments show that our system is able to learn more comprehensive rule sets than the evaluated baselines . 2 Inductive KG Completion . In the context of this paper , a knowledge graph is a set of triples represented in a standard format for graph-structured data such as RDF [ 9 ] . More formally , a signature consists of pairwise disjoint sets of types ( i.e. , unary predicates ) , relations ( i.e. , binary predicates ) , and constants . A knowledge graph ( KG ) is a finite set of triples of the form ( c , type , t ) , where c is a constant and t is a type , and triples of the form ( c , r , d ) , where c , d are constants and r is a relation.1 The signature Sig ( K ) of a KG K is the set of types , relations , and constants used in K ; then , Pred ( K ) is the subset of Sig ( K ) consisting of all its types and relations . The problem of KG completion can be loosely understood as that of extending an incomplete KG K to its complete version K∗ by adding triples over Sig ( K ) , where a triple is added if there is sufficient evidence that it holds given the triples in K. We formalise inductive KG completion as the following ML problem . Given arbitrary ( but fixed ) finite sets Types and Rels of types and relations , respectively , the aim in inductive KG completion is to learn a Boolean completion function f ( · , · ) applicable to each pair of a KG K with Pred ( K ) ⊆ Types ∪ Rels and a triple λ with Sig ( λ ) ⊆ Sig ( K ) such that f ( K , λ ) is true if λ is in K∗ . Note that , in this formalisation , transductive KG completion is the particular case of inductive completion where the function to be learnt is applicable only to a fixed KG K—that is , can be seen as fK ( · ) such that fK ( λ ) is true for a triple λ if λ belongs to K∗ . A key limitation of transductive approaches based on graph embeddings is that the completion function learnt for a given KG is not applicable to any other KG ( for this one needs to independently learn a new function ) . In other words , a trained instance of such an approach can not make predictions for triples involving constants unseen during training ; for example , a model trained by such a system on a graph in the frame in Fig . 1 can not make predictions for any triple with a constant outside this frame . In contrast , once trained on a KG K , inductive GNN-based systems , such as GraIL and our system , can make predictions without re-training on every KG and triple over the same types and relations as K , regardless of the constants they use . 3 A GNN-Based Architecture for Inductive KG Completion . 3.1 Overview . Our inductive approach relies on the completion function f realised by the following three steps . 1 . Encoding , which takes an ( incomplete ) KG K and a set Λ of candidate triples ( of the same signature ) as input and returns a node-annotated graph GΛK of the form specified in Definition 1 ; the encoding is pair-wise : nodes in GΛK correspond to pairs of constants occurring in K and Λ . 2 . GNN application , which updates the annotations of the graph nodes ; in our approach , we use a graph convolutional network ( GCN ) , which is a GNN variant popular in applications [ 8 ] . 3 . Decoding , which extracts the predictions f ( K , λ ) for each λ ∈ Λ from the updated graph output by the GNN ; this step is essentially mirroring the encoding . The details of these steps are given in Sections 3.2 , 3.3 , and 3.4 , respectively . The key conceptual difference distinguishing our approach from existing inductive systems , such as R-GCN and GraIL , lies in the encoding and decoding steps . On the one hand , existing approaches typically encode each constant in K by a unique node in the graph , where the node ’ s feature vector is randomly initialised ; in contrast , in our approach each node in the graph encodes a pair of constants , and each element in 1In the KG literature , constants and types are sometimes collectively referred to as entities and the type relation is treated as an ordinary relation . the initial feature vector of this node directly captures a fact in the KG involving these constants . On the other hand , existing approaches typically rely on ad-hoc scoring functions to decode the output of the GNN , whereas in our approach the predicted triples can be read out directly from the feature vectors in the final layer of the GNN . Furthermore , in our approach , we are able to make predictions for a set Λ of candidate triples at once ( for the same KG ) rather than for a single triple ; this facilitates training and testing in many practical scenarios , including those reflected in existing benchmarks . Next , we define the node-annotated graphs which are used on all three steps of our approach . There are significant differences between KGs , which represent graph-structured data , and annotated graphs , which are the graphs manipulated by GNNs . Nodes in a KG represent constants and types , and edges are labelled with relations between such entities ; in contrast , nodes in annotated graphs are annotated with feature vectors , and ( undirected and unlabelled ) edges between nodes indicate that the values of their vectors influence each other during the execution of the GNN . Definition 1 . For annotation dimension δ ∈ N , a δ-annotated graph is an undirected graph where each node u is associated with a feature vector u ∈ Rδ . The i-th element of u is denoted by ( u ) i . In our approach , the annotation dimension δ is determined by the relevant sets Rels of relations and Types of types ( which are assumed known and fixed before training in the inductive setting ) ; in particular , we take δ = |Types|+ 2 · |Rels| . This allows us to associate positions in feature vectors to types , relations , and inverses of relations , which will be instrumental for encoding and decoding . In particular , we fix an ( arbitrary ) enumeration id assigning a unique natural number from 1 to δ to each t ∈ Types , each r ∈ Rels , and the inverse r− of each r ∈ Rels , and assume that , for every type , relation , or inverse p , element id ( p ) of a node feature vector corresponds to p. Observe that the dimension δ does not depend on the constants of the input KG ; so our approach is indeed inductive : once trained , our model is applicable to any KG over the fixed sets of types and relations , but using arbitrary constants . For the rest of this section , we assume that Rels , Types , id , and δ are fixed .

Summary: Knowlegde graph completion (KGC) has traditionally been attempted in transductive mode: a fixed (incomplete) graph is provided, the algorithm analyzes it, and proposes edges between existing nodes.  The inductive version, where new nodes never seen during training are then presented, and (subject, relation, object) queries have to be scored with unseen subject and/or object, is less studied, and is the subject of this paper.  The paper proposes a new encoding/transformation of the training+test graph, new in the KGC task but not particularly striking as far as graph representation goes; it is a variation of a product graph.  Then it proposes to run a fairly standard GCN on top of the transformed graph.  The output from the GCN is used to predict new edges in the original KG.  Experiments show frequent, sometimes dramatic gains over some recent systems.  The core idea (Defn 2) is to build a new graph from the KG, where each node is a pair of (perhaps identical) entities.  Roughly speaking, each node $(e,e)$ in the new graph, where $e$ is an entity in the original KG is associated with a bit vector over the entity types of the KG.  If $e$ is an instance of type $t$, then the $t$th bit is set.  Also, each node pair $(s,o)$ in the new graph, where $(s,r,o)$ is in the original KG, is associated with a bit vector over the relation types of the KG, with the $r$th bit set.   Two nodes in the new graph are connected by an edge if any of their component entities in the original KG is common.  On this new graph they run a GCN.  The GCN representation of a node is used to infer a softmax over types or relations as appropriate.  This makes the KGC scheme inductive.