A Topological View of Rule Learning in Knowledge Graphs

1 INTRODUCTION . Knowledge graphs ( KGs ) are graph-structured knowledge bases that integrate human knowledge through relational triplets . In a KG , nodes represent entities and edges represent relational triplets connecting them . A relational triplet is defined as ( eh , r , et ) , where eh and et are the head and tail entities respectively , and r is the relation between them . KGs have been used in many problems such as recommendation systems ( Wang et al. , 2018 ) , question answering ( Huang et al. , 2019 ; Zhang et al. , 2018 ) , biomedical research ( Zhao et al. , 2020b ; Zhu et al. , 2020 ) , and zero-shot learning ( Kampffmeyer et al. , 2019 ) . Due to the limitation of human knowledge and data extraction algorithms , we can not thoroughly define and excavate all the entities and relations in a KG ( Chen et al. , 2020 ) . The incomplete structures and contents of KGs can significantly benefit from an automatic completion algorithm . Early works ( Bordes et al. , 2013 ; Yang et al. , 2014 ; Sun et al. , 2019 ) focus on incorporating the attributes of entities . Recent works develop models that are agnostic of entity attributes . They can handle new entities and dynamic KGs , which are quite common . These entity-agnostic works ( Yang et al. , 2017 ; Sadeghian et al. , 2019 ; Teru et al. , 2020 ) are called inductive relation prediction models . They predict missing triplets by learning logical rules in KGs . For example , from the KG shown in Figure 1 ( a ) , we can learn the rule : ∃X , ( X , part of , Y ) ∧ ( X , lives in , Z ) → ( Y , located in , Z ) . ( 1 ) Based on the learned rule , in Figure 1 ( b ) , we can induce the missing triplet ( ManchesterUnited , located in , Manchester ) due to the existence of the two-hop path ( Cristiano , lives in , Manchester ) and ( Cristiano , part of , ManchesterUnited ) .1 Existing inductive relation prediction works ( Yang et al. , 2017 ; Sadeghian et al. , 2019 ) mainly view rules as paths , i.e. , sequences of relations connecting two entities of interest . For example , to express the rule shown in Equation 1 , these methods need to enumerate all possible 2-hop paths . As Yang et al . ( 2017 ) pointed out , the number of learnable parameters of these methods is O ( |R|T ) , where |R| denotes the number of relation types , and T denotes the maximum length of rules . To avoid 1Technically speaking , these methods can only learn the “ and ” operation between relations . We will be interested in expanding to more sophisticated rules , but this is beyond the scope of this paper . the exponential-size search space of paths , we tackle the problem from a new perspective : to view logical rules as cycles , and then learn the rules in the space of cycles . In fact , any rule can be considered a cycle by including both the relation path and the target relation itself . The benefit of using cycles is that there is an intrinsic structure in the space of cycles . Based on the theory of algebraic topology ( Munkres , 2018 ) , the space of cycles is a vector space under certain assumptions . We can exploit the linear structure of cycle space for efficient rule learning . In particular , we focus on a basis , i.e. , a set of linearly independent cycles that can represent any cycle . Taking Figure 1 ( a ) as an example , if we choose the red cycle and the blue cycle as the cycle basis , then the green cycle can be represented as their sum . Here “ sum ” means modulo-2 sum ( Figure 1 ( d ) ) , thus the overlapped triplet ( Lebron , part of , Lakers ) is canceled . By focusing on the cycle basis that spans the cycle space , we decrease the parameter space from exponential to linear2 . However , the choice of a useful cycle basis is not trivial . Not all cycles/rules are “ good ” , or say , providing a valid relation reasoning . For example , the green cycle in Figure 1 ( a ) reveals the rule : ∃X1 , X2 , ( X1 , support , Y ) ∧ ( X1 , fan of , X2 ) ∧ ( X2 , lives in , Z ) → ( Y , located in , Z ) . ( 2 ) This rule is not considered good . There is no strong reason that a basketball team ( Y ) should be located in a city ( Z ) just because there is a fan ( X1 ) that supports both the team and a player ( X2 ) living in the city Z . The goal of relation learning is to efficiently comb through all cycles/rules and learn the good ones . A desirable cycle basis should facilitate this process . Intuitively , the elements of the basis should be as close to good cycles as possible . Revisiting the example of Figure 1 ( a ) , we prefer the basis consisting of the red and blue cycles ( both are good rules ) and avoid the green cycle . To find suitable cycle bases for relation learning , we propose an approach inspired by the theory of topological data analysis ( Dey et al. , 2010 ; Chen & Freedman , 2010 ; Busaryev et al. , 2012 ) . These works show that cycle bases generated from shortest path trees have favorable theoretical guarantees . In the context of relation learning , we argue that the good cycles are generally short , and can be easily learned using shortest-path-tree-based cycle bases . We also propose to sample multiple shortest path trees and collect a family of cycle bases . This provides sufficient redundancy and coverage to ensure we learn good cycles/rules efficiently , as experiments will show . To fully exploit the cycle bases to learn all the good rules , we propose Cycle Basis Graph Neural Network ( CBGNN ) . We build a GNN on a new graph whose nodes represent cycles in the chosen bases , and edges represent their interaction . Through the message passing of the GNN , we are running implicit algebraic operations over the space of cycles . Our method will efficiently learn to represent good cycles , and learn to predict relations in KGs . Experiments on popular inductive relation prediction benchmarks show the efficacy of our method . 2A cycle basis of size β spans a cycle space of size 2β . In summary , our contribution is three-fold : 1 . We propose , for the first time , to investigate the inductive relation prediction problem through a cycle-based perspective . 2 . Inspired by the theory of topological data analysis , we propose to exploit the linear structure of the cycle space , and to compute suitable cycle bases that can best express the rules . This empowers us to explore rule space more efficiently than previous path-based approaches . 3 . We propose a novel graph neural network , CBGNN . It runs implicit algebraic operations in the cycle space through the message passing of a GNN , and learns the representation of good cycles and good rules . Experiments show that CBGNN achieves state-of-the-art results on various inductive relation prediction benchmarks . 2 RELATED WORKS . Graph Learning with Topology . Graph structural information have been shown to enhance graph representation learning ( Kipf & Welling , 2016 ; You et al. , 2019 ; Ye et al. , 2019 ) . In recent years advanced topological information , i.e. , persistent homology ( Edelsbrunner et al. , 2000 ; Edelsbrunner & Harer , 2010 ) , have been applied to graph learning problem . Persistent-homology-based features encode multiscale topological information regarding graph structures in view of different filter functions . They provide additional discriminative power for various tasks such as graph classification ( Hofer et al. , 2020 ; Carrière et al. , 2020 ; Hofer et al. , 2017 ; Zhao & Wang , 2019 ) , link prediction ( Yan et al. , 2021 ; Bhatia et al. , 2019 ) , and node classification ( Zhao et al. , 2020a ) . From a different perspective , new graph neural networks have been proposed for high-order graphs , treated as simplicial or cell complexes ( Bodnar et al. , 2021b ; a ) . Unlike these approaches , we focus on a different task , inductive relation prediction . We exploit the space of cycles and its underlying algebraic structure for better graph representation learning . We believe the cycle-centric design of our graph neural network is generic and can extend to many other tasks beyond relation prediction . Finally , we note that in topological data analysis , homology localization , including computing short cycles representatives of a homology class and computing short cycle bases representing the whole homology group , is well studied theoretically ( Chambers et al. , 2009 ; Chen & Freedman , 2011 ; Dey et al. , 2011 ; Busaryev et al. , 2012 ; Dey et al. , 2010 ; Dey & Wang , 2022 ) . In recent years , new questions have been raised regarding finding short representative cycles for classes in persistent homology ( Wu et al. , 2017 ; Dey et al. , 2020 ) . Cycle information can complement persistent homology information to achieve better learning power . Inductive relation prediction methods . Inductive relation prediction methods can be divided into two categories : path-based methods , and GNN-based methods . Among path-based methods , AMIE ( Galárraga et al. , 2013 ) and RuleN ( Meilicke et al. , 2018 ) are classic rule learning methods . These two methods prune the process of rule searching based on strong assumptions on the attribute of rules , thus their performances are not satisfying . To deal with the problem , NeuralLP ( Yang et al. , 2017 ) and DRUM ( Sadeghian et al. , 2019 ) are proposed to learn the rules in a data-driven manner . However , as pointed in ( Sadeghian et al. , 2019 ) , these works can not learn all the rules correctly with a small number of learnable parameters . GNN-based methods such as GraIL ( Teru et al. , 2020 ) and CoMPILE ( Mai et al. , 2021 ) predict missing triplets with graph neural networks ( GNNs ) . These methods are appealing as GNNs are able to learn better representations and achieve state-of-the-art performance in general . To predict whether a certain triplet exists in the KG , these methods first extract the corresponding vicinity graph of the triplet and then learn the rules through message passing and GNN scoring . Therefore , they can only predict the triplets one by one , with a rather low computational efficiency . In the experiment part , we will empirically show the efficiency of our framework by comparing it with these models . 3 CYCLE SPACE , CYCLE BASIS , AND THE PURSUIT OF SUITABLE BASES . In this section , we explain how to find suitable cycle bases that can facilitate the learning of good cycles/rules . We first introduce the background of the cycle space and cycle basis . Next , we explain our choice of suitable cycle bases , which will be the foundation of our model . By no means our exposition is comprehensive . For more details , we refer the readers to textbooks on algebraic topology and computational topology ( Munkres , 2018 ; Edelsbrunner & Harer , 2010 ; Dey & Wang , 2022 ) . We focus on cycles in undirected graphs , while the definitions generalize to higher dimensions , e.g. , simplicial complexes . Furthermore , we focus on the algebraic structures over Z2 field , which has two elements , 0 and 1 , under modulo-2 addition and multiplication . Over Z2 field , the structure of the space of cycles is simpler and more friendly to computation . For the rest of the paper , regarding the input KG , we will use node , vertex , and entity interchangeably . We will also use edge and triplet interchangeably . Within this section , we temporarily ignore the relation associated with each triplet . We treat the input KG as an undirected graph G = ( V , E ) , where V and E denote the sets of vertices and edges , respectively .

Summary: The paper proposes a GNN framework to solve the inductive relation prediction problem, namely CBGNN. The framework consists of three modules: 1)  a cycle extraction module that generates cycles graphs with clustering and SPT tree search; 2) a bi-directional LSTM that converts a cycle of triplets into a feature vector, and 3) a GNN model that learns the triplet confidence using the feature vectors. In experiments, the proposed method is evaluated with a set of differentiable ILP methods and GNN methods. Some ablation studies are conducted to validate the cycle extraction module and feature generation module.