Differentiable Learning of Graph-like Logical Rules from Knowledge Graphs

Logical rules inside a knowledge graph ( KG ) are essential for reasoning , logical inference , and rule mining . However , existing works can only handle simple , i.e. , chain-like and tree-like , rules and can not capture KG ’ s complex semantics , which can be better captured by graph-like rules . Besides , learning graph-like rules is very difficult because the graph structure exhibits a huge discrete search space . To address these issues , observing that the plausibility of logical rules can be explained by how frequently it appears in a KG , we propose a score function that represents graph-like rules with learnable parameters . The score also helps relax the discrete space into a continuous one and can be uniformly transformed into matrix form by the Einstein summation convention . Thus , it allows us to learn graph-like rules in an efficient , differentiable , and end-to-end training manner by optimizing the normalized score . We conduct extensive experiments on real-world datasets to show that our method outperforms previous works due to logical rules ’ better expressive ability . Furthermore , we demonstrate that our method can learn high-quality and interpretable graph-like logical rules . 1 INTRODUCTION . Knowledge graph ( KG ) refers to a special type of directed graphs including various entities as nodes and relations as directed edges representing a large number of facts ( Auer et al. , 2007 ; Bollacker et al. , 2008 ) . In KG , logical rules are a set of compositional logical relations within a specific structure , which are important for reasoning ( Cohen et al. , 2019 ; Zhang et al. , 2019a ; Qu & Tang , 2019 ) , logical inference ( Dhingra et al. , 2020 ; Das et al. , 2018 ; Xiong et al. , 2017 ) , rule mining ( Sadeghian et al. , 2019 ; Yang et al. , 2017 ; Yang & Song , 2020 ) , theorem proving ( Rocktäschel & Riedel , 2017 ; Minervini et al. , 2018 ; 2020 ) , etc . Learning logical rules ( Galárraga et al. , 2015 ; Chen et al. , 2016 ) , as an important task , aims to infer a structural logical rule for logical query or relation , which can support logical query or link prediction while providing interpretable logical rules . The structure of logical queries can be various with very different semantics , as shown in Figure 1 , including chain-like , tree-like and graph-like rules . Learning the logical rules , especially the graph-like rules , are very difficult because both the logical structure and the relations assigned on each edge are unknown requiring to be inferred from input-output pairs , which compose a huge discrete searching space . In this paper , we dive into the problem of learning graph-like logical rules , including both the logical structure representing how logic connects and the relations assigned on different edges . Recently , a series of works on learning logical rule ( Yang et al. , 2017 ; Sadeghian et al. , 2019 ; Yang & Song , 2020 ) has been proposed , which not only can support tasks including logical query and link prediction , but as a side effect , can also provide the mined logical rules with high interpretability . As shown in Figure 1 , all these works are limited to learning chain-like rules ( the left case ) ( Yang et al. , 2017 ; Sadeghian et al. , 2019 ) or tree-like rules ( the middle case ) ( Hamilton et al. , 2018 ; Ren et al. , 2020 ; Yang & Song , 2020 ) . However , there are widely-existed graph-like logical rules , which the existing works can not handle due to their limited expressive ability about logical rules . Learning graph-like logical rules is very important in many scenarios such as recommendation systems , question-answering system and KG completion , while learning such complex rules is still an open and challenging problem . Graph-like rule Which book has two common readers with the book X while the two readers are friends ? Tree-like rule What is the address of the university that both the students X1 and X2 study at ? Chain-like rule Who is X ’ s friend ’ s supervisor ? Semantic Questions We propose a novel method that can explicitly learn the structural logical rules , including a logical structure and the relations assigned on each edge , and we can use the inferred logical rules for conducting inductive logical query with unseen entities and graphs . All the structural logical rules construct a discrete search space to explore , and searching for that is an NP-hard problem . To tackle with this problem , our method constructs a continuous space including both the structural information and the relational information to learn , which allows us to train our model in an endto-end differentiable manner . Specifically , as shown in Figure 1 , we take the frequency of a logical rule in KG as its score to estimate how likely a logical rule stands . After optimizing w.r.t . the normalized score , our model yields interpretable logical rules of high quality , and support inductive logical query and link prediction , which has been demonstrated by our extensive experiments on real-world datasets . Our contributions can be summarized as following three aspects , • We first propose the problem of learning graph-like rules and design an end-to-end differentiable model that can learn graph-like logical rules instead of only chain-like or tree-like rules , modeling both the logical structure describing how the logic connects and relations assigned on edges . • We provide a uniform expression by Einsum to represent the score of all graph-like logical rules , including the ones that can not be represented by a combination of matrix/element-wise addition/product , which is elegant for expression and convenient for implementation . • We conduct extensive experiments to demonstrate that our model has better expressive ability for graph-like logical rules and show our model can mine high-quality logical rules with high interpretability . 2 PROBLEM FORMULATION . Here , we formally introduce the definition of logical score , and based on that , we further introduce our model ’ s main focus , relation inference ( Yang et al. , 2017 ; Sadeghian et al. , 2019 ) and structural rule learning , and our evaluation task , logical query ( Hamilton et al. , 2018 ; Ren et al. , 2020 ) . Definition 1 ( Logical Score ) Logical rule is formulated by ∧ni=1Ri → Rcpx : sr where sr is the score for ∧ni=1Ri , andRi is a relationRi = Ri ( Vi , V ′i ) , Vi , V ′i ∈ { { Xj } , Y , { Zk } } for i = 1 , · · · , n and Rcpx is a relation Rcpx ( { Xj } , Y ) , { Xj } are input nodes , { Zk } are free-variable nodes , Y is output node . For strict logical query , for any Rcpx ( { Xj } , Y ) , there exists ( Z1 , · · · , ZK ) that make ∧ni=1Ri be true , we can draw the conclusion ∧ni=1Ri → Rcpx . However , because KG is usually noisy and incomplete , for learning logical rules , our key insight is to design the score as the number of freevariable tuples ( Z1 , · · · , ZK ) that make ∧ni=1Ri be true , which can capture the correlation between logical rules and the input-output pairs of a logical query . For example , for the case in the middle of Figure 1 , Rstudy at ( X1 , Z ) ∧ Rstudy at ( X2 , Z ) ∧ Raddress of ( Z , Y ) → Rcpx ( X1 , X2 , Y ) ; for the case in the right of Figure 1 , we have Rread ( X , Z1 ) ∧ Rread ( X , Z2 ) ∧ Rfriend ( Z1 , Z2 ) ∧ Rread ( inv ) ( Z1 , Y ) ∧ Rread ( inv ) ( Z2 , Y ) → Rcpx ( X , Y ) . Note that , Rcpx can both be a relation that exists in the KG and the human-defined logic rule for a query , which tends to be more complex . The score sr serves as two roles : ( i ) . when input-output pairs are given , it can measure how likely a logical rule is , which corresponds to the scenario of Task 1 and Task 2 ; ( ii ) . when logical rules for query and inputs are given , it can measure how much a output node fits the query , which corresponds to Task 3 . Task 1 ( Relation Inference ) GivenRcpx ( { Xj } , Y ) is satisfied and a logical structure composed by G = { e1 ( V1 , V ′1 ) , e2 ( V2 , V ′2 ) , · · · } , we need to infer how to assign relation Ri on each edge ei to form a logical rule ∧ni=1Ri ( Vi , V ′i ) that will make the score sr of Rcpx ( { Xj } , Y ) high . For this task , previous relation inference works ( Yang et al. , 2017 ; Sadeghian et al. , 2019 ) can also conduct this task but they limit the G to be a chain-like . We model the relation between the inputoutput pairs behind the query as Rcpx and infer its graph-like logical rule . Task 2 ( Structural Rule Learning ) Given Rcpx ( { Xj } , Y ) is satisfied and the possible max nodes number n̂ ≥ ne , where ne is the size of { { Xj } , Y } , we need to infer what structure G = { e1 ( V1 , V ′1 ) , e2 ( V2 , V ′2 ) , · · · , en ( Vn , V ′n ) } where ne ≤ n ≤ n̂ and the relations assigned on edges ∧ni=1Ri that will make the score sr high . For this task , logical structures in previous works ( Yang et al. , 2017 ; Sadeghian et al. , 2019 ; Yang & Song , 2020 ) are limited to chains or trees , and the number of input entities are limited to 1 . However , we can infer both the logical structure and the relations assigned on edges for graph-like rules . Task 3 ( Logical Query ) Given input nodes { Xj } and the query relation , the target nodes of this query can be represented by q = { Y |Rcpx ( { Xj } , Y ) } . Note that , in previous works ( Hamilton et al. , 2018 ; Ren et al. , 2020 ) , the logical ruleRcpx = ∧ni=1Ri is given , different from those works , we need to infer the ∧ni=1Ri for logic query . Our model targets at the inference of complex logical rules , and use the inferred logic rules to conduct logical query as evaluation task . For evaluation , we regard Task 3 as the main task and the other two tasks as side products . 3 RELATED WORKS . 3.1 LOGICAL QUERY FROM KNOWLEDGE GRAPHS . Logical rules learning ( Teru et al. , 2020 ; Evans & Grefenstette , 2018 ; Manhaeve et al. , 2018 ; Wang et al. , 2019 ; Ho et al. , 2018 ) is to learn logical rules ( Task 1 ) for logical query ( Task 3 ) in an inductive setting . Neural-LP ( Yang et al. , 2017 ) design an end-to-end differentiable framework to learn the probability of different logical rules . Furthermore , DRUM ( Sadeghian et al. , 2019 ) improve NeuralLP ( Yang et al. , 2017 ) by introducing the low-rank matrix decomposition . However , these two works can only tackle chain-like logical rules . Different from our model , they mainly focus on relatively simple logical rules such as chain-like or tree-like rules . To the best of our knowledge , our model is the first one that can learn to infer graph-like complex logical rule including structure and relations assigned on different edges . Logical queries ( Serge et al. , 1995 ) aims to learn how to accurately query an entity ( Task 3 ) according to given input entities and relations representing the logical rules in a transductive setting . According to Task 3 , the logic rules representing the semantics of query explicitly given at both training and testing stages in this branch of works , but in our paper , the logic rules require to be inferred in the training stage . For most of these works , the main idea is to project entities into the embedding space ( Bordes et al. , 2013 ; Trouillon et al. , 2016 ; Sun et al. , 2018 ; Balažević et al. , 2019 ) and transform the relations into a type of manipulation in embedding space , such as a linear projection . Hamilton et al . ( 2018 ) first proposes an embedding-based method for conduct query with tree-like logical rules . Ren et al . ( 2020 ) further improves Hamilton et al . ( 2018 ) by modeling the entities as box embedding rather than vector embedding , which is more natural for the manipulation for the conjunction of sets . Different from our model , these methods require explicit given logical structures with given relations on edges .

Summary: This paper proposes techniques that generate logical rules out of knowledge graphs; the idea is to produce more complex rules than usual by exploiting a differentiable formulation of the associated learning process. This is a relevant theme as rule learning from knowledge graphs is important in practice due to its potential interpretability (as compared to black-box schemes based on embeddings). The solution is relatively simple to describe, with a score that leads to differentiable learning, and some needed insights to obtain useful results. The empirical testing seems fine and does indicate that the method is useful in practice.