LPRules: Rule Induction in Knowledge Graphs Using Linear Programming

1 INTRODUCTION . Knowledge graphs ( KG ) are used to represent a collection of known facts via labeled directed edges . Each node of the graph represents an entity , and a labeled directed edge from one node to another indicates that the pair of nodes satisfies a binary relation given by the edge label . A fact in the knowledge graph is a triplet of the form ( a , r , b ) where a and b are nodes , and r is a binary relation labeling a directed edge from a to b indicating that r ( a , b ) is true . Consider a KG where the nodes correspond to distinct cities , states , and countries and the relations are one of capital_of , shares_border_with , or part_of . A fact ( a , part_of , b ) in such a graph corresponds to a directed edge from a to b labeled by part_of implying that a is a part of b . Practical knowledge graphs are often incomplete ( they do not contain all true representable facts ) and noisy ( they can have inconsistencies or errors ) . Knowledge graph completion ( KGC ) , triple classification , entity recognition , and relation extraction are common tasks for extracting implied information from KGs . See Ji et al . ( 2021 ) for a recent survey on KGs . KGC involves using known facts in a KG to infer additional ( missing ) facts . One approach for KGC is to learn first-order logic rules that encode known facts . In the example above , we could learn a “ rule ” capital_of ( X , Y ) → part_of ( X , Y ) where X , Y are variables that take on entity values . Then if we find a pair of entities P , Q such that ( P , capital_of , Q ) is a fact in the graph , we could infer that ( P , part_of , Q ) is also true and augment the set of facts with this new fact if not originally present . A more complex rule of length two is capital_of ( X , Y ) and part_of ( Y , Z ) → part_of ( X , Z ) . Again , applying it to entities P and Q , if there exists a third entity R such that capital_of ( P , R ) is a fact in the graph , and so is part_of ( R , Q ) , then we infer that P is a part of Q. KG link prediction deals with finding answers to queries of the form part_of ( P , ? ) , and we focus on finding first-order logic rules of the type above for this task . Instead of learning one rule for a relation , it is common to learn a set of rules along with rule weights , where larger weights indicate more important or more certain rules . Dealing with uncertainty or noise is essential for KG reasoning . Learning logic rules is a well-studied area . In a paper ( Lao & Cohen , 2010 ) on path ranking algorithms and another ( Richardson & Domingos , 2006 ) on Markov logic networks , candidate logic rules are obtained via relational path enumeration , and then rule weights are calculated . Yang et al . ( 2017 ) use neural logic programming to simultaneously obtain sets of rules and the weights of individual rules . Qu et al . ( 2021 ) , separately find rules and rule weights , but add a feedback loop from the latter learning problem to the former . The use of recursive neural networks ( RNN ) to learn rules is common nowadays , though traditional rule-mining approaches remain popular ( Meilicke et al. , 2019 ) . Embedding-based methods for KGC consist of representing nodes by vectors , and relations by vector transformations that are consistent with the facts in the knowledge graph . They exhibit better scaling with KG size , and yield more accurate predictions . See the surveys by Ji et al . ( 2021 ) and Wang et al . ( 2017 ) . On the other hand , rule-based methods can yield more interpretable explanations of associated reasoning steps ( Qu & Tang , 2019 ) , especially if one obtains compact rule sets ( with few rules and few relations per rule ) . Furthermore , entity-independent rules are more generalizeable ( Teru et al. , 2020 ) and can be applied in an inductive setting : they can be applied to entities not considered in the learning process . Embedding-based methods work mostly in a transductive setting . We propose a novel approach to learning entity-independent , weighted first-order logic rules for knowledge graph reasoning . Our approach combines rule enumeration with linear programming ( LP ) , and completely avoids the solution of difficult nonconvex optimizaton models inherent in training RNNs . We describe a linear programming ( LP ) formulation with exponentially many variables corresponding to first-order rules and associated weights . Nonzero variable values correspond to chosen rules and their weights . We deal with the exponential number of variables/rules by column generation ideas from linear optimization , where we start off with some small initial set of rules and associated variables , find the best subset of these and associated weights via the partial LP defined on the initial set of rules , and then generate new rules which can best augment the existing set of rules . Our final output rule-based scoring functions resemble those in NeuralLP ( Yang et al. , 2017 ) , DRUM ( Sadeghian et al. , 2019 ) , and RNNLogic ( Qu et al. , 2021 ) in that we form a linear combination of rule scores ( calculated differently from the above papers ) . As in RNNLogic , our iterative process of adding new rules is influenced by previous rules . Our algorithm has better scaling with KG size than a many existing rule-based methods . In addition , we obtain state-of-the-art results on three out of four standard KG datasets while taking significantly less running time than some competing codes . Furthermore , we are able to obtain results of reasonable quality – when compared to embedding-based methods – on YAGO3-10 , a large dataset which is difficult for existing rule-based methods . As an important goal of rule-based methods is to obtain interpretable solutions , we promote interpretability by adding a constraint in our LP formulation limiting the complexity of the chosen rule set ( and this hyperparameter is tuned per dataset ) . In some cases , we obtain more accurate results ( higher MRR ) for the same level of complexity as other codes , and less complex rules for the same level of accuracy in other cases . 2 RELATED WORK . 2.1 RULE-BASED METHODS . There is a rich body of literature on rule-based methods for knowledge graph reasoning . The motivation for learning rules is that they form an explicit symbolic representation of existing knowledge and are amenable to inspection and verification . Further , compact rule sets are interpretable , an efficient way to store knowledge , and useful for transfer learning . For KG applications , when rules are entity-independent , they can be used in an inductive setting ( Yang et al . ( 2017 ) , Teru et al . ( 2020 ) ) . Inductive Logic Programming ( ILP ) . In this approach , one takes as input positive and negative examples and learns logic programs that entail all positive examples and none of the negative examples . See for example Cropper & Muggleton ( 2016 ) and Cropper & Morel ( 2020 ) . First-order Logic ( FOL ) programs in the form of a collection of chain-like Horn clauses are a popular output format . Negative examples are not available in typical knowledge graphs , and some of the positive examples can be mutually inconsistent . Evans & Grefenstette ( 2018 ) developed a differential ILP framework to generate rules for noisy data . Statistical Relational Learning ( SRL ) . SRL aims to learn FOL formulas from data and to quantify their uncertainty . Markov logic , which is a probabilistic extension of FOL , is a popular framework for SRL . In this framework , one learns a set of weighted FOL formulas ; see , for example , Kok & Domingos ( 2005 ) where beam search is used to find a set of FOL rules , and rule weights are learned via standard numerical methods . In knowledge graph reasoning , chain-like rules which correspond to relational paths ( and to chain-like Horn clauses ) are widely studied . In Lao & Cohen ( 2010 ) , a weighted linear combination of rule-based functions ( e.g. , the function could return a probability that the rule implies a link between a pair of entities ) is used as a scoring function for KG link completion . An initial set of rules is created and the weights are obtained via regression . A recent , bottom-up rule-learning algorithm with excellent predictive performance is AnyBURL ( Meilicke et al. , 2019 ) . We learn FOL rules corresponding to relational paths and rule weights ; our scoring function , and learning model/algorithm are different from prior work . The column generation aspect of our work has similarities to cutting plane inference methods ( Riedel , 2008 ; Noessner et al. , 2013 ) for MAP inference in SRL . While we deal with an exponential number of possible rules/Horn clauses via column generation , the above papers use constraint generation/cutting plane ideas to deal with an exponential number of constraints corresponding to ground clauses . Neuro-symbolic methods . In NeuralLP ( Yang et al. , 2017 ) , rules and rule weights are learned simultaneously by training an appropriate RNN . Further improvements in this paradigm can be found in DRUM ( Sadeghian et al . ( 2019 ) ) . Another neuro-symbolic approach is implemented in the Neural Theorem Prover ( NTP , Rochstätel & Riedel ( 2017 ) ) . More general rules ( than the chain-like rules found in NeuralLP ) are obtained in NLIL ( Yang & Song ) along with better scaling behavior . Simultaneously solving for rules and rule-weights is difficult , and a natural question is how well the associated optimization problem can be solved , and how scalable such methods are . We use an easier-to-solve LP formulation . Hybrid methods . In RNNLogic ( Qu et al. , 2021 ) , the rules are generated using an RNN , and rule weights are later calculated via a probabilistic model . Such a separation of rule generation and weight calculation can be found in earlier work ( e.g. , Kok & Domingos ( 2005 ) ) , but in RNNLogic new rule generation is influenced by the calculated weights of previously generated rules . Our column generation method and AnyBURL have the same property . Reinforcement Learning . Recent attempts to use reinforcement learning ( RL ) to search for rules can be found in MINERVA ( Das et al. , 2018 ) , MultiHopKG ( Lin et al. , 2018 ) , M-Walk ( Shen et al. , 2018 ) and DeepPath ( Xiong et al. , 2017 ) . The first three papers use RL to explore relational paths conditioned on a specific query , and use RNNs to encode and construct a graph-walking agent . Rule types/Rule combinations . AnyBURL generates rules corresponding to different types of paths ( acyclic or cyclic , in their notation ) which may be entity-dependent or independent . As mentioned before , NLIL goes beyond simple chain-like rules . Recently Teru et al . ( 2020 ) use subgraphs to perform reasoning , and not just paths . We generate weighted , entity-independent , chain-like rules as in NeuralLP . Our scoring function combines rule scores via a linear combination . For a rule r and a pair of entities a , b , the rule score is just 1 if there exists a relational path from a to b following the rule r and 0 otherwise . We use rule weights as a measure of importance but they are not probabilities . For Neural LP , DRUM , RNNLogic and other comparable codes , the scoring functions depend on the set of paths from a to b associated with r. Finally , AnyBURL uses maximum confidence scores rather than sums of confidence scores ; further , its scoring function returns a vector of sorted scores , and two score vectors are compared lexicographically . Scalability/Compact Rule sets . As noted above , many recent papers use RNNs in the process of finding chain-like rules and this can lead to expensive computation times . On the other hand , bottom-up rule-learners such as AnyBURL are much faster . The main focus of our work is obtaining compact rule sets for the sake of interpretability while maintaining scalability via LP models and column generation . We impose explicit constraints to maintain compactness , and these constraints influence new rule generation . The new rules must perform well with previously generated/selected rules . NeuralLP , for example , usually returns compact rule sets while AnyBURL returns a very large number of rules ( and does not prune discovered rules for interpretability ) , and RNNLogic is somewhere in between ( though the number of output rules can be controlled ) .

Summary: This paper proposed a simple linear programming model for learning logical rules for KG completion. The model selects candidate rules from KG with explicit constraints and then solves a linear programming problem. The authors conduct experiments on several public datasets and the model has better efficiency than baseline models.