LPRules: Rule Induction in Knowledge Graphs Using Linear Programming

1 INTRODUCTION . Knowledge graphs ( KG ) are used to represent a collection of known facts via labeled directed edges . Each node of the graph represents an entity , and a labeled directed edge from one node to another indicates that the pair of nodes satisfies a binary relation given by the edge label . A fact in the knowledge graph is a triplet of the form ( a , r , b ) where a and b are nodes , and r is a binary relation labeling a directed edge from a to b indicating that r ( a , b ) is true . Consider a KG where the nodes correspond to distinct cities , states , and countries and the relations are one of capital_of , shares_border_with , or part_of . A fact ( a , part_of , b ) in such a graph corresponds to a directed edge from a to b labeled by part_of implying that a is a part of b . Practical knowledge graphs are often incomplete ( they do not contain all true representable facts ) and noisy ( they can have inconsistencies or errors ) . Knowledge graph completion ( KGC ) , triple classification , entity recognition , and relation extraction are common tasks for extracting implied information from KGs . See Ji et al . ( 2021 ) for a recent survey on KGs . KGC involves using known facts in a KG to infer additional ( missing ) facts . One approach for KGC is to learn first-order logic rules that encode known facts . In the example above , we could learn a “ rule ” capital_of ( X , Y ) → part_of ( X , Y ) where X , Y are variables that take on entity values . Then if we find a pair of entities P , Q such that ( P , capital_of , Q ) is a fact in the graph , we could infer that ( P , part_of , Q ) is also true and augment the set of facts with this new fact if not originally present . A more complex rule of length two is capital_of ( X , Y ) and part_of ( Y , Z ) → part_of ( X , Z ) . Again , applying it to entities P and Q , if there exists a third entity R such that capital_of ( P , R ) is a fact in the graph , and so is part_of ( R , Q ) , then we infer that P is a part of Q. KG link prediction deals with finding answers to queries of the form part_of ( P , ? ) , and we focus on finding first-order logic rules of the type above for this task . Instead of learning one rule for a relation , it is common to learn a set of rules along with rule weights , where larger weights indicate more important or more certain rules . Dealing with uncertainty or noise is essential for KG reasoning . Learning logic rules is a well-studied area . In a paper ( Lao & Cohen , 2010 ) on path ranking algorithms and another ( Richardson & Domingos , 2006 ) on Markov logic networks , candidate logic rules are obtained via relational path enumeration , and then rule weights are calculated . Yang et al . ( 2017 ) use neural logic programming to simultaneously obtain sets of rules and the weights of individual rules . Qu et al . ( 2021 ) , separately find rules and rule weights , but add a feedback loop from the latter learning problem to the former . The use of recursive neural networks ( RNN ) to learn rules is common nowadays , though traditional rule-mining approaches remain popular ( Meilicke et al. , 2019 ) . Embedding-based methods for KGC consist of representing nodes by vectors , and relations by vector transformations that are consistent with the facts in the knowledge graph . They exhibit better scaling with KG size , and yield more accurate predictions . See the surveys by Ji et al . ( 2021 ) and Wang et al . ( 2017 ) . On the other hand , rule-based methods can yield more interpretable explanations of associated reasoning steps ( Qu & Tang , 2019 ) , especially if one obtains compact rule sets ( with few rules and few relations per rule ) . Furthermore , entity-independent rules are more generalizeable ( Teru et al. , 2020 ) and can be applied in an inductive setting : they can be applied to entities not considered in the learning process . Embedding-based methods work mostly in a transductive setting . We propose a novel approach to learning entity-independent , weighted first-order logic rules for knowledge graph reasoning . Our approach combines rule enumeration with linear programming ( LP ) , and completely avoids the solution of difficult nonconvex optimizaton models inherent in training RNNs . We describe a linear programming ( LP ) formulation with exponentially many variables corresponding to first-order rules and associated weights . Nonzero variable values correspond to chosen rules and their weights . We deal with the exponential number of variables/rules by column generation ideas from linear optimization , where we start off with some small initial set of rules and associated variables , find the best subset of these and associated weights via the partial LP defined on the initial set of rules , and then generate new rules which can best augment the existing set of rules . Our final output rule-based scoring functions resemble those in NeuralLP ( Yang et al. , 2017 ) , DRUM ( Sadeghian et al. , 2019 ) , and RNNLogic ( Qu et al. , 2021 ) in that we form a linear combination of rule scores ( calculated differently from the above papers ) . As in RNNLogic , our iterative process of adding new rules is influenced by previous rules . Our algorithm has better scaling with KG size than a many existing rule-based methods . In addition , we obtain state-of-the-art results on three out of four standard KG datasets while taking significantly less running time than some competing codes . Furthermore , we are able to obtain results of reasonable quality – when compared to embedding-based methods – on YAGO3-10 , a large dataset which is difficult for existing rule-based methods . As an important goal of rule-based methods is to obtain interpretable solutions , we promote interpretability by adding a constraint in our LP formulation limiting the complexity of the chosen rule set ( and this hyperparameter is tuned per dataset ) . In some cases , we obtain more accurate results ( higher MRR ) for the same level of complexity as other codes , and less complex rules for the same level of accuracy in other cases . 2 RELATED WORK . 2.1 RULE-BASED METHODS . There is a rich body of literature on rule-based methods for knowledge graph reasoning . The motivation for learning rules is that they form an explicit symbolic representation of existing knowledge and are amenable to inspection and verification . Further , compact rule sets are interpretable , an efficient way to store knowledge , and useful for transfer learning . For KG applications , when rules are entity-independent , they can be used in an inductive setting ( Yang et al . ( 2017 ) , Teru et al . ( 2020 ) ) . Inductive Logic Programming ( ILP ) . In this approach , one takes as input positive and negative examples and learns logic programs that entail all positive examples and none of the negative examples . See for example Cropper & Muggleton ( 2016 ) and Cropper & Morel ( 2020 ) . First-order Logic ( FOL ) programs in the form of a collection of chain-like Horn clauses are a popular output format . Negative examples are not available in typical knowledge graphs , and some of the positive examples can be mutually inconsistent . Evans & Grefenstette ( 2018 ) developed a differential ILP framework to generate rules for noisy data . Statistical Relational Learning ( SRL ) . SRL aims to learn FOL formulas from data and to quantify their uncertainty . Markov logic , which is a probabilistic extension of FOL , is a popular framework for SRL . In this framework , one learns a set of weighted FOL formulas ; see , for example , Kok & Domingos ( 2005 ) where beam search is used to find a set of FOL rules , and rule weights are learned via standard numerical methods . In knowledge graph reasoning , chain-like rules which correspond to relational paths ( and to chain-like Horn clauses ) are widely studied . In Lao & Cohen ( 2010 ) , a weighted linear combination of rule-based functions ( e.g. , the function could return a probability that the rule implies a link between a pair of entities ) is used as a scoring function for KG link completion . An initial set of rules is created and the weights are obtained via regression . A recent , bottom-up rule-learning algorithm with excellent predictive performance is AnyBURL ( Meilicke et al. , 2019 ) . We learn FOL rules corresponding to relational paths and rule weights ; our scoring function , and learning model/algorithm are different from prior work . The column generation aspect of our work has similarities to cutting plane inference methods ( Riedel , 2008 ; Noessner et al. , 2013 ) for MAP inference in SRL . While we deal with an exponential number of possible rules/Horn clauses via column generation , the above papers use constraint generation/cutting plane ideas to deal with an exponential number of constraints corresponding to ground clauses . Neuro-symbolic methods . In NeuralLP ( Yang et al. , 2017 ) , rules and rule weights are learned simultaneously by training an appropriate RNN . Further improvements in this paradigm can be found in DRUM ( Sadeghian et al . ( 2019 ) ) . Another neuro-symbolic approach is implemented in the Neural Theorem Prover ( NTP , Rochstätel & Riedel ( 2017 ) ) . More general rules ( than the chain-like rules found in NeuralLP ) are obtained in NLIL ( Yang & Song ) along with better scaling behavior . Simultaneously solving for rules and rule-weights is difficult , and a natural question is how well the associated optimization problem can be solved , and how scalable such methods are . We use an easier-to-solve LP formulation . Hybrid methods . In RNNLogic ( Qu et al. , 2021 ) , the rules are generated using an RNN , and rule weights are later calculated via a probabilistic model . Such a separation of rule generation and weight calculation can be found in earlier work ( e.g. , Kok & Domingos ( 2005 ) ) , but in RNNLogic new rule generation is influenced by the calculated weights of previously generated rules . Our column generation method and AnyBURL have the same property . Reinforcement Learning . Recent attempts to use reinforcement learning ( RL ) to search for rules can be found in MINERVA ( Das et al. , 2018 ) , MultiHopKG ( Lin et al. , 2018 ) , M-Walk ( Shen et al. , 2018 ) and DeepPath ( Xiong et al. , 2017 ) . The first three papers use RL to explore relational paths conditioned on a specific query , and use RNNs to encode and construct a graph-walking agent . Rule types/Rule combinations . AnyBURL generates rules corresponding to different types of paths ( acyclic or cyclic , in their notation ) which may be entity-dependent or independent . As mentioned before , NLIL goes beyond simple chain-like rules . Recently Teru et al . ( 2020 ) use subgraphs to perform reasoning , and not just paths . We generate weighted , entity-independent , chain-like rules as in NeuralLP . Our scoring function combines rule scores via a linear combination . For a rule r and a pair of entities a , b , the rule score is just 1 if there exists a relational path from a to b following the rule r and 0 otherwise . We use rule weights as a measure of importance but they are not probabilities . For Neural LP , DRUM , RNNLogic and other comparable codes , the scoring functions depend on the set of paths from a to b associated with r. Finally , AnyBURL uses maximum confidence scores rather than sums of confidence scores ; further , its scoring function returns a vector of sorted scores , and two score vectors are compared lexicographically . Scalability/Compact Rule sets . As noted above , many recent papers use RNNs in the process of finding chain-like rules and this can lead to expensive computation times . On the other hand , bottom-up rule-learners such as AnyBURL are much faster . The main focus of our work is obtaining compact rule sets for the sake of interpretability while maintaining scalability via LP models and column generation . We impose explicit constraints to maintain compactness , and these constraints influence new rule generation . The new rules must perform well with previously generated/selected rules . NeuralLP , for example , usually returns compact rule sets while AnyBURL returns a very large number of rules ( and does not prune discovered rules for interpretability ) , and RNNLogic is somewhere in between ( though the number of output rules can be controlled ) .

Summary: The paper presents a method to obtain weights for a knowledge graph scoring model for link prediction. There are numerous good algorithms for mining rules from knowledge graphs (e.g. AnyBURL and AMIE). Less research has focused on the problem of creating scoring functions based on an (implicit) list of rules. This is what the present paper proposes.   The core idea is to formulate a linear program whose solution corresponds to a scoring method. Instead of incorporating all rules (there are typically many!) into the LP formulation, the authors propose a column generation approach.