Differentiable learning of numerical rules in knowledge graphs

1 INTRODUCTION . Due to the availability of vast amounts of knowledge on the web , advances in information extraction have led to large graph-structured knowledge bases , also known as knowledge graphs ( KGs ) , which are widely used in web search , question answering , and data analytics . Such KGs represent data as a graph of entities ( e.g. , john , article1 ) connected via relations ( e.g. , citedIn ) , or more formally as a set of binary grounded atoms ( e.g. , citedIn ( john , article1 ) ) . A common task in such settings is that of link prediction , determining whether a relation exists between two entities in the graph even if the relation is not included explicitly in the graph . Although most work on this topic has focused on statistical rule-extraction techniques ( Meilicke et al . ( 2019 ) ; Galárraga et al . ( 2015 ) ; Ortona et al . ( 2018a ) ) , recent methods have shown the benefit of using deep learning approaches for this link prediction task ( see Wang et al . ( 2017 ) for overview ) . And while most deep approaches ( for example , those based upon graph embedding methods ) are inherently difficult to interpret , the Neural LP method of Yang et al . ( 2017 ) is particularly appealing in that it allows for interpretable resulting rules for the link prediction task while still preserving the flexibility of a learning approach . Unfortunately , Neural LP is also quite limited in the types of rules it is capable of representing , and notably no rules that depend on numerical features can be efficiently learned within this framework . In this paper , we propose an extension to Neural LP that allows for fast learning of numerical rules . Specifically , although numerical rules would result in dense matrix operations in the generic Neural LP framework , we show that using dynamic programming and cumulative sum operations , we can efficiently express the operators for numerical comparators within the Neural LP framework . By defining the relevant operators implicitly in this manner , we show that we can extend Neural LP to efficiently learn rules that make use of numerical features , while retaining the interpretability of the Neural LP framework . More generally , this is an instance of integrating so-called “ aggregates ” ( i.e . external oracle queries , in this case binary queries that reflect numerical comparison ) within a rule-learning framework . Learning such rules with aggregates is very much an open problem in the KG community ( Galárraga & Suchanek ( 2014 ) ) , and our approach is the first work to learn rules with these numerical aggregates . We apply our approach to several knowledge graph datasets , and show that we are able to answer queries more accurately than the previous Neural LP approach , as well as more accurately than a state-of-the-art rule extraction method , the AnyBurl package proposed by Meilicke et al . ( 2019 ) . Specifically , we show on two synthetic and two real-world datasets that our extension to Neural LP is able to more accurately recover rules that depend on numerical information , and thus make much more accurate link predictions in the knowledge graph . Further , the extracted rules are still interpretable as in the original Neural LP framework , and unlike the pure graph embedding strategies ( Bordes et al . ( 2013 ) ) . 2 RELATED WORK . Relational Data Mining . The problem of learning rules from the data has been traditionally addressed in the area of relational data mining Raedt ( 2017 ) and inductive logic programming ( ILP ) Muggleton ( 1995 ) . Works most related to ours concern learning decision trees with aggregates Vens et al . ( 2006 ) from relational data . However , these methods typically do not scale well , and modern knowledge graphs are far beyond what they can handle . In the context of KGs , the problem of rule learning has recently gained a lot of attention . In Ortona et al . ( 2018b ) rules with negation , which also support numerical comparison as we do have been considered . Contrary to our approach , however , Ortona et al . ( 2018b ) is designed to find a small set of rules that cover the majority of positive and as few negative examples as possible , which differs from our goal of learning rules in an unsupervised fashion . Neural-based Rule Learning . Several works Yang et al . ( 2017 ) ; Manhaeve et al . ( 2018 ) ; Rocktäschel & Riedel ( 2017 ) ; Evans & Grefenstette ( 2018 ) ; Zhang et al . ( 2019 ) ; Ho et al . ( 2018 ) ; Weber et al . ( 2019 ) utilize embedding models and neural architectures for rule learning . The closest to ours is the work Yang et al . ( 2017 ) , which reduces the rule learning problem to algebraic operations on neural-embedding-based representations of a given KG . However , Yang et al . ( 2017 ) is restricted to non-numerical rules in contrast to our work . Embedding Models with Numericals . The problem of KG incompleteness has been tackled by methods that predict missing relational edges between existing entities . Several approaches rely on statistics and include tensor factorization ( e.g. , Nickel et al . ( 2011 ) ) . Other models are based on neural embeddings ( e.g. , Bordes et al . ( 2013 ) ) . For overview see Wang et al . ( 2017 ) . The most relevant for us is the work Garcı́a-Durán & Niepert ( 2018 ) , which presents a novel approach to combining relational , latent ( learned ) and numerical features , i.e . features taking large or infinite number of real values for the KG completion task . While Garcı́a-Durán & Niepert ( 2018 ) operates on KGs with numerical values , it ’ s results like in the case of most knowledge graph embedding models are not interpretable . 3 PRELIMINARIES . Knowledge Graphs . We assume countable sets C of constants ( a.k.a . entities ) , N ⊂ R of numerical values andR of binary relations ( a.k.a . predicates ) . A KG G is defined by a finite set of ground atoms ( a.k.a . facts ) , of the form p ( x , y ) , where p ∈ R , x ∈ C and y ∈ C∪N ( e.g. , citedIn ( john , article1 ) ) . The set Rn ⊆ R stores all numerical predicates p , such that p ( x , y ) ∈ G , where x ∈ C and y ∈ N . The set of numerical facts , i.e . facts over numerical predicates , is denoted by Gn ⊆ G. We use lower-case letters for constants and upper-case letters for variables . As KGs are incomplete , one can assume that missing facts , i.e . facts that are not in G , are either unknown or false . Typically the open world assumption ( OWA ) is employed , which means that missing facts are considered to be unknown rather than false . Alternatively , the local closed world assumption ( LCWA ) can be considered to generate negative facts by assuming that the KG is locally complete as data is usually added to KGs in batches . More precisely , it means that for any x ∈ C we can conclude that p ( x , y ) is false if z ∈ C ∪ N exists such that p ( x , z ) ∈ G and p ( x , y ) /∈ G. Numerical Rules . A rule is an expression of the form p ( X , Y ) ← q1 ( X , Z1 ) ∧ . . . ∧ qn ( Zn , Y ) , ( 1 ) where p , q1 , . . . , qn ∈ R , left-hand side of the rule is referred to as the rule head , and right-hand side as the rule body , and every conjunct in the rule head or body is referred to as an atom . The rule influences ( X , Y ) ← colleagueOf ( X , Z ) ∧ supervisorOf ( Z , Y ) intuitively states that typically students are influenced by colleagues of their supervisors . Aside from conventional rules , we can also have numerical rules , i.e . rules that contain numerical comparison among variables ( e.g. , number of citations of two people ) , or a variable and a numerical constant . To simplify presentation , numerical values inN linked to an entity from C are sometimes treated as its “ features ” , and numerical relations inRn as functions that depend on those features . In this case , for p ( X , Y ) we also use a shortcut notation X.p = Y . For instance , john.hasCitation = 124 stands for hasCitation ( john , 124 ) , and for compactness , r◦pq ( X , Y ) stands for X.p ◦ Y.q , where ◦ ∈ { ≤ , > } . The second subscript in r◦pq is omitted if it is clear from the context that p = q . Example 1 . For example , consider a KG in Figure 1 and the following rule influences ( X , Y ) ← colleagueOf ( X , Z ) ∧ supervisorOf ( Z , Y ) ∧ r > hasCitation ( X , Z ) . This rule states that students are influenced by colleagues of their supervisors with a higher number of citations1 . We can also define a classification relation mapping the feature to the probability of a logistic classification , σ ( wTX.features + b ) , where w and b are parameters and σ is the sigmoid function . As we demonstrate later such rules can be integrated and learnt naturally in the Neural LP framework . Rule Learning . Given a KG G the goal of rule learning is to extract rules from G , such that their application to G results in an approximation of the ideal KG , which stores all correct facts . 1I.e . influences ( X , Y ) ← colleagueOf ( X , Z ) ∧ supervisorOf ( Z , Y ) ∧ hasCitation ( X , V1 ) ∧ hasCitation ( Z , V2 ) ∧ V1 > V2 . The Neural LP method ( Yang et al . ( 2017 ) ) is among rule learning proposals , which learns a distribution over rules of the form in Eq . ( 1 ) without comparison operators in an end-to-end fashion by making use of gradient-based optimization . This approach relies on the TensorLog framework ( Cohen et al . ( 2017 ) ) , which connects rule application with sparse matrix multiplications . In TensorLog all entities are mapped to integers , and each entity i is associated with a one-hot encoded vector vi ∈ { 0 , 1 } |C| such that only its i-th entry is 1 . For example , every KG entity c ∈ C in Fig . 1 is encoded as a 0/1 vector of length 5 , since |C| = 5 . For every relation p ∈ R \ Rn and every pair of entities x , y ∈ C a matrix Mp ∈ { 0 , 1 } |C|×|C| is defined such that its ( y , x ) entry , denoted by ( Mp ) yx , is 1 iff p ( x , y ) ∈ G. For example , by considering the KG in Fig . 1 , for the relation p = citedIn we have Mp = john pete bob article1 article2 0 0 0 0 0 john 0 0 0 0 0 pete 0 0 0 0 0 bob 1 1 0 0 0 article1 1 0 0 0 0 article2 The idea of TensorLog is to imitate application of rules for any entity X = x by performing matrix multiplications Mqn . . .Mq2Mq1vx = s , where vx is the indicator of entity x. Nonzero entries in the vector s point to the entities y for which p ( x , y ) is derived by applying the above rule on G. For example , the inference for the following rule influences ( X , Y ) ← colleagueOf ( X , Z ) , supervisorOf ( Z , Y ) can be translated to MsupervisorOf McolleagueOf vx = s . By setting vx = [ 1 , 0 , 0 , 0 , 0 ] > as indicator of john and applying the matrix multiplications , we obtain s = [ 0 , 0 , 1 , 0 , 0 ] > , the indicator of bob . As Mq1 , . . . , Mqn are sparse , the matrix-vector multiplication can be done efficiently , and the inference process is parallelizable on GPUs . In Neural LP ( Yang et al . ( 2017 ) ) above operators are used to learn for every head the formula f ( α ) = ∑ i αi ∏ j∈βi Mqj ( 2 ) where i indexes over all possible rules , αi is the confidence associated with the rule ri and βi is an ordered list of all relations appearing in these rules . The rules are read off from the solution of the following optimization problem max { αi , βi } log ( ∑ { x , y } v > y ( ∑ i αi ( ∏ j∈βj Mqj ) vx ) ) .

Summary: This paper proposed several extensions to the Neural LP work. Specifically, this paper addresses several limitations, including numerical variables, negations, etc. To efficiently compute these in the original Neural LP framework, this paper proposed several computation tricks to accelerate, as well as to save memory. Experiments on benchmark datasets show significant improvements over previous methods, especially in the case where numerical variables are required.