Scalable Neural Theorem Proving on Knowledge Bases and Natural Language

Reasoning over text and Knowledge Bases ( KBs ) is a major challenge for Artificial Intelligence , with applications in machine reading , dialogue , and question answering . Transducing text to logical forms which can be operated on is a brittle and error-prone process . Operating directly on text by jointly learning representations and transformations thereof by means of neural architectures that lack the ability to learn and exploit general rules can be very data-inefficient and not generalise correctly . These issues are addressed by Neural Theorem Provers ( NTPs ) ( Rocktäschel & Riedel , 2017 ) , neuro-symbolic systems based on a continuous relaxation of Prolog ’ s backward chaining algorithm , where symbolic unification between atoms is replaced by a differentiable operator computing the similarity between their embedding representations . In this paper , we first propose Neighbourhoodapproximated Neural Theorem Provers ( NaNTPs ) consisting of two extensions to NTPs , namely a ) a method for drastically reducing the previously prohibitive time and space complexity during inference and learning , and b ) an attention mechanism for improving the rule learning process , deeming them usable on real-world datasets . Then , we propose a novel approach for jointly reasoning over KB facts and textual mentions , by jointly embedding them in a shared embedding space . The proposed method is able to extract rules and provide explanations—involving both textual patterns and KB relations—from large KBs and text corpora . We show that NaNTPs perform on par with NTPs at a fraction of a cost , and can achieve competitive link prediction results on challenging large-scale datasets , including WN18 , WN18RR , and FB15k-237 ( with and without textual mentions ) while being able to provide explanations for each prediction and extract interpretable rules . 1 INTRODUCTION . The main focus in Artificial Intelligence is building systems that exhibit intelligent behaviour ( Levesque , 2014 ) . In particular , Natural Language Understanding ( NLU ) and Machine Reading ( MR ) aim at building models and systems with the ability to read text , extract meaningful knowledge , and actively reason with it ( Etzioni et al. , 2006 ; Hermann et al. , 2015 ; Weston et al. , 2015 ; McCallum et al. , 2017a ) . This ability enables both the synthesis of new knowledge and the possibility to verify and update a given assertion . For example , given the following statement : The River Thames is in the United Kingdom .. and the following supporting text : London is the capital and most populous city of England and the United Kingdom . Standing on the River Thames in the south east of the island of Great Britain , London has been a major settlement for two millennia . a reader can verify that the statement is consistent since London is standing on the River Thames and London is in the United Kingdom . Automated reasoning applied on text requires Natural Language Processing ( NLP ) tools capable of extracting meaningful knowledge from free-form text and compiling it into KBs ( Niklaus et al. , 2018 ) . However , the compiled KBs tend to be incomplete , ambiguous , and noisy , impairing the application of standard deductive reasoners ( Huang et al. , 2005 ) . A rich and broad literature in MR has approached this problem within a variety of frameworks , including Natural Logic ( MacCartney & Manning , 2007 ; Angeli & Manning , 2014 ) and Semantic Parsing ( Dong & Lapata , 2016 ; Bos , 2008 ) , and by framing the problem as Natural Language Inference—also referred to as Recognising Textual Entailment ( Fyodorov et al. , 2000 ; Condoravdi et al. , 2003 ; Dagan et al. , 2005 ; Bowman et al. , 2015 ; Rocktäschel et al. , 2015 ) —and Question Answering ( Hermann et al. , 2015 ) . However , such methods suffer from several limitations . For instance , they rely on significant amounts of annotated data to suitably approximate the implicit distribution from which training and test data are drawn , and thus are often unable to generalise correctly in the absence of a sufficient quantity of training data or appropriate priors on model paramaters ( e.g . via regularisation ) . Orthogonally , even if accurate , such methods also can not provide explanations for a given prediction ( Evans & Grefenstette , 2018 ; Marcus , 2018 ) . A promising strategy for overcoming these issues consists of combining neural models and symbolic reasoning , given their complementary strengths and weaknesses ( Rocktäschel & Riedel , 2017 ; Evans & Grefenstette , 2018 ) . While symbolic models can generalise well from a small number of examples when the problem domain fits the inductive biases presented by the symbolic system at hand , they are brittle and prone to failure when the observations are noisy and ambiguous , or when the domain ’ s properties are not known or formalisable , all of which being the case for natural language ( Raedt & Kersting , 2008 ) . On the other hand , neural models are robust to noise and ambiguity but prone to overfitting ( Marcus , 2018 ) and not easily interpretable ( Lipton , 2018 ) , making them incapable of providing explanations or incorporating background knowledge . Recent work in neuro-symbolic systems ( Garcez et al. , 2015 ) has made progress in learning neural representations that allow for comparison of symbols not on the basis of identity , but of their semantics ( as learned in continuous representations of said symbols ) , while maintaining interpretability and generalisation , thereby inheriting the best of both worlds . Among such systems , NTPs ( Rocktäschel & Riedel , 2017 ) are end-to-end differentiable deductive reasoners based on Prolog ’ s backward chaining algorithm , where unification between atoms is replaced by a differentiable operator computing their similarity between their embedding representations . NTPs are especially interesting since they allow learning interpretable rules from data , by back-propagating a KB reconstruction error to the rule representations . Furthermore , NTPs are explainable : by looking at the proof tree associated with the highest proof score , it is possible to know which rules are activated during the reasoning process—this enables providing explanations for a given reasoning outcome , performing error analysis , and driving modelling choices . So far , due to their computational complexity , NTPs have only been successfully applied to learning tasks involving very small KBs . However , most human knowledge is still stored in large KBs and natural language corpora , which are difficult to reason over automatically . With this paper we aim at addressing these issues , by proposing : a ) An efficient method for significantly reducing the time and space complexity required by NTPs by reducing the number of candidate proof scores and by using an attention mechanism for reducing the number of parameters required for learning new rules ( Section 3 ) , and b ) An extension of NTPs towards text , by jointly embedding predicates and textual surface patterns in a shared embedding space by means of an efficient reading component ( Section 4 ) . 2 NEURAL THEOREM PROVING . Mimicking backward chaining , NTPs recursively build a neural network enumerating all the possible proof states for proving a goal over the KB . NTPs rely on three modules for building this neural network ; the Unification module , which compares subsymbolic representations of symbols , and mutually recursive OR and AND modules , which jointly enumerate all the proof paths , before the final aggregation choses the single , highest scoring state . We briefly overview these modules and the training procedure in the following . Unification Module . In backward chaining , unification is the operator that matches two atoms like locatedIn ( LONDON , UK ) and situatedIn ( X , Y ) . Discrete unification checks for equality between the elements of the atom ( e.g . locatedIn 6= situatedIn ) and binds variables to symbols via substitution ( e.g . { X/LONDON , Y/UK } ) . Unification in NTPs matches two atoms by comparing their embedding representations via a differentiable similarity function , which enables matching different symbols with similar semantics , such as locatedIn and situatedIn . The unifyθ ( H , G , S ) operator creates a neural network module that does exactly that . Given two atoms H = [ locatedIn , LONDON , UK ] and G = [ situatedIn , X , Y ] , and a proof state S = ( Sψ , Sρ ) consisting of a set of substitutions Sψ and a proof score Sρ , the unify module compares the embedding representations θlocatedIn : and θsituatedIn : with a Radial Basis Function ( RBF ) kernel k , updates the variable binding substitution set S′ψ = Sψ ∪ { X/LONDON , Y/UK } , and calculates the new proof score S′ρ = min ( Sρ , k ( θlocatedIn : ,θsituatedIn : ) ) . The resulting proof state S′ = ( S′ψ , S ′ ρ ) is further expanded with the or and and modules . OR Module . The or module unifies the goal with all the facts and rules in a KB . Concretely , for each rule H : – B in KB K , orKθ ( G , d , S ) unifies the goal G with the rule head H , and invokes the and module to prove atoms in the body B of the rule H , keeping track of the maximum proof depth d. For example , given a goal G = [ situatedIn , Q , UK ] , a rule H : – B with H = [ locatedIn , X , Y ] and B = [ [ locatedIn , X , Z ] , [ locatedIn , Z , Y ] ] , the model would unify the goal G with the head H of the rule , and instantiate and modules , to prove sub-goals in the body G of the rule . Note that each fact F can be represented as a rule F : – [ ] with no body atoms . AND Module . The and module recursively tries to prove a list of sub-goals for a rule body . Concretely , given the first sub-goal G and the following sub-goals G , the andKθ ( G : G , d , S ) module will substitute variables in G with constants according to the substitutions in S , and invoke the or module on G. The resulting state be used to prove the atoms in G , by recursively invoking the and module . For example , when invoked on the rule body B mentioned above , the and module will first substitute variables with constants for the sub-goal [ locatedIn , X , Z ] and invoke the or module , whose resulting state will be the basis of the next invocation of and module on [ locatedIn , Z , Y ] . Proof Aggregation . After building a neural network that enumerates all the proof paths of the goal G on a KB K , NTPs select the proof path with the maximum proof score : ntpKθ ( G , d ) = argmax S ∈ orKθ ( G , d , ( ∅,1 ) ) S 6=FAIL Sρ where d is a predefined maximum proof depth . The initial proof state is set to ( ∅ , 1 ) , an empty substitution set , and a proof score of 1 . Training In NTPs , predicate and constant embeddings are learned by optimising a cross-entropy loss on the final proof score , by iteratively masking facts in the KB and trying to prove them using available facts and rules ( Rocktäschel & Riedel , 2017 ) . Negative examples are sampled from the positive ones by corrupting the entities ( Nickel et al. , 2016 ) . Other than learning embeddings of predicates and constants , NTPs can also learn interpretable rules from data . Rocktäschel & Riedel ( 2017 ) show that it is possible to learn rules from data by specifying rule templates H : – B , with H = [ θp : , X , Y ] and B = [ [ θq : , X , Z ] , [ θr : , Z , Y ] ] , where θp : ,θq : ,θr : ∈ Rk are free parameters . Note that θp : ,θq : ,θr : can be learned from data , and decoded by searching the closest representation of known predicates .

Summary: This paper scales NTPs by using approximate nearest neighbour search over facts and rules during unification. Additionally, the paper incorporates mentions as additional facts where the predicate is the text that the entities of the mention are contained in. The paper also suggests parameterizing predicates using attention over known predicates. The increments presented are reasonable and justified, but the experimental results, specifically on the larger datasets, warrant further investigation.