Learn to Explain Efficiently via Neural Logic Inductive Learning

1 INTRODUCTION The recent years have witnessed the growing success of deep learning models in a wide range of applications . However , these models are also criticized for the lack of interpretability in its behavior and decision making process ( Lipton , 2016 ; Mittelstadt et al. , 2019 ) , and for being data-hungry . The ability to explain its decision is essential for developing a responsible and robust decision system ( Guidotti et al. , 2019 ) . On the other hand , logic programming methods , in the form of first-order logic ( FOL ) , are capable of discovering and representing knowledge in explicit symbolic structure that can be understood and examined by human ( Evans & Grefenstette , 2018 ) . In this paper , we investigate the learning to explain problem in the scope of inductive logic programming ( ILP ) which seeks to learn first-order logic rules that explain the data . Traditional ILP methods ( Galárraga et al. , 2015 ) rely on hard matching and discrete logic for rule search which is not tolerant for ambiguous and noisy data ( Evans & Grefenstette , 2018 ) . A number of works are proposed for developing differentiable ILP models that combine the strength of neural and logicbased computation ( Evans & Grefenstette , 2018 ; Campero et al. , 2018 ; Rocktäschel & Riedel , 2017 ; Payani & Fekri , 2019 ; Dong et al. , 2019 ) . Methods such as ∂ILP ( Evans & Grefenstette , 2018 ) are referred to as forward-chaining methods . It constructs rules using a set of pre-defined templates and evaluates them by applying the rule on background data multiple times to deduce new facts that lie in the held-out set ( related works available at Appendix A ) . However , general ILP problem involves several steps that are NP-hard : ( i ) the rule search space grows exponentially in the length of the rule ; ( ii ) assigning the logic variables to be shared by predicates grows exponentially in the number of arguments , which we refer as variable binding problem ; ( iii ) the number of rule instantiations needed for formula evaluation grows exponentially in the size of data . To alleviate these complexities , most works have limited the search length to within 3 and resort to template-based variable assignments , limiting the expressiveness of the learned rules ( detailed discussion available at Appendix B ) . Still , most of the works are limited in small scale problems with less than 10 relations and 1K entities . On the other hand , multi-hop reasoning methods ( Guu et al. , 2015 ; Lao & Cohen , 2010 ; Lin et al. , 2015 ; Gardner & Mitchell , 2015 ; Das et al. , 2016 ) are proposed for the knowledge base ( KB ) completion task . Methods such as NeuralLP ( Yang et al. , 2017 ) can answer the KB queries by searching for a relational path that leads from the subject to the object . These methods can be interpreted in the ILP domain where the learned relational path is equivalent to a chain-like first-order rule . Compared to the template-based counterparts , methods such as NeuralLP is highly efficient in variable binding and rule evaluation . However , they are limited in two aspects : ( i ) the chain-like rules represent a subset of the Horn clauses , and are limited in expressing complex rules such as those shown in Figure 1 ; ( ii ) the relational path is generated while conditioning on the specific query , meaning that the learned rule is only valid for the current query . This makes it difficult to learn rules that are globally consistent in the KB , which is an important aspect of a good explanation . In this work , we propose Neural Logic Inductive Learning ( NLIL ) , a differentiable ILP method that extends the multi-hop reasoning framework for general ILP problem . NLIL is highly efficient and expressive . We propose a divide-and-conquer strategy and decompose the search space into 3 subspaces in a hierarchy , where each of them can be searched efficiently using attentions . This enables us to search for x10 times longer rules while remaining x3 times faster than the state-of-theart methods . We maintain the global consistency of rules by splitting the training into rule generation and rule evaluation phase , where the former is only conditioned on the predicate type that is shared globally . And more importantly , we show that a scalable ILP method is widely applicable for model explanations in supervised learning scenario . We apply NLIL on Visual Genome ( Krishna et al. , 2016 ) dataset for learning explanations for 150 object classes over 1M entities . We demonstrate that the learned rules , while maintaining the interpretability , have comparable predictive power as densely supervised models , and generalize well with less than 1 % of the data . 2 PRELIMINARIES . Supervised learning typically involves learning classifiers that map an object from its input space to a score between 0 and 1 . How can one explain the outcome of a classifier ? Recent works on interpretability focus on generating heatmaps or attention that self-explains a classifier ( Ribeiro et al. , 2016 ; Chen et al. , 2018 ; Olah et al. , 2018 ) . We argue that a more effective and humanintelligent explanation is through the description of the connection with other classifiers . For example , consider an object detector with classifiers Person ( X ) , Car ( X ) , Clothing ( X ) and Inside ( X , X ′ ) that detects if certain region contains a person , a car , a clothing or is inside another region , respectively . To explain why a person is present , one can leverage its connection with other attributes , such as “ X is a person if it ’ s inside a car and wearing clothing ” , as shown in Figure 1 . This intuition draws a close connection to a longstanding problem of first-order logic literature , i.e . Inductive Logic Programming ( ILP ) . 2.1 INDUCTIVE LOGIC PROGRAMMING . A typical first-order logic system consists of 3 components : entity , predicate and formula . Entities are objects x ∈ X . For example , for a given image , a certain region is an entity x , and the set of all possible regions is X . Predicates are functions that map entities to 0 or 1 , for example Person : x 7→ { 0 , 1 } , x ∈ X . Classifiers can be seen as soft predicates . Predicates can take multiple arguments , e.g . Inside is a predicate with 2 inputs . The number of arguments is referred to as the arity . Atom is a predicate symbol applied to a logic variable , e.g . Person ( X ) and Inside ( X , X ′ ) . A logic variable such as X can be instantiated into any object in X . A first-order logic ( FOL ) formula is a combination of atoms using logical operations { ∧ , ∨ , ¬ } which correspond to logic and , or and not respectively . Given a set of predicates P = { P1 , ... , PK } , we define the explanation of a predicate Pk as a first-order logic entailment ∀X , X ′ ∃Y1 , Y2 ... Pk ( X , X ′ ) ← A ( X , X ′ , Y1 , Y2 ... ) , ( 1 ) where Pk ( X , X ′ ) is the head of the entailment , and it will become Pk ( X ) if it is a unary predicate . A is defined as the rule body and is a general formula , e.g . conjunction normal form ( CNF ) , that is made of atoms with predicate symbols from P and logic variables that are either head variables X , X ′ or one of the body variables Y = { Y1 , Y2 , ... } . By using the logic variables , the explanation becomes transferrable as it represents the “ lifted ” knowledge that does not depend on the specific data . It can be easily interpreted . For example , Person ( X ) ← Inside ( X , Y1 ) ∧ Car ( Y1 ) ∧ On ( Y2 , X ) ∧ Clothing ( Y2 ) ( 2 ) represents the knowledge that “ if an object is inside the car with clothing on it , then it ’ s a person ” . To evaluate a formula on the actual data , one grounds the formula by instantiating all the variables into objects . For example , in Figure 1 , Eq . ( 2 ) is applied to the specific regions of an image . Given a relational knowledge base ( KB ) that consists of a set of facts { 〈xi , Pi , x′i〉 } Ni=1 wherePi ∈ P and xi , x′i ∈ X . The task of learning FOL rules in the form of Eq . ( 1 ) that entail target predicate P ∗ ∈ P is called inductive logic programming . For simplicity , we consider unary and binary predicates for the following contents , but this definition can be extended to predicates with higher arity as well . 2.2 MUTLI-HOP REASONING . The ILP problem is closely related to the multi-hop reasoning task on the knowledge graph ( Guu et al. , 2015 ; Lao & Cohen , 2010 ; Lin et al. , 2015 ; Gardner & Mitchell , 2015 ; Das et al. , 2016 ) . Similar to ILP , the task operates on a KB that consists of a set of predicates P . Here the facts are stored with respect to the predicate Pk which is represented as a binary matrix Mk in { 0 , 1 } |X |×|X| . This is an adjacency matrix , meaning that 〈xi , Pk , xj〉 is in the KB if and only if the ( i , j ) entry of Mk is 1 . Given a query q = 〈x , P ∗ , x′〉 . The task is to find a relational path x P ( 1 ) −−−→ ... P ( T ) −−−→ x′ , such that the two query entities are connected . Formally , let vx be the one-hot encoding of object x with dimension of |X | . Then , the ( t ) th hop of the reasoning along the path is represented as v ( 0 ) = vx , v ( t ) = M ( t ) v ( t−1 ) , where M ( t ) is the adjacency matrix of the predicate used in ( t ) th hop . The v ( t ) is the path features vector , where the jth element v ( t ) j counts the number of unique paths from x to xj ( Guu et al. , 2015 ) . After T steps of reasoning , the score of the query is computed as score ( x , x′ ) = v > x′ T∏ t=1 M ( t ) · vx . ( 3 ) For each q , the goal is to ( i ) find an appropriate T and ( ii ) for each t ∈ [ 1 , 2 , ... , T ] , find the appropriate M ( t ) to multiply , such that Eq . ( 3 ) is maximized . These two discrete picks can be relaxed as learning the weighted sum of scores from all possible paths , and weighted sum of matrices at each step . Let κ ( sψ , Sϕ ) ≡ T∑ t′=1 s ( t′ ) ψ t′∏ t=1 K∑ k=1 s ( t ) ϕ , k Mk ( 4 ) be the soft path selection function parameterized by ( i ) the path attention vector sψ = [ s ( 1 ) ψ , ... , s ( T ) ψ ] > that softly picks the best path with length between 1 to T that answers the query , and ( ii ) the operator attention vectors Sϕ = [ s ( 1 ) ϕ , ... , s ( T ) ϕ ] > , where s ( t ) ϕ softly picks the M ( t ) at ( t ) th step . Here we omit the dependence on Mk for notation clarity . These two attentions are generated with a model sψ , Sϕ = T ( x ; w ) ( 5 ) with learnable parameters w. For methods such as ( Guu et al. , 2015 ; Lao & Cohen , 2010 ) , T ( x ; w ) is a random walk sampler which generates one-hot vectors that simulate the random walk on the graph starting from x . And in NeuralLP ( Yang et al. , 2017 ) , T ( x ; w ) is an RNN controller that generates a sequence of normalized attention vectors with vx as the initial input . Therefore , the objective is defined as arg max w ∑ q v > x′κ ( sψ , Sϕ ) vx , ( 6 ) Learning the relational path in the multi-hop reasoning can be interpreted as solving an ILP problem with chain-like FOL rules ( Yang et al. , 2017 ) P ∗ ( X , X ′ ) ← P ( 1 ) ( X , Y1 ) ∧ P ( 2 ) ( Y1 , Y2 ) ∧ ... ∧ P ( T ) ( Yn−1 , X ′ ) . Compared to the template-based ILP methods such as ∂ILP , this class of methods is efficient in rule exploration and evaluation . However , ( P1 ) generating explanations for supervised models puts a high demand on the rule expressiveness . The chain-like rule space is limited in its expressive power because it represents a constrained subspace of the Horn clauses rule space . For example , Eq . ( 2 ) is a Horn clause and is not chain-like . And the ability to efficiently search beyond the chain-like rule space is still lacking in these methods . On the other hand , ( P2 ) the attention generator T ( x ; w ) is dependent on x , the subject of a specific query q , meaning that the explanation generated for target P ∗ can vary from query to query . This makes it difficult to learn FOL rules that are globally consistent in the KB .

Summary: This paper presents a model for effectively hierarchically ‘searching’ through the space of (continuously relaxed) FOL formulas that explain the underlying dataset. The model presented employs a three-level architecture to produce logic entailment formulas, skolemized with a set of skolem functions, i.e. ‘operators’. The three-level search, which is implemented via a stack of transformers, first searches through a space of variable and predicate embeddings to select operators, after which it searches through the space of predicates to form primitive statements of predicates and operators, and finally it generates a number of formulas from the previously ‘selected’ primitive statements. The model is applied on a toy task to showcase its speed, a knowledge base completion task, and modeling the visual genome dataset, where the model shows the ability to induce meaningful rules that operate on visual inputs. The presented benefit of the model is scalability, ability to induce explainable rules, and the ability to induce full FOL rules.