R5: Rule Discovery with Reinforced and Recurrent Relational Reasoning

1 INTRODUCTION . While deep learning has achieved great success in various applications , it was pointed out that there is a debate over the problem of systematicity in connectionist models ( Fodor & Pylyshyn , 1988 ; Fodor & McLaughlin , 1990 ; Hadley , 1994 ; Jansen & Watter , 2012 ; Dong et al. , 2018 ) . To concretely explain systematicity ( Hupkes et al. , 2020 ) , let us consider the kinship problem shown in Figure 1 . By training on the small-scale observed examples of family relationships where the rule paths are short ( in Figure 1a & b , both of the rule paths have the length of 2 ) , our objective is to extract the underlying rules as general principles and generalize to large-scale tasks where the rule paths are long ( in Figure 1c , the rule path has the length of 3 ) . The rules in the training examples are a ) Mother ( X , Y ) ←− Mother ( X , Z ) , Sister ( Z , Y ) , b ) Grandma ( X , Y ) ←− Mother ( X , Z ) , Father ( Z , Y ) . And the necessary rule in order to conclude the relation between Mary and Ann is c ) Grandma ( X , Y ) ←− Mother ( X , U ) , Sister ( U , Z ) , Father ( Z , Y ) , which does not explicitly show up in the training examples . Successfully giving prediction in c ) shows the ability of systematicity , i.e. , model ’ s ability to recombine known parts and rules to form new sequences . Some attempts on the graph relational data grounded in Graph Neural Networks ( Schlichtkrull et al. , 2018 ; Sinha et al. , 2020 ; Pan et al. , 2020 ) rely on learning embedding vectors of entities and relations , and have shown some capability of systematicity , However , since the rules are implicitly encapsulated in the neural networks , these models are lack of interpretability . Inductive Logic Programming ( ILP ) ( Muggleton , 1992 ; Muggleton et al. , 1996 ; Getoor , 2000 ; Yang et al. , 2017 ; Evans & Grefenstette , 2018 ) is a subfield of symbolic rule learning , which naturally learns symbolic rules and provides interpretable explanations for labels prediction as well as being able to generalize to other tasks . A traditional ILP system learns a set of rules from a collection of positive and negative examples , which entails all the positives and none of the negatives . However , these methods face the challenge that the search space of the compositional rules is exponentially large , making it hard to scale beyond small rule sets . Extending the idea of ILP , Neural-Symbolic Learning ( Garcez et al. , 2015 ; Besold et al. , 2017 ) seeks to integrate principles from neural networks learning and logical reasoning , among which Neural Logic Machines ( Dong et al. , 2018 ; Zimmer et al. , 2021 ) and Theorem Provers ( Rocktäschel & Riedel , 2017 ; Minervini et al. , 2020a ; b ) aim to overcome the problems of systematicity . On one hand , the Neural Logic Machines approximate the logic predicates using a probabilistic tensor representation , and the conclusion tensors are generated by applying the neural operators over the premise tensors . These approaches have done a great job in reasoning the decision-making tasks including sorting tasks , finding the shortest paths , and so on . However , in terms of the relational reasoning tasks on graphs , Neural Logic Machines reason only the single relation prediction tasks , i.e . determining whether a single relation exists between all the queried entities or entity pairs . On the other hand , Theorem Provers jointly learn representations and rules from data via backpropagation , given a pre-defined task-specific rule template and a set of possible rules following the template . Theorem Provers are able to solve and reason the relation prediction tasks instead of only determining the existence of a single relation . However , the performance is not satisfying . In this paper , we focus on model ’ s systematicity for relational reasoning over graph relational prediction tasks and present a new reasoning framework named R5 , i.e. , Rule discovery with Reinforced and Recurrent Relational Reasoning , for rule induction and reasoning on relational data with strong systematicity . R5 formulates the problem of relational reasoning from the perspective of sequential decision-making , and performs rule extraction and logical reasoning with deep reinforcement learning equipped with a dynamic rule memory . More concretely , R5 learns the short definite clauses that are in the form u ←− pi ∧ pj . Since long Horn clauses ( Horn , 1951 ) can be decomposed into short Horn clauses , a long definite clause outcome ←− p0 ∧ p1 ... pi ∧ pj ... pk used in prediction tasks is represented with a sequence of short definite clauses while performing decision-making , i.e . pi ∧ pj is replaced by u . Specifically , we make the following contributions : First , R5 performs explicit reasoning for relation prediction via composition and achieves explainability . Instead of learning embeddings for entities and relations , ( Hamilton et al. , 2017 ; Schlichtkrull et al. , 2018 ; Wang et al. , 2019 ; Pan et al. , 2020 ) and performing implicit reasoning , we perform explicit relational reasoning by modeling it as sequential decision-making . Specifically , given a query ( usually consists of two queried entities ) and the related relationship graph , the agent recurrently selects a relation pair ri , rj from the input graph to combine into a compound relation rk and updates the graph , until the target relation for the query is reached . Trained by reinforcement learning , the agent learns to take actions , i.e. , which pair of relations to combine , given the state representations of all possible pairs of relations . Second , we propose a dynamic rule memory module to maintain and score candidate rules during the training of the reasoning agent . Each item in the rule memory is a candidate rule in the format of rk ←− ri , rj . ri , rj serve as the key , and rk serves as the value . In each step of decisionmaking , the agent queries the rule memory to re-use already stored rules for reasoning , i.e. , deducing compound relations , or insert new candidate rules into it if no existing rule matches the agent ’ s action . Each rule is associated with a score that indicates the confidence for the rule . The rules and their scores in the memory are dynamically updated during training . Finally , a set of candidate rules with scores above a threshold are kept . By recurrently applying the learned rules , R5 demonstrates strong compositional generalization ability ( i.e . systematicity ) in relational reasoning tasks . Third , rather than only taking the observed relations into account , we introduce extra hidden ( or invented ) relations into the reasoning process . For example , R5 may learn to combine r1 and r2 by a rule r ← r1 , r2 , where r is an intentionally introduced unknown relation . The invented relations can appear on both the left or right side of a rule clause . Such a design enables our model to learn intermediate relations that do not explicitly appear in the training data to model underlying complex relations between entities , which are sometimes necessary to discover complex rules in the relational data . Furthermore , we design a backtrack rewriting mechanism that replaces an invented relation with an observed one when R5 finds they are actually equivalent . Our goal is to explain the observed data with rules that are as simple as possible . Backtrack rewriting merges redundant relations and rules to enforce the principle of Occam ’ s razor , which prevents overfitting and improves the robustness of our approach . We perform extensive evaluations based on two public relation prediction datasets , CLUTRR ( Sinha et al. , 2019 ) and GraphLog ( Sinha et al. , 2020 ) , and compare R5 with a variety of baseline methods . The experimental results demonstrate that our approach significantly outperforms state-of-the-art methods in terms of relation prediction accuracy and recall rate in rule discovery . Moreover , R5 exhibits a strong ability of compositional generalization and robustness to data noise . 2 PROPOSED APPROACH Table 1 : Notations M The set of known relations types N The set of invented relations types r a relation type qj a query L number of paths to sample for each query Lj actual number of paths for qj ai The ith action in in the sequence aB , i The ith relation in a ’ s body aH a ’ s head si The ith current state in in the sequence zi The ith reward in in the sequence Drl Dictionary of rules Drls Dictionary of rules scores c an entry in Drls Bunkn Buffer of unused invented relation types vi a score value In this section , we introduce our Rule Discovery with Reinforced and Recurrent Relational Reasoning framework , namely R5 , to solve the inductive relation prediction problem . We first formally define the relation prediction problem discussed in this paper . Let pdata ( G , q , a ) be a training data distribution , where G is the set of training graphs , q = ( X , Y ) is a query , and a = r is the answer . The graphs consist of nodes in a set N , which are connected by relations in a set R. X , Y ∈ N are nodes , and r ∈ R is a relation . Given G and the query q , the goal is to predict the correct answer a. Relational prediction task is actually program induction , which is a hard problem that has been studied for many years in the area of Inductive Logic Programming and Statistical Relational Learning , especially for large-scale tasks in noisy domains . Figure 2 shows an overview of R5 . Our framework takes a relational graph as input , and outputs the relationship between two queried entities based on extracted rules . R5 first transforms a relation graph into a set of paths connecting the queried node ( entity ) pair . After that , it recurrently applies learned rules to merge a relation pair in the paths to form a compound relation , until it outputs a final relation between the two queried nodes . The reasoning agent is trained with deep reinforcement learning and MCTS , while a dynamic rule memory module is utilized to extract rules from observations during training . The notations we used in this paper are summarized in Table 1 . Note that an action a : = aH ← ( aB,0 , aB,1 ) and a rule rl : = rlH ← ( rlB,0 , rlB,1 ) share the same structure , where different a and rl are all relations . aH and aB represent the head and body in a rule , respectively . Next we introduce R5 in detail . 2.1 RECURRENT RELATIONAL REASONING . Path sampling . To predict the relation between two entities in graphs , we preprocess the data by sampling paths that connect the two entities from the graph . When the total number of paths is small , we enumerate all the paths . Otherwise , we randomly sample L paths at maximum . Each path purely consists of relations . For instance , in Figure 2 , given the input graph , we can get 4 paths between query nodes X and Y , including r1-r2-r5-r3-r7-r6 and so on . Reasoning as sequential decision-making . Our method solves the problem of relational reasoning in terms of sequential decision-making . Specifically , we train a reasoning agent based on a policy value network combined with MCTS ( Silver et al. , 2017a ) to recurrently reason over the extracted relation paths . The policy value network fθ ( s ) is a neural network with parameters θ . It takes the current state s as input , and outputs the action probability distribution ρ and a state value ν. MCTS utilizes the policy network fθ ( s ) to guide its simulations , and outputs a vector π representing the improved search probabilities of the available actions . The policy network ( ρ , ν ) = fθ ( s ) is then trained to minimize the error between the predicted state value ν and the reward z received after an episode , as well as maximize the similarity between two probability distributions ρ and π . The loss function l is l = ( z − ν ) 2 − π⊺ logρ+ a ∥θ∥2 , where a is a hyper-parameter . Action . At each step of an episode , the MCTS outputs an action ( aB,0 , aB,1 ) , which is a relation pair denoted as aB . Furthermore , by looking up the dynamic rule memory , it obtains a : aH ← ( aB,0 , aB,1 ) that contains a head relation aH to be deducted to , which means that the relation pair ( aB,0 , aB,1 ) in the path will be substitute with aH . For example , in Figure 2 at step 1 , MCTS outputs an action a : r3 ← ( r1 , r2 ) , and the path r1-r2-r5-r3-r7-r6 is transformed into r3-r5-r3-r7-r6 . By recurrently applying different actions to a path between the query nodes , it will be transformed into a single relation at the end of an episode . State . Instead of encoding the walking paths between the query nodes as other RL-based rule induction methods ( Shen et al. , 2018 ; Das et al. , 2017 ; Xiong et al. , 2017 ) , we make use of the features of the possible relations pairs at the current state . As shown in Figure 2 , we define the state s ∈ R ( m+n ) × ( m+n ) ×k as the representation of all the possible relations pairs among the current paths , where m is the number of observed relations , n is the number of invented relations we assumed , and k is the dimension of features of each relation pair . As previously discussed , a relation pair may be deducted to an invented relation r∗ ∈ N . Besides , r∗ can be paired with another relation r , and maybe further deducted to another invented relation r′∗ ∈ N . Our state space design enables us to represent such complicated rules with the invented relations that serve as intermediate predicates , which is essential to model complex relational data . n invented relations are allowed in our design , where n depends on the complexity of input . An example is presented in Appendix A.4 . Even if sometimes the input data can be explained by rules without invented relation , the invented relations in our state design can actually help to speed up model training . For example , when we observe that a relation pair r1 and r2 shows up frequently in the training data , we can sample an invented relation r∗ as the head , and form a candidate rule r∗ ← ( r1 , r2 ) . In this way , our model learns to merge r1 and r2 quickly without the need to figure out what exactly r∗ is . The value of r∗ will be inferred by our model with Backtrack Rewriting mechanism , which we will introduce in more detail later . Without these invented relations , a reasoning agent has to infer what exactly r∗ is at an early state , otherwise it can not proceed to merge the relation pairs in a path correctly . Thus , the invented relations actually serve as a buffering mechanism to allow an agent to learn “ what to merge ” first , rather than inferring the exact rule at an early stage . The features we utilized to represent each relation pair can be summarized into two groups . The first group includes the general and statistical features of the pair , such as the number of occurrences , the index of the most viewed position among the paths , the number of occurrences at the most viewed position , and the types of the two relations consist the pair . The other group includes the rulesrelated features , such as whether the relation pair is in the obtained rules , whether the relation pair consists of two known relations , whether the rule head is a known relation if the pair is in the rules memory , and the score of the corresponding rule if the pair is in the rules memory . Reward . During an episode , actions are applied to a relation path recurrently , until the path is deducted to a single relation r , and a reward zT ∈ { −1 , 0 , +1 } will be assigned to all the states in this episode . If r is an known relation , but is not the target relation , zT = −1 . If r is an invented relation , zT = 0 . If r is the target relation , zT = 1 .

Summary: This paper presents a novel method for rule induction. The main idea is to apply reinforcement learning in the task of relational pathfinding within a finite Herbrand base. The reinforcement learner uses MCTS to find the best routes to establish the path in between the two arguments of training examples, while the useful actions (which are length-two relational paths) are maintained in a hash table as the induced ruleset. Experimental results has shown the effectiveness of the proposed method.