Learning Symbolic Rules over Abstract Meaning Representations for Textual Reinforcement Learning
Year: 2023

Abstract:
Text-based reinforcement learning agents have predominantly been neural network-based models with embeddings-based representation, learning uninterpretable policies that often do not generalize well to unseen games. On the other hand, neuro-symbolic methods, specifically those that leverage an intermediate formal representation, are gaining significant attention in language understanding tasks. This is because of their advantages ranging from inherent interpretability, the lesser requirement of training data, and being generalizable in scenarios with unseen data. Therefore, in this paper, we propose a modular, NEuro-Symbolic Textual Agent (NESTA) that combines a generic semantic parser with a rule induction system to learn abstract interpretable rules as policies. Our experiments on established textbased game benchmarks show that the proposed NESTA method outperforms deep reinforcement learning-based techniques by achieving better generalization to unseen test games and learning from fewer training interactions.

text: Our main contributions in this work are: (1) We propose a novel and modular neuro-symbolic agent named NESTA. To the best of our knowledge, NESTA is the first to use a generic semantic parser with a rule learning system for TBGs, (2) Our empirical analysis of commonsense-aware textworld games shows that NESTA outperforms deep RL methods by a significant margin. We also show that NESTA has better sample efficiency compared to traditional text-based RL agents obtaining better test performance with up to 5× lesser training interactions, and (3) Our method produces interpretable abstract rules from the rule induction system.

heading: 2 NEuro-Symbolic Textual Agent
text: Text-based RL agents for TBGs interact with the environment using text-only action commands and obtain feedback solely as textual observations. As the agent does not have access to global state information, it is modeled as a Partially Observable Markov Decision Process (POMDP) (Kaelbling et al., 1998) represented as (S,A, T , R,Ω,O), where (S,A, T , R) represent a Markov Decision Process. Ω represents the finite set of all observations, and O represents the observation function representing the conditional distribution over observations for a given action and next state. The goal of the agent is to learn optimal action probabilities
AMR-to-triples: We design an AMR-to-triples
module to extract a set of symbolic facts consisting of generic domain-agnostic predicates from the AMR semantic representation. Fig. 2 shows the extraction of facts from AMR. The AMR-totriples module performs a set of graph operations to extract propbank nodes as the predicates and the children entities as the arguments. In the example, the two operands of the “and” node are converted into two symbolic facts with be-located-at predicate with two separate entities of “brown golf shoe” and “blue moccasins”. We convert the symbolic facts to unary predicates. For example, we convert the be-located-at predicate with two arguments into single argument facts. These simplifications result in some loss of representational power but make the task of rule learning simpler. We also add the commonsense predicates from conceptnet subgraph (Speer et al., 2017) provided by the TWC environment (Murugesan et al., 2020).

heading: 2.2 Rule Learner: ILP from Rewards
text: In order to learn interpretable rules that can be debugged by humans, we use the symbolic representation obtained from the above step. Such symbolic rules are learned from reward signals by interacting with the environment. For this purpose, we use Inductive Logic Programming (ILP) in an RL setting with the objective of expected future reward maximization. We use Logical Neural Networks (LNN) as the differentiable rule learning engine.
Logical Neural Networks: LNN (Riegel et al., 2020) proposes a differentiable rule learning framework that retains the benefits of both neural networks and symbolic learners. It proposes a logi-
cal neuron that has the core properties of gradientbased learning similar to a standard neuron but adds logic-aware forward functions and constrained optimization making it suitable for logical operations. This can be illustrated on 2-input logical conjunction (AND) neuron with (x, y) as two logical inputs to the conjunction node. The LNN conjunction neuron generalizes the classical AND logic for realvalued logic by defining a noise threshold (α). The real-values in [α, 1] and [0, 1− α] signify a logical high and logical low respectively. To emulate an AND neuron, LNN uses the standard truth table of the conjunction (AND) gate to obtain the following constraints,
f(x, y) ≤ 1− α, ∀x, y ∈ [0, 1− α] f(x, y) ≤ 1− α, ∀x ∈ [0, 1− α], y ∈ [α, 1] f(x, y) ≤ 1− α, ∀x ∈ [α, 1], y ∈ [0, 1− α] f(x, y) ≥ α, ∀x, y ∈ [α, 1] .
LNN uses the forward function as the weighted Łukasiewicz t-norm, f(x, y;β,w1, w2) = β − w1(1 − x) − w2(1 − y), where β,w1, w2 are the bias and weights of the inputs. Given a target label, the weights and biases are tuned to learn the logical rule that best describes the data.
ILP-based reward maximization: Our ILP rule learner is based on the LNN rule learning implementation in Sen et al. (2022). However, our rulelearning model makes significant modifications to adapt the previous algorithm for model-free policy optimization suitable for text-based RL. Consider the state transition at time step t as (ot, at, rt, ot+1), where ot represents the textual observation, at is
the action command that yields the reward rt and takes the agent to the next state with observation ot+1. AMR-to-triples semantic parser is used to obtain the symbolic state st (list of symbolic facts) from ot as shown in Figure 2. At each step, the agent has to choose from a set of admissible action commands which are also converted to their symbolic form. Starting from an initial random policy π, we sample trajectories τ ∼ π and store the transitions (st, at, rt, st+1) in a buffer B. We also store the admissible actions set admt and the discounted future reward gt = ∑T k=t γ
k−trk, for each step in the buffer, where γ is the discount factor.
From the buffer B, we find a set of template predicates P = {p | p ∈ st, for st ∈ B}, where p ∈ st operation states whether facts with predicate p exist in the symbolic state st. We also obtain a set of action predicates A = {a | a ∈ admt, for admt ∈ B} finding all action predicates in the admissible action set. We initialize ILP rule learner πa(θ) for each action predicate a ∈ A. Action predicates for TBGs typically coincide with the action verbs. The LNN policy is formulated as a weighted conjunction operation over the template predicates P . The likelihood of action a for abstract lifted variables x, y is given as a conjunction template over the predicate list as follows: unary action likelihood is given as L(a(x)|st) = ∧ k wkpk(x) and binary action likelihood is formulated as L(a(x, y)|st) =∧ k wkpk(x) ∧ mwmqm(x, y). The predicates pk and qm are 1 and 2 arity predicates in P respectively and ∧ represents the LNN’s logical conjunction operator. The weights wk and wm constitute the LNN parameters θ that are updated during training. At any given step, the likelihood of each action is normalized over all actions in the admissible action set to obtain the action probabilities.
For training the rule learning model πa(θ) for a specific action a, we only extract transitions from the buffer containing the action a and store it in a sub-buffer Ba. The model is updated following the policy gradient loss, L = ∇θE(st,gt)∼Ba log(πa(at = a|st)gt, where the trajectories are sampled from Ba. We assume that πa gives normalized probabilities for this loss formulation. Therefore, this training procedure yields separate rules learned for each action predicate. Figure 2 shows the learned rules for each action.
Generalization under Distribution Shift: Having learned the action rules for each action predi-
cate using dedicated ILP models, NESTA uses the rules for obtaining the action probabilities at each step. This process consists of three steps: (a) For each action in the admissible action list, invoke the learned rule for that action predicate, (b) Assign the abstract variables with the symbolic action arguments, and (c) Match the symbolic facts using entity alignment by root noun matching (instead of an exact match). The probabilities are then obtained by the LNN conjunction node feed-forward operation based on the current weights. This procedure is also used for sampling during training.
Figure 2 shows the reasoning steps for fixed weights after training is complete. Since the rules learned by NESTA abstract out the entities in the form of lifted variables, human interpretability and generalization to unseen entities is a natural advantage of our method. In addition to this, since we modularize language understanding and RL policy learning into separate modules, our LNN symbolic learner can solely focus on optimal reward performance leading to sample-efficient learning.

heading: 2.3 Pruner: Irrelevant Action Pruning by Look-Ahead
text: The third module in NESTA, tackles the large action space problem in TBGs by removing actions from the admissible commands that do contribute to future rewards in the games. A large number of possible actions at each step can increase the branching factor of the agent at each step during the training and testing leading to a combinatorially large search problem. We employ a look-ahead strategy to find out which actions do not contribute to future reward accumulation. For example, the action examine(x) returns the description of the entity x, but does not change the state of the game and does not contribute to future rewards. However, for the action take(x), although an immediate reward is not obtained on execution, it leads to a future reward when the object x is put in the correct container y using the put(x, y) command. Therefore, the action command of type examine(x) can be pruned but take(x) is essential and hence cannot be pruned. This can be computed by looking ahead from the current step and comparing the future reward if that particular action was removed from the trajectory.
Due to AMR error propagation and undesirable credit assignment (for example, examine(x) command issued just before a rewarded action), the rule
learner can assign high action probabilities to noncontributing actions. Therefore, the pruner module is desirable to remove such action predicates (a) by evaluating the total reward in action trajectories with and without the particular action predicate a. More specifically, for each episodic trajectory, we remove the action predicates a and re-evaluate the episodic reward obtained from the environment. If the average episodic reward in both cases, with and without removal of a is the same then the action predicate is not contributing to the future reward. Therefore, it can be removed from the original action predicate set, A to obtain the pruned action set Apruned for which LNN models are learned.

heading: 3 Outlier Rejection in Policy Training
text: The training samples that NESTA collects from interacting with the environment can be noisy and this can affect learning a good policy. There exists two sources of noise: (a) AMR noise, where AMR incorrectly parses the surface text resulting in erroneous identification of entity extraction or relationships between entities, and (b) RL credit assignment noise, where discounted reward gives reward to a suboptimal action taken right before a correct action. Although symbolic reasoners have the advantages of learning from fewer data and better generalization, they are not robust to noise. We mitigate the effect of noise in LNN policy training by using a consensus-based noise rejection method.
Our noise rejection method trains the LNN Policy on multiple subsets of training data and selects the model with the smallest training error as the best model. The multiple subsets of training data are prepared as follows - for each training subset, a particular predicate p from the predicate list P is given priority. We only choose state transitions that contain the predicate p ensuring that this predicate will be part of the final learned rule, thus elimi-
nating the source of AMR noise for this predicate (such a subset is rejected if the number of such transitions is less than some threshold percentage). Subsequently, the resulting transitions are sorted by the discounted reward gt and we only retain the top first k% of this sorted data as training data. This encourages action transition with more immediate average reward gains to constitute the training data.

heading: 4 Experimental results
text: Our experiments are designed to answer these questions that analyze if NESTA can overcome the common drawbacks of deep RL methods: (i) Can NESTA enable better generalization in test environments? (ii) Does NESTA improve upon sample efficiency while still maintaining good reward performance, (iii) Are the rules learned by NESTA, human interpretable? For comparing the performance of various methods, we use the metrics of normalized score (total reward from the games normalized by maximum reward) and number of steps to reach the goal (lower is better). Our experiments were conducted on Ubuntu 18.04 operating system with NVidia V100 GPUs.

heading: 4.1 Environment
text: We use the textworld commonsense (TWC) environment (Murugesan et al., 2020) for empirical evaluation of our method. The goal here is to clean up a messy room by placing the objects in the correct containers. The game provides conceptnet sub-graphs relating the game entities which are used as commonsense graphs. TWC provides two splits of testing games: (i) in-distribution games that have the same entities as training games but unseen object-container configuration, and (ii) outof-distribution games that use new objects not seen during training. This provides a systematic framework for measuring generalization in NESTA and
other baseline agents for both within-training distribution and out-of-training distributions. Since we are focusing on generalization aspects, we do not use other textworld games (Côté et al., 2018; Hausknecht et al., 2020) because these environments primarily focus on the agent’s exploration strategies and are therefore not suitable to evaluate the agent’s generalization ability.

heading: 4.2 Agents
text: For baseline agents, we report performance by these deep RL-based methods: (1) Text-based agent that uses a GRU network for observation representation and action scorer units, (2) TWCAgent (Text + CS) that uses combined textual and commonsense embeddings for action scoring, (3) KG-A2C (Ammanabrolu and Hausknecht, 2020) that uses extracted knowledge graphs as input, (4) BiKE (Murugesan et al., 2021) which leverages graph structures in both textual and commonsense information and (5) CBR (Atzeni et al., 2021) which is the SOTA method using case-based reasoning for improving generalization in text-based agents. We did not compare with previous neurosymbolic methods (Kimura et al., 2021; Chaudhury et al., 2021) because they use a hand-crafted game-specific predicate design scheme that was not available for TWC.

heading: 4.3 Generalization to Test Games
text: We evaluate the generalization ability of NESTA on TWC easy, medium and hard games. Table 1 and Table 2 shows the performance of baseline and our agents on in-distribution and out-of-distribution games, including the human performance from Murugesan et al. (2020). For the baseline models, we report scores from Atzeni et al. (2021). For NESTA, we report the mean of 5 independent runs.
For easy games, NESTA gets a perfect score outperforming previous games with similar steps as human performance. For medium and hard games, NESTA greatly surpasses the SOTA agent and needs a lesser number of steps for both indistribution and out-of-distribution games. For medium out-of-distribution games, NESTA outperforms humans in terms of the number of steps. This might be due to the fact that during human annotation, the subjects would take a larger number of steps for the initial few games due to trial-and-error, thus increasing the average number of steps.
While easy and medium games have a singleroom setting, hard games present a two-room setting where the agent might require picking up an object in room 1 and putting it in a container in room 2. This requires learning a complex strategy especially for generalizing to unseen entities. Our method NESTA scores significantly higher compared to SOTA on hard games, thus exhibiting the ability of our method to generalize in complex settings while deep RL methods fail to generalize due to overfitting the training data. Furthermore, our outlier rejection model helps improve the number of steps to reach the goal for both in-distribution and out-of-distribution games.

heading: 4.4 Ablation Results with Action Pruning
text: To study the effect of our action pruning module on deep RL agents, we implemented action pruning on the publicly available TWCAgent code from Murugesan et al. (2020). We follow the exact same methodology for TWCAgent that we used for the NESTA agent. Using the look-ahead method, we obtain Aretain, the list of action verbs to retain at a specific episode (episode num 10 for this result). For all subsequent training steps, only action verbs a ∈ Aretain were retained from the admissible list.
We also follow the same strategy for the test games. Table 3 shows the results for action pruning for both TWCAgent and NESTA. Firstly, even without action pruning, NESTA outperforms the TWCAgent with action pruning. NESTA+AP shows a higher gain in performance compared to NESTA only, whereas TWCAgent did not exhibit such large improvements. We found that even without AP, TWCAgent learns to avoid sub-optimal actions. However, it suffers from overfitting and hence cannot generalize to unseen configurations and entities.

heading: 4.5 Human-in-the-loop Rule Debugging
text: NESTA enables the user to verify all the learned rules. It provides the facility to add new rules that might be missing or edit the rules if they are suboptimal. The ability of human-in-the-loop debugging is what sets NESTA apart from other methods that tend to provide some level of explainability. Table 4 shows the human-interpretable learned rules for a particular training on hard games. The rule for take(x, y) can be identified as sub-optimal because it implies that the agent should take any object that is present in a container y present in the current room. The human-corrected rule implies the agent should only “take” objects that are not in their assigned location according to conceptnet facts. The human-corrected rule perfectly solves the out-of-distribution hard games in close to the optimal number of steps. This demonstrates that NESTA’s human-in-the-loop rule debugging feature can be readily used to achieve favorable performance gains.

heading: 4.6 Sample efficient learning
text: We hypothesize that deep RL policies require a large number of training interactions because they learn both language understanding and action scoring from rewards ignoring external language pretraining. NESTA, on the other hand, decouples language understanding to AMR-based semantic representations while the LNN-ILP rule learner can focus on RL policy optimization resulting in learning from fewer samples. Figure 3 shows that the NESTA model obtains better scores for both in-distribution and out-distribution games at much fewer training interactions compared to the deep RL text agent. In fact, NESTA can outperform text agents even when it learns from 5× lesser training interactions.
We also computed computational time for NESTA compared to neural agents. Average computational times (out-of-distribution) required for each step for NESTA compared to neural agents. For easy games, the average computation time for neural agents was 0.12 ± 0.06 s, and that for NESTA was 0.16± 0.05. The corresponding numbers for medium games were 0.17 ± 0.06 and 0.22 ± 0.06 respectively. NESTA requires extra time due to parsing. However, since it has a lower overall number of steps (almost 5 times lower for easy/medium games from Table 2), time per game would be lower or comparable.

heading: 5 Related Work
text: Text-only Agents: Early work on text-based reinforcement learning agents used an LSTM-
based representation learning from textual observations (Narasimhan et al., 2015b), and Q-learning (Watkins and Dayan, 1992) in the action scorer of LSTM-DQN to assign probability scores to the possible actions. Yuan et al. (2018) used LSTM units in the action scorer of LSTM-DRQN to handle the better generalization. Chaudhury et al. (2020) further improved generalization and reduced overfitting by training a bootstrapped model, named CREST, on context-relevant observation text. Adolphs and Hofmann (2020) presented one of the winning strategies in the First-TextWorld Competition using the actor-critic algorithm (Mnih et al., 2016) for training the policy. Unlike these text-only models, NESTA uses symbolic reasoning over the lifted rules for better generalization and interpretability.
Graph-based Agents: Instead of relying on the neural models to capture the structure of the observed text , recent works considered the graph representation of the observed text to guide the agent for better exploration. Graph-based agents from (Ammanabrolu and Riedl, 2019; Ammanabrolu and Hausknecht, 2020) build a knowledge graph representation of the textual state for efficient exploration and handling large action space. Adhikari et al. (2020) learns a dynamic belief graph from raw observations using adversarial learning on the First Textworld Problems (FTWP). Atzeni et al. (2021) proposed a case-based reasoning approach that improves upon existing graph-based methods
by reusing the past positive experiences stored in the agent’s memory. Unlike NESTA, these graphbased methods suffer from noise in the observation as the graphs are generated from the observed text.
Reasoning-based Agents: Both text-only and graph-based methods use only the texts observed during the game interaction. Murugesan et al. (2020) introduced Textworld commonsense (TWC), text-based cleanup games that require commonsense reasoning-based knowledge about everyday household objects Recent works tried to enrich text-only agents with commonsense reasoning for exploiting readily-available external knowledge graphs (Murugesan et al., 2021) and images generated from the observed texts using pre-trained models (Murugesan et al., 2022). These methods suffer from noisy features extracted from the external knowledge thus hindering the learning ability of the text-based RL agents. Unlike the traditional deep RL agents, Chaudhury et al. (2021); Kimura et al. (2021); Basu et al. (2021) proposed neurosymbolic agents for TBGs that show near-perfect performance. Related work from Li et al. (2021) uses the world model as a symbolic representation to capture the current state of the game. These approaches require hand-engineering of domainspecific symbolic state representation. On the other hand, NESTA presents a generic domainindependent symbolic logic representation with an automatic symbolic rule learner that handles large action spaces and noisy observation with ease.
In other symbolic methods, there are works (Petersen et al.; Costa et al., 2020) which employ deep learning for neuro-symbolic regression. Compared to these methods, NESTA aims to improve the generalization to unseen cases, whereas these methods train and test in the same setting. Additionally, neuro-symbolic regression methods have limited interaction with the environment in intermediate steps, and reward is obtained at the terminal state. However, for NESTA we use the symbolic representation from intermediate steps to learn action rules from partially-observable symbolic states.

heading: 6 Conclusion
text: In this paper, we present NESTA, a neuro-symbolic policy learning method that modularizes language understanding using an AMR-based semantic parsing module and RL policy optimization using an ILP rule learner. NESTA benefits from prior advances in AMR-based generic parsers for symbolic
fact extraction allowing the ILP symbolic learner to solely learn interpretable action rules. NESTA outperforms SOTA models on TBGs by showing better generalization while learning from a fewer number of training interactions. We believe our model is one of the first works combining advances in neural semantic parsing and efficient symbolic planning for text-based RL. We hope this work will encourage future research in this direction.

heading: 7 Limitations
text: The neuro-symbolic rule learning presented in the paper can handle most generic text-based games. Only in a few specific use cases, additional training of the AMR parser would be required. Since AMR is used for symbolic representation for text-based games, the vocabulary of the extracted triples is limited by the vocabulary of PropBank semantic roles. For applications in a very specific kind of domain where the predicates and entities do not match with this pre-defined vocabulary (for example, specific financial, legal domains, etc.), the AMR semantic parsing engine needs to be retrained first on such specific data before using it for rule learning. However, even in the cases where the testing environment requires additional rules, NESTA allows human-in-the-loop debugging to conveniently add them making it adaptable to generic environments.

heading: 8 Ethics Statement
text: Our method uses a constrained set of action samples to generate the textual actions in each step. Since this action set is generated from a controlled vocabulary of actions and entities, the produced actions cannot contain harmful content like hate speech and racial biases. Furthermore, our neurosymbolic model produces human interpretable rules for the action policy thereby making the model transparent and easier to control. Due to these reasons, the ethical risk from this work is low.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7 is the limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 8

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Grammar check assistant that corrected spellings on the paper
B 3 Did you use or create scientific artifacts? Section 2.1, Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 2.1, Section 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We will discuss the details of the license in the final code distribution
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 2.1, Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Partially in Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We hope this work will encourage future research in this direction