SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To compute the normalization term, we ground  $R$ , i.e. we replace variables with ground terms. We consider all of the possible groundings. Let  $\mathcal{T}$  be the set of all of the possible variables substitutions to ground  $R$ . For each  $\tau \in \mathcal{T}$ , we get a ground rule  $R\tau = X_A\tau : -X_S^{(1)}\tau, \ldots, X_S^{(n)}\tau$ , where  $X\tau$  represents the result of applying substitution  $\tau$  to atom  $X$ . Let  $\mathcal{J} = \{j_1, \ldots, j_n\}$  be indices of the ground atoms  $X_S^{(1)}\tau, \ldots, X_S^{(n)}\tau$  in ordered set of ground atoms  $\mathcal{G}$ . Then, the normalization term is computed as:
$$
N (R, \mathcal {X}) = \sum_ {\tau \in \mathcal {T}} \sum_ {s \in \mathcal {X}} \prod_ {j \in \mathcal {J}} = \mathbf {v} _ {s} ^ {(0)} [ j ], \tag {3}
$$
where  $\mathbf{v}_s^{(0)}$  is an initial valuation vector for state  $s$ , i.e.  $f_{\Theta}^{\text{perceive}}(s)$ . Eq. 3 quantifies how often the body atoms of the ground rule  $R\tau$  are activated on the given set of states  $\mathcal{X}$ . Simple rules with fewer atoms in their body tend to have large values, and thus their evaluation scores in Eq. 2 tend to be small. After scoring all of the new rules, NUDGE select top- $k$  rules to refine them in the next step. To this end, all of the top- $k$  rules in each step will be returned as the candidate ruleset  $\mathcal{C}$  for the policy (cf. App. A for more details about our algorithm).
We perform the action-rule generation for each action. In practice, NUDGE maintains the cached history  $\mathcal{H}$  of states and actions produced by the neural policy. For a given action, the search quality increases together with the amount of times the action was selected, i.e. if  $\mathcal{H}$  does not contain any record of the action, then all action rules (with this action as head) would get the same scores, leading to a random search.
NUDGE has thus produced candidate action rules  $\mathcal{C}$ , that will be associated with  $\mathbf{W}$  to form untrained differentiable logic policy  $\pi_{(\mathcal{C},\mathbf{W})}$ , as the ones described in Section 3.1.2.
3.2.2 Learning the Rules' Weights using the Actor-critic Algorithm
In the following, we consider a (potentially pretrained) actor-critic neural agent, with  $v_{\phi}$  its differentiable state-value function parameterized by  $\phi$  (critic). Given a set of action rules  $\mathcal{C}$ , let  $\pi_{(\mathcal{C},\mathbf{W})}$  be a differentiable logic policy. NUDGE learns the weights of the action rules in the following steps. For each non-terminal state  $s_t$  of each episode, we store the actions sampled from the policy  $(a_t \sim \pi_{(\mathcal{C},\mathbf{W})}(s_t))$  and the next states  $s_{t+1}$ . We update the value function and the policy as follows:
$$
\delta = r + \gamma v _ {\phi} \left(s _ {t + 1}\right) - v _ {\phi} \left(s _ {t}\right) \tag {4}
$$
$$
\phi = \phi + \delta \nabla_ {\phi} v _ {\phi} (s _ {t}) \tag {5}
$$
$$
\mathbf {W} = \mathbf {W} + \delta \nabla_ {\mathbf {W}} \ln \pi_ {(\mathcal {C}, \mathbf {W})} (s _ {t}). \tag {6}
$$
Figure 3: NUDGE outperforms neural and logic baselines on the the 3 logic environments. Returns (avg.  $\pm$  std.) obtained by NUDGE, neural PPO and logic-based agents without abstraction through training. NUDGE (Top-  $k$  rf.), with  $k\in \{1,3,10\}$  uses neurally-guided symbolic abstraction repeatedly until they get  $k$  rules for each action predicate. NUDGE (with E.S.) uses rule set  $\mathcal{C}$  supervised by an expert. Neural Logic RL composes logic-based policies by generating all possible rules without neurally-guided symbolic abstraction [Jiang and Luo, 2019]. Random and human baselines are also provided.
The logic policy  $\pi_{(\mathcal{C},\mathbf{W})}$  thus learn to maximize the expected return, potentially bootstrapped by the use of a pretrained neural critic. Moreover, to ease interpretability, NUDGE can prune the unused action rules (i.e. with low weights) by performing top- $k$  selection on the optimized rule weights after learning.
We here compare neural agents' performances to NUDGE ones, showcase NUDGE interpretable policies and its ability to report the importance of each input on their decisions, i.e. explainable logic policies. We use DQN agents (on Atari environments) and PPO actor-critic (on logic-ones) as neural baselines, for comparison and PPO as pretrained agents to guide the symbolic abstraction. All agent types receive object-centric descriptions of the environments. For clarity, we annotate action predicates on action rules with specific names on purpose, e.g. right $_{[to_key]}$  instead of right $^{(1)}$  (when the rule describes an action right motivated to get the key).
We intend to compare agents with object-centric information bottlenecks. We thus had to extract object-centric states of the Atari environments. To do so, we make use of the Object-Centric Atari library [Delfosse et al., 2023]. As Atari games do not embed logic challenges, but are rather designed to test the reflexes of human players, we also created 3 logic-oriented environments. We thus have modified environments from the Progen [Mohanty et al., 2020] environments that are open-sourced along with our evaluation to have object-centric representations. Our environments are easily hackable. We provide variations of these environments also to evaluate the ease of adaptation of every agent type. In GetOut, the goal is to obtain a key, and then go to a door, while avoiding a moving enemy. GetOut+ is a more complex variation with a larger world containing 5 enemies (among which 2 are static). In 3Fishes, the agent controls a fish and is confronted with 2 other fishes, one smaller (that the agent needs to "eat", i.e. go to) and one bigger, that the agent needs to dodge. A variation is 3Fishes-C, where the agent can eat green fishes and dodge red ones. Finally, in Loot, the agent can open 1 or 2 chests and their corresponding (i.e. same color) keys. In Loot-C, the chests have different colors. Further details and hyperparameters are provided in App. D.
We aim to answer the following research questions: Q1. How does NUDGE compare with neural and logic baselines? Q2. Can NUDGE agents easily adapt to environmental changes? Q3. Are NUDGE agents interpretable and explainable?
NUDGE competes with existing methods (Q1). We compare NUDGE with different baselines regarding their scores (or returns). First, we present scores obtained by trained DQN, Random and NUDGE agents (with expert supervision) on 2 Atari games (cf. Table 1). Our result show that NUDGE obtain better (Asterix) or similar (Freeway) scores than DQN. However, as said, Atari games are not logically challenging. We thus evaluate NUDGE on 3 logic environments. Figure 3 shows the returns in GetOut, 3Fishes, and Loot, with descriptions for each baseline in the caption. NUDGE obtains better performances than neural baselines (Neural PPO) on 3Fishes, is more stable on GetOut, i.e. less variance, and achieves faster convergence on Loot. This shows that NUDGE successfully distillates logic-based policies competing with neural baselines in different complex environments.
Score (↑)	Random	DQN	NUDGE
Asterix	235 ±134	124.5	6259 ±1150
Freeway	0.0 ±0	25.8	21.4 ±0.8
Score (↑)	Random	Classic	NUDGE
GetOut	-22.5±0.41	11.59±4.29	17.86±2.86
3Fish	-0.73±0.05	-0.24±0.29	0.05±0.27
Loot	0.57±0.39	0.51±0.74	5.66±0.59
We also evaluate agents with a baseline without symbolic abstraction, where candidate rules are generated not being guided by neural policies, i.e. accept all of the generated rules in the rule refinement steps. This setting corresponds to the template-based approach [Jiang and Luo, 2019], but we train the agents by the actor-critic method, while vanilla policy gradient [Williams, 1992] is employed in [Jiang and Luo, 2019]. For the no-abstraction baseline and NUDGE, we provide initial action rules with basic type information, e.g.  $\text{jump}^{(1)}(\text{agent})$ : - type(01, agent), type(02, enemy), for each action rule. For this baseline, we generate 5 rules for GetOut, 30 rules for 3Fishes, and 40 rules for Loot in total to define all of the actual actions. NUDGE agents with small  $k$  tend to have less rules, e.g. 5 rules in Getout, 6 rules in 3Fishes, and 8 rules in Loot for the top-1 refinement version. In Figure 3, the no-abstraction baselines perform worse than neural PPO and NUDGE in each environment, even though they have much more rules in 3Fishes and Loot. NUDGE thus composes efficient logic-based policies using neurally-guided symbolic abstraction. In App. B.1, we visualize the transition of the distribution of the rule weights in the GetOut environment.
To demonstrate the necessity of differentiability within symbolic reasoning in continuous RL environments, we simulated classic (i.e. discrete logic) agents, that reuse NUDGE agents' set of action rules, but with all weights fixed to 1.0. Such agents can still play better than ones with pure symbolic reasoners, as they still use fuzzy (i.e., continuous) concepts, such as closeby. In all our environments, NUDGE clearly outperforms the classic counterparts (cf. Table 1 right). This experiment shows the superiority of differentiable policy reasoning (embedding weighted action rules) over policy without weighted rules (classic).
To further reduce the gap between our environments and those from environments Cao et al. [2022], we have adapted our loot environment (Loot-hard). In this environment, the agent first has to pick up keys and open their corresponding saves, before being able to exit the level (by going to a final exit tile). Our results in Table 2 (bottom), as well as curves on Figure 10 in the appendix show the dominance of NUDGE agents.
NUDGE agents adapt to environment changes (Q2).
We used the agents trained on the basic environment for this experimental evaluation, with no retraining or finetuning. For 3Fishes-C, we simply exchange the atom is_bigger with the atom same_color. This easy modification is not applicable on the black-box networks of neural PPO agents. For GetOut+ and Loot-C, we do not apply any modification to the agents. Our results are summarized in Table 2. Note that the agents obtain better performances in the 3Fishes-C variation, dodging a (small) red fish is eas
ier than a big one. For GetOut+, NUDGE performances have decreased as avoiding 5 enemies drastically increases the difficulty of the game. On Loot-C, the performances are similar to the ones obtained in the original game. Our experiments show that NUDGE logic agents can easily adapt to environmental changes.
Table 1: Left: NUDGE agents can learn successful policies. Trained NUDGE agents (with expert supervision) scores (avg. ± std.) on 2 OCatari games [Delfosse et al., 2023]. Random and DQN (from van Hasselt et al. [2016]) are also provided. Right: NUDGE outperforms non-differentiable symbolic reasoning baselines. Returns obtained by NUDGE, classic and random agents our the 3 logic environments.
Score (↑)	3Fishes-C	GetOut+	Loot-C
Random	-0.6 ±0.2	-22.5 ±0.4	0.6 ±0.3
Neural PPO	-0.4 ±0.1	-20.9 ±0.6	0.8 ±0.5
NUDGE	3.6 ±0.2	3.6 ±3.0	5.6 ±0.3
Table 2: NUDGE agents adapt to environmental changes and solve logically-challenging environments. Returns obtained by NUDGE, neural PPO and random agents on our modified environments and on Loot-hard.
NUDGE agents are interpretable and explainable (Q3). We show that NUDGE agents are interpretable and explainable by showing that (1) NUDGE produces interpretable policy as a set of weighted rules, and (2) NUDGE can show the importance of each atom, explaining its action choices.
The efficient neurally-guided learning on NUDGE enables the system to learn rules without inventing predicates with no specific interpretations, which are unavoidable in template-based approaches [Evans and Grefenstette, 2018, Jiang and Luo, 2019]. Thus, the policy can easily be read out by extracting action rules with high weights. Figure 5 shows some action rules discovered by NUDGE in GetOut. The first rule says: "The agent should jump when the enemy is close to the agent (to avoid the enemy)". The produced NUDGE is an interpretable policy
0.57: jump(agent): -type(01, agent), type(02, enemy), closeby(01, 02).
0.29: right[to_key](agent): -type(01, agent), type(02, key), on_right(02, 01), -has_key(01).
0.56: right[to Door](agent): -type(01, agent), type(02, door), on_right(02, 01), has_key(01).
0.32:left[to_key](agent):- type(01,agent),type(02,key),on_right(01,02),has_key(01).
0.30:left[to-door](agent):-type(01,agent),type(02,door),on_left(02,01),has_key(01).
with its set of weighted rules using interpretable predicates. For each state, we can also look at the valuation of each atom and the selected rule.
Moreover, contrary to classic logic policies, the differentiable ones produced by NUDGE allow us to compute the attribution values over the logical representations via backpropagation of the policies' gradients. We can compute the action gradients w.r.t. input atoms, i.e.  $\partial \mathbf{v}_A / \partial \mathbf{v}^{(0)}$ , as shown in Figure 4, which represent the relevance scores of the probabilistic input atoms  $\mathbf{v}^{(0)}$  for the actions given a specific state. The explanation is computed on the state shown in Figure 1, where the agent takes right as its action. Important atoms receive large gradients, e.g. has_key(agent) and on_right(obj2,obj1). By extracting relevant atoms with large gradients, NUDGE can produce clear explanations for the action selection. For example, by
Figure 5: NUDGE produces interpretable policies as sets of weighted rules. A subset of the weighted action rules from the Getout environment. Full policies for every logic environment are provided in App. B.3.
Gradients of actions w.r.t input atoms
Figure 4: Explanation using inputs' gradients. The action gradient w.r.t. input atoms, i.e.  $\partial \mathbf{v}_A / \partial \mathbf{v}^{(0)}$ , on the state shown in Figure 1. right was selected, due to the highlighted relevant atoms (with large gradients).
extracting the atoms wrapped in orange in Figure 4, NUDGE can explain the motivation: "The agent decides to go right because it does not have the key and the key is located on the right-side of it." These results show that NUDGE policies are interpretable and explainable, i.e. each action predicate is defined by rules that humans can fully understand, and gradient-based explanations for the action selections can be efficiently produced.
We also computed explanations of a neural agent on the same state, also using the input-gradients method. The concepts which received the highest explanation scores are: the key x-axis position (1.22), the key x-axis velocity (1.21), the enemy y-axis velocity (1.10), and the door x-axis position (0.77). For a consistent explanation, the agent x-axis position should also be among the highest scores. For this specific state, the neural agent seems to be placing its explanations on non relevant concepts (such as the key's velocity). Stammer et al. [2021] have shown that neural networks trained on logical concepts tend to produce wrong explanations often, and additional regularization/supervision about explanations during training is necessary to force neural networks to produce correct explanations. This approach requires additional efforts to label each state with its correct explanation. NUDGE policies produces better explanations than neural-based agents. More explanations produced by NUDGE are provided in Figure 11 in the appendix.
Relational RL [Dzeroski et al., 2001, Kersting et al., 2004, Kersting and Driessens, 2008, Lang et al., 2012] has been developed to tackle RL tasks in relational domains. Relational RL frameworks incorporate logical representations and use probabilistic reasoning. With NUDGE, we make use of differentiable logic programming. The Neural Logic Reinforcement Learning (NLRL) [Jiang and Luo, 2019] framework is the first to integrate Differentiable Inductive Logic Programming  $(\partial \mathrm{ILP})$  [Evans and Grefenstette, 2018] to RL domain.  $\partial \mathrm{ILP}$  learns generalized logic rules from examples by gradient-based optimization. NLRL adopts  $\partial \mathrm{ILP}$  as a policy
function. We extend this approach by proposing neurally-guided symbolic abstraction embracing extensive work of  $\partial$ ILP [Shindo et al., 2021b,a, 2023b] for learning complex programs including functors in visual scenes, allowing agents to learn interpretable action rules efficiently for complex environments.
FOL	N.G.	Int.	Exp.
NLRL	✓	✗	✓	✗
NeSyRL	✓	✗	✓	✗
DiffSES	✗	✓	✓	✗
NUDGE	✓	✓	✓	✓
Table 3: Logic-based RL methods comparison: First Order Logic (FOL), neurally-guided search (N.G.), interpretability (Int.), and explainability (Exp.).
GALOIS [Cao et al., 2022] is a framework to represent policies as logic programs using the sketch setting [Solar-Lezama, 2008], where programs are learned to fill blanks. NUDGE performs structure learning from scratch using policy gradients. KoGun [Zhang et al., 2020] integrates human knowledge as a prior for RL agents. NUDGE learns a policy as a set of weighted rules and thus also can integrate human knowledge. Neuro-Symbolic RL (NeSyRL) [Kimura et al., 2021] uses Logical Neural Networks (LNNs) [Riegel et al., 2020] for the policy computation. LNNs parameterize the soft logical operators while NUDGE parameterizes rules with their weights. Deep Relational RL approaches [Zambaldi et al., 2018] achieve relational reasoning as a neural network, but NUDGE explicitly encodes relations in logic. Many languages for planning and RL tasks have been developed [Fikes and Nilsson, 1971, Fox and Long, 2003]. Our approach is inspired by situation calculus [Reiter, 2001], which is an established framework to describe states and actions in logic.
Symbolic programs within RL have been investigated, e.g. program guided agent [Sun et al., 2020], program synthesis [Zhu et al., 2019], PIRL [Verma et al., 2018], SDRL [Lyu et al., 2019], interpretable model-based hierarchical RL Xu and Fekri [2021], deep symbolic policy Landajuela et al. [2021], and DiffSES Zheng et al. [2021]. These approaches use domain specific languages or propositional logic, and address either of interpretability or explainability of RL. To this end, in Table 3, we compare NUDGE with the most relevant approaches that share at least 2 of the following aspects: supporting first-order logic, neural guidance, interetability and explainability. NUDGE is the first method to use neural guidance for differentiable first-order logic policies and to address both interpretability and explainability. Specifically, PIRL develops functional programs with neural guidance using sketches, which define a grammar of programs to be generated. In contrast, NUDGE performs structure learning from scratch using a neural guidance with a language bias of mode declarations [Muggleton, 1995, Cropper et al., 2022] restricting the search space. As NUDGE uses first-order logic, it can incorporate background knowledge in a declarative form, e.g. with a few lines of relational atoms and rules. To demonstrate this, NUDGE has been evaluated on challenging environments where the main interest is relational reasoning.
We proposed NUDGE, an interpretable and explainable policy reasoning and learning framework for reinforcement learning. NUDGE uses differentiable forward reasoning to obtain a set of interpretable weighted rules as policy. It performs neurally-guided symbolic abstraction, which efficiently distillates symbolic representations from a neural policy, and incorporate an actor-critic method to perform gradient-based policy optimization. We empirically demonstrated that NUDGE agents (1) can compete with neural based policies, (2) use logical representations to produce both interpretable and explainable policies and (3) can automatically adapt or easily be modified and are thus robust to environmental changes.
Societal and environmental impact. As NUDGE agents can explain the importance of each the input on their decisions, and as their rules are interpretable, it can help us understanding the decisions of RL agents trained in sensitive complicated domains, and potentially help discover biases and misalignments of discriminative nature. While the distillation of the learned policy into a logic ones augments the computational resources needed for completing the agents learning process, we believe that the logic the agents' abstraction capabilities will overall reduce the need of large neural networks, and will remove the necessity of retraining agents for each environmental change, leading to an overall reduction of necessary computational power.
Limitation and Future Work. NUDGE is only complete if provided with a sufficiently expressive language (in terms of predicates and entities) to approximate neural policies. Interesting lines of future research can include: (i) an automatic discovery of predicates, using e.g. predicate invention [Muggleton et al., 2015], (ii) augmenting the number of accessible entities to reason on. Explainable interactive learning [Teso and Kersting, 2019] in RL can integrate NUDGE agents, since it produces explanations on logical representations. Causal RL [Madumal et al., 2020] and meta learning [Mishra et al., 2018] also constitute interesting avenues for NUDGE's development. Finally, we could incorporate objects extraction methods [Delfosse et al., 2022] within our NUDGE agents, to obtain logic agents that extract object and their properties from images.
Acknowledgements. The authors thank the anonymous reviewers for their valuable feedback. This research work has been funded by the German Federal Ministry of Education and Research, the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK) within their joint support of the National Research Center for Applied Cybersecurity ATHENE, via the "SenPai: XReLeaS" project. We gratefully acknowledge support by the Federal Ministry of Education and Research (BMBF) AI lighthouse project "SPAICER" (01MK20015E), the EU ICT-48 Network of AI Research Excellence Center "TAILOR" (EU Horizon 2020, GA No 952215), and the Collaboration Lab "AI in Construction" (AICO).