SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Quentin Delfosse*
Technical University of Darmstadt  
National Research Center for Applied Cybersecurity  
quentin.delfosse@tu-darmstadt.de
Eindhoven University of Technology†  
Hessian Center for AI (hessian.AI)  
d.s.dhami@tue.nl
Hikaru Shindo*
Technical University of Darmstadt hikaru.shindo@tu-darmstadt.de
The limited priors required by neural networks make them the dominating choice to encode and learn policies using reinforcement learning (RL). However, they are also black-boxes, making it hard to understand the agent's behaviour, especially when working on the image level. Therefore, neuro-symbolic RL aims at creating policies that are interpretable in the first place. Unfortunately, interpretability is not explainability. To achieve both, we introduce Neurally gUided Differentiable IoGic policiEs (NUDGE). NUDGE exploits trained neural network-based agents to guide the search of candidate-weighted logic rules, then uses differentiable logic to train the logic agents. Our experimental evaluation demonstrates that NUDGE agents can induce interpretable and explainable policies while outperforming purely neural ones and showing good flexibility to environments of different initial states and problem sizes.
Deep reinforcement learning (RL) agents use neural networks to take decisions from the unstructured input state space without manual engineering [Mnih et al., 2015]. However, these black-box policies lack interpretability [Rudin, 2019], i.e. the capacity to articulate the thinking behind the action selection. They are also not robust to environmental changes [Pinto et al., 2017, Wulfmeier et al., 2017]. Although performing object detection and policy optimization independently can get over these issues Devin et al. [2018], doing so comes at the cost of the aforementioned issues when employing neural networks to encode the policy.
As logic constitutes a unified symbolic language that humans use to compose the reasoning behind their behavior, logic-based policies can tackle the interpretability problems for RL. Recently proposed Neural Logic RL (NLRL) agents [Jiang and Luo, 2019] construct logic-based policies using differentiable rule learners called  $\partial ILP$  [Evans and Grefenstette, 2018], which can then be integrated with gradient-based optimization methods for RL. It represents the policy as a set of weighted rules, and performs policy gradients-based learning to solve RL tasks which require relational reasoning. It successfully produces interpretable rules, which describe each action in terms of its preconditions and outcome. However, the number of potential rules grows exponentially with the number of considered actions, entities, and their relations. NLRL is a memory-intensive approach, i.e. it generates a set of potential simple rules based on rule templates and can only be evaluated on simple abstract
$\dagger$  DSD contributed while being with hessian.AI and TU Darmstadt before joining TU\e.
Figure 1: Overview of NUDGE. Given a state (depicted in the image), NUDGE computes the action distribution using relational state representation and differentiable forward reasoning. NUDGE provides interpretable and explainable policies, i.e. derives policies as sets of interpretable weighted rules, and can produce explanations using gradient-based attribution methods.
environments, created for the occasion. This approach can generate many newly invented predicates without their specification of meaning [Evans and Grefenstette, 2018], making the policy challenging to interpret for complex environments. Moreover, the function of explainability is absent, i.e. the agent cannot explain the importance of each input on its decision. Explainable agents should adaptively produce different explanations given different input states. A question thus arises: How can we build interpretable and explainable RL agents that are robust to environmental changes?
To this end, we introduce Neurally gUided Differentiable loGic policiEs (NUDGE), illustrated in Figure 1, that embody the advantages of logic: they are easily adaptable to environmental changes, composable, interpretable and explainable (because of our differentiable logic module). Given an input state, NUDGE extracts entities and their relations, converting raw states to a logic representations. This probabilistic relational states are used to deduce actions, using differentiable forward reasoning [Evans and Grefenstette, 2018, Shindo et al., 2023a]. NUDGE produces a policy that is both interpretable, i.e. provides a policy as a set of weighted interpretable rules that can be read out by humans, and explainable, i.e. explains which input is important using gradient-based attribution methods [Sundararajan et al., 2017] over logical representations.
To achieve an efficient learning with logic policies, we provide an algorithm to train NUDGE agents based on the PPO actor-critic framework. Moreover, we propose a novel rule-learning approach, called Neurally-Guided Symbolic Abstraction, where the candidate rules for the logic-based agents are obtained efficiently by being guided by neural-based agents. NUDGE distillates abstract representations of neural policies in the form of logic rules. Rules are assigned with their weights, and we perform gradient-based optimization using the PPO actor-critic framework.
1. We propose NUDGE<sup>3</sup>: differentiable logical policies that learn interpretable rules and produce explanations for their decisions in complex environments. NUDGE uses neurally-guided symbolic abstraction to efficiently find a promising ruleset using pretrained neural-based agents guidance.
2. We empirically show that NUDGE agents: (i) can compete with neural-based agents, (ii) adapt to environmental changes, and (iii) are interpretable and explainable, i.e. produce interpretable policies as sets of weighted rules and provide explanations for their action selections.
3. We evaluate NUDGE on 2 classic Atari games and on 3 proposed object-centric logically challenging environments, where agents need relational reasoning in dynamic game-playing scenarios.
We start off by introducing the necessary background. Then we explain NUDGE's inner workings and present our experimental evaluation. Before concluding, we touch upon related work.
We now describe the necessary background before formally introducing our NUDGE method.
3Code publicly available: https://github.com/k4ntz/NUDGE.
Deep Reinforcement Learning. In RL, the task is modelled as a Markov decision process,  $\mathcal{M} = < S, \mathcal{A}, P, R>$ , where, at every timestep  $t$ , an agent in a state  $s_t \in S$ , takes action  $a_t \in \mathcal{A}$ , receives a reward  $r_t = R(s_t, a_t)$  and a transition to the next state  $s_{t+1}$ , according to environment dynamics  $P(s_{t+1}|s_t, a_t)$ . Deep agents attempt to learn a parametric policy,  $\pi_\theta(a_t|s_t)$ , in order to maximize the return (i.e.  $\sum_t \gamma^t r_t$ , with  $\gamma \in [0,1]$ ). The desired input to output (i.e. state to action) distribution is not directly accessible, as RL agents only observe returns. The value  $V_{\pi_\theta}(s_t)$  (resp. Q-value  $Q_{\pi_\theta}(s_t, a_t)$ ) function provides the return of the state (resp. state/action pair) following the policy  $\pi_\theta$ . Policy-based methods directly optimize  $\pi_\theta$  using the noisy return signal, leading to potentially unstable learning. Value-based methods learn to approximate the value functions  $\hat{V}_\phi$  or  $\hat{Q}_\phi$ , and implicitly encode the policy, e.g. by selecting the actions with the highest Q-value with a high probability [Mnih et al., 2015]. To reduce the variance of the estimated Q-value function, one can learn the advantage function  $\hat{A}_\phi(s_t, a_t) = \hat{Q}_\phi(s_t, a_t) - \hat{V}_\phi(s_t)$ . An estimate of the advantage function can be computed as  $\hat{A}_\phi(s_t, a_t) = \sum_{i=0}^{k-1} \gamma^i r_{t+i} + \gamma^k \hat{V}_\phi(s_{t+k}) - \hat{V}_\phi(s_t)$  [Mnih et al., 2016]. The Advantage Actor-critic (A2C) methods both encode the policy  $\pi_\theta$  (i.e. actor) and the advantage function  $\hat{A}_\phi$  (i.e. critic), and use the critic to provide feedback to the actor, as in [Konda and Tsitsiklis, 1999]. To push  $\pi_\theta$  to take actions that lead to higher returns, gradient ascent can be applied to  $L^{PG}(\theta) = \hat{\mathbb{E}}[\log \pi_\theta(a|s) \hat{A}_\phi]$ . Proximal Policy Optimization (PPO) algorithms ensure minor policy updates that avoid catastrophic drops [Schulman et al., 2017], and can be applied to actor-critic methods. To do so, the main objective constraints the policy ratio  $r(\theta) = \frac{\pi_\theta(a|s)}{\pi_{\theta_{\mathrm{old}}}(a|s)}$ , following  $L^{PR}(\theta) = \hat{\mathbb{E}}[\min(r(\theta) \hat{A}_\phi, \operatorname{clip}(r(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_\phi)]$ , where clip constrains the input within  $[1-\epsilon, 1+\epsilon]$ . PPO actor-critic algorithm's global objective is  $L(\theta, \phi) = \hat{\mathbb{E}}[L^{PR}(\theta) - c_1 L^{VF}(\phi)]$ , with  $L^{VF}(\phi) = (\hat{V}_\phi(s_t) - V(s_t))^2$  being the value function loss. An entropy term can also be added to this objective to encourage exploration.
First-Order Logic (FOL). In FOL, a Language  $\mathcal{L}$  is a tuple  $(\mathcal{P},\mathcal{D},\mathcal{F},\mathcal{V})$ , where  $\mathcal{P}$  is a set of predicates,  $\mathcal{D}$  a set of constants,  $\mathcal{F}$  a set of function symbols (functors), and  $\mathcal{V}$  a set of variables. A term is either a constant (e.g. obj1, agent), a variable (e.g. 01), or a term which consists of a function symbol. An atom is a formula  $\mathfrak{p}(\mathfrak{t}_1,\ldots ,\mathfrak{t}_{\mathfrak{n}})$ , where  $\mathfrak{p}$  is a predicate symbol (e.g. closeby) and  $\mathfrak{t}_1,\ldots ,\mathfrak{t}_{\mathfrak{n}}$  are terms. A ground atom or simply a fact is an atom with no variables (e.g. closeby(obj1, obj2)). A literal is an atom  $(A)$  or its negation  $(\neg A)$ . A clause is a finite disjunction  $(\lor)$  of literals. A ground clause is a clause with no variables. A definite clause is a clause with exactly one positive literal. If  $A,B_{1},\ldots ,B_{n}$  are atoms, then  $A\lor \neg B_1\lor \dots \lor \neg B_n$  is a definite clause. We write definite clauses in the form of  $A: - B_{1},\ldots ,B_{n}$ . Atom  $A$  is called the head, and set of negative atoms  $\{B_1,\dots ,B_n\}$  is called the body. We call definite clauses as rules for simplicity in this paper.
Differentiable Forward Reasoning is a data-driven approach of reasoning in FOL [Russell and Norvig, 2010]. In forward reasoning, given a set of facts and a set of rules, new facts are deduced by applying the rules to the facts. Differentiable forward reasoning [Evans and Grefenstette, 2018, Shindo et al., 2023a] is a differentiable implementation of the forward reasoning with tensor-based differentiable operations.
3 Neurally Guided Logic Policies
Figure 2 illustrates an overview of RL on NUDGE. They consist of a policy reasoning module and a policy learning module. NUDGE performs end-to-end differentiable policy reasoning based on forward reasoning, which computes action distributions given input states. On top of the reasoning module, policies are learned using neurally-guided symbolic abstraction and an actor-critic framework.
3.1 Policy Reasoning: Selecting Actions using Differentiable Forward Reasoning.
To realize NUDGE, we introduce a language to describe actions and states in FOL. Using it, we introduce differentiable policy reasoning using forward chaining reasoning.
3.1.1 Logic Programs for Actions
In RL, states and actions are key components since the agent performs the fundamental iteration of observing the state and taking an action to maximize its expected return. To achieve an efficient computation on first-order logic in RL settings, we introduce a simple language suitable for reasoning about states and actions.
We split the predicates set  $\mathcal{P}$  into two different sets, i.e., action predicates  $(\mathcal{P}_A)$ , which define the actions and state predicates  $(\mathcal{P}_S)$  used for the observed states. If an atom  $A$  consists of an action predicate,  $A$  is called an action atom. If  $A$  consists of a state predicate,  $A$  is called state atom.
Figure 2: NUDGE-RL. Policy Reasoning (bottom): NUDGE agents incorporate end-to-end reasoning architectures from raw input based on differentiable forward reasoning. In the reasoning step, (1) the raw input state is converted into a logical representation, i.e. a set of atoms with probabilities. (2) Differentiable forward reasoning is performed using weighted action rules. (3) The final distribution over actions is computed using the results of differentiable reasoning. Policy Learning (top): Using the guidance of a pretrained neural policy, a set of candidate action rules is searched by neurally-guided symbolic abstraction, where promising action rules are produced. Then, randomly initialized weights are assigned to the action rules and are optimized using the critic of an actor-critic agent.
Definition 1 Action-state Language is a tuple of  $(\mathcal{P}_A, \mathcal{P}_S, \mathcal{D}, \mathcal{V})$ , where  $\mathcal{P}_A$  is a set of action predicates,  $\mathcal{P}_S$  is a set of state predicates,  $\mathcal{D}$  is a set of constants for entities, and  $\mathcal{V}$  is a set of variables.
For example, for Getout illustrated in Figures 1 and 2, actual actions are: left, right, jump, and idle. We define action predicates  $\mathcal{P}_A = \{\text{left}^{(1)}, \text{left}^{(2)}, \text{right}^{(1)}, \text{right}^{(2)}, \text{jump}^{(1)}, \text{idle}^{(1)}, \ldots\}$  and state predicates  $\mathcal{P}_S = \{\text{type}, \text{closeby}, \ldots\}$ . To encode different reasons for a given game action, we can generate several action predicates (e.g. right $^{(1)}$  and right $^{(2)}$  for right). By using these predicates, we can compose action atoms, e.g. right $^{(1)}$  (agent), and state atoms, e.g. type(obj1, agent). An action predicate can also be a state predicate, e.g. in multiplayer settings. Now, we define rules to describe actions in the action-state language.
Definition 2 Let  $X_A$  be an action atom and  $X_S^{(1)}, \ldots, X_S^{(n)}$  be state atoms. An action rule is a rule, written as  $X_A: -X_S^{(1)}, \ldots, X_S^{(n)}$ .
For example, for action right, we define an action rule as:
$$
\operatorname {r i g h t} ^ {(1)} (\text {a g e n t}): - \text {t y p e} (0 1, \text {a g e n t}), \text {t y p e} (0 2, \text {k e y}), \neg \text {h a s} _ {\text {k e y}} (0 1), \text {o n} _ {\text {r i g h t}} (0 2, 0 1).
$$
which can be interpreted as "The agent should go right if the agent does not have the key and the key is located on the right of the agent." Having several action predicates for an actual action (in the game) allows our agents to define several action rules that describe different reasons for the action.
3.1.2 Differentiable Logic Policies
We denote the set of actual actions by  $\mathcal{A}$ , the set of action rules by  $\mathcal{C}$ , the set of all of the facts by  $\mathcal{G} = \mathcal{G}_A \cup \mathcal{G}_S$  where  $\mathcal{G}_A$  is a set of action atoms and  $\mathcal{G}_S$  is a set of state atoms.  $\mathcal{G}$  contains all of the facts produced by a given FOL language. We here consider ordered sets, i.e. each element has its index. We also denote the size of the sets as:  $A = |\mathcal{A}|$ ,  $C = |\mathcal{C}|$ ,  $G = |\mathcal{G}|$ , and  $G_A = |\mathcal{G}_A|$ .
We propose Differentiable Logic Policies, which perform differentiable forward reasoning on action rules to produce a probability distribution over actions. The policy computation consists of 3 components: the relational perception module (1), the differentiable forward-reasoning module (2), and the action-extraction module (3).
The policy  $\pi_{(\mathcal{C},\mathbf{W})}$  parameterized by a set of action rules  $\mathcal{C}$  and the rules' weights  $\mathbf{W}$  is computed as follows:
$$
\pi_ {(\mathcal {C}, \mathbf {W})} (s _ {t}) = p (a _ {t} | s _ {t}) = f ^ {\text {a c t}} \left(f _ {(\mathcal {C}, \mathbf {W})} ^ {\text {r e a s o n}} \left(f _ {\Theta} ^ {\text {p e r c e i v e}} (s _ {t})\right)\right), \tag {1}
$$
with  $f_{\Theta}^{\text{perceive}} \colon \mathbb{R}^N \to [0,1]^G$  a perception function that maps the raw input state  $s_t \in \mathbb{R}^N$  into a set of probabilistic atoms,  $f_{(\mathcal{C},\mathbf{W})}^{\text{reason}} \colon [0,1]^G \to [0,1]^{G_A}$  a differentiable forward reasoning function parameterized by a set of rules  $\mathcal{C}$  and rule weights  $\mathbf{W}$ , and  $f^{\text{act}} \colon [0,1]^{G_A} \to [0,1]^A$  an action-selection function, which computes the probability distribution over the action space.
Relational Perception. NUDGE agents take an object-centric state representations as input, obtained by e.g. using object detection [Redmon et al., 2016] or discovery [Lin et al., 2020, Delfosse et al., 2022] methods. These models return the detected objects and their attributes (e.g. class and positions). They are then converted into a probabilistic logic form with their relations, i.e. a set of facts with their probabilities. An input state  $s_t \in \mathbb{R}^N$  is converted to a valuation vector  $\mathbf{v} \in [0,1]^G$ , which maps each fact to a probabilistic value. For example, let  $\mathcal{G} = \{\text{type(obj1, agent)}, \text{type(obj2, enemy)}, \text{closeby(obj1, obj2)}, \text{jump(agent)}\}$ . A valuation vector  $[0.8, 0.6, 0.3, 0.0]^\top$  maps each fact to a corresponding probabilistic value. NUDGE performs differentiable forward reasoning by updating the initial valuation vector  $\mathbf{v}^{(0)}T$  times to obtain  $\mathbf{v}^{(T)}$ .
Initial valuation vector  $\mathbf{v}^{(0)}$  is computed as follows. For each ground state atom  $\mathfrak{p}(\mathfrak{t}_1,\dots ,\mathfrak{t}_{\mathfrak{n}})\in \mathcal{G}_S$ , e.g. closeby(obj1, obj2), a differentiable function is called to compute its probability, which maps each term  $\mathfrak{t}_1,\ldots ,\mathfrak{t}_{\mathfrak{n}}$  to vector representations according to the interpretation, e.g. obj1 and obj2 are mapped to their positions, then perform binary classification using the distance between them. For action atoms, zero is assigned as its initial probability (e.g. for  $\mathrm{jump}^{(1)}(\mathrm{agent})$ ).
Differentiable Forward Reasoning. Given a set of candidate action rules  $\mathcal{C}$ , we create the reasoning function  $f_{(\mathcal{C},\mathbf{W})}^{reason}:[0,1]^G\to [0,1]^{G_A}$ , which takes the initial valuation vector and induces action atoms using weighted action rules. We assign weights to the action rules of  $\mathcal{C}$  as follows: We fix the target programs' size,  $M$ , i.e. select  $M$  rules out of  $C$  candidate action rules. To do so, we introduce  $C$ -dimensional weights  $\mathbf{W} = [\mathbf{w}_1,\dots ,\mathbf{w}_M]$  where  $\mathbf{w}_i\in \mathbb{R}^C$  (cf. Figure 6 in the appendix). We take the softmax of each weight vector  $\mathbf{w}_i\in \mathbf{W}$  to select  $M$  action rules in a differentiable manner.
We perform  $T$ -step forward reasoning using action rules  $\mathcal{C}$  with weights  $\mathbf{W}$ . We compose the differentiable forward reasoning function following Shindo et al. [2023a]. It computes soft logical entailment based on efficient tensor operations. Our differentiable forward reasoning module computes new valuation  $\mathbf{v}^{(T)}$  including all induced atoms given weighted action rules  $(\mathcal{C}, \mathbf{W})$  and initial valuation  $\mathbf{v}^{(0)}$ . Finally, we compute valuations on action atoms  $\mathbf{v}_A \in [0,1]^{G_A}$  by extracting relevant values from  $\mathbf{v}^{(T)}$ . We provide details in App. E.
Compute Action Probability. Given valuations on action atoms  $\mathbf{v}_A$ , we compute the action distribution for actual actions. Let  $\mathbf{a}_i \in \mathcal{A}$  be an actual action, and  $v_1', \ldots, v_n' \in \mathbf{v}_A$  be valuations which are relevant for  $\mathbf{a}_i$  (e.g. valuations of right $^{(1)}$  (agent) and right $^{(2)}$  (agent) in  $\mathbf{v}_A$  for the action right). We assign scores to each action  $\mathbf{a}_i$  based on the log-sum-exp approach of Cuturi and Blondel [2017]:  $val(\mathbf{a}_i) = \gamma \log \sum_{1 \leq i \leq n} \exp(v_i' / \gamma)$ , that smoothly approximates the maximum value of  $\{v_1', \ldots, v_n'\}$ . We use  $\gamma > 0$  as a smoothing parameter. The action distribution is then obtained by taking the softmax over the evaluations of all actions.
3.2 Policy Learning
So far, we have considered that candidate rules for the policy are given, requiring human experts to handcraft potential rules. To avoid this, template-based rule generation [Evans and Grefenstette, 2018, Jiang and Luo, 2019] can be applied, but the number of generated rules increases exponentially with the number of entities and their potential relations. This technique is thus difficult to apply to complex environments where the agents need to reason about many different relations of entities.
To mitigate this problem, we propose an efficient learning algorithm for NUDGE that consists of two steps: neurally-guided symbolic abstraction and gradient-based optimization. First, NUDGE obtains a symbolic abstract set of rules, aligned with a given neural policy. The set of candidate rules is selected by neurally-guided top-k search, i.e., we generate a set of promising rules using a neural policy as an oracle. Then we assign randomized weights to each selected rule and perform differentiable reasoning. We finally optimize the rule weights using a "logic actor - neural critic" algorithm that aims at maximizing the expected return. Let us elaborate on each step.
3.2.1 Neurally Guided Symbolic Abstraction
Given a well-performing neural policy  $\pi_{\theta}$ , promising action rules for an RL task entail the same actions as the ones selected by  $\pi_{\theta}$ . We generate such rules by performing top- $k$  search-based abstraction, which uses the neural policy to evaluate rules efficiently. The inputs are initial rules  $\mathcal{C}_0$ , neural policy  $\pi_{\theta}$ . We start with elementary action rules and refine them to generate better action rules.  $\mathcal{C}_{to\_open}$  is a set of rules to be refined, and initialized as  $\mathcal{C}_0$ . For each rule  $C_i \in \mathcal{C}_{to\_open}$ , we generate new rules by refining them as follows. Let  $C_i = X_A \gets X_S^{(1)}, \ldots, X_S^{(n)}$  be an already selected general action rule. Using a randomly picked ground or non-ground state atom  $Y_S$  ( $\neq X_S^{(i)} \forall i \in [1, \ldots, n]$ ), we refine the selected rule by adding a new state atom to its body, obtaining:  $X_A \gets X_S^{(1)}, \ldots, X_S^{(n)}, Y_S$ .
We evaluate each newly generated rule to select promising rules. We use the neural policy  $\pi_{\theta}$  as a guide for the rule evaluation, i.e. rules that entail the same action as the neural policy  $\pi_{\theta}$  are promising action rules. Let  $\mathcal{X}$  be a set of states. Then we evaluate the rule  $R$ , following:
$$
e v a l (R, \pi_ {\theta}, \mathcal {X}) = \frac {1}{N (R , \mathcal {X})} \sum_ {s \in \mathcal {X}} \pi_ {\theta} (s) ^ {\top} \cdot \pi_ {(\mathcal {R}, \mathbf {1})} (s), \tag {2}
$$
where  $N(R, \mathcal{X})$  is a normalization term,  $\pi_{\mathcal{R},1}$  is the differentiable logic policy with rules  $\mathcal{R} = \{R\}$  and rule weights  $\mathbf{1}$ , which is an  $1 \times 1$  identity matrix (for consistent notation), and  $\cdot$  is the dot product. Intuitively,  $\pi_{(\mathcal{R},1)}$  is the logic policy that has  $R$  as its only action rule. If  $\pi_{(\mathcal{R},1)}$  produces a similar action distribution as the one produced by  $\pi_{\theta}$ , we regard the rule  $R$  as a promising rule. We compute similarity scores between the neural policy  $\pi_{\theta}$  and the logic one  $\pi_{(\mathcal{R},1)}$  using the dot product between the two action distributions, and average them across the states of  $\mathcal{X}$ . The normalization term avoids high scores for simple rules.