Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction
Authors: Quentin Delfosse
Hikaru Shindo
Devendra Singh Dhami
Kristian Kersting
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
First-Order-Logic
Symbolic Abstraction
Interpretable Reinforcement Learning
Logic Reinforcement Learning
TL;DR: We introduce neurally-guided differentiable logic policies for reinforcement learning.

Abstract: The limited priors required by neural networks make them the dominating choice to encode and learn policies using reinforcement learning (RL). However, they are also black-boxes, making it hard to understand the agent's behavior, especially when working on the image level. Therefore, neuro-symbolic RL aims at creating policies that are interpretable in the first place.
Unfortunately, interpretability is not explainability. To achieve both, we introduce Neurally gUided Differentiable loGic policiEs (NUDGE). NUDGE exploits trained neural network-based agents to guide the search of candidate-weighted logic rules, then uses differentiable logic to train the logic agents. Our experimental evaluation demonstrates that NUDGE agents can induce interpretable and explainable policies while outperforming purely neural ones and showing good flexibility to environments of different initial states and problem sizes.

Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction
Quentin Delfosse"
Hikaru Shindo'
Technical University of Darmstadt
Technical University of Darmstadt
National Research Center for Applied Cybersecurity quentin. www.sssstddddrdssaaateeeeeeee de
hikaru.shindo www.ssshontdddastttttaaaeeeeeeee de
Devendra Singh Dhami
Kristian Kersting
Eindhoven University of Technology! Hessian Center for Al (hessian.Al) d. S. dhami @tue. .nl
Technical University Darmstadt Hessian Center for AI (hessian.AD) German Research Center for AI (DFKI) kerstingecs. tu-darmstadt.cm de
Abstract
The limited priors required by neural networks make them the dominating choice to encode and learn policies using reinforcement learning (RL). However, they are also black- boxes, making it hard to understand the agent's behaviour, especially when working the image level. Therefore, neuro-symbolic RL aims creating policies that are interpretable in the first place. Unfortunately, interpretability is not explainability. To achieve both, we introduce Neurally gUided Differentiable loGic policiEs (NUDGE). NUDGE exploits trained neural network- based agents guide the search of candidate- candidate-weighted logic rules, then uses differentiable logic to train the logic agents. Our experimental evaluation demonstrates that NUDGE agents induce interpretable and explainable policies while outperforming purely neural ones and showing good flexibility to environments of different initial states and problem sizes.
1 Introduction
Deep reinforcement learning (RL) agents use neural networks to take decisions from the unstructured input state space without manual engineering [Mnih al., 20015). However, these black-box policies lack interpretabil- ity [Rudin] 2019], i.e. the capacity to articulate the thinking behind the action selection. They are also not robust environmental changes [Pinto et al. 2017 Wulfmeier al. 2017 Although performing object detection and policy optimization independently can get over these issues Devin et al. (2018). doing so comes at the cost of the aforementioned issues when employing neural networks encode the policy.
As logic constitutes a unified symbolic language that humans use to compose the reasoning behind their behavior, logic-based policies can tackle the interpretability problems for RL. Recently proposed Neural Logic RL (NLRL) agents [Jiagg and Luo 2019] construct logic-based policies using differentiable rule learners called 0ILP [Evans and Grefenstette 2018), which
then be integrated with gradient-based optimization methods for RL. lt represents the policy as a set of weighted rules, and performs policy gradients-based learning to solve RI. tasks which require relational reasoning. It successfully produces interpretable rules, which describe each action in terms of its preconditions and outcome. However, the number of potential rules grows exponentially with the number of considered actions, entities, and their relations. NLRL a memory-intensiv approach, i.e. it generates a set of potential simple rules based on rule templates and can only be evaluated on simple abstract
"Equal contribution.
DSD contributed while being with hessian.Al and TU Darmstadt before joining TUle.
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

input state
NUDGE
interpretation as weighted rules
relational state
0. (e.ri a tyype key),
98 sage (ol agyaac
-bos (o1 (tt.://oooo_rg000000000000 agentt) 0. 0.93: (a:junt(agenn)):= closeby(02,00) (ype))
9, sape tolo enneyyy 5.040cloeeyy eie olo Lolg223
Ow
agent
key
enemy
Differentiable Forward Reasoning
explanation by input gradients
Why right?
(agent))))
on on_ right (door, eagent)
action space
action distribution
closeby aggent ydoor) on (keg, agent]
jump
right
left idle
0 gonp
" "Because the agent doesn 1 have the key, and the key on right of the agent."
Figure 1: Overview of NUDGE. Given state (depicted in the image), NUDGE computes the action distribution using relational state representation and differentiable forward reasoning. NUDGE provides interpretable and explainable policies, i.e. derives policies sets of interepretable weighted rules, and can produce explanations using gradient-basse attribution methods.
environments, created for the occasion. This approach can generate many newly invented predicates without their specification of meaning [Evans and Grefenstette, 2018], making the policy challenging to interpret for complex environments. Moreover, the function of explainability is absent, i.e. the agent cannot explain the importance each input on its decision. Explainable agents should adaptively produce different explanations given different input states. A question thus arises: How can we build interpretable and explainable RL agents that are robust to environmental changes?
To this end, we introduce Neurally gUided Differentiable loGic policiEs (NUDGE), illustrated in Figure I that embody the advantages of logic: they are easily adaptable to environmental changes, composable, interpretable and explainable (because our differentiable logic module). Given an input state, NUDGE extracts entities and their relations, converting raw states logie representations. This probabilistic relational states used to deduce actions, using differentiable forward reasoning [Evans and Grefenstette, 2018 [Shndo et al. [203aa]. NUDGE produces policy that is both interpretable, i.e. provides policy set of weighted interpretable rules that can be read out by humans, and explainable, i.e. explains which input is important using gradient-based attribution methods (Sundararajan et al. 2017] over logical representations.
To achieve an efficient learning with logic policies, we provide an algorithm to train NUDGE agents based on the PPO actor-critic framework. Moreover, we propose novel rule-learning approach, called Neurally-Guidee Symbolic Abstraction, where the candidate rules for the logic-based agents are obtained efficiently by being guided by neural-based agents. NUDGE distillates abstract representations of neural policies in the form of logic rules. Rules are assigned with their weights, and we perform gradient-bar based optimization using the PPO actor-critic framework.
Overall, we make the following contributions:
1. We propose 3 differentiable logical policies that learn interpretable rules and produce explanations for their decisions in complex environments. NUDGE uses neurally guilee symbolic abstraction to efficiently find promising ruleset using pretrained neural-based agents guidance.
2. We empirically show that NUDGE agents: (i) can compete with neural-based agents, (ii) adapt to environmen- tal changes, and (iii) are interpretable and explainable, i.e. produce interpretable policies sets of weighted rules and provide explanations for their action selections.
3. We evaluate NUDGE on classic Atari games and 3 proposed object-centric logically challenging environments, where agents need relational reasoning in dynamic game- same--paying scenarios.
We start off by introducing the necessary background. Then explain NUDGE's inner workings and present our experimental evaluation. Before concluding, we touch upon related work.
2 Background
We now describe the necessary background before formally introducing our NUDGE method.
Cooee publicly available:
com htm//4Anz//UUDGE

---

Deep Reinforcement Learning. In RL, the task is modelled as Markov decision process, -M where, at every timestep t, an agent in state 8 S. takes action or E A, receives reward r2 R(s at) and a transition to the next state St+1, according to environment dynamics P(S++1 |st, at). Deep agents attempt to learn a parametrie policy, ( 8t), order to maximize the return (i.e. E,> 7',, with y E [0. 1). The desired input to output (i.e. state to action) distribution is not directly accessible, as RI agents only observe returns. The value V= (s:) (resp. Q-value QT, ae)) function provides the return of the state (resp. statc/action pair) following the policy wa. Policy-base methods directly optimize то using the noisy return signal, leading to potentially unstable learning. Value-based methods learn to approximate the value functions Vo or Qo. and implicitly encode the policy, e.g. by selecting the actions with the highest Q-value with a high probability [Mnih al., 2015]. To reduce the variance of the estimated Q-value function, one can learn the advantage function Ag (+t, az) = Qo, ae) - Vo(s)). An estimate of the advantage function can be computed as A a+) = Σi=o 7"re+i + 7 (V((()))) - V (sx) [Mnih et al. 2016). The Advantage Actor-critic (A2C) methods both encode the policy my (i.e. actor) and the advantage function Åo (i.e. critic), and use the critic to provide feedback to the actor, as in [Konda and T Sitsiklis 1999]. To push wy to take actions that lead to higher returns, gradient ascent can be applied to L PG (0) = EE Eloog Te s)AA). Proximal Policy Optimization (PPO) algorithms ensure minor policy updates that avoid catastrophic drops (Schulman al. 2017], and can be applied to actor-critic methods. To do so, the main objective constraints the policy ratio r r(0)
xo(a)s) Kew (a]s) following
L (0) É [[min(r(0)))) (ro clip(r(0), 1 - (..+) where clip constrains the input within [1 1- [ PPO
actor-critic algorithm's 's global objective is L(e, Φ) E[L (R) LV [Ø)]. with L
being the value function loss. An entropy term can also be added to this objective to encourage exploration.
First-Order Logic (FOL). In FOL., Language L is tuple (P, D, F.F.), where P is set of predicates, 'D set of constants, F set of function symbols (functors), and V a set of variables. A term is either a constant (e.g. obj agent), a variable (e.g. 01), or term which consists of function symbol. An atom is formula p(t],....; t.). where p is a predicate symbol (e.g. closeby) and t,....; t are terms. A ground atom or simply fact is an atom with no variables (e.g. closeby(obj1. obj2)). A literal is atom (A) or its negation (-A). A clause is a finite disjunction (V) of literals. A ground clause is a clause with no variables. A definite clause is a clause with exactly one positive literal. If A, B1, B, are atoms, then A V -B1 V... V BB, is definite clause. We write definite clauses in the form of A :- BI...., Bn Atom A is called the head, and set of negative atoms {B...., } } is called the body. We call definite clauses rules for simplicity in this paper.
Differentiable Forward Reasoning is data-driven approach of reasoning in FOI. [Russell and Norvig 2010 In forward reasoning, given set of facts and set of rules, new facts are deduced by applying the rules to the facts. Differentiable forward reasoning [Evans and Grefenstette, 2018 Shindo et al. [2023a] is a differentiable implementation of the forward reasoning with tensor- based differentiable operations.
3 Neurally Guided Logic Policies
Figure 2 illustrates an overview of RL on NUDGE. They consist of policy reasoning module and a policy learning module. NUDGE performs end-to-end differentiable policy reasoning based on forward reasoning, which computes action distributions given input states. On top of the reasoning module, policies are learned using neurally-guides symbolic abstraction and an actor-critic framework.
3.1 Policy Reasoning: Selecting Actions using Differentiable Forward Reasoning.
To realize NUDGE, introduce language describe actions and states in FOL. Using it, introduce differentiable policy reasoning using forward chaining reasoning.
3.1.1 Logic Programs for Actions
In RI., states and actions are key components since the agent performs the fundamental iteration of observing the state and taking an action to maximize its expected return. To achieve an efficient computation on first-order logic in RL settings, we introduce a simple language suitable for reasoning about states and actions.
We split the predicates set P into two different sets, i.e. action predicates (PA), which define the actions and state predicates (Ps) used for the observed states. If an atom A consists of an action predicate, A is called an action atom. If A consists of state predicate, A is called state atom.

---

Policy Learning
Neurally-Guided
Differentiable Logic Policy (Actor)
Symbolic Abstraction
0.981right" (agent) (ggntt,, (11)) -bas
Critic
Policy
Neural
Gonerated Rules
0.961richt (agent) 11 (yp agent) 0ype
has (ey (00),,n www.rnght(Oaa (02,010 01),
0.65:jump" =tt ,yent)) enerry)
(02,01),
Policy Reasoning
Logical State Representation
Induced Action Atoms
Action Distribution
Input State
0. top , 0. (.99:type(cbi2,emee enneyy 0.99:type(obj3. 0.96 (oor))
0.00 left
0.OBrright:""agenn
2
0 D. ( (agent)) o. D.321jump" (agent) 0.00:id1a" (agent:)
3
0.95 right
0.05 jump
0.90:cloeeby((bbil.cbb.
0.00 Idle
0.89: has kay(ab)1)
Relational Perception
Differentiable Forward Reasoning
Action Extractor
Figure 2: NUDGE-RL. Policy Reasoning (bottom): NUDGE agents incorporate end-to-end reasoning architec- tures from input based on differentiable forward reasoning. In the reasoning step, (1) the raw input state is converted into logical representation, i.e. a set atoms with probabilities. (2) Differentiable forward reasoning is performed using weighted action rules. (3) The final distribution over actions is computed using the results of differentiable reasoning. Policy Learning (top): Using the guidance of a pretrained neural policy, a set of candidate action rules is searched by maurall-gguider symbolic abstraction, where promising action rules are produced. Then, randomly initialized weights are assigned to the action nules and are optimized using the critic of an actor-critic agent.
Definition 1 Action-state Language is tuple of (PA,Ps, V), where PA of action predicates, Ps s is a set of state predicates, 'D is a set of constants for entities, and V is a set of variables.
For example, for Getout illustrated in Figures and 2 actual actions are: left, right, jump, and idle. We define action predicates PA = (left(1),
( (eff (ight right(?) jump(()) idle( (ddee(),...) and state predicates P3- (type, closeby, ...). To encode different reasons for given game action, can generate several action predicates (e.g. right(1) and right( for right). By using these predicates, e.g. righ right(" (agent), and state atoms, e.g. type(obj1, agent). An action predicate can also be state predicate, e.g. in multiplayer settings. Now, we define rules to describe actions in the action- state language.
can compose action atoms,
:.... x!n) be state atoms. An action rule is rule, written Definition 2 Let XA be action atom and x (1)
X ,
x(().
For example, for action right, we define an action rule as:
right (3) (agent):-a -type(O1,, agent), type key), hhssaaaaaekeeee (aa_kee((01), on2right
which can be interpreted as "The agent should go right if the agent does not have the key and the key is located on the right of the agent.", ". Having several action predicates for an actual action (in the game) allows our agents to define several action rules that describe different reasons for the action.
3.1.2 Differentiable Logic Policies
We denote the set of actual actions by A, the set of action rules by C. the set of all of the facts by G = G U Gs where GA a set of action atoms and Gs is set of state atoms. G contains all of the facts produced by a given FOL language. We here consider ordered sets, i.e. each element has its index. We also denote the size of the sets as: A AA, = C), = (g], and G = (Gal-
We propose Differentiable Logic Policies, which perform differentiable forward reasoning on action rules to produce probability distribution over actions. The policy computation consists of 3 components: the relational perception module (1), the differentiable forward-reasoning module (2), and the extionnextractton module (3).

---

The policy (c..w parameterized by set of action rules C and the rules' weights W is computed as follows:
(s) = p(ar|st) = fact (Fre.w) ( (Je (Fe (81))
(1)
with fg : NRN + [0, I]c perception function that maps the raw input state & E R into a set of probabilistic atoms, [CC, W) [0, _|c + [0, 1| a differentiable forward reasoning function parameterized by a set of rules C' and rule weights W, and fact; [0, .jaa + [0, 1 action-selection function, which computes the probability distribution over the action space.
Relational Perception. NUDGE agents take an object- state representations as input, obtained by e.g. using object detection Redmon et al., 2016) or discovery [Lin et al. 2020 Delfosse et al., 2022] methods. These models return the detected objects and their attributes (e.g. class and positions). They are then converted into a probabilistic logic form with their relations, Lee a set of facts with their probabilities. An input state 8 E TR- is converted to valuation vector v E [0, 1]c which maps each fact to probabilistic value. For example, obj2), jump (agent)) A valuation vector
let G = (type(obj1, agent), typeeobj2, enemy), closeby(ob)) [0.8, 0.6, 0.3, 0.0] maps each fact to a corresponding probabilistic value. NUDGE performs differentiable forward reasoning by updating the initial valuation vector (0)) T times to obtain (TT)
Initial valuation vector (0) is computed as follows. For each ground state atom p(t....., ta)) Gs. egg. closeby(obj1, obj2), a differentiable function is called to compute its probability, which maps each term f,.... t, to vector representations according to the interpretation, e.g. obj1 and obj2 are mapped to their positions, then perform binary classification using the distance between them. For action atoms, zero is assigned as its initial probability (e.g. for jump(t)
jump("(agent)). (agent)).
Differentiable Forward Reasoning, Given a set of candidate action rules C. we create the reasoning function freasom
().W) [0, 1j6 + [0, 1j which takes the initial valuation vector and induces action atoms using weighted action rules. We assign weights to the action rules of C as follows: We fix the target programs' size, M, i.e. select M rules out of C' candidate action rules. To do so, we introduce C-dimensional weights W [WI,..., Was] where w, E RC (cf. Figure 6 in the appendix). We take the softmax of each weight vector w; E W to select M action rules in a differentiable manner.
We perform stes forward reasoning using action rules C with weights W. We compose the differentiable forward reasoning function following Shindo et al. [2023a]. It computes soft logical entailment based on efficient tensor operations. Our differentiable forward reasoning module computes new valuation v(T) including all induced atoms given weighted action rules (C, W) and initial valuation (()), Finally, we compute valuations on action atoms A E [0, by extracting relevant values from (T), We provide details in App. E
Compute Action Probability. Given valuations on action atoms VA, we compute the action distribution for actual actions. Let a, E AA be an actual action, and v'..... ... on VA be valuations which are relevant for a, (e.g. valuations of right()) (agent) and (2ght(2) (agent) in v A for the action right). We assign scores to each action a based on the log- log-sum-exp approach of Cuturi and Blondel [2017]: val (a.) = ry log Eisign exp(r</7) that smoothly approximates the maximum value of ('',..., Vn J. We use > 0 as smoothing parameter. The action distribution is then obtained by taking the softmax over the evaluations of all actions.
3.2 Policy Learning
So far, we have considered that candidate rules for the policy are given, requiring human experts to handcraft potential rules. To avoid this, template- based rule generation (Evans and Grefenstette 2018 Tiang and Luo 2019] can be applied, but the number of generated rules increases exponentially with the number of entities and their potential relations. This technique is thus difficult to apply to complex environments where the agents need to reason about many different relations of entities.
To mitigate this problem, we propose an efficient learning algorithm for NUDGE that consists of steps: sourally-guider symbolic abstraction and gradient-based optimization. First, NUDGE obtains a symbolic abstract set of rules, aligned with given neural policy. The set of candidate rules is selected by neurally-guu search, i.e. we generate a set of promising rules using a neural policy as an oracle. Then we assign randomized weights to each selected rule and perform differentiable reasoning. We finally optimize the rule weights using a "logic actor neural critic" algorithm that aims at maximizing the expected return. Let us elaborate on each step.
top-k

---

3.2.1 Neurally Guided Symbolic Abstraction
Given a welf-performing neural policy To, promising action rules for an RL task entail the same actions as the ones selected by Tg. We generate such rules by performing top-k search-based abstraction, which uses the neural policy to evaluate rules efficiently. The inputs are initial rules Co. neural policy no. We start with elementary action rules and refine them to generate better action rules. Cto open is a set of rules to be refined, and initialized as Co. For each rule C₁ E Cia_ Cto_open- we generate new rules by refining them as follows. Let
.. be an already selected general action rule. Using a randomly picked ground or non-ground state atom Ys (+ x Vi [l,...,,]), we refine the selected rule by adding a new state atom to its body, obtaining: XA + X (1)
C = XA* + X ,.....'' x
..... x Ys.
We evaluate each newly generated rule to select promising rules. We use the neural policy wo as guide for the rule evaluation, Lee. rules that entail the same action as the neural policy xo are promising action rules. Let X' be a set of states. Then we evaluate the rule R, following:
eval R, ne, X))
Σ TT(s (TR,I)())) SEX
N NR, X')
where (R,, X)) is normalization term, TR,1 is the differentiable logic policy with rules R {R) and rule weights 1, which is an 1 x 1 identity matrix (for consistent notation), and is the dot product. Intuitively, T(R,1) is the logic policy that has R as its only action rule. If ((RR1) produces similar action distribution as the one produced by ma, we regard the rule R as a promising rule. We compute similarity scores between the neural policy me and the logic one T(R.1) using the dot product between the two action distributions, and average them across the states of X. The normalization term avoids high scores for simple rules.
To compute the normalization term, we ground R. i.e. we replace variables with ground terms. We consider all of the possible groundings. Let T be the set of all of the possible variables substitutions to ground R. For each T, we get ground rule Rr = X X ,,... x!"),, substitution T to atom X. Let J = {ji...... ... In } be indices of the ground atoms X²! x(!),,.. .,.... x(a), in ordered set of ground atoms G. Then, the normalization term is computed as:
......,g"),, r, where X represents the result of applying
......,""""
N(R, = ΣI = v (O'' TET SEX
(3)
where ( is an initial valuation vector for state 8, i.e. f f.erceioe (s). Eq. quantifies how often the body atoms of the ground rule Rr are activated on the given set of states X. Simple rules with fewer atoms in their body tend to have large values, and thus their evaluation scores in Eq, 2 tend to be small. After scoring all of the new rules, NUDGE select top-k rules to refine them in the next step. To this end, all of the top-k rules in each step will be returned as the candidate ruleset C for the policy (cf. App. A for more details about our algorithm).
We perform the action-rule generation for each action. In practice, NUDGE maintains the cached history H of states and actions produced by the neural policy. For given action, the search quality increases together with the amount of times the action was selected, f.e. if H. does not contain any record of the action, then all action rules (with this action as head) would get the same scores, leading to a random search.
NUDGE has thus produced candidate action rules C, that will be associated with W to form untrained differen- tiable logic policy T(C,W)- as the ones described in Section 3.1.2
3.2.2 Learning the Rules' Weights using the Actor-critic Algorithm
In the following, we consider a (potentially pretrained) actor-critic neural agent, with Us its differentiable state-value function parameterized by (critic). Given set of action rules C. let Tee w) be differentiable logic policy. NUDGE learns the weights of the action rules in the following steps. For each non-terminal state of each episode, we store the actions sampled from the policy (a, (s)) and the next states st+1. We update the value function and the policy as follows:
5 = r + (00(8441) (6(())
(4)
(5)
W = W + WS
(6)

---

3Fishes
GetOut
2.5
20
10
-0.5
-10
-20
-1.0
400K
BOOK
10
10"
steps
steps
steps llog)
Neural PPO
MUDGE (top-1 rt!) NUDGE Itop33 ml
NUDGE (top-5 rt) NUDGE (with E.S.)
Human
Neural Logic RL
Random
Figure 3: NUDGE outperforms neural and logic baselines on the the 3 logic environments. Returns std..= obtained by NUDGE, neural PPO and logic-basek agents without abstraction through training. NUDGE (Top-k rf.), with k {1,3, 10) uses neurally-guided symbolic abstraction repeatedly until they get k rules for each action predicate. NUDGE (with E.S.) uses rule set C supervised by an expert. Neural Logic RL composes logic-based policies by generating all possible rules without neurally-guide symbolic abstraction (Jiang and Luo 2019]. Random and human baselines are also provided.
The logic policy WiC,w) thus learn to maximize the expected return, potentially bootstrapped by the use of a pretrained neural critic. Moreover, to ease interpretability. NUDGE can prune the unused action rules (i.e. with low weights) by performing top-k selection on the optimized rule weights after learning.
4 Experimental Evaluation
We here compare neural agents' performances to NUDGE ones, showcase NUDGE interpreiable policies and its ability to report the importance of each input on their decisions, i.e. explainable logic policies. We use DQN agents (on Atari environments) and PPO actor-critic (on logic-ones) as neural baselines, for comparison and PPO as pretrained agents to guide the symbolic abstraction. All agent types receive object-centric descriptions of the environments. For clarity, we annotate action predicates action rules with specific names purpose, e.g. ri right[to_key] instead of right()) (when the rule describes an action right motivated to get the key).
We intend to compare agents with object-centric information bottlenecks. We thus had extract object-centric states of the Atari environments. To do so, we make use of the Object-Centric Atari library [Delfosse er al. 2023 As Atari games do not embed logic challenges. but are rather desgined test the reflexes of human players, we also created 3 logic-oriented environments. We thus have modified environments from the Procgen Mohanty et al.
2020] environments that are open-sourced along with our evaluation have object-centric representations. Our environments are easily hackable. We provide variations of these environments also to evaluate the ease of adaptation of every agent type. In GetOut, the goal is to obtain key, and then go to a door, while avoiding moving enemy. GetOut+ is a more complex variation with larger world containing 5 enemies (among which 2 are static). In 3Fishes, the agent controls a fish and is confronted with 2 other fishes, one smaller (that the agent needs to "eat", i.e. go to) and one bigger, that the agent needs to dodge. A variation is 3Fishes-C. C, where the agent can eat green fishes and dodge red ones. Finally, Loot, the agent can open 1 or 2 chests and their corresponding (iee same color) keys. In Loot-C, the chests have different colors. Further details and hyperparameters are provided in App. D
We aim to answer the following research questions: Q1. How does NUDGE compare with neural and logic baselines? Q2. Can NUDGE agents easily adapt to environmental changes? Q3. Are NUDGE agents interpretable and explainable?
NUDGE competes with existing methods (Q1). We compare NUDGE with different baselines regarding their scores (or returns). First, we present scores obtained by trained DQN, Random and NUDGE agents (with expert supervision) on 2 Atari games (cf. Table I]. Our result show that NUDGE obtain better (Asterix) similar (Freeway) scores than DQN. However, as said, Atari games are not logically challenging. We thus evaluate NUDGE on logic environments. Figure shows the returns in GetOut, 3Fishes, and Loot, with descriptions for each baseline in the caption. NUDGE obtains better performances than neural baselines (Neural PPO) on 3Fishes, is more stable on GetOut, i.e. less variance, and achieves faster convergence on Loot. This shows that NUDGE successfully distillates lasic-based policies competing with neural baselines in different complex environments.

---

Score (t) Random
DQN
NUDGE
Score (1)
Random
Classic
NUDGE
Asterix
235 1134
124.5
6259
1115
GetOut
-22.510.41
11.5914.29
17 17.8612..6
Freeway
0.0 +0
25.8
21.4 =0.8
3Fish
-0.73+0005
-0.24=0.29
0.05++0.27
Loot
0.37
0.51
0.66
Table 1: Left: NUDGE agents can learn successful policies. Trained NUDGE agents (with expert supervision) scores (avg. + std.) on 2 OCAtari games (Delfosse et al.) [023]] Random and DQN (from van Hasselt et al.) (2016)) are also provided. Right: NUDGE outperforms non-differentiable symbolic reasoning baselines. Returns obtained by NUDGE, classic and random agents our the 3 logic environmnents.
We also evaluate agents with baseline without symbolic abstraction, where candidate rules are generated not being guided by neural policies, i.e. accept all of the generated rules in the rule refinement steps. This setting corresponds to the cemplate-based approach [Jiang and Luo 2019), but train the agents by the actor-critic method, while vanilla policy gradient [Williams] 1992 is employed in fJangg and Luo 2019]. For the no-abstraction baseline and NUDGE, we provide initial action rules with basic type information, e.g. (agent): typ agent), (02 enemy), for each action rule. For this baseline, we generate 5 rules for GetOut, 30 rules for 3Fishes, and 40 rules for Loot in total to define all of the actual actions. NUDGE agents with small & tend to have less rules, e.g. 5 rules in Getout, 6 rules in 3Fishes, and rules in Loot for the top-1 refinement version. In Figure 3 the no-abstraction baselines perform worse than neural PPO and NUDGE in each environment, even though they have much more rules in 3Fishes and Loot. NUDGE thus composes efficient logic-based policies using surrall-guided symbolic abstraction. In App. B.I we visualize the transition of the distribution of the rule weights in the GetOut environment.
To demonstrate the necessity of differentiability within symbolic reasoning in continuous RL. environments, we simulated classic (i.e. discrete logic) agents, that reuse NUDGE agents' set of action rules, but with all weights fixed to 1.0. Such agents can still play better than ones with pure symbolic reasoners, as they still use fuzzy (i.e. continuous) concepts, such closeby. In all our environments, NUDGE clearly outperforms the classic counterparts (cf. Table I right). This experiment shows the superiority of differentiable policy reasoning (embedding weighted action rules) over policy without weighted rules (classic).
To further reduce the gap between our environments and those from enviromments Cao et al. [022]] we have
adapted our loot environment (Loot-hard). In this environment, the agent first has to pick up keys and open their corresponding saves, before being able to exit the level (by going to a final exit tile). Our results in Table 2
(bottom), as well as curves on Figure 10 in the appendix show the dominance of NUDGE agents.
NUDGE agents adapt to environment changes (Q2). We used the agents trained on the basic environment for this experimental evaluation, with no retraining or finetuning. For 3Fishes-C, we simply exchange the atom is bigger with the atom same color. This easy modification is not applicable on the black-box networks of neural PPO agents. For GetOut+ and Loot-C, we do not apply any modification to the agents. Our results are summarizee in Table Note that the agents obtains better performances in the 3Fishes-C variation, dodging a (small) red fish is eas- ier than a big one. For GetOut+, NUDGE performances have decreased as avoiding 5 enemies drastically increases the difficulty of the game, On Loot-C, the performances are similar to the ones obtained in the original game. Our experiments show that NUDGE logic agents can easily adapt to environmental changes.
Score (1)
3Fishes-C
GetOut+
Loot-C
Random
-0.6 +0.2
-22.5 40.4
0.6 #0.3
Neural PPO NUDGE
-0.4 10.1
-20.9 #0.6
0.8 #0.5
3.6 =0.2
3.6 #3.0
5.6 #0.3
Table 2: NUDGE agents adapt to environmental changes and solve logically-challenging environments. Returns obtained by NUDGE, neural PPO and random agents on modified environments and on Loot-hard.
NUDGE agents are interpretable and explainable (Q3). We show that NUDGE agents are interpretable and explainable by showing that (1) NUDGE produces interpretable policy as set of weighted rules, and (2) NUDGE can show the importance of each atom, explaining its action choices.
The efficient neurally-guidee learning on NUDGE enables the system to learn rules without inventing predicates with no specific interpretations, which are unavoidable in remplate-based approachs [Evans and Grefenstette 2018 Jiang and Luo] 2019). Thus, the policy can easily be read out by extracting action rules with high weights. Figure 5 shows some action rules discovered by NUDGE in GetOut. The first rule says: "The agent should jump when the enemy is close to the agent (to avoid the enemy).". The produced NUDGE is an interpretable policy

---

0.57: jump (agent) :-type (0rp (02,
0.29 122g:righ/joo (oo (to_key
(agent):
-type agentt) koy)
01), -has hha__ey (01).
0.56:right [to_d (agent): :type (01,) (0g,t) type (02, oooo (02, 01)) _as__ey (01).
/to (agent): cype ttype oxpp
-has_key(00) (01).
0.30:1eft tood (agent): :-type (01,,antt)))))) ttype , (0__1eft(022,01) _aa__kyy (01).
Figure 5: NUDGE produces interpretable policies sets of weighted rules. A subset of the weighted action rules from the Getout environment. Full policies for every logic environment are provided in App. B.3
with its set of weighted rules using interpretable predicates. For each state, we can also look at the valuation of each atom and the selected rule.
Gradients of actions w.r.t input atoms
Moreover, contrary to classic logic policies, the differ- entiable ones produced by NUDGE allow to com- pute the attribution values over the logical representa- tions via backpropagation of the policies' gradients. We can compute the action gradients w.r.t. input atoms, i.e. dv / (0), as shown in Figure which represent the relevance scores of the probabilistic input atoms v()) for the actions given a specific state. The expla- nation is computed on the state shown in Figure [T] where the agent takes right as its action. Important atoms receive large gradients, e.g. -has key(agent))) and on ooorriiio right(objjl obj1). By extracting relevant atoms with large gradients, NUDGE can produce clear explanations for the action selection. For example, by extracting the atoms wrapped in orange in Figure NUDGE can explain the motivation: "The agent decides to go right because it does not have the key and the key is located on the right-side of it.". These results show that NUDGE policies are interpretable and explainable, i.e. each action predicate is defined by rules that humans can fully understand, and gradient-based explanations for the action selections be efficiently produced.
jump .23
.04
.04
(eft(e
.01
.01
.01
right
[eft
right door)
.01
dooseby (obi1 .bbj4)
has. keylagent
-has, kee (agent)
on mn_lefl(obye sehtobj4,ob11) obb1)
on on..right rightt objii
type(obi], a,gent)
type(obj2, key)
typetobj4 .enemy)
typel (bb)4 key)
Figure 4: Explanation using inputs' gradients. The action gradient w.r.t. input atoms, i.e. åv (0v((0)) on the state shown in Figure [ right was selected, due to the highlighted relevant atoms (with large gradients).
We also computed explanations of a neural agent on the same state, also using the input-gradient method. The concepts which received the highest explanation scores are: the key x-axis position (1.22), the key x-axis velocity (1.21), the enemy y-axis velocity (1.10), and the door x-axis position (0.77). For a consistent explanation, the agent x-axis position should also be among the highest scores. For this specific state, the neural agent seems to be placing its explanations on non relevant concepts (such as the key's 's velocity). Stammer et al. 2202 have shown that neural networks trained on logical concepts tend to produce wrong explanations often, and additional regularization/supervisii about explanations during training is necessary to force neural networks to produce correct explanations. This approach requires additional efforts to label each state with its correct explanation. NUDGE policies produces better explanations than neural-based agents. More explanations produced by NUDGE are provided in Figure 11 in the appendix.
5 Related Work
Relational RL Dzeroski et al. 2001 Kersting et al. 2004 Kersting
FOL
N.G.
Int. Exp.
and Driessens 2008 Lang et al. 2012] has been developed to
NLRL NeSyRL
X
X
tackle RL tasks in relational domains. Relational RL frameworks incorporate logical representations and use probabilistic reasoning.
X
X
DifTSES NUDGE
X
x
With NUDGE, we make use of differentiable logic programming.
The Neural Logic Reinforcement Learning (NLRL) (Jang and Luo
Table Logic-based RL methods comparison: First Order Logic (FOL), neurally-guided search (N.G.), inter- pretability (Int.). and explainability (Exp.).
2019 framework is the first to integrate Differentiable Inductive
Logic Programming (AILP) [Evans and Grefenstette 2018 to RI. domain. @ILP learns generalized logic rules from examples by gradient-based optimization. NLRL adopts ØILP a policy function. We extend this approach by proposing neurally-guided symbolic abstraction embracing extensive work of ØIL [Shindo et al. 2021b 2023b) for learning complex programs including functors in visual scenes, allowing agents to learn interpretable action rules efficiently for complex environments.

---

GALOIS C Cao et al. 2022 is a framework to represent policies as logic programs using the sketch setting [Solar Lezama 2008], where programs are learned to fill blanks. NUDGE performs structure learning from scratch
using policy gradients. KoGun [Zhang et al. 2020] integrates human knowledge as prior for RI. agents. NUDGE learns policy as a set of weighted rules and thus also can integrate human knowledge. Neuro- Symbolic RL (NeSyRL.) [Kimura et al. [021] uses Logical Neural Networks (LNNs) [Riegel et al. 2020 for the policy computation. LNNS parameterize the soft logical operators while NUDGE parameterizes rules with their weights. Deep Relational RL approaches (Zambaldi et al. 2018 achieve relational reasoning as a neural network, but NUDGE explicitly encodes relations in logic. Many languages for planning and RL tasks have been developed (Fikes and Nilsson 1971 Fox and Long 2003]. Our approach is inspired by situation calculus [Reiter 2001], which is an established framework to describe states and actions in logic.
Symbolic programs within RL have been investigated, e.g. program guided agent (Sun et al.
2020 program
synthesis (Zhu et al. [2019) PIRL. |Verma et al. 2018). SDRI. [L.yu et al. 2019]. interpretable model- hier- archical Ri, and Fekri| 202T], deep symbolic policy Landajuela et al. [202T]. and DiftSES (Zheng et al. [2021] These approaches use domain specific languages or propositional logic, and address either of interpretability or
explainability of RL. To this end, in Table 3 we compare NUDGE with the most relevant approaches that share at least 2 of the following aspects: supporting first-order logic, neural guidance, interetability and explainability. NUDGE is the first method to use neural guidance for differentiable first-order logic policies and to address both interpretability and explainability. Specifically, PIRL develops functional programs with neural guidance using sketches, which define a grammar of programs to be generated. In contrast, NUDGE performs structure learning from scratch using a neural guidance with a language bias of mode declarations (Muggleton) [995| CCroperr et al.) 2022 restricting the search space. As NUDGE uses first-order logic, it can incorporate background knowledge in a declarative form, e.g. with a few lines of relational atoms and rules. To demonstrate this, NUDGE has been evaluated challenging environments where the main interest is relational reasoning.
6 Conclusion
We proposed NUDGE, an interpretable and explainable policy reasoning and learning framework for reinforce- ment learning. NUDGE uses differentiable forward reasoning to obtain a set of interpretable weighted rules as policy. It performs ceurally-guided symbolic abstraction, which efficiently distillates symbolic representations from a neural policy, and incorporate an actor-critic method to perform gradient- based policy optimization. We empirically demonstrated that NUDGE agents (1) can compete with neural based policies, (2) use logical representations to produce both interpretable and explainable policies and (3) can automatically adapt or easily be modified and are thus robust to environmental changes.
Societal and environmental impact. As NUDGE agents can explain the importance of each the input on their decisions, and as their rules are interpretable, it can help us understanding the decisions of RL agents trained in sensitive complicated domains, and potentially help discover biases and misalignments of discriminative nature. While the distillation of the learned policy into a logic ones augments the computational resources needed for completing the agents learning process, we believe that the logic the agents' abstraction capabilities will overall reduce the need of large neural networks, and will remove the necessity of retraining agents for each environmental change, leading to an overall reduction of necessary computational power.
Limitation and Future Work. NUDGE is only complete if provided with a sufficiently expressive language (in
terms of predicates and entities) to approximate neural policies. Interesting lines of future research can include: (i) an automatic discovery of predicates, using e.g. predicate invention [Muggleton et al. (2015), (ii) augmenting the number of accessible entities to reason on. Explainable interactive learning [Teso and Kersting 2019 in RL can integrate NUDGE agents, since it produces explanations on logical representations. Causal RL MMadumal et al.
2020 and meta learning [Mishra et al., 2018 also constitute interesting avenues for NUDGE's 's development. Finally. we could incorporate objects extraction methods [Delfosse et al. 2022 within our NUDGE agents, to obtain logic agents that extract object and their properties from images.
Acknowledgements. The authors thank the anonymous reviewers for their valuable feedback. This research work has been funded by the German Federal Ministry of Education and Research, the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK) K) within their joint support of the National Research Center for Applied Cybersecurity ATHENE, via the "SenPai: XReLeaS" project. We gratefully acknowledge support by the Federal Ministry of Education and Research (BMBF) AI lighthouse project "SPAICER" (0IMK20015E), the EU ICT-48 Network of AI Research Excellence Center "TAILOR" (EU Horizon 2020, GA No 952215), and the Collaboration Lab "Al in Construction" (AICO).
10

---

References
Yushi Cao, Zhiming Li, Tianpei Yang, Hao Zhang, Yan Zheng, Yi Li, Jianye Hao, and Yang Liu. GALOIS: boosting deep reinforcement learning via generalizable logic synthesis. CoRR, 2022.
Andrew Cropper. Sebastijan Dumancic, Richard Evans, and Stephen H. Muggleton. Inductive logic programming 30. Mach. Learn., 2022.
Marco Cuturi and Mathieu Blondel. Soft-DTW: a differentiable loss function for time-series. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Quentin Delfosse, Patrick Schramowski, Martin Mundt, Alejandro Molina, and Kristian Kersting. Adaptive rational activations to boost deep reinforcement learning. CoRR, 2021.
Quentin Delfosse, Wolfgang Stammer, Thomas Rothenbacher, Dwarak Vittal, and Kristian Kersting. Boosting object representation learning via motion and object continuity. CoRR, 2022.
Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sebastian Szwwiertnia, and Kristian Kersting, Ocatari: Object- centric atari 2600 reinforcement learning environments. CoRR, 2023.
Coline Devin, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric representations for generalizable robot learning. IEEE International Conference on Robotics and Automation, 2018.
Saso Dzeroski, Luc De Raedt, and Kurt Driessens. Relational reinforcement learning. Mach. Learn., 2001.
Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. J. Artif. Intell. Res., 2018.
Richard Fikes and Nils J. Nilsson. STRIPS: A new approach to the application of theorem proving to problem solving. Artif. Intell., 1971.
Maria Fox and Derek Long. PDDL2.1: an extension to PDDL. for expressing temporal planning domains. J. Artif. Intell. Res., 2003.
Zhengyao Jiang and Shan Luo. Neural logic reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, International Conference on Muchine Learning, 2019.
Kristian Kersting and Kurt Driessens. Non-parametric policy gradients: a unified treatment of propositional and relational domains. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, International Conference on Machine Learning. 2008
Kristian Kersting, Martijn Otterlo, and Lue De Raedt. Bellman goes relational. In Carla E, Brodley, editor, International Conference on Machine Learning, 2004.
Daiki Kimura, Masaki Ono, Subhajit Chaudhury, Ryosuke Kohita, Akifumi Wachi, Don Joven Agravante, Michiaki Tatsubori, Asim Munawar, and Alexander Gray. Neuro-symbolic reinforcement learning with first-order logic. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Conference Empirical Methods in Natural Language Processing, 2021.
Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In Sara A. Solla, Todd K. Leen, and Müller, editors, Advances in Neural Information Processing Systems 12, 1999.
Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep reinforcement learning. In International Conference on Machine Learning, 2021.
Tobias Lang, Mare Toussaint, and Kristian Kersting. Exploration in relational domains for model-based reinforcement learning. J. Mach. Learn. Res., 2012.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposi- tion. In International Conference on Learning Representations, 2020.
Daoming Lyu, Fangkai Yang, Bo Liu, and Steven Gustafson. SDRL:
interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In The Thirty-Third AAAI Conference on Artificial Intelligence, 2019.

---

Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. The Phirty-Fourds AAAI conference on Artificial Intelligence (AAAI), 2020.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In 6th International Conference on Learning Representations, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nat., 2015.
Volodymyr Mnih, Adrià Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria- Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33rd International Conference on Machine Learning, 2016.
Sharada P. Mohanty, Jyotish Poonganam, Adrien Gaidon, Andrey Kolobov, Blake Wulfe, Dipam Chakraborty, Grazvydas Semetulskis, Jo
textascittilde ao Schapke, Jonas Kubilius, Jurgis Pasukonis, Linas Klimas, Matthew J. Hausknecht, Patrick MacAlpine, Quang Nhat Tran, Thomas Tumiel, Xiaocheng Tang. Xinwei Chen, Christopher Hesse, Jacob Hilton, William Hebgen Guss, Sahika Genc, John Schulman, and Karl Cobbe. Measuring sample efficiency and generalization reinforcement learning benchmarks: Neurips 2020 procgen benchmark. In Hugo Jair Escalante and Katja Hofmann, editors, NeurIPS 2020 Competition and Demonstration Track, 2020.
S. Muggleton. Inverse Entailment and Progol. New Generation Computing, Special issue on Inductive Logic Programming, 1995.
Stephen H. Muggleton, Dianhuan Lin, and Alireza Tamaddoni-Nezhial Meta-interpretive learning of higher-order dyadic datalog: predicate invention revisited. Mach. Learn., 2015.
Shan-Hwei Nienhuys-Cheng and Ronald de Wolf. Foundations of Inductive Logic Programming. Springer, 1997.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning- In Doina Precup and Yee Whye Teh, editors, International Conference Machine Learning. 2017.
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Computer Vision and Patlern Recognition, 2016.
Raymond Reiter. Knowledge in Action: Logical Foundations for Specifying and Implementing Dynamical Systems. MIT Press, 2001.
Ryan Riegel, Alexander G. Gray, Francois P. S. Luus, Naweed Khan, Ndivhuwo Makondo, Ismail Yunus Akhalwaya, Haifeng Qian, Ronald Fagin, Francisco Barahona, Udit Sharma, Shajith Ikbal, Hima Karanam, Sumit Neelam, Ankita Likhyani, and Santosh K. Srivastava. Logical neural networks. CORR, 2020.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell., 2019.
Stuart Russell and Peter Norvig. Artificial intelligence a modern approach. Prentice Hall, 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 2017.
Hikaru Shindo, Devendra Singh Dhami, and Kristian Kersting. Neuro-symbolic forward reasoning. CoRR, 2021a.
Hikaru Shindo, Masaaki Nishino, and Akihiro Yamamoto. Differentiable inductive logic programming for structured examples. In Fhitty-Fifth AAAI Conference on Artificial Intelligence (AAAI), 2021b.
Hikaru Shindo, Viktor Pfanschilling, Devendra Singh Dhami, and Kristian Kersting, ailp: thinking visual scenes differentiable logic programs. Mach. Learn., 2023a.
Hikaru Shindo, Viktor Pfanschilling, Devendra Singh Dhami, and Kristian Kersting. Learning differentiable logic programs for abstract visual reasoning. CoRR, 2023b.

---

Armando Solar-Lezama. Program Synthesis by Sketching. PhD thesis, University of California at Berkeley, 2008.
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Revising neuro- symbolic concepts by interacting with their explanations. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Shao-Hua Sun, Te-Lin Wu, and Joseph J. Lim. Program guided agent. In International Conference Learning Representations, 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning. 2017.
Stefano Teso and Kristian Kersting. Explanatory interactive machine learning. Vincent Conitzer, Gillian K. Hadfield, and Shannon Vallor, editors, AAAI/ACM Conference on Al, Ethics, and Society, 2019.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Dale Schuurmans and Michael P. Wellman, editors, AAAI Conference on Artificial Intelligence. 2016
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmati- cally interpretable reinforcement learning. In International Conference Machine Learning, 2018.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 1992.
Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual aligmment transfer learning. In Conference on Robot Learning, 2017.
Duo Xu and Faramarz Fekri. Interpretable model-based hierarchical reinforcement learning using inductive logic programming. CoRR, abs/2106.1 1417, 2021.
Vinícius Flores Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David P. Reichert, Timothy P. Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew M. Botvinick, Oriol Vinyals, and Peter W. Battaglia. Relational deep reinforcement learning. CoRR, 2018.
Peng Zhang, Jianye Hao, Weixun Wang, Hongyao Tang, Ma, Yihai Duan, and Yan Zheng, Kogun: Accelerating deep reinforcement learning via integrating human suboptimal knowledge. In International Joint Conference on Artificial Intelligence, 2020.
Wenqing Zheng, S P Sharan, Zhiwen Fan, Kevin Wang, Yihan Xi, and Zhangyang Wang. Symbolic visual reinforcement learning: A scalable framework with object-level abstraction and differentiable expression search. CoRR, abs/2106.11417 2021.
He Zhu, Zikang Xiong, Stephen Magill, and Suresh Jagannathan. An inductive synthesis framework for verifiable reinforcement learning. In ACM-SIGPLAN Symposium on Programming Language Design and Implementation, 2019.

---

Supplemental Materials
A
Details on Neurally-Guided Symbolic Abstraction
We here provide details on the neurally-guided symbolic abstraction algorithm.
A.1 Algorithm of Neurally-Guided Symbolic Abstraction
We show the algorithm of neurally-guided symbolic abstraction in Algorithm T
Algorithm Neurally- Guided Symbolic Abstraction
Input: Co, Ne, hyperparameters ( Meam: Ieam.)
1: Cto open f Co
2: C+ 0
3: ==0
while <. Tham do
5:
Choum t- 0
6:
for C2 E Cio_ open do
7:
C= CUC
8:
for R ((C,) do
# Evaluate each clause score = eval(R, To)
9:
# select top-k rules
10:
Caram top kCCeam score, Naam) selected rules refined next
11:
Cto_open Cdeam
12:
return
A.2
Rule Generation
At line 8 in Algorithm given action rule C, we generate new action rules using the following refinement operation:
= (XA x(1)
x Ys Ys E G3 A Ys + x
where G3 is non-ground state atoms. This operation is specification of (downward) refinement operator, which a fundamental technique for rule learning in ILP [NienhuysCCheng and de Wolf 1997], for action rules to solve RL tasks.
define the search space, i.e. G3 in Eq. 7 which are defined as follows. A mode declaration is either head declaration modeh(I; p(mdt,,......................... mdt,)) or body declaration modeb(r, p(mátı,... mdt,)), where datatype. A mode datatype a tuple (pm, dt), where pm is a place-marker and dt is datatype. A place-marker is either #. which represents constants, or + (resp. -), which represents input (resp. output) variables. represents the number of the usages of the predicate compose solution. Given set of mode declarations, we can determine finite set of rules be generated by the rule refinement.
We use mode declarations [Mgggleton 1995 Cropper et 2022
an integer, p a predicate, and måt; mode
Now we describe mode declarations we used in our experiments. For Getout, we used the following mode declarations:
modeb(2, bopet
nodee(t,havv_kkey(oooep

---

For 3Fishes, we used the following mode declarations:
modeb(2, sype(-objjec,
m1deb(1, olo modeb(1, (o0
modeb(1, maceb((:a._botoooo--0000ee000000
modeb(1, mn
modeb(1, mocebblbagger.tafffooooooo-ooooo
modeb(1, moeel(((iagaLeve((ooooeeeoooooo
For Loot, we used the following mode declarations:
modeb(2, sypel-object,
modeb(2, coloc//))oojojjtttccoororr))
modeb(1, moleb1.0.._ogg(-bjjcccrrroooe))j
modeb(1, (aabf(((-o tobjectt)
modeb 1, have_key (oddab)
B Additional Results
B.1 Weights learning
Fig. shows the NUDGE agent ((C.W) parameterized by rules C and weights W before training (top) and after training (bottom) on the GetOut environment. Each element on x-axis the plots corresponds to an action rule. In this examples, we have 10 action rules C (Co, Ci...., Co). and we assign M = 5 weights i.e. W Woo Wi,..., wa). The distributions of rule weighs with softmax are getting peaked by learning maximize the return. The right 4 rules are redundant rules, and theses rules get low weights after learning.
B.2 Deduction Pipeline
Fig. provvdde the deduction pipeline of NUDGE agent on 3 different states. Facts can be deduced from object detection method, or directly given by the object centric environment. For state #1, the agent chooses to jump the jump action is prioritized over the other ones and all atoms that compose this rules' body have high valuation (including closeby). In state #2, the agent chose to go left as the rule left left_key is selected. In state #3, the agent selects right as the rule right door has the highest forward chaining evaluation.
B.3 Policies of every logic environment.
We show the logic policies obtained by NUDGE GetOut, 3Fishes, and Loot in Fig. X e.g. the first line of GetOut, "0.574 jump(X):: -closeb(011, 02), type agent), type(02, enemy).", represents that the action rule chosen by the weight vector Wi with value 0.574. NUDGE agents have several weight vectors Wi,..., War and thus several chosen action rules are shown for each environment.
C Illustrations of our environments
We showcase in Fig. 9 one state of the 3 object-centric environments and their variations. In GetOut (blue humanoid agent), the goal is to obtain a key, then to a door, while avoiding a moving enemy. GetOut-2En

---

Weights on action rules (before training)
w 0
W
W
3
Weight
00
W 4
00
Jump C_0
Left Jey C_1
Right bey C_2
Left dolr C_3
Right doer
Stay
Jamp doer C_6
Left nothing
Rghi eeneryy
Stay, m.nnhhiinng C_9
C_4
C_5
C_7
C_8
Weights on action rules (after training)
0_0
08
3
06
W_4
Weights
00
Jump C_
Left key
Right kay C_2
Left laft
Right door
Stay
jump_door
jump, door
Left nothing light, lightenerr Stay not.intiigg
C
C
C_4
C_5
C_6
C_7
C_8
C_9
Figure 6: Weights on action rules via softmax before training (top) and after training (bottom) on NUDGE in GetOut. Each element on the x-axis of the plots corresponds an action rule. NUDGE learns to get high returns while identifying useful action rules to solve the RL task. The right 5 rules are redundant rules, and theses rules get low weights after learning.
observartion
object-centric state
induced facts
induced actions
final actions
1.0: agge()1.agent)
0.99: closebyl01,0
State #1
type(02.key) 1.0: type(O3,door) door)
0.99: on an__eeff(22001.
0.99: left(03,01)
0.983: jumplagent)
right, key(agent)
0.89; jump(agent)
4
1.0: type(04,enent) enemm)
10: Oos 2)
0.99: not has key(ageey))))
0.048: has keylagent)
not_has___
0.051: left keylagent)
0.051: daor, right(agentt
0.01: leftlagent)
0.09: rightlagent)
O
1.0: O11 10)
0.048: on right(02,
door leftlagent)
0.01: idle(agent)
post
0.048: on right(O3.01)
induction
*:: softor+ softmax
State #2
0.02: jumplagent)
state #3
0. 02: jump(agent)
0.94: right(agent)
>>>>>>>>>>>>>
0 0.94: right(agent)
0.03: leftlagent)
03: left(agent)
0.01: idle(agent)
0. 0.01: idle(agent)
Figure The logic reasoning of NUDGE agents makes them interpretable. The detailed logic pipeline for the input state #1 of the Getout environment and the condensed action selection for state #2 and state #3.
is a variation with 2 enemies. In 3Fishes, the agent controls green fish and confronted with 2 other fishes, one smaller (that the agent need to "eat", i.e. go to) and one bigger, that the agent needs to dodge. A variation is 3Fishes-C, where the agent can eat green fishes and dodge red ones, all fishes have the same size. Finally, in
Loot, the (orange) agent is exposed with or 2 chests and their corresponding (i.e. same color) keys. Loot-C. the chests have different colors. All 3 environment are stationary in the sense of Delfosse et al. 2021

---

GetOut
0. (0100ebb0110b
0. http://wwwaa.aaaaa...oaaaaaeaaaa enemyyy.
att type key).
type
0. http://aaaaaaaaaaaaa.aaaaaaaaaa.
(02, door).
0. 0.291:right..o..ge (X) --not__ave__ey o, agentt) t02, key).
0.562:right.go. to_ door -have_key snwve__key)),,n___.ee.). ott.
door).
#3Fishes
0.779:right_to_.er (X) :-10:_ssbbll wtw.sssbgee__hhaeeeeeeeeeee 00, (tppe()).ypp@)...... (op.
(0pe(02,fish))
1.445:down_to_ddodee ott.puaggeei.aaaaa...oooooooaaaa .n__eeft (ype
than (12,01)
(0pe((22fiihh)
o.t79:doom_to_eat (X) ____
(00 agentt,,,
(0pp
o.699:up.to_dodgeehh oagp://aaoo_olaaaa.....o.aaaaaaa
(02, agge (02
0. 000.:.a..................aaaaaa. on (02, (02,e1))type@01 (0),,
type
1.B.:.aff..c.aaaa000:ddeee0000.. oggpa:ae____aaall00000.aaaa.aa11 art. (ype(02, fish).
# Loot
w._.44up_ttodooee door :0) (aa..2)),hae__kyy(000 (o__)))))_o_p
typpe(01, (.yp)
(02,
0. ro. 02)) on_right (02, typp type (Op)
0.7222rrighh.todddo0000000000000 (X) :101 hhweel),,aaeeeeeaeeeaeeee (e (ggent))
agent),
(02,
01), tttp:0/.......ccell...aaaaroo.op
http://eeeeccooooaaaeeeeoaollppp agentt))
type do2,
(0_top(01,022) ayyp(( (t
(Q :00o haave_key (02 tyyp
(02,
:tt.::/fttt.ookkkyoooooooooooood on_ on_leetttontt0000e (02,01) type agent) type (02,key).
Figure 8: NUDGE produces an interpretable policy as set of weighted rules. Weighted action rules discovered
NUDGE in the each logic environment.
D Hyperparameters and rules sets
D.I
Hyperparameters
We here provide the hyperparameters used in our experiments. set the clip parameter <elip = 0.2, the discount factor 0.99. We use the Adam optimizer, with 1e - 3 as actor learning rate, 3e 4 as critic learning rate. The episode length is 500 timesteps. The policy is updated every 1000 steps We train every algorithm for 800k steps on each environment, apart from neural PPO, that needed 5.M steps on Loot. We use epsilon greedy strategy with e = max(e -piande 0.02).
D.2
Rules set
All the rules set C of the different NUDGE and logic agents are available at https://githhb: httptttuu...om//k/a k4ntz/ NUDGE the folder nsfr/nsfr/ http///ss///aaaaaaannaaaaaaaaaaa
E Details of Differentiable Forward Reasoning
We provide details differentiable forward reasoning used in NUDGE, We denote valuation vector at time step v(t) [0, _jc, We also denote the i-th element of vector by x|i], and the (i, element of matrix X by X(i, j]. The same applies to higher dimensional tensors.

---

GetOut
GetOut-2En
Loot
Loot-C
3issess
3Fishes-C
Figure 9: Pictures of our environments (GetOut, Loot and 3Fishes) and their variations (GetOut-2En. Loot-C and 3Fishes-C). these environments can provide object-centric state descriptions (instead pixel-based states).
E.1 Differentiable Forward Reasoning
We compose the reasoning function freason J0, 1 - 10, which takes the initial valuation vector and returns valuation vector for induced action atoms. We describe each step in detail.
(Step 1) Encode Logic Programs to Tensors. To achieve differentiable forward reasoning, each action rule is encoded tensor representation. Let S be the number of the maximum number of substitutions for existentially quantified variables in C, and be the maximum length of the body rules in C. Each action rule C, E is encoded to tensor I E NGXSXL, which contain the indices of body atoms. Intuitively, I, k. (] is the index of the 1-th fact (subgoal) the body the i-th rule to derive the j-th fact with the k-th substitution for existentially quantified variables.
For example. let Rg jumplagent) 001, (0,, agent), type(02, enemy), closeby(01, 02) E C and F- j jump(agent) E G. and we assume that constants for objects are {obj1, obj2]. Ro has existentially quantified variables and 02 on the body, so we obtain ground rules by substituting constants. By considering the possible substitutions for 01 and 02, namely (01/obj 1, 02/ o2/obj2) and (01/obj2, 02/obj1}, we have two ground rules, as shown in top of Table 4 Bottom rows of Table shows elements of tensor Io..o.: and Lo.. Facts G and the indices are represented on the upper rows in the table. For example, 10,,0,, [3, 6, 7] because Ro entails jump(agent) with the first (k- = 0) substitution T = (01/obj1, 02/obj2). Then the subgoal atoms are (type(obji agent), sype(obj2 enemy), (eloseby(objia. obj]), which have indices [3, 6, 7]. respectively. The atoms which have a different predicate, e.g., (ooseby(obji,, obj2). will never be entailed by clause Ry- Therefore, the corresponding values are filled with 0, which represents the index of the false atom.
(Step 2) Assign Rule Weights. We assign weights to compose the policy with several action rules as follows: (i) We fix the target programs' size as M, i.e. where we try to find policy with M action rules. (ii) We introduce C-dim weights W [w....., w M) (iii) We take the sofimax each weight vector w, E W and softly choose M action rules out of C action rules to compose the policy.
(Step 3) Perform Differentiable Inference. We compute 1-step forward reasoning using weighted action rules, then we recursively perform reasoning to compute T T-step reasoning.

---

(k 0)
j jump(agent)) :tyyee
: aggnttt),.gg...... (yppeoobj,,enmmy. enemy), cloaeby
(k 1)
jump(agent)) : tyypeeob22..... agent), type(obji, e closeby(obj2.or) obj1).
0
G
(ump(agent)
type(abjie agent)
type agent)
Sype(obj1, enemy)
I0,50,
0, 0,0,0
1.
3, 6, 7
0, 0,0,80 0
0,0,0 0
0, 0,0
10
00,0,0
11 1.1.1
8,8
00,0,0
00,0,0
[0, 0,00,
6
8
type enemy)
closeby(obji, obj2)
closeby(obj2.ob obj1)
Lo,j.0;
0,000 0
0,
0,0
Io
0,,0,
0,0,0
00.0.0
Table 4: Example of ground rules (top) and elements in the index tensor (bottom). Each fact has its index, and the index tensor contains the indices of the facts to compute forward inferences.
((i) Reasoning using an action rule] First, for each action rule C, E C, we evaluate body atoms for different grounding of C2 by computing (2)
1. [0,1]:
II
pather(v(0.II)))
(8)
ISISL
where gather [0, 17 x NGX8XL + [0, 1111xxxXX is:
gather(x, -
(9)
The gather function replaces the indices of the body state atoms by the current valuation values ((), To take logical and across the subgoals in the body, we take the product across valuations. bij.x represents the valuation of body atoms for i-th rule using k-th substitution for the existentially quantified variables to deduce j-th fact at time
(t) E [0. 1]: Now we take logical or softly to combine all of the different grounding for C by computing i.
softor 2 (1) (0.....-.... i.j.s)
(10)
where softor" is smooth logical or function:
softor (I (I1....,
kn) log Σ
exp(x1/7),
(11)
1gisn
where y > 0 is a smooth parameter. Eq. 11 is an approximation of the max function over probabilistic values based on the log-sum-exp approach (Cuturi and Blondel| (007).
((i)) Combine results from different action rules] Now we apply different action rules using the assigned weights by computing 1. [0, 1]:
(1)) j.m
Σ uimi
G.j
(12)
15i5
where w exp(wm,i)/E) exp(wm.e) ) and Wm,i Wm]i). Note that w interpreted as a probability that action rule C, C is the m-th component of the policy. Now we complete the 1-step forward reasoning by combining the results from different weights:
(4) softor (h) (.......... M.
(13)
Taking softor" means that we compose the policy using M softly chosen action rules out of C' candidates of rules.
(!) ((ii) Multi-step reasoning] We perform T-step forward reasoning by computing rj recursively for T times: (t+1) softor (1) , (.))). Finally, we compute (() E [0, 1] and returns A E [0, 1| by extracting only
output for action atoms from (T), The whole reasoning computation Eq. 13 can be implemented using only
efficient tensor operations. See App. E,2 for a detailed description.

---

E.2 Implementation Details
Here we provide implementational details of the differentiable forward reasoning. The whole reasoning computa- tions in NUDGE can be implemented as a neural network that performs forward reasoning and can efficiently process a batch of examples in parallel on GPUs, which non-trivial function of logical reasoners.
Each clause C E C is compiled into a differentiable function that performs forward reasoning using the tensor. The clause function is computed as:
c (1) softor? (prod ( T)))
(14)
where gather,(X (X, Y). = Y.s.Iikk obtains valuations for body atoms of the clause C2 from the valuation tensor and the index tensor. prod, returns the product along dimension 2, i.e. the product of valuations of body atoms for each grounding of C.. The softor" function is applied along dimension 3, on all the grounding
(or possible substitutions) of C₁.
softor] is function for taking logical or softly along dimension d:
softor](X) = 7 log (sum, exp((X/1)),
(15)
where y > is a smoothing parameter, sumg is the sum function along dimension d. The results from each clause C R Bx is stacked into tensor C(O) Rx
Finally, the T-time step inference computed by amalgamating the inference results recursively. We take the softmax of the clause weights, W e RMXC and softly choose M clauses out of C' clauses to compose the logic program:
W = softmaz (W).
(16)
where softmar, is a softmax function over dimension 1. The clause weights W E MxC and the output of the clause function ((a) xCxxxx are expanded (via copy) to the same shape W C(() E R MRMxxxBXXC BxG The
tensor (t) E MxBG is computes as
H summ (W" @ C),
(17)
where @ is element-wise multiplication. Each value H jj represents the weight of k-th ground atom using i-th clause weights for the j-th example in the batch. Finally, we compute tensor (t) R BxG corresponding to the fact that logie program is set of clauses:
R = softor (H)
(18)
With the 1-step forward-chaining reasoning function:
r(V(",; W) = R()))
(19)
we compute the 7'-step reasoning using:
((+ softori (stack] (V(+), r( (V())) w)))
(20)
where I E NCXGXSXL is a precomputed index tensor, and W e MxC is clause weights. After T-step reasoning, the probabilities over action atoms GA are extracted from V(T) as VA E 1]
F Figures for Ablation Study
Here we provide figures used for our ablation study.
'one with howpoooocc.ooy/d/ss///occhgggeee
20

---

Loot-hard
lagic
10
9etum
Steps
Figure 10: Returns lavg.:ssdl obtained by NUDGE and neural PPO on Loot-hard.
impol atoms
Distribution over action atoms
05
04-
left
here Joyrppo) spert
at Llrttrors eeji
hertnaz aepur)
Jeep
teft.
right
left
rggh dear
Gradients of actions input
.07
left key
13
.15
14
.15
032
right key
.01
04
01
doo-
right_don
ctesely (ob)1 .0044)
lave sey (Lagont)
mit fave, kayi (
on left (eft(oh)2, (ooj (oo_22__11) (de1))
on Left(ob oo (3,23, (b)1)
tyee ous ogent)
(811)
Eypel (oo) 12, key)
type (ob33 dear)
typere ony eneay)
type on key)
Left
right
bert deor
Figh Bat
right Lob
Figure 11: Explanations and action distributions produced by NUDGE for different states of GetOut.
21


Reviews:
[{"review_id": "tVS7EkBF0k", "reviewer": "Reviewer_AnR8", "summary": "In this work, the authors propose neurally guided differentiable logic policies (NUDGE), which learns differentiable logical policies that are represented by interpretable rules. It is capable of producing explanations for their decisions in complex environments. NUDGE utilizes neurally guided symbolic abstraction to boost the model's performance. Specifically, the candidate rules for the logic-based agents are obtained efficiently by being guided by neural-based agents. Experimental results show that NUDGE can achieve competitive performance compared with the neural baselines. Besides, the NUDGE agent can adapt to environmental changes and its learned policies are interpretable and explainable.", "questions": "1. What's the input representation for the neural baselines, namely, DQN agents and PPO agents? Is it the same with the neural logic methods?\n2. Wonder how does NUDGE perform under more logically challenging environments, for example, grid environments which are utilized in previous work [1][2].\n3. How are the head atoms interpreted, for example, **right_go_to_door** used in the GetOut task and **right_to_eat** in the Fishes tasks. Are they automatically learned or interpreted by developers manually?\n\n[1] Cao, Yushi, et al. \"GALOIS: boosting deep reinforcement learning via generalizable logic synthesis.\" Advances in Neural Information Processing Systems 35 (2022): 19930-19943.\n[2] Yichen Yang, Jeevana Priya Inala, Osbert Bastani, Yewen Pu, Armando Solar-Lezama, and Martin Rinard.\nProgram synthesis guided reinforcement learning for partially observed environments. Advances in Neural\nInformation Processing Systems, 34, 2021.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors propose using a novel mechanism called neurally guided symbolic abstraction, to boost the neural logic model's performance.\n2. The writing of the paper is well-organized and clear, and the evaluations are quite comprehensive.", "weaknesses": "It is unclear how are the abstract head atoms being interpreted, for example, **right_go_to_door** used in the GetOut task and **right_to_eat** in the Fishes tasks. Are they automatically learned or interpreted by developers manually?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comment and for acknowledging that the paper proposes a novel neuro-symbolic RL method, and the writing and presentation are well-organized and quite comprehensive.\n\nWe address the concerns next.\n\n***Unclear how are the abstract head atoms being interpreted***\n\nAs stated on lines 240-242, we renamed the predicates for clarity using the obtained explanatory rules after training. \nHowever, rules with interpretable atoms are automatically learned, and the renaming process does not require much additional resources and effort. \n\nYour point led us to think that these interpretations could be automatically identified. Let us illustrate how to do this.\nFor example, after training, NUDGE obtained rules about the action to go right:\n* right$^{(1)}$(agent):-type(O1,agent),type(O2,door),on-right(O2,O1),has_key(O1).\n* right$^{(2)}$(agent):-type(O1,agent),type(O2,key),on_right(O2,O1),not_has_key(O1).\n\nAs these two rules have the same semantic head atom, a program could automatically search for differences in the bodies of the rules. Here, the right$^{(1)}$ head could be relabelled right$^{(door)}$ and right$^{(2)}$ relabelled  right$^{(key)}$, as these rules are respectively concerned with objects of types door and key. We will add the discussion in the next version.\n\n\n***Input representation for the neural baselines***\n\n\nYes, we used object-centric representations for both neural and logic agents. Continuous values (e.g. x, y) are unchanged. For the logic agent, the categorical values are one-hot encoded to fit the αILP framework. \n\n\n***NUDGE in more logically challenging environments***\n\n\nNUDGE can handle as complex rule as those used in the GALOIS paper [1]. For example, \nthe synthesized programs shown in the paper [1] are:\n* (1.00) gt_box() :- ¬ has_key(X), is_agent(X), ¬ has_key(Y), is_env(Y) \n* (1.00) gt_key() :- ¬ has_key(X), is_agent(X), has_key(Y), is_env(Y) \n* (1.00) gt_door() :- has_key(X), is_agent(X), ¬ is_open(Y), is_door(Y)\n* (1.00) gt_goal() :- has_key(X), is_agent(X), is_open(Y), is_door(Y)\n\nThese rules have similar complexity as the ones learned by NUDGE in our experiments  (e.g. loot)  in terms of the number of relations and variables. Thus, NUDGE will perform well in the environment used in GALOIS.\n\nTo further reduce the gap between our environments and those from [1], we have adapted our loot environment (Loot-hard). In this environment, the agent first has to pick up keys and open their corresponding saves, before being able to exit the level (by going to a final exit tile). \nWe provide the results hereafter, as well as curves on **Fig.1 in the attached PDF**.\n|   PPO   |  NUDGE  | Random  |\n|---------|---------|---------|\n| 8.4±3.2 | 9.2±2.0 | 1.2±0.4 |\n\n\n[1] GALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesis. NeurIPS 2022\n"}, {"review_id": "QkJoZXiUbh", "reviewer": "Reviewer_NLZb", "summary": "This paper introduces NUDGE, a framework for interpretable and explainable policy reasoning and learning in reinforcement learning. NUDGE employs differentiable forward reasoning during inference to derive a set of weighted rules that form an interpretable policy. During training, NUDGE leverages neurally-guided symbolic abstraction to efficiently distill rule-based symbolic representations from neural-based policies. It then employs gradient-based policy optimization using actor-critic methods to learn the weights of these rules. Empirical results demonstrate that NUDGE achieves comparable performance to neural-based policies while providing interpretable and explainable logical representations. Furthermore, the rule-based symbolic policies in NUDGE exhibit adaptability and flexibility in addressing environmental changes. Overall, NUDGE offers a promising approach to achieving interpretable and adaptive policies in reinforcement learning.", "questions": "There is confusion regarding the paper's claim on policy explainability in relation to NUDGE. The approach used for explainability appears to be fairly standard and not inherently tied to differentiable logical programs. In fact, it can be applied to explain a neural network as well e.g. annotating regions of an image that mostly influence policy output. Therefore, it remains unclear how NUDGE specifically enhances policy explainability compared to existing methods. Could the authors elaborate on that?\n\n", "limitations": "Although the paper is well executed and integrates existing components effectively, it does not present advancements that significantly differentiate it from prior work PIRL [Verma et al., 2018].", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ NUDGE introduces end-to-end reasoning architectures based on differentiable forward reasoning, allowing for comprehensive policy generation directly from raw input. \n+ Its neurally guided symbolic abstraction leverages a pre-trained neural policy to induce a set of candidate action rules, which are subsequently optimized using the critic component of an actor-critic agent. \n+ Notably, NUDGE policies are differentiable, enabling the utilization of action gradients with respect to input atoms. This unique feature facilitates the explanation of state features that influence the decision-making process, shedding light on how specific actions are chosen in a given state. \n\nThese aspects collectively contribute to NUDGE's interpretability and explainability, providing insights into the underlying mechanisms of the learned policies.", "weaknesses": "It appears that the individual components of NUDGE are not inherently novel on their own. The paper's main contribution lies in the integration of existing components to form the NUDGE framework. By bringing together existing components such as differentiable forward reasoning, neurally guided symbolic abstraction, and policy optimization, NUDGE creates a framework for interpretable and explainable policy learning. Although the individual components are not novel on their own, the paper would have been acceptable if it can still demonstrate the novel insights that arise from their integration. However, the novelty of NUDGE in comparison to prior work such as PIRL [Verma et al., 2018] appears to be very limited. Both NUDGE and PIRL employ a pretrained neural policy as a guide for learning policies. Additionally, both NUDGE and PIRL utilize random search to update policy structures and employ optimization techniques to learn policy parameters. The primary distinction lies in the fact that NUDGE learns programmatic policies within logical programs, whereas PIRL learns policies within functional programs. While NUDGE can learn from raw image inputs, this aspect is not considered a core contribution, as it relies on off-the-shelf tools for object extraction from visual inputs. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback and for finding that the combination of differentiable reasoning and RL contributes to the interpretability and explainability issues in RL and provides insights into the underlying mechanism of RL agents.\n\nWe address the concerns next.\n\n***NUDGE limited novelty wrt PIRL.***\n\nThe main difference is that PIRL develops programs based on restricted functional language, but NUDGE searches over a more general language of first order logic. \nPIRL requires *sketch* as input to synthesize programs, which is a specification of the structures of programs to be generated, then optimize the internal parameters in the sketch. In contrast, NUDGE performs structure learning from scratch using neural guidance on symbolic abstraction and weight optimization. We provide a language bias (mode declaration) to restrict the search space, but not the structure of the programs.\nAs NUDGE uses first-order logic, it can incorporate background knowledge in a declarative form (e.g., with a few lines of relational atoms and rules), which is difficult with the functional programming language.\n\n\nPIRL nicely incorporates recursive calls of learned programs. We have not developed this for NUDGE, but labeling together combinations of logic atoms would ease the understanding of the rules. For example, creating an eatable atom in case of a 3fishes environment where fishes are only eatable if they are both green and smaller.\nHowever, NUDGE makes it easy to incorporate background knowledge (e.g. closeby)\n\nFinally, PIRL is developed for and evaluated on continuous/control tasks, e.g. TORCS, where the main interest is not on relational reasoning.\n\n\n\n***How NUDGE specifically enhances policy explainability compared to existing methods.***\n\nNUDGE uses gradient-based explanation methods, which can compute explanations very efficiently using automatic differentiation.  Explainability is thus produced efficiently in comparison to classical logic-based policies (that are not differentiable and thus require additional hard coding to produce explanations).\n\nMoreover, heatmaps used on DNNs can highlight raw inputs and not over concepts. The symbolic abstraction of our logic policies allows us to differentiate between the importance of the size and of the color of a fish for the decision of, e.g., dodging. Heatmaps of DNNs would not be able to make such distinctions. \nThank you for raising this point, it helped us improve the quality of our explanation on this topic.\n"}, {"review_id": "B0JxVkxyJC", "reviewer": "Reviewer_AhLx", "summary": "This paper aims to learn more interpretable and explainable logical policies. It modifies NLRL, Neural Logical RL, with (1) an extra neural-guided heuristic, i.e., alignments with a pretrained neural policy, for filtering logical rules during search; (2) changing policy learning algorithm to actor-critic from policy gradients; (3) pre-defining / hardcoding the meanings of symbols for more interpretability; and (4) explanation of policies according to the gradients. It conducted experiments in five simple games. ", "questions": "* Why are the policies more interpretable and explainable? How can the proposed methods contribute to them? \n* Are there results of classic symbolic methods? Can NUDGE perform better than them? \n* Are there results on more complex and difficult games? ", "limitations": "N/A", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This paper uses pre-trained policy networks as neural guidance for faster search of logical rules (filtering according to their alignments). This could be an interesting direction to explore. However, experiments that evaluate/support its effectiveness are not adequate in the paper. ", "weaknesses": "This paper claims that their methods improve the interpretability and explainability of logical policy learning. However, as far as I understand, this is not true because \n* The proposed methods (compared with existing ones) are mainly about optimization and logical rule mining, instead of about interpretability and explainability.\n* Policies become more interpretable only because they change the problem setting to an easier and more interpretable one. They assume predefined/hardcoded meanings of all the predicates/symbols (e.g., `close_by(obj1, obj2)`), as well as a pretrained neural module to detect the objects and those meaningful attributes. \n  - Previous methods do not assume those and instead, jointly learn the symbols and their meanings with logical rules. \n  - Any model with access to those predefined meanings and those pretrained detection modules is more interpretable and explainable. For example, when given access to those symbolic inputs (i.e., memory states), even DQN can adapt to environmental changes (as defined in Q2, Sec. 4) and would be explainable by performing the gradient analysis. \n\nFrom the optimization perspective, the new problem setting is less interesting and challenging as well. With access to symbolic inputs (e.g., the memory states of Atari games), classic ILP methods may even work, especially when considering the relatively small search space in the five simple games. The authors should compare with those baselines as well. \n\nThere are other minor concerns such as \n* If the neural policies are much worse than the symbolic ones as reported, why and how would it be helpful to distill/use them as neural guidance?\n* The games are relatively simple and easy. \n* There should be experiments analyzing the effects of actor critics v.s. policy gradients. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback and for highlighting that our proposed approach is interesting. \nWe address the concerns next.\n\n***Proposed methods not about interpretability and explainability.***\n\nOne of our main contributions is indeed the neural guidance incorporated in our NUDGE framework. This is why the neural guidance is part of the name of our framework, whereas Interpretability and Explainability are not.\n\nHowever, our motivation is to develop interpretable and explainable policies for RL, and the optimization is a means to an end. \nTable 2 shows that NUDGE, is the first interpretable and explainable RL framework using differentiable logic reasoning. We demonstrate empirical support for these features (Fig. 4 for interpretability, Fig. 5 for explainability).\nThe system's methodology and outcome are orthogonal, i.e., we perform differentiable rule learning to obtain policies that address the interpretability and explainability problems. \n\n***Changing the problem setting to easier and more interpretable.***\n\nAdapting the problem settings by focusing on object-centric representation is a common way of obtaining interpretability.\nAs explained in the paper, we believe that obtaining an interpretation over the RL agents’ behaviors requires this problem-setting change. We are not hiding this point, as stated l. 147, “NUDGE agents take an object-centric state representation as input, obtained e.g. using object detection or discovery methods.” We have not incorporated such techniques yet. \n\nFor a fair comparison, we used object-centric input representations for neural policies. However, structured inputs are not sufficient to obtain interpretability, as the trained neural baselines are not interpretable (due to their black-box nature), whereas NUDGE policies are.\n\nThe use of predefined (and meaningful) symbols is a fundamental feature, and often a drawback, of any logic-based system.\nAs mentioned in the limitation, predicate invention could be an alternative that would lower the priors provided to logic-based agents. We left this for future work. \n\nFinally, NUDGE has an explicit logical reasoning structure and thus can produce the reasoning steps of action selection, explaining the reasoning behind decision-making. Building precise reasoning steps using neural networks is still an open problem.\n\n***Classic ILP methods may work + comparison to these***\n\nLearning in RL tasks using symbolic ILP solvers is very inefficient, as their non-differentiable nature prevents the use of efficient RL algorithms (such as policy-gradients techniques). One needs to construct inputs that can be processed by ILP solvers i.e., positive/negative labeled examples, namely transforming a RL task to a binary classification task. \n\nMoreover, even with a provided set of action rules obtained from trained NUDGE agents, the classic ILP framework’s deterministic reasoning will not allow for rule prioritization. \nFor example, the weighted rules of NUDGE policies in Getout are:\n\n* 0.57:jump():-...\n* 0.29:right_key():-...\n\nThese weights show that the agent has learned to prioritize jump over right_key. These adaptive action selections cannot be done easily by classic ILP.\n\n\nTo demonstrate this, we conducted reasoning experiments using trained NUDGE agents and Classic logic benchmark, where the set of action rules are given but with all weights set to 1.0, which simulates discrete logic. Note that such agents can still play better than ones with pure symbolic reasoners, as our classic baselines still use fuzzy (i.e., continuous) concepts, such as closeby.\nWe show the average (and std) returns of 5 randomly seeded agents on GetOut, 3Fish, and Loot. We used action rules obtained by NUDGE.\n\n| Score | NUDGE          | Classic        | Random         |\n|-|-|-|-|\n| GetOut         | 17.86±2.86 | 11.59±4.29 | $-22.5±0.41 |\n| 3Fish          | 0.05±0.27 | -0.24±0.29 | $-0.73±0.05 |\n| Loot           | 5.66±0.59  | 0.51±0.74  | $0.57±0.39  |\n\nIn all environments, NUDGE clearly outperforms the Classic counterparts. \nThis experiment already shows the superiority of differentiable policy reasoning (embedding weighted action rules) over policy without weighted rules (Classic). \n\nFinally, learning using classic ILP relies on policy reasoning, and thus will lead to (at best) brittle learning. \n\nAs shown above and explained in [1], it is neither trivial nor efficient to use classic ILP solvers in such RL tasks, we do not see why we should develop and compare with these methods. Adding these results and discussions will improve the manuscript.\n\n[1] Zhengyao Jiang and Shan Luo: Neural logic reinforcement learning. ICML 2019\n\n***Why and how would it be helpful to distill/use neural policies as guidance***\n\n\nNeural policies are used to find the set of promising rules that NUDGE agents will use when learning to solve the task.  Of course, the better the neural agents are, the more likely is the set of rule to be successful.\nNUDGE agents then have the advantage of relying on logic rules, and their embedded priors, which allows for faster and better generalization.\n\n***The games are relatively simple and easy.***\n\n\nWe have added experiments on a harder version of loot (loot-hard, *see answer to reviewer AnR8 or the attached PDF for details*).\nThe set of games used in our experiments are meant to back up our claims, i.e., demonstrate that we can produce explainable and interpretable logic based policies that solve RL tasks that involve relational reasoning. Scaling it to more complex games (such as starcraft) is left for future work.\n\n***Experiments analyzing the effects of actor critics v.s. policy gradients.***\n\n\nWe directly incorporated the actor-critic framework because we use the critic in the neurally guided symbolic abstraction. The critic stabilizes the gradients backpropagated to the policy compared to direct backpropagation from the return. The focus of our paper is not to compare the advantages of existing RL methods.\n"}, {"review_id": "yponomN0nV", "reviewer": "Reviewer_Jkdb", "summary": "This paper presents NUDGE, an RL technique that combines neural network and logic program learning to learn policies that are interpretable and explainable by humans. NUDGE produces agents that match or exceed the performance of agents trained using standard RL techniques like PPO.", "questions": "- Line 3: What do you mean by “articulate the thinking” in this definition?\n- How do you envision this work translating to real world settings? This work makes a lot of sense in the rule-based testbed, where much of the non-rule-based information is extraneous. Can naturalistic inputs be represented in this way? I could see an argument for this work as automating the pipeline approaches in vision-based control settings. \n- Line 142: I’m curious whether there are analogues in cognitive science for these modules you’ve created.\n- Line 143: Is $\\Theta$ learned? \n- Line 204: What is $\\theta$? I don’t see $\\theta$ anywhere else.\n\nNits:\n- Line 1: “dominating” —> “dominant?", "limitations": "The authors have adequately addressed limitations.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written, clear, and addresses the significant challenge of uninterpretable reinforcement learning policies. The paper proposes to solve this by learning policies that are interpretable and explainable by design. \n\nI am unqualified to assess the originality of this paper because I am unfamiliar with the logic program learning literature.", "weaknesses": "This work can be improved with stronger definitions of interpretability and explainability. Currently, it’s unclear what criteria make a model interpretable or uninterpretable (line 43). What exactly makes a policy readable by humans? Is this person an expert or a layperson? I suspect that interpretability here means that an expert can assign meanings to the policy. Similarly, what constitutes an explanation (line 44)? From their definition, it seems that the explanation need not be causal? Do all methods from explainable AI count?\n\nThere are also existing definitions of both terms from the community. I would like to see the authors engage with the existing literature on definitions of interpretability and explainability. See Lipton 2016 for a discussion on interpretability. I think that having set definitions is important for this work in order to avoid moving goalposts.\n\nLine 141: Another weakness is that the relational perception module maps all entities to a particular index in the valuation vector. This method does not seem like it would scale to new objects. Similarly, the identity of entities seem to be fixed given a particular validation vector. It seems like the parallel version of this weakness (parallel in the sense of scaling issues) for the forward reasoning module is solved by neurally guided symbolic abstraction (line 193).\n\nI also found some parts on what parameters are and aren't learned to be confusing. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback and for finding that the paper is well-written, clear, and addresses the significant challenge of uninterpretable reinforcement learning policies.\n\nWe address concerns next.\n\n\n***Engaging with the existing literature on definitions of interpretability and explainability***\n\n\nThank you for this resource. Explainability and interpretability are indeed used interchangeably in the literature. We thus set their definition (following [1]):\n* **interpretability**, i.e. the capacity to articulate the thinking behind the action selection. (l.16)\n* **explainability**, i.e. [the ability of] the agent can explain the importance of each input on its decision. (l.33)\nThese definitions align with the ones discussed by Lipton (2017). We engage more research works (including the provided one), to improve our discussion on these terms for the final version.\n\n[1] Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5), 206-215.\n\n\n***Relational perception module maps all entities to a particular index in the valuation vector.***\n\n\nIndeed, to perform reasoning, we need to name entities and relations. It would be very interesting to investigate how we can remove mappings by incorporating deep representations obtained by, e.g., object-discovery models. We left it for future work. Thank you for the insight.\n\n\n***Some parts on parameter learning confusing***\n\nWe appreciate your detailed feedback, we clarify hereafter and did improve the paper based on your questions:\n* **What do you mean by “articulate the thinking” in this definition? (l. 3)**\n   -> explain the reasoning behind [the action selection]\n* **I’m curious whether there are analogues in cognitive science for these modules you’ve created (l. 142).**\nThis is a very exciting question, we have found [1] and [2] and more recently [3] (also more focus or AI) that provides some light on this matter. It seems that such module are simplifications of what is happening inside biological nervous systems. The work of Daniel Kahneman is also of relevance here. \n* **Is $\\Theta$ learned ? (l. 143)**\n$\\Theta$ is a parameter for the perception networks (e.g. an object detector), and we assumed that it is pre-trained on the environment. We will clarify this more in the manuscript. \n * **What is $\\theta$, I don’t see it anywhere else. (l. 204)**\n$\\theta$ represents a parameter of neural policy $\\pi_\\theta$, i.e. the set of learnable weights in the neural agent. We assume $\\theta$ is pre-trained, and NUDGE produces differentiable logic policy $\\pi_{(\\mathcal{C}, \\mathbf{W})}$, which is parameterized action rules $\\mathcal{C}$ and their weights $\\mathbf{W}$. It is introduced l. 194. \n * **How do you envision this work translating to real world settings? This work makes a lot of sense in the rule-based testbed, where much of the non-rule-based information is extraneous. Can naturalistic inputs be represented in this way? I could see an argument for this work as automating the pipeline approaches in vision-based control settings.**\nOur work rely on an accurate object-centric and/or concept extraction method to produce successful and explainable policies. For general real world agents, this would be tedious, but techniques to ease the extraction of concepts based on, e.g., natural language are already emerging. You are right that the automatic pipeline approach in industrial, based on, e.g., vision sensors is a very interesting real world use case that would fit NUDGE policies’ advantages.  \n\n[1] Alvarez, M. (2018). Reasons for action, acting for reasons, and rationality. Synthese, 195, 3293-3310.\n\n\n[2] Ryan, R. M., & Connell, J. P. (1989). Perceived locus of causality and internalization: examining reasons for acting in two domains. Journal of personality and social psychology, 57(5), 749.\n\n\n[3] Huang, J., Zhu, W. Y., Jia, B., Wang, Z., Ma, X., Li, Q., & Huang, S. (2022). Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representation. \n\n"}, {"review_id": "9xykqCW7rD", "reviewer": "Reviewer_hGCz", "summary": "The paper tackles interpretable RL with a neurosymbolic method (NUDGE) separating perception and action prediction. Given a pre-trained actor-critic model and a pre-trained/given perception module, NUDGE checks randomly generated rules and assess their quality by comparing them to the pre-trained actor. It then performs rule weight learning using an A2C-like algorithm using the pre-trained critic.", "questions": "- It is not immediately clear what the differences between the Differentiable Forward Reasoning module are with AlphaILP.\n- The object-centric states are similar to concept-based models (for instance, [2], which also performs rule learning). It might be worth discussing. \n- 3.2.1: NUDGE never removes state atoms?\n- 3.2.1: If $\\pi_\\theta$ rarely picks action $A$ in states $X$, is the eval rule still reliable?\n- Experiments:\n\t- If all networks get object-centric descriptions, how are those in $[0, 1]$? Or are all these binary?\n\t- What is the actor-critic/PPO model you use to train NUDGE?\n\t- What is the motivation behind adding basic rules? It might be worth highlighting this is an option to improve performance or mention that it does not matter.\n\t- Table 1 Left: Why not compare to PPO? Why did you use expert supervision here? That means you did not test your actual method. \n\t- Figure 3: The comparison by steps is somewhat unfair because NUDGE uses a pretrained model at initialisation. Why not train all models until convergence?\n\t- The conclusion that NUDGE adapts to environmental changes is misleading: Rather, it is easy to adapt by changing the logic, but NUDGE is not adaptive itself.\n\t- The explainability claims are only tested with a single example. It is unclear why NUDGE would be more explainable with LIME-style diagnostics than a neural network trained on object-centric states.\n- Table 2 claims NUDGE is FOL, but it is weaker: It does not support infinite domains or function symbols\n\nMinor comments:\n- Make sure to use \\citep consistently\n- The statement that humans use symbolic languages to compose their reasoning is something I'd like to see a citation for\n- Throughout the paper, you refer to the rules and their weights as 'probabilistic'. However, $\\alpha$ILP is not a probabilistic logic (in fact, the softOr is not fuzzy either, as it is not a t-conorm [3]). The invalid use of the word probability is confusing.\n- Typos:\n\t- 123 reads wrong to me\n\t- 188: a given neural policy\n\t- Equation 3 seems invalid. What does the = mean between the $\\prod$ and $\\mathbf{v}$?\n\t- Figure 3: Is the largest $k$ 5 or 10? The legend and caption don't agree.\n\t- 245: designed to\n\t- 255: Agents can 'open' keys?\n\t- 298: opening quotes\n\n[1]: Hikaru Shindo, Viktor Pfanschilling, Devendra Singh Dhami, Kristian Kersting: alphaILP: Thinking Visual Scenes as Differentiable Logic Programs.  Machine Learning (Special Issue on Learning and Reasoning 2022), vol. 112, pp. 1465–1497, 2023\n\n[2] Barbiero, P., Ciravegna, G., Giannini, F., Zarlenga, M. E., Magister, L. C., Tonda, A., ... & Marra, G. (2023). Interpretable Neural-Symbolic Concept Reasoning. arXiv preprint arXiv:2304.14068.\n\n[3] van Krieken, E., Acar, E., & van Harmelen, F. (2022). Analyzing differentiable fuzzy logic operators. Artificial Intelligence, 302, 103602.", "limitations": "The method requires a (pretrained / hand-designed) perception module that can be hard to come by. The paper does not discuss this limitation thoroughly. Otherwise, limitations are addressed.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I like the ideas, and the paper offers an excellent combination of methods with clear benefits in terms of explainability and possibly generalisation (although I think the verdict is out on this). The paper is well-written, easy to understand and complete. \nThe explicit parameter $k$ allows limiting the complexity of policies, generating interpretable policies as highlighted in Figure 4.", "weaknesses": "The experimental setup varies significantly throughout the different settings, leaving me with questions about the experimental methodology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback and for acknowledging that the paper is well-written, clear, and complete.\nWe would like to answer questions and concerns raised in the review.\n\n***How is the reasoning module different from alphaILP’s one?***\n\n\nNUDGE focuses on differentiable policy learning, i.e. we modified the reasoning module to produce a valid action distribution. Differentiable forward reasoners, in general, output all consequent atoms, but NUDGE outputs only action atoms to achieve a valid action distribution, differentiating action atoms and state atoms.  \n\nThe object-centric states are similar to concept-based models (for instance, [2], which also performs rule learning). It might be worth discussing.\nThis is true, thank you. We are adding the reference and following discussion to the manuscript:\nConcept-based models aim to make DNNs more interpretable by encoding human-understandable concepts into the neural network. NUDGE could rely on concept-based models to extract (weighted) logical representations from raw inputs, e.g., images. \n\n***NUDGE never removes state atoms?***\n\n\nYou are right, NUDGE never removes state atoms. Adding such a capability is easily doable and would be very interesting in situations for which state atoms are not needed. It would simplify the interpretations of the decision-making process.\n\n***If the neural policy rarely picks action A in states X, is the eval rule still reliable?***\n\n\nGood question. If action A is rarely picked in states X, rules defining action A will get low scores, as the score is based on the action distribution. Every action thus gets a score for every state X that is used in the rule evaluation. \n\n\n***If all networks get object-centric descriptions, how are those in [0,1]? Or are all these binary?***\n\n\nWe add (small) noise to the object-centric description of NUDGE input system, their values thus lie in ]0, 1[. \n\n\n***What is the actor-critic/PPO model you use to train NUDGE?***\n\n\nContrary to the neural model that uses an MLP both for actor and critic, NUDGE agents encompass αILP (a differentiable rule learning system) -based actor. Both agent types follow the PPO algorithm (i.e., use clipping to stabilize policy optimization).\n\n***What is the motivation behind adding basic rules? Does it improve performance?***\n\n\nExcellent question. NUDGE can start from the set of empty rules for each action.\nWe provided the first rule-mining step in our manuscript as it helps to understand how the rules are expanded.\n\n***Tab. 1 Left: Why not compare to PPO? Why did you use expert supervision here?***\n\n\nWe used DQN, as it is the most used baseline in the literature for this benchmark. The PPO results reported in its introductory paper ([1]) do not change our conclusions. \nAtari's games are a common RL benchmark. Compared to our implemented environments, they rely less on relational and logic reasoning, but require accurate moves. Our Atari experiments are meant to show that αILP agents can learn on this benchmark too. We have observed that NUDGE produces equivalent rules as the one provided by the game experts in all other tested environments, explaining why we have not run the rule search yet.\nFor completeness, we are currently setting up and running these experiments to be added to the final version.\n\n\n[1] Schulman, J., et al.: Proximal policy optimization algorithms. (2017)\n\n***Fig. 3: Why not train all models until convergence?***\n\n\nThe reported results in the tables are from agents trained until convergence. NUDGE does not use a pretrained model, starts with a randomly initialized critic, too. However, its actor is composed of already curated rules, as “NUDGE exploits trained neural network-based agents.” (l. 7)\nAgents that use expert priors do not make use of this advantage but incorporate expert knowledge. We show the learning through steps allows the reader to see how much prior knowledge can help to boost learning.\n\n***The conclusion that NUDGE adapts to environmental changes is misleading: Rather, it is easy to adapt by changing the logic, but NUDGE is not adaptive itself.***\n\n\nVery important insight. It depends on the environment.\nIn our experiments, in the GetOut environment, the agent that is trained using 1 moving enemy can perform well in the variant with more moving and static enemies. This is because of the NUDGE’s symbolic abstraction and concepts, e.g., closeby, are good enough to adapt to more enemies. The modified Loot environments did not need any change either.\nIndeed, in 3Fishes-C, the agent requires a change in its logic to adapt. This change was made manually, but the new one could be obtained by deriving from the existing one. It would be faster than fully retraining an agent. We will clarify this in the paper.\n\n\n***Why would NUDGE be more explainable than neural networks with LIME?***\n\n\nDue to the space limitation, we put only one example in the manuscript.\nHowever, we conducted the experiments for different actions producing different explanations. In Fig.2 in the attached PDF, we show 2 more explanations and action distributions produced by NUDGE. For each state, NUDGE produced human-readable explanations using gradients and logical representations, which are difficult to be produced by neural networks. Thank you for pointing it out. We will add these results to the appendix.\n\n***Tab 2 claims NUDGE is FOL, but it is weaker: It does not support infinite domains or function symbols***\n\n\nGood point, thank you for raising this. Currently, we are not providing experiments using functors or infinite domains. NUDGE can handle these as it incorporates the differentiable ILP techniques for functors [1], where necessary symbols are adaptively enumerated according to input. \nWe will add this discussion to the manuscript.\n\n[1] H. Shindo, et al.: Differentiable Inductive Logic Programming for Structured Examples AAAI 2021\n\nFinally, we greatly appreciate all the minor comments. We address them in the updated version.\n"}]

Author rebuttal:
Dear reviewers,

Thank you all for your helpful feedback that helped us improve our manuscript's clarity and quality.


Here we provide a PDF that contains 2 figures:
* Figure 1: Returns (avg.±std.) obtained by NUDGE and neural PPO on the newly created more logically challenging environment (*Loot-hard*).
* Figure 2: Additional examples of explanations and action distributions produced by NUDGE for 2 different states of GetOut. 

Meta-review: This paper proses NUDGE: Neurally gUided Differentiable loGic policiEs. NUDGE requires a trained actor-critic model and a trained perception module, and uses those to guide the search for (interpretable) rules.

First an acknowledgement. The discussion of this paper was *extremely* contentious, and serves, in my opinion, as an example of how *not* to discuss a paper. I did my best to make my decision objectively in spite of this.

Most reviewers praised the proposed method for symbolic abstraction (and e.g. that the complexity and perhaps interpretability of the policies can be controlled), the writing and the evaluations (even though the experimental settings vary). Multiple reviewers found it unclear from the current draft that the method requires a *given* perception module, and given this the authors need to make that limitation very clear in the final manuscript. 

While some reviewers point out that the components of NUDGE are not particularly novel on their own, the combination in the proposed framework still adds to, and deviates enough from, previous work such as PIRL in my opinion. (The authors should, as they promised in discussion, include more discussion regarding PIRL, but I am confident that they can do that.) 

Despite much back and forth, the reviewers did not reach a consensus, yet 4/5 reviewers recommend acceptance to varying degrees, and I don't see enough reason to override that.
Decision: Accept (poster)