Title: Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

First-Order-Logic

Symbolic Abstraction

Interpretable Reinforcement Learning

Logic Reinforcement Learning
Abstract: The limited priors required by neural networks make them the dominating choice to encode and learn policies using reinforcement learning (RL). However, they are also black-boxes, making it hard to understand the agent's behavior, especially when working on the image level. Therefore, neuro-symbolic RL aims at creating policies that are interpretable in the first place.
Unfortunately, interpretability is not explainability. To achieve both, we introduce Neurally gUided Differentiable loGic policiEs (NUDGE). NUDGE exploits trained neural network-based agents to guide the search of candidate-weighted logic rules, then uses differentiable logic to train the logic agents. Our experimental evaluation demonstrates that NUDGE agents can induce interpretable and explainable policies while outperforming purely neural ones and showing good flexibility to environments of different initial states and problem sizes.

Interpretable and Explainable Logical Policies via Neurally Guided Symbolic Abstraction

Abstract

The limited priors required by neural networks make them the dominating choice to encode and learn policies using reinforcement learning (RL). However, they are also black-boxes, making it hard to understand the agent's behaviour, especially when working on the image level. Therefore, neuro-symbolic RL aims at creating policies that are interpretable in the first place. Unfortunately, interpretability is not explainability. To achieve both, we introduce Neurally gUided Differentiable loGic policiEs (NUDGE). NUDGE exploits trained neural networkbased agents to guide the search of candidate-weighted logic rules, then uses differentiable logic to train the logic agents. Our experimental evaluation demonstrates that NUDGE agents can induce interpretable and explainable policies while outperforming purely neural ones and showing good flexibility to environments of different initial states and problem sizes.

1 Introduction

Deep reinforcement learning (RL) agents use neural networks to take decisions from the unstructured input state space without manual engineering [Mnih et al., 2015]. However, these black-box policies lack interpretability [Rudin, 2019], i.e. the capacity to articulate the thinking behind the action selection. They are also not robust to environmental changes [Pinto et al., 2017, Wulfmeier et al., 2017]. Although performing object detection and policy optimization independently can get over these issues Devin et al. [2018], doing so comes at the cost of the aforementioned issues when employing neural networks to encode the policy.

As logic constitutes a unified symbolic language that humans use to compose the reasoning behind their behavior, logic-based policies can tackle the interpretability problems for RL. Recently proposed Neural Logic RL (NLRL) agents [Jiang and Luo, 2019] construct logic-based policies using differentiable rule learners called ∂ ILP [Evans and Grefenstette, 2018], which can then be integrated with gradient-based optimization methods for RL. It represents the policy as a set of weighted rules, and performs policy gradients-based learning to solve RL tasks which require relational reasoning. It successfully produces interpretable rules, which describe each action in terms of its preconditions and outcome. However, the number of potential rules grows exponentially with the number of considered actions, entities, and their relations. NLRL is a memory-intensive approach, i.e. it generates a set of potential simple rules based on rule templates and can only be evaluated on simple abstract

NUDGE

interpretation

as weighted rules

explanation

by input gradients

0.65:right(agent):-type(O1,agent),type(O2,key),

¬has_key(O1),on_right(O2,O1).

⁝

¬have_key(agent)

Why

right

?

on_right(key,agent)

on_right(door,agent)

closeby(agent,door)

⁝

'Because the agent doesn't have the key,

and the key is on right of the agent.'

relational state

action distribution

Differentiable

Forward

Reasoning

0.10:jump(agent)

0.85:right(agent)

0.04:left(agent)

0.01:idle(agent)

0.98:type(obj1,agent)

0.96:type(obj2,enemy)

0.99:¬has_key(agent)

0.04:closeby(obj1,obj2)

⁝

left

jump

action space

right

idle

agent

key

enemy

door

input

state

environments, created for the occasion. This approach can generate many newly invented predicates without their specification of meaning [Evans and Grefenstette, 2018], making the policy challenging to interpret for complex environments. Moreover, the function of explainability is absent, i.e. the agent cannot explain the importance of each input on its decision. Explainable agents should adaptively produce different explanations given different input states. A question thus arises: How can we build interpretable and explainable RL agents that are robust to environmental changes?

To this end, we introduce Neurally gUided Differentiable loGic policiEs (NUDGE), illustrated in Figure 1, that embody the advantages of logic: they are easily adaptable to environmental changes, composable, interpretable and explainable (because of our differentiable logic module). Given an input state, NUDGE extracts entities and their relations, converting raw states to a logic representations. This probabilistic relational states are used to deduce actions, using differentiable forward reasoning [Evans and Grefenstette, 2018, Shindo et al., 2023a]. NUDGE produces a policy that is both interpretable , i.e. provides a policy as a set of weighted interpretable rules that can be read out by humans, and explainable , i.e. explains which input is important using gradient-based attribution methods [Sundararajan et al., 2017] over logical representations.

To achieve an efficient learning with logic policies, we provide an algorithm to train NUDGE agents based on the PPO actor-critic framework. Moreover, we propose a novel rule-learning approach, called Neurally-Guided Symbolic Abstraction , where the candidate rules for the logic-based agents are obtained efficiently by being guided by neural-based agents. NUDGE distillates abstract representations of neural policies in the form of logic rules. Rules are assigned with their weights, and we perform gradient-based optimization using the PPO actor-critic framework.

Overall, we make the following contributions:

We propose NUDGE 3 : differentiable logical policies that learn interpretable rules and produce explanations for their decisions in complex environments. NUDGE uses neurally-guided symbolic abstraction to efficiently find a promising ruleset using pretrained neural-based agents guidance.

We empirically show that NUDGE agents: (i) can compete with neural-based agents, (ii) adapt to environmental changes, and (iii) are interpretable and explainable, i.e. produce interpretable policies as sets of weighted rules and provide explanations for their action selections.

We evaluate NUDGE on 2 classic Atari games and on 3 proposed object-centric logically challenging environments, where agents need relational reasoning in dynamic game-playing scenarios.

We start off by introducing the necessary background. Then we explain NUDGE's inner workings and present our experimental evaluation. Before concluding, we touch upon related work.

2 Background

We now describe the necessary background before formally introducing our NUDGE method.

Deep Reinforcement Learning . In RL, the task is modelled as a Markov decision process, M = < S , A , P, R> , where, at every timestep t , an agent in a state s t ∈ S , takes action a t ∈A , receives a reward r t = R ( s t , a t ) and a transition to the next state s t +1 , according to environment dynamics P ( s t +1 | s t , a t ) . Deep agents attempt to learn a parametric policy, π θ ( a t | s t ) , in order to maximize the return ( i.e. ∑ t γ t r t , with γ ∈ [0 , 1] ). The desired input to output ( i.e. state to action) distribution is not directly accessible, as RL agents only observe returns. The value V π θ ( s t ) (resp. Q-value Q π θ ( s t , a t ) ) function provides the return of the state (resp. state/action pair) following the policy π θ . Policy-based methods directly optimize π θ using the noisy return signal, leading to potentially unstable learning. Value-based methods learn to approximate the value functions ˆ V ϕ or ˆ Q ϕ , and implicitly encode the policy, e.g. by selecting the actions with the highest Q-value with a high probability [Mnih et al., 2015]. To reduce the variance of the estimated Q-value function, one can learn the advantage function ˆ A ϕ ( s t , a t ) = ˆ Q ϕ ( s t , a t ) -ˆ V ϕ ( s t ) . An estimate of the advantage function can be computed as ˆ A ϕ ( s t , a t ) = ∑ k -1 i =0 γ i r t + i + γ k ˆ V ϕ ( s t + k ) -ˆ V ϕ ( s t ) [Mnih et al., 2016]. The Advantage Actor-critic (A2C) methods both encode the policy π θ ( i.e. actor) and the advantage function ˆ A ϕ ( i.e. critic), and use the critic to provide feedback to the actor, as in [Konda and Tsitsiklis, 1999]. To push π θ to take actions that lead to higher returns, gradient ascent can be applied to L PG ( θ ) = ˆ E [log π θ ( a | s ) ˆ A ϕ ] . Proximal Policy Optimization (PPO) algorithms ensure minor policy updates that avoid catastrophic drops [Schulman et al., 2017], and can be applied to actor-critic methods. To do so, the main objective constraints the policy ratio r ( θ ) = π θ ( a | s ) π θ old ( a | s ) , following L PR ( θ ) = ˆ E [min( r ( θ ) ˆ A ϕ , clip( r ( θ ) , 1 -ϵ, 1+ ϵ ) ˆ A ϕ )] , where clip constrains the input within [1 -ϵ, 1+ ϵ ] . PPO actor-critic algorithm's global objective is L ( θ, ϕ ) = ˆ E [ L PR ( θ ) -c 1 L V F ( ϕ )] , with L V F ( ϕ )=( ˆ V ϕ ( s t ) -V ( s t )) 2 being the value function loss. An entropy term can also be added to this objective to encourage exploration.

First-Order Logic (FOL). In FOL, a Language L is a tuple ( P , D , F , V ) , where P is a set of predicates, D a set of constants, F a set of function symbols (functors), and V a set of variables. A term is either a constant ( e.g. obj1 , agent ), a variable ( e.g. O1 ), or a term which consists of a function symbol. An atom is a formula p ( t 1 , . . . , t n ) , where p is a predicate symbol ( e.g. closeby ) and t 1 , . . . , t n are terms. A ground atom or simply a fact is an atom with no variables ( e.g. closeby ( obj1 , obj2 ) ). A literal is an atom ( A ) or its negation ( ¬ A ). A clause is a finite disjunction ( ∨ ) of literals. A ground clause is a clause with no variables. A definite clause is a clause with exactly one positive literal. If A,B 1 , . . . , B n are atoms, then A ∨ ¬ B 1 ∨ . . . ∨ ¬ B n is a definite clause. We write definite clauses in the form of A :-B 1 , . . . , B n . Atom A is called the head , and set of negative atoms { B 1 , . . . , B n } is called the body . We call definite clauses as rules for simplicity in this paper.

Differentiable Forward Reasoning is a data-driven approach of reasoning in FOL [Russell and Norvig, 2010]. In forward reasoning, given a set of facts and a set of rules, new facts are deduced by applying the rules to the facts. Differentiable forward reasoning [Evans and Grefenstette, 2018, Shindo et al., 2023a] is a differentiable implementation of the forward reasoning with tensor-based differentiable operations.

3 Neurally Guided Logic Policies

Figure 2 illustrates an overview of RL on NUDGE. They consist of a policy reasoning module and a policy learning module. NUDGE performs end-to-end differentiable policy reasoning based on forward reasoning, which computes action distributions given input states. On top of the reasoning module, policies are learned using neurally-guided symbolic abstraction and an actor-critic framework.

3.1 Policy Reasoning: Selecting Actions using Differentiable Forward Reasoning.

To realize NUDGE, we introduce a language to describe actions and states in FOL. Using it, we introduce differentiable policy reasoning using forward chaining reasoning.

3.1.1 Logic Programs for Actions

In RL, states and actions are key components since the agent performs the fundamental iteration of observing the state and taking an action to maximize its expected return. To achieve an efficient computation on first-order logic in RL settings, we introduce a simple language suitable for reasoning about states and actions.

We split the predicates set P into two different sets, i.e. , action predicates ( P A ), which define the actions and state predicates ( P S ) used for the observed states. If an atom A consists of an action predicate, A is called an action atom . If A consists of a state predicate, A is called state atom .

2 Differentiable Forward Reasoning

Critic

Differentiable Logic Policy (Actor)

0.00 : left

0.95 : right

0.05 : jump

0.00 : idle

Logical State

Representation

Induced

Action Atoms

Action

Distribution

Neurally-Guided

Symbolic Abstraction

Policy Reasoning

Policy Learning

Neural

Policy

1

1

Relational Perception

3 Action Extractor

0.98:right (1) (agent):-type(O1,agent),type(O2,key)

￢

0.96:right (2) (agent):-type(O1,agent),type(O2,door)

0.65:jump (1) (agent):-type(O1,agent),type(O2,enemy)

0.99:type(obj1,agent)

0.99:type(obj2,enemy)

0.99:type(obj3,key)

0.96:type(obj4,door)

0.93:closeby(obj1,obj2)

0.89:

￢

has_key(obj1)

...

0.89:right (1) (agent)

0.08:right (2) (agent)

0.00:left (1) (agent)

0.00:left (2) (agent)

0.32:jump (1) (agent)

0.00:idle (1) (agent)

...

Generated

Rules

2

3

1

...

Definition 1 Action-state Language is a tuple of ( P A , P S , D , V ) , where P A is a set of action predicates, P S is a set of state predicates, D is a set of constants for entities, and V is a set of variables.

For example, for Getout illustrated in Figures 1 and 2, actual actions are: left , right , jump , and idle . We define action predicates P A = { left ( 1 ) , left ( 2 ) , right ( 1 ) , right ( 2 ) , jump ( 1 ) , idle ( 1 ) , ... } and state predicates P S = { type , closeby , ... } . To encode different reasons for a given game action, we can generate several action predicates ( e.g. right ( 1 ) and right ( 2 ) for right ). By using these predicates, we can compose action atoms, e.g. right ( 1 ) ( agent ) , and state atoms, e.g. type ( obj1 , agent ) . An action predicate can also be a state predicate, e.g. in multiplayer settings. Now, we define rules to describe actions in the action-state language.

Definition 2 Let X A be an action atom and X (1) S , . . . , X ( n ) S be state atoms. An action rule is a rule, written as X A :-X (1) S , . . . , X ( n ) S .

For example, for action right , we define an action rule as:

which can be interpreted as 'The agent should go right if the agent does not have the key and the key is located on the right of the agent." . Having several action predicates for an actual action (in the game) allows our agents to define several action rules that describe different reasons for the action.

3.1.2 Differentiable Logic Policies

We denote the set of actual actions by A , the set of action rules by C , the set of all of the facts by G = G A ∪ G S where G A is a set of action atoms and G S is a set of state atoms. G contains all of the facts produced by a given FOL language. We here consider ordered sets, i.e. each element has its index. We also denote the size of the sets as: A = |A| , C = |C| , G = |G| , and G A = |G A | .

We propose Differentiable Logic Policies , which perform differentiable forward reasoning on action rules to produce a probability distribution over actions. The policy computation consists of 3 components: the relational perception module (1), the differentiable forward-reasoning module (2), and the action-extraction module (3).

The policy π ( C , W ) parameterized by a set of action rules C and the rules' weights W is computed as follows:

with f perceive Θ : R N → [0 , 1] G a perception function that maps the raw input state s t ∈ R N into a set of probabilistic atoms, f reason ( C , W ) : [0 , 1] G → [0 , 1] G A a differentiable forward reasoning function parameterized by a set of rules C and rule weights W , and f act : [0 , 1] G A → [0 , 1] A an action-selection function, which computes the probability distribution over the action space.

Relational Perception. NUDGE agents take an object-centric state representations as input, obtained by e.g. using object detection [Redmon et al., 2016] or discovery [Lin et al., 2020, Delfosse et al., 2022] methods. These models return the detected objects and their attributes ( e.g. class and positions). They are then converted into a probabilistic logic form with their relations, i.e. a set of facts with their probabilities. An input state s t ∈ R N is converted to a valuation vector v ∈ [0 , 1] G , which maps each fact to a probabilistic value. For example, let G = { type ( obj1 , agent ) , type ( obj2 , enemy ) , closeby ( obj1 , obj2 ) , jump ( agent ) } . A valuation vector [0 . 8 , 0 . 6 , 0 . 3 , 0 . 0] ⊤ maps each fact to a corresponding probabilistic value. NUDGE performs differentiable forward reasoning by updating the initial valuation vector v (0) T times to obtain v ( T ) .

Initial valuation vector v (0) is computed as follows. For each ground state atom p ( t 1 , . . . , t n ) ∈ G S , e.g. closeby ( obj1 , obj2 ) , a differentiable function is called to compute its probability, which maps each term t 1 , . . . , t n to vector representations according to the interpretation, e.g. obj1 and obj2 are mapped to their positions, then perform binary classification using the distance between them. For action atoms, zero is assigned as its initial probability ( e.g. for jump ( 1 ) ( agent ) ).

Differentiable Forward Reasoning. Given a set of candidate action rules C , we create the reasoning function f reason ( C , W ) : [0 , 1] G → [0 , 1] G A , which takes the initial valuation vector and induces action atoms using weighted action rules. We assign weights to the action rules of C as follows: We fix the target programs' size, M , i.e. select M rules out of C candidate action rules. To do so, we introduce C -dimensional weights W = [ w 1 , . . . , w M ] where w i ∈ R C ( cf. Figure 6 in the appendix). We take the softmax of each weight vector w i ∈ W to select M action rules in a differentiable manner.

We perform T -step forward reasoning using action rules C with weights W . We compose the differentiable forward reasoning function following Shindo et al. [2023a]. It computes soft logical entailment based on efficient tensor operations. Our differentiable forward reasoning module computes new valuation v ( T ) including all induced atoms given weighted action rules ( C , W ) and initial valuation v (0) . Finally, we compute valuations on action atoms v A ∈ [0 , 1] G A by extracting relevant values from v ( T ) . We provide details in App. E.

Compute Action Probability. Given valuations on action atoms v A , we compute the action distribution for actual actions. Let a i ∈ A be an actual action, and v ′ 1 , . . . , v ′ n ∈ v A be valuations which are relevant for a i ( e.g. valuations of right ( 1 ) ( agent ) and right ( 2 ) ( agent ) in v A for the action right ). We assign scores to each action a i based on the log-sum-exp approach of Cuturi and Blondel [2017]: val ( a i ) = γ log ∑ 1 ≤ i ≤ n exp( v ′ i /γ ) , that smoothly approximates the maximum value of { v ′ 1 , . . . , v ′ n } . We use γ > 0 as a smoothing parameter. The action distribution is then obtained by taking the softmax over the evaluations of all actions.

3.2 Policy Learning

So far, we have considered that candidate rules for the policy are given, requiring human experts to handcraft potential rules. To avoid this, template-based rule generation [Evans and Grefenstette, 2018, Jiang and Luo, 2019] can be applied, but the number of generated rules increases exponentially with the number of entities and their potential relations. This technique is thus difficult to apply to complex environments where the agents need to reason about many different relations of entities.

To mitigate this problem, we propose an efficient learning algorithm for NUDGE that consists of two steps: neurally-guided symbolic abstraction and gradient-based optimization. First, NUDGE obtains a symbolic abstract set of rules, aligned with a given neural policy. The set of candidate rules is selected by neurally-guided topk search, i.e. , we generate a set of promising rules using a neural policy as an oracle. Then we assign randomized weights to each selected rule and perform differentiable reasoning. We finally optimize the rule weights using a 'logic actor - neural critic' algorithm that aims at maximizing the expected return. Let us elaborate on each step.

3.2.1 Neurally Guided Symbolic Abstraction

̸

Given a well-performing neural policy π θ , promising action rules for an RL task entail the same actions as the ones selected by π θ . We generate such rules by performing topk search-based abstraction, which uses the neural policy to evaluate rules efficiently. The inputs are initial rules C 0 , neural policy π θ . We start with elementary action rules and refine them to generate better action rules. C to _ open is a set of rules to be refined, and initialized as C 0 . For each rule C i ∈ C to _ open , we generate new rules by refining them as follows. Let C i = X A ← X (1) S , . . . , X ( n ) S be an already selected general action rule. Using a randomly picked ground or non-ground state atom Y S ( = X ( i ) S ∀ i ∈ [1 , ..., n ] ), we refine the selected rule by adding a new state atom to its body, obtaining: X A ← X (1) S , . . . X ( n ) S , Y S .

We evaluate each newly generated rule to select promising rules. We use the neural policy π θ as a guide for the rule evaluation, i.e. rules that entail the same action as the neural policy π θ are promising action rules. Let X be a set of states. Then we evaluate the rule R , following:

where N ( R, X ) is a normalization term, π R , 1 is the differentiable logic policy with rules R = { R } and rule weights 1 , which is an 1 × 1 identity matrix (for consistent notation), and · is the dot product. Intuitively, π ( R , 1 ) is the logic policy that has R as its only action rule. If π ( R , 1 ) produces a similar action distribution as the one produced by π θ , we regard the rule R as a promising rule. We compute similarity scores between the neural policy π θ and the logic one π ( R , 1 ) using the dot product between the two action distributions, and average them across the states of X . The normalization term avoids high scores for simple rules.

To compute the normalization term, we ground R , i.e. we replace variables with ground terms. We consider all of the possible groundings. Let T be the set of all of the possible variables substitutions to ground R . For each τ ∈ T , we get a ground rule Rτ = X A τ :-X (1) S τ, . . . , X ( n ) S τ , where Xτ represents the result of applying substitution τ to atom X . Let J = { j 1 , . . . , j n } be indices of the ground atoms X (1) S τ, . . . , X ( n ) S τ in ordered set of ground atoms G . Then, the normalization term is computed as:

where v (0) s is an initial valuation vector for state s , i.e. f perceive Θ ( s ) . Eq. 3 quantifies how often the body atoms of the ground rule Rτ are activated on the given set of states X . Simple rules with fewer atoms in their body tend to have large values, and thus their evaluation scores in Eq. 2 tend to be small. After scoring all of the new rules, NUDGE select topk rules to refine them in the next step. To this end, all of the topk rules in each step will be returned as the candidate ruleset C for the policy ( cf. App. A for more details about our algorithm).

We perform the action-rule generation for each action. In practice, NUDGE maintains the cached history H of states and actions produced by the neural policy. For a given action, the search quality increases together with the amount of times the action was selected, i.e. if H does not contain any record of the action, then all action rules (with this action as head) would get the same scores, leading to a random search.

NUDGE has thus produced candidate action rules C , that will be associated with W to form untrained differentiable logic policy π ( C , W ) , as the ones described in Section 3.1.2.

3.2.2 Learning the Rules' Weights using the Actor-critic Algorithm

In the following, we consider a (potentially pretrained) actor-critic neural agent, with v ϕ its differentiable state-value function parameterized by ϕ (critic). Given a set of action rules C , let π ( C , W ) be a differentiable logic policy. NUDGE learns the weights of the action rules in the following steps. For each non-terminal state s t of each episode, we store the actions sampled from the policy ( a t ∼ π ( C , W ) ( s t ) ) and the next states s t +1 . We update the value function and the policy as follows:

steps (log)

3Fishes

GetOut

Loot

Neural PPO

Neural Logic RL

NUDGE (top-1 rf.)

NUDGE (top-3 rf.)

NUDGE (top-5 rf.)

NUDGE (with E.S.)

Human

Random

steps

steps

return

The logic policy π ( C , W ) thus learn to maximize the expected return, potentially bootstrapped by the use of a pretrained neural critic. Moreover, to ease interpretability, NUDGE can prune the unused action rules ( i.e. with low weights) by performing topk selection on the optimized rule weights after learning.

4 Experimental Evaluation

We here compare neural agents' performances to NUDGE ones, showcase NUDGE interpretable policies and its ability to report the importance of each input on their decisions, i.e. explainable logic policies. We use DQN agents (on Atari environments) and PPO actor-critic (on logic-ones) as neural baselines, for comparison and PPO as pretrained agents to guide the symbolic abstraction. All agent types receive object-centric descriptions of the environments. For clarity, we annotate action predicates on action rules with specific names on purpose, e.g. right [ to _ key ] instead of right (1) (when the rule describes an action right motivated to get the key).

We intend to compare agents with object-centric information bottlenecks. We thus had to extract object-centric states of the Atari environments. To do so, we make use of the Object-Centric Atari library [Delfosse et al., 2023]. As Atari games do not embed logic challenges, but are rather desgined to test the reflexes of human players, we also created 3 logic-oriented environments. We thus have modified environments from the Procgen [Mohanty et al., 2020] environments that are open-sourced along with our evaluation to have object-centric representations. Our environments are easily hackable. We provide variations of these environments also to evaluate the ease of adaptation of every agent type. In GetOut , the goal is to obtain a key, and then go to a door, while avoiding a moving enemy. GetOut+ is a more complex variation with a larger world containing 5 enemies (among which 2 are static). In 3Fishes , the agent controls a fish and is confronted with 2 other fishes, one smaller (that the agent needs to 'eat', i.e. go to) and one bigger, that the agent needs to dodge. A variation is 3Fishes-C , where the agent can eat green fishes and dodge red ones. Finally, in Loot , the agent can open 1 or 2 chests and their corresponding ( i.e. same color) keys. In Loot-C , the chests have different colors. Further details and hyperparameters are provided in App. D.

We aim to answer the following research questions: Q1. How does NUDGE compare with neural and logic baselines? Q2. Can NUDGE agents easily adapt to environmental changes? Q3. Are NUDGE agents interpretable and explainable?

NUDGE competes with existing methods (Q1) . We compare NUDGE with different baselines regarding their scores (or returns). First, we present scores obtained by trained DQN, Random and NUDGE agents (with expert supervision) on 2 Atari games ( cf. Table 1). Our result show that NUDGE obtain better (Asterix) or similar (Freeway) scores than DQN. However, as said, Atari games are not logically challenging. We thus evaluate NUDGE on 3 logic environments. Figure 3 shows the returns in GetOut, 3Fishes, and Loot, with descriptions for each baseline in the caption. NUDGE obtains better performances than neural baselines (Neural PPO) on 3Fishes, is more stable on GetOut, i.e. less variance, and achieves faster convergence on Loot. This shows that NUDGE successfully distillates logic-based policies competing with neural baselines in different complex environments.

We also evaluate agents with a baseline without symbolic abstraction, where candidate rules are generated not being guided by neural policies, i.e. accept all of the generated rules in the rule refinement steps. This setting corresponds to the template-based approach [Jiang and Luo, 2019], but we train the agents by the actor-critic method, while vanilla policy gradient [Williams, 1992] is employed in [Jiang and Luo, 2019]. For the no-abstraction baseline and NUDGE, we provide initial action rules with basic type information, e.g. jump ( 1 ) ( agent ) :-type ( O1 , agent ) , type ( O2 , enemy ) , for each action rule. For this baseline, we generate 5 rules for GetOut, 30 rules for 3Fishes, and 40 rules for Loot in total to define all of the actual actions. NUDGE agents with small k tend to have less rules, e.g. 5 rules in Getout, 6 rules in 3Fishes, and 8 rules in Loot for the top1 refinement version. In Figure 3, the no-abstraction baselines perform worse than neural PPO and NUDGE in each environment, even though they have much more rules in 3Fishes and Loot. NUDGE thus composes efficient logic-based policies using neurally-guided symbolic abstraction. In App. B.1, we visualize the transition of the distribution of the rule weights in the GetOut environment.

To demonstrate the necessity of differentiability within symbolic reasoning in continuous RL environments, we simulated classic ( i.e. discrete logic) agents, that reuse NUDGE agents' set of action rules, but with all weights fixed to 1 . 0 . Such agents can still play better than ones with pure symbolic reasoners, as they still use fuzzy ( i.e. , continuous) concepts, such as closeby . In all our environments, NUDGE clearly outperforms the classic counterparts ( cf. Table 1 right). This experiment shows the superiority of differentiable policy reasoning (embedding weighted action rules) over policy without weighted rules (classic).

To further reduce the gap between our environments and those from environments Cao et al. [2022], we have adapted our loot environment (Loot-hard). In this environment, the agent first has to pick up keys and open their corresponding saves, before being able to exit the level (by going to a final exit tile). Our results in Table 2 (bottom), as well as curves on Figure 10 in the appendix show the dominance of NUDGE agents.

NUDGEagents adapt to environment changes (Q2).

We used the agents trained on the basic environment for this experimental evaluation, with no retraining or finetuning. For 3Fishes-C, we simply exchange the atom is _ bigger with the atom same _ color . This easy modification is not applicable on the black-box networks of neural PPO agents. For GetOut+ and Loot-C, we do not apply any modification to the agents. Our results are summarized in Table 2. Note that the agents obtains better performances in the 3Fishes-C variation, dodging a (small) red fish is eas-

ier than a big one. For GetOut+, NUDGE performances have decreased as avoiding 5 enemies drastically increases the difficulty of the game. On Loot-C, the performances are similar to the ones obtained in the original game. Our experiments show that NUDGE logic agents can easily adapt to environmental changes.

NUDGE agents are interpretable and explainable (Q3). We show that NUDGE agents are interpretable and explainable by showing that (1) NUDGE produces interpretable policy as a set of weighted rules, and (2) NUDGE can show the importance of each atom, explaining its action choices.

The efficient neurally-guided learning on NUDGE enables the system to learn rules without inventing predicates with no specific interpretations, which are unavoidable in template-based approachs [Evans and Grefenstette, 2018, Jiang and Luo, 2019]. Thus, the policy can easily be read out by extracting action rules with high weights. Figure 5 shows some action rules discovered by NUDGE in GetOut. The first rule says: 'The agent should jump when the enemy is close to the agent (to avoid the enemy). ' . The produced NUDGE is an interpretable policy

with its set of weighted rules using interpretable predicates. For each state, we can also look at the valuation of each atom and the selected rule.

Moreover, contrary to classic logic policies, the differentiable ones produced by NUDGE allow us to compute the attribution values over the logical representations via backpropagation of the policies' gradients. We can compute the action gradients w.r.t. input atoms, i.e. ∂ v A /∂ v (0) , as shown in Figure 4, which represent the relevance scores of the probabilistic input atoms v (0) for the actions given a specific state. The explanation is computed on the state shown in Figure 1, where the agent takes right as its action. Important atoms receive large gradients, e.g. ¬ has _ key ( agent ) and on _ right ( obj2 , obj1 ) . By extracting relevant atoms with large gradients, NUDGE can produce clear explanations for the action selection. For example, by

left

[to_key]

right

[to_key]

Gradients of actions w.r.t input atoms

jump

left

[to_door]

right

[to_door]

closeby(obj1,obj4)

has_key(agent)

¬has_key(agent)

on_left(obj4,obj1)

on_right(obj2,obj1)

type(obj1,agent)

type(obj2,key)

type(obj4,enemy)

type(obj4,key)

extracting the atoms wrapped in orange in Figure 4, NUDGE can explain the motivation: 'The agent decides to go right because it does not have the key and the key is located on the right-side of it. ' . These results show that NUDGE policies are interpretable and explainable , i.e. each action predicate is defined by rules that humans can fully understand, and gradient-based explanations for the action selections can be efficiently produced.

We also computed explanations of a neural agent on the same state, also using the input-gradients method. The concepts which received the highest explanation scores are: the key x-axis position (1.22), the key x-axis velocity (1.21), the enemy y-axis velocity (1.10), and the door x-axis position (0.77). For a consistent explanation, the agent x-axis position should also be among the highest scores. For this specific state, the neural agent seems to be placing its explanations on non relevant concepts (such as the key's velocity). Stammer et al. [2021] have shown that neural networks trained on logical concepts tend to produce wrong explanations often, and additional regularization/supervision about explanations during training is necessary to force neural networks to produce correct explanations. This approach requires additional efforts to label each state with its correct explanation. NUDGEpolicies produces better explanations than neural-based agents. More explanations produced by NUDGE are provided in Figure 11 in the appendix.

5 Related Work

Relational RL [Dzeroski et al., 2001, Kersting et al., 2004, Kersting and Driessens, 2008, Lang et al., 2012] has been developed to tackle RL tasks in relational domains. Relational RL frameworks incorporate logical representations and use probabilistic reasoning. With NUDGE, we make use of differentiable logic programming. The Neural Logic Reinforcement Learning (NLRL) [Jiang and Luo, 2019] framework is the first to integrate Differentiable Inductive Logic Programming ( ∂ ILP) [Evans and Grefenstette, 2018] to RL domain. ∂ ILP learns generalized logic rules from examples by gradient-based optimization. NLRL adopts ∂ ILP as a policy

FOL

N.G.

Int.

Exp.

NLRL

✓

✗

✓

✗

NeSyRL

✓

✗

✓

✗

DiffSES

✗

✓

✓

✗

NUDGE

✓

✓

✓

✓

function. We extend this approach by proposing neurally-guided symbolic abstraction embracing extensive work of ∂ ILP [Shindo et al., 2021b,a, 2023b] for learning complex programs including functors in visual scenes, allowing agents to learn interpretable action rules efficiently for complex environments.

GALOIS [Cao et al., 2022] is a framework to represent policies as logic programs using the sketch setting [SolarLezama, 2008], where programs are learned to fill blanks. NUDGE performs structure learning from scratch using policy gradients. KoGun [Zhang et al., 2020] integrates human knowledge as a prior for RL agents. NUDGE learns a policy as a set of weighted rules and thus also can integrate human knowledge. NeuroSymbolic RL (NeSyRL) [Kimura et al., 2021] uses Logical Neural Networks (LNNs) [Riegel et al., 2020] for the policy computation. LNNs parameterize the soft logical operators while NUDGE parameterizes rules with their weights. Deep Relational RL approaches [Zambaldi et al., 2018] achieve relational reasoning as a neural network, but NUDGE explicitly encodes relations in logic. Many languages for planning and RL tasks have been developed [Fikes and Nilsson, 1971, Fox and Long, 2003]. Our approach is inspired by situation calculus [Reiter, 2001], which is an established framework to describe states and actions in logic.

Symbolic programs within RL have been investigated, e.g. program guided agent [Sun et al., 2020], program synthesis [Zhu et al., 2019], PIRL [Verma et al., 2018], SDRL [Lyu et al., 2019], interpretable model-based hierarchical RL Xu and Fekri [2021], deep symbolic policy Landajuela et al. [2021], and DiffSES Zheng et al. [2021]. These approaches use domain specific languages or propositional logic, and address either of interpretability or explainability of RL. To this end, in Table 3, we compare NUDGE with the most relevant approaches that share at least 2 of the following aspects: supporting first-order logic, neural guidance, interetability and explainability. NUDGE is the first method to use neural guidance for differentiable first-order logic policies and to address both interpretability and explainability . Specifically, PIRL develops functional programs with neural guidance using sketches , which define a grammar of programs to be generated. In contrast, NUDGE performs structure learning from scratch using a neural guidance with a language bias of mode declarations [Muggleton, 1995, Cropper et al., 2022] restricting the search space. As NUDGE uses first-order logic, it can incorporate background knowledge in a declarative form, e.g. with a few lines of relational atoms and rules. To demonstrate this, NUDGE has been evaluated on challenging environments where the main interest is relational reasoning.

6 Conclusion

We proposed NUDGE, an interpretable and explainable policy reasoning and learning framework for reinforcement learning. NUDGE uses differentiable forward reasoning to obtain a set of interpretable weighted rules as policy. It performs neurally-guided symbolic abstraction, which efficiently distillates symbolic representations from a neural policy, and incorporate an actor-critic method to perform gradient-based policy optimization. We empirically demonstrated that NUDGE agents (1) can compete with neural based policies, (2) use logical representations to produce both interpretable and explainable policies and (3) can automatically adapt or easily be modified and are thus robust to environmental changes.

Societal and environmental impact. As NUDGE agents can explain the importance of each the input on their decisions, and as their rules are interpretable, it can help us understanding the decisions of RL agents trained in sensitive complicated domains, and potentially help discover biases and misalignments of discriminative nature. While the distillation of the learned policy into a logic ones augments the computational resources needed for completing the agents learning process, we believe that the logic the agents' abstraction capabilities will overall reduce the need of large neural networks, and will remove the necessity of retraining agents for each environmental change, leading to an overall reduction of necessary computational power.

Limitation and Future Work. NUDGE is only complete if provided with a sufficiently expressive language (in terms of predicates and entities) to approximate neural policies. Interesting lines of future research can include: (i) an automatic discovery of predicates, using e.g. predicate invention [Muggleton et al., 2015], (ii) augmenting the number of accessible entities to reason on. Explainable interactive learning [Teso and Kersting, 2019] in RL can integrate NUDGE agents, since it produces explanations on logical representations. Causal RL [Madumal et al., 2020] and meta learning [Mishra et al., 2018] also constitute interesting avenues for NUDGE's development. Finally, we could incorporate objects extraction methods [Delfosse et al., 2022] within our NUDGE agents, to obtain logic agents that extract object and their properties from images.

Acknowledgements. The authors thank the anonymous reviewers for their valuable feedback. This research work has been funded by the German Federal Ministry of Education and Research, the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK) within their joint support of the National Research Center for Applied Cybersecurity ATHENE, via the 'SenPai: XReLeaS' project. We gratefully acknowledge support by the Federal Ministry of Education and Research (BMBF) AI lighthouse project 'SPAICER' (01MK20015E), the EU ICT-48 Network of AI Research Excellence Center 'TAILOR' (EU Horizon 2020, GA No 952215), and the Collaboration Lab 'AI in Construction' (AICO).

References

Yushi Cao, Zhiming Li, Tianpei Yang, Hao Zhang, Yan Zheng, Yi Li, Jianye Hao, and Yang Liu. GALOIS: boosting deep reinforcement learning via generalizable logic synthesis. CoRR , 2022.

Andrew Cropper, Sebastijan Dumancic, Richard Evans, and Stephen H. Muggleton. Inductive logic programming at 30. Mach. Learn. , 2022.

Marco Cuturi and Mathieu Blondel. Soft-DTW: a differentiable loss function for time-series. In Proceedings of the 34th International Conference on Machine Learning , 2017.

Quentin Delfosse, Patrick Schramowski, Martin Mundt, Alejandro Molina, and Kristian Kersting. Adaptive rational activations to boost deep reinforcement learning. CoRR , 2021.

Quentin Delfosse, Wolfgang Stammer, Thomas Rothenbacher, Dwarak Vittal, and Kristian Kersting. Boosting object representation learning via motion and object continuity. CoRR , 2022.

Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sebastian Sztwiertnia, and Kristian Kersting. Ocatari: Objectcentric atari 2600 reinforcement learning environments. CoRR , 2023.

Coline Devin, Pieter Abbeel, Trevor Darrell, and Sergey Levine. Deep object-centric representations for generalizable robot learning. In IEEE International Conference on Robotics and Automation , 2018.

Saso Dzeroski, Luc De Raedt, and Kurt Driessens. Relational reinforcement learning. Mach. Learn. , 2001.

Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. J. Artif. Intell. Res. , 2018.

Richard Fikes and Nils J. Nilsson. STRIPS: A new approach to the application of theorem proving to problem solving. Artif. Intell. , 1971.

Maria Fox and Derek Long. PDDL2.1: an extension to PDDL for expressing temporal planning domains. J. Artif. Intell. Res. , 2003.

Zhengyao Jiang and Shan Luo. Neural logic reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, International Conference on Machine Learning , 2019.

Kristian Kersting and Kurt Driessens. Non-parametric policy gradients: a unified treatment of propositional and relational domains. In William W. Cohen, Andrew McCallum, and Sam T. Roweis, editors, International Conference on Machine Learning , 2008.

Kristian Kersting, Martijn van Otterlo, and Luc De Raedt. Bellman goes relational. In Carla E. Brodley, editor, International Conference on Machine Learning , 2004.

Daiki Kimura, Masaki Ono, Subhajit Chaudhury, Ryosuke Kohita, Akifumi Wachi, Don Joven Agravante, Michiaki Tatsubori, Asim Munawar, and Alexander Gray. Neuro-symbolic reinforcement learning with first-order logic. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Conference on Empirical Methods in Natural Language Processing , 2021.

Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In Sara A. Solla, Todd K. Leen, and Klaus-Robert Müller, editors, Advances in Neural Information Processing Systems 12 , 1999.

Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep reinforcement learning. In International Conference on Machine Learning , 2021.

Tobias Lang, Marc Toussaint, and Kristian Kersting. Exploration in relational domains for model-based reinforcement learning. J. Mach. Learn. Res. , 2012.

Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations , 2020.

Daoming Lyu, Fangkai Yang, Bo Liu, and Steven Gustafson. SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In The Thirty-Third AAAI Conference on Artificial Intelligence , 2019.

Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In The Thirty-Fourth AAAI conference on Artificial Intelligence (AAAI) , 2020.

Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In 6th International Conference on Learning Representations , 2018.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nat. , 2015.

Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In MariaFlorina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33rd International Conference on Machine Learning , 2016.

Sharada P. Mohanty, Jyotish Poonganam, Adrien Gaidon, Andrey Kolobov, Blake Wulfe, Dipam Chakraborty, Grazvydas Semetulskis, Jo textasciitilde ao Schapke, Jonas Kubilius, Jurgis Pasukonis, Linas Klimas, Matthew J. Hausknecht, Patrick MacAlpine, Quang Nhat Tran, Thomas Tumiel, Xiaocheng Tang, Xinwei Chen, Christopher Hesse, Jacob Hilton, William Hebgen Guss, Sahika Genc, John Schulman, and Karl Cobbe. Measuring sample efficiency and generalization in reinforcement learning benchmarks: Neurips 2020 procgen benchmark. In Hugo Jair Escalante and Katja Hofmann, editors, NeurIPS 2020 Competition and Demonstration Track , 2020.

Muggleton. Inverse Entailment and Progol. New Generation Computing, Special issue on Inductive Logic Programming , 1995.

Stephen H. Muggleton, Dianhuan Lin, and Alireza Tamaddoni-Nezhad. Meta-interpretive learning of higher-order dyadic datalog: predicate invention revisited. Mach. Learn. , 2015.

Shan-Hwei Nienhuys-Cheng and Ronald de Wolf. Foundations of Inductive Logic Programming . Springer, 1997.

Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In Doina Precup and Yee Whye Teh, editors, International Conference on Machine Learning , 2017.

Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Computer Vision and Pattern Recognition , 2016.

Raymond Reiter. Knowledge in Action: Logical Foundations for Specifying and Implementing Dynamical Systems . MIT Press, 2001.

Ryan Riegel, Alexander G. Gray, Francois P. S. Luus, Naweed Khan, Ndivhuwo Makondo, Ismail Yunus Akhalwaya, Haifeng Qian, Ronald Fagin, Francisco Barahona, Udit Sharma, Shajith Ikbal, Hima Karanam, Sumit Neelam, Ankita Likhyani, and Santosh K. Srivastava. Logical neural networks. CoRR , 2020.

Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. , 2019.

Stuart Russell and Peter Norvig. Artificial intelligence - a modern approach . Prentice Hall, 2010.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , 2017.

Hikaru Shindo, Devendra Singh Dhami, and Kristian Kersting. Neuro-symbolic forward reasoning. CoRR , 2021a.

Hikaru Shindo, Masaaki Nishino, and Akihiro Yamamoto. Differentiable inductive logic programming for structured examples. In Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI) , 2021b.

Hikaru Shindo, Viktor Pfanschilling, Devendra Singh Dhami, and Kristian Kersting. α ilp: thinking visual scenes as differentiable logic programs. Mach. Learn. , 2023a.

Hikaru Shindo, Viktor Pfanschilling, Devendra Singh Dhami, and Kristian Kersting. Learning differentiable logic programs for abstract visual reasoning. CoRR , 2023b.

Armando Solar-Lezama. Program Synthesis by Sketching . PhD thesis, University of California at Berkeley, 2008.

Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Revising neurosymbolic concepts by interacting with their explanations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Shao-Hua Sun, Te-Lin Wu, and Joseph J. Lim. Program guided agent. In International Conference on Learning Representations , 2020.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning , 2017.

Stefano Teso and Kristian Kersting. Explanatory interactive machine learning. In Vincent Conitzer, Gillian K. Hadfield, and Shannon Vallor, editors, AAAI/ACM Conference on AI, Ethics, and Society , 2019.

Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Dale Schuurmans and Michael P. Wellman, editors, AAAI Conference on Artificial Intelligence , 2016.

Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning , 2018.

Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. , 1992.

Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. In Conference on Robot Learning , 2017.

Duo Xu and Faramarz Fekri. Interpretable model-based hierarchical reinforcement learning using inductive logic programming. CoRR , abs/2106.11417, 2021.

Vinícius Flores Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David P. Reichert, Timothy P. Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew M. Botvinick, Oriol Vinyals, and Peter W. Battaglia. Relational deep reinforcement learning. CoRR , 2018.

Peng Zhang, Jianye Hao, Weixun Wang, Hongyao Tang, Yi Ma, Yihai Duan, and Yan Zheng. Kogun: Accelerating deep reinforcement learning via integrating human suboptimal knowledge. In International Joint Conference on Artificial Intelligence , 2020.

Wenqing Zheng, S P Sharan, Zhiwen Fan, Kevin Wang, Yihan Xi, and Zhangyang Wang. Symbolic visual reinforcement learning: A scalable framework with object-level abstraction and differentiable expression search. CoRR , abs/2106.11417, 2021.

He Zhu, Zikang Xiong, Stephen Magill, and Suresh Jagannathan. An inductive synthesis framework for verifiable reinforcement learning. In ACM-SIGPLAN Symposium on Programming Language Design and Implementation , 2019.

Supplemental Materials

A Details on Neurally-Guided Symbolic Abstraction

We here provide details on the neurally-guided symbolic abstraction algorithm.

A.1 Algorithm of Neurally-Guided Symbolic Abstraction

We show the algorithm of neurally-guided symbolic abstraction in Algorithm 1.

Algorithm 1 Neurally-Guided Symbolic Abstraction

A.2 Rule Generation

At line 8 in Algorithm 1, given action rule C , we generate new action rules using the following refinement operation:

̸

where G ∗ S is a non-ground state atoms. This operation is a specification of (downward) refinement operator , which a fundamental technique for rule learning in ILP [Nienhuys-Cheng and de Wolf, 1997], for action rules to solve RL tasks.

We use mode declarations [Muggleton, 1995, Cropper et al., 2022] to define the search space, i.e. G ∗ S in Eq. 7 which are defined as follows. A mode declaration is either a head declaration modeh ( r , p ( mdt 1 , . . . , mdt n )) or a body declaration modeb ( r , p ( mdt 1 , . . . , mdt n )) , where r ∈ N is an integer, p is a predicate, and mdt i is a mode datatype. A mode datatype is a tuple ( pm , dt ) , where pm is a place-marker and dt is a datatype. A place-marker is either # , which represents constants, or + (resp. -), which represents input (resp. output) variables. r represents the number of the usages of the predicate to compose a solution. Given a set of mode declarations, we can determine a finite set of rules to be generated by the rule refinement.

Now we describe mode declarations we used in our experiments. For Getout, we used the following mode declarations:

For 3Fishes, we used the following mode declarations:

For Loot, we used the following mode declarations:

B Additional Results

B.1 Weights learning

Fig. 6 shows the NUDGE agent π ( C , W ) parameterized by rules C and weights W before training (top) and after training (bottom) on the GetOut environment. Each element on the x-axis of the plots corresponds to an action rule. In this examples, we have 10 action rules C = { C 0 , C 1 , . . . , C 9 } , and we assign M = 5 weights i.e. W = [ w 0 , w 1 , . . . , w 4 ] . The distributions of rule weighs with softmax are getting peaked by learning to maximize the return. The right 4 rules are redundant rules, and theses rules get low weights after learning.

B.2 Deduction Pipeline

Fig. 7 provides the deduction pipeline of a NUDGE agent on 3 different states. Facts can be deduced from an object detection method, or directly given by the object centric environment. For state #1, the agent chooses to jump as the jump action is prioritized over the other ones and all atoms that compose this rules' body have high valuation (including closeby ). In state #2, the agent chose to go left as the rule left _ key is selected. In state #3, the agent selects right as the rule right _ door has the highest forward chaining evaluation.

B.3 Policies of every logic environment.

We show the logic policies obtained by NUDGE in GetOut, 3Fishes, and Loot in Fig. 8, e.g. the first line of GetOut, ' 0 . 574 : jump ( X ) :-closeby ( O1 , O2 ) , type ( O1 , agent ) , type ( O2 , enemy ) . ', represents that the action rule is chosen by the weight vector w 1 with a value 0 . 574 . NUDGE agents have several weight vectors w 1 , . . . , w M and thus several chosen action rules are shown for each environment.

C Illustrations of our environments

We showcase in Fig. 9 one state of the 3 object-centric environments and their variations. In GetOut (blue humanoid agent), the goal is to obtain a key, then go to a door, while avoiding a moving enemy. GetOut-2En

Weights on action rules (after training)

w_1

w_2

w_3

w_4

w_1

w_2

w_3

w_4

C_0

C_1

C_2

C_3

C_4

C_5

C_6

C_7

C_8

C_9

C_0

C_1

C_2

C_3

C_4

C_5

C_6

C_7

C_8

C_9

Weights on action rules (before training)

1.0

0.8

Weights

0.6

0.4

0.2

0.0

1.0

0.8

Weights

0.6

0.4

0.2

0.0

Left_door

Right_door

observartion

state #1

2

state #2

ght(02,01)

0.048:doorleft(agen

ght(03,01)

*>:induction

state#3

1

is a variation with 2 enemies. In 3Fishes , the agent controls a green fish and is confronted with 2 other fishes, one smaller (that the agent need to 'eat', i.e. go to) and one bigger, that the agent needs to dodge. A variation is 3Fishes-C , where the agent can eat green fishes and dodge red ones, all fishes have the same size. Finally, in Loot , the (orange) agent is exposed with 1 or 2 chests and their corresponding ( i.e. same color) keys. In Loot-C , the chests have different colors. All 3 environment are stationary in the sense of Delfosse et al. [2021].

D Hyperparameters and rules sets

D.1 Hyperparameters

We here provide the hyperparameters used in our experiments. We set the clip parameter ϵ clip = 0 . 2 , the discount factor γ = 0 . 99 . We use the Adam optimizer, with 1 e -3 as actor learning rate, 3 e -4 as critic learning rate. The episode length is 500 timesteps. The policy is updated every 1000 steps We train every algorithm for 800 k steps on each environment, apart from neural PPO, that needed 5 M steps on Loot. We use an epsilon greedy strategy with ϵ = max ( e -episode 500 , 0 . 02) .

D.2 Rules set

All the rules set C of the different NUDGE and logic agents are available at https://github.com/k4ntz/ NUDGE in the folder nsfr/nsfr/data/lang .

E Details of Differentiable Forward Reasoning

We provide details of differentiable forward reasoning used in NUDGE. We denote a valuation vector at time step t as v ( t ) ∈ [0 , 1] G . We also denote the i -th element of vector x by x [ i ] , and the ( i, j ) -th element of matrix X by X [ i, j ] . The same applies to higher dimensional tensors.

Loot

Loot-C

3Fishes

3Fishes-C

GetOut

GetOut-2En

E.1 Differentiable Forward Reasoning

We compose the reasoning function f reason ( C , W ) : [0 , 1] G → [0 , 1] G A , which takes the initial valuation vector and returns valuation vector for induced action atoms. We describe each step in detail.

(Step 1) Encode Logic Programs to Tensors. To achieve differentiable forward reasoning, each action rule is encoded to a tensor representation. Let S be the number of the maximum number of substitutions for existentially quantified variables in C , and L be the maximum length of the body of rules in C . Each action rule C i ∈ C is encoded to a tensor I i ∈ N G × S × L , which contain the indices of body atoms. Intuitively, I i [ j, k, l ] is the index of the l -th fact (subgoal) in the body of the i -th rule to derive the j -th fact with the k -th substitution for existentially quantified variables.

For example. let R 0 = jump ( agent ) :-type ( O1 , agent ) , type ( O2 , enemy ) , closeby ( O1 , O2 ) ∈ C and F 2 = jump ( agent ) ∈ G , and we assume that constants for objects are { obj1 , obj2 } . R 0 has existentially quantified variables O1 and O2 on the body, so we obtain ground rules by substituting constants. By considering the possible substitutions for O1 and O2 , namely { O1 / obj1 , O2 / obj2 } and { O1 / obj2 , O2 / obj1 } , we have two ground rules, as shown in top of Table 4. Bottom rows of Table 4 shows elements of tensor I 0 , : , 0 , : and I 0 , : , 1 , : . Facts G and the indices are represented on the upper rows in the table. For example, I 0 , 2 , 0 , : = [3 , 6 , 7] because R 0 entails jump ( agent ) with the first ( k = 0 ) substitution τ = { O1 / obj1 , O2 / obj2 } . Then the subgoal atoms are { type ( obj1 , agent ) , type ( obj2 , enemy ) , closeby ( obj1 , obj2 ) } , which have indices [3 , 6 , 7] , respectively. The atoms which have a different predicate, e.g., closeby ( obj1 , obj2 ) , will never be entailed by clause R 0 . Therefore, the corresponding values are filled with 0 , which represents the index of the false atom.

(Step 2) Assign Rule Weights. We assign weights to compose the policy with several action rules as follows: (i) We fix the target programs' size as M , i.e. , where we try to find a policy with M action rules. (ii) We introduce C -dim weights W = [ w 1 , . . . , w M ] . (iii) We take the softmax of each weight vector w j ∈ W and softly choose M action rules out of C action rules to compose the policy.

(Step 3) Perform Differentiable Inference. We compute 1 -step forward reasoning using weighted action rules, then we recursively perform reasoning to compute T -step reasoning.

( k = 1) jump ( agent ) :-type ( obj2 , agent ) , type ( obj1 , enemy ) , closeby ( obj2 , obj1 ) .

[(i) Reasoning using an action rule] First, for each action rule C i ∈ C , we evaluate body atoms for different grounding of C i by computing b ( t ) i,j,k ∈ [0 , 1] :

where gather : [0 , 1] G × N G × S × L → [0 , 1] G × S × L is:

The gather function replaces the indices of the body state atoms by the current valuation values in v ( t ) . To take logical and across the subgoals in the body, we take the product across valuations. b ( t ) i,j,k represents the valuation of body atoms for i -th rule using k -th substitution for the existentially quantified variables to deduce j -th fact at time t .

Now we take logical or softly to combine all of the different grounding for C i by computing c ( t ) i,j ∈ [0 , 1] :

where softor γ is a smooth logical or function:

where γ > 0 is a smooth parameter. Eq. 11 is an approximation of the max function over probabilistic values based on the log-sum-exp approach [Cuturi and Blondel, 2017].

[(ii) Combine results from different action rules] Now we apply different action rules using the assigned weights by computing h ( t ) j,m ∈ [0 , 1] :

where w ∗ m,i = exp( w m,i ) / ∑ i ′ exp( w m,i ′ ) , and w m,i = w m [ i ] . Note that w ∗ m,i is interpreted as a probability that action rule C i ∈ C is the m -th component of the policy. Now we complete the 1 -step forward reasoning by combining the results from different weights:

Taking softor γ means that we compose the policy using M softly chosen action rules out of C candidates of rules.

[(iii) Multi-step reasoning] We perform T -step forward reasoning by computing r ( t ) j recursively for T times: v ( t +1) j = softor γ ( r ( t ) j , v ( t ) j ) . Finally, we compute v ( T ) ∈ [0 , 1] G and returns v A ∈ [0 , 1] G A by extracting only output for action atoms from v ( T ) . The whole reasoning computation Eq. 8-13 can be implemented using only efficient tensor operations. See App. E.2 for a detailed description.

E.2 Implementation Details

Here we provide implementational details of the differentiable forward reasoning. The whole reasoning computations in NUDGE can be implemented as a neural network that performs forward reasoning and can efficiently process a batch of examples in parallel on GPUs, which is a non-trivial function of logical reasoners.

Each clause C i ∈ C is compiled into a differentiable function that performs forward reasoning using the tensor. The clause function is computed as:

where gather 1 ( X , Y ) i,j,k,l = X i, Y i,j,k,l ,k,l 4 obtains valuations for body atoms of the clause C i from the valuation tensor and the index tensor. prod 2 returns the product along dimension 2 , i.e. the product of valuations of body atoms for each grounding of C i . The softor γ function is applied along dimension 3, on all the grounding (or possible substitutions) of C i .

softor γ d is a function for taking logical or softly along dimension d :

where γ > 0 is a smoothing parameter, sum d is the sum function along dimension d . The results from each clause C t i ∈ R B × G is stacked into tensor C ( t ) ∈ R C × B × G .

Finally, the T -time step inference is computed by amalgamating the inference results recursively. We take the softmax of the clause weights, W ∈ R M × C , and softly choose M clauses out of C clauses to compose the logic program:

where softmax 1 is a softmax function over dimension 1 . The clause weights W ∗ ∈ R M × C and the output of the clause function C ( t ) ∈ R C × B × G are expanded (via copy) to the same shape ˜ W ∗ , ˜ C ( t ) ∈ R M × C × B × G . The tensor H ( t ) ∈ R M × B × G is computes as

where ⊙ is element-wise multiplication. Each value H ( t ) i,j,k represents the weight of k -th ground atom using i -th clause weights for the j -th example in the batch. Finally, we compute tensor R ( t ) ∈ R B × G corresponding to the fact that logic program is a set of clauses:

With r the 1-step forward-chaining reasoning function:

we compute the T -step reasoning using:

where I ∈ N C × G × S × L is a precomputed index tensor, and W ∈ R M × C is clause weights. After T -step reasoning, the probabilities over action atoms G A are extracted from V ( T ) as V A ∈ [0 , 1] B × G A .

F Figures for Ablation Study

Here we provide figures we used for our ablation study.

Loot-hard

12

ppo

0

10

8

100

101

102

103

Steps

Gradientsofactionsw.r.t.inputatoms

Distributionoveractionatoms

jump

.54

.53

0.5

left_key

0.4

0.3

right_key

0.2

left_door

26

9Z

0.1

right_door

.05

.06

.06

0.0

_key(agent)

closeby(obj1,obj4)

have_k

on

on

type

typ

Gradientsofactionsw.r.t.inputatoms

Distribution over action atoms

jump

.51

.07

.07

0.16

0.14

left_key

.13.15

.14.15

0.12

0.10

right_key

.01

.04.01

.06

0.08

0.06

left_door-

.01

.01

.01

0.04

right_door

0.02

0.00

,obj

type(obj4,key)

closeby(obj1,obj4)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles interpretable RL with a neurosymbolic method (NUDGE) separating perception and action prediction. Given a pre-trained actor-critic model and a pre-trained/given perception module, NUDGE checks randomly generated rules and assess their quality by comparing them to the pre-trained actor. It then performs rule weight learning using an A2C-like algorithm using the pre-trained critic.\n\nSTRENGTHS:\nI like the ideas, and the paper offers an excellent combination of methods with clear benefits in terms of explainability and possibly generalisation (although I think the verdict is out on this). The paper is well-written, easy to understand and complete. \nThe explicit parameter $k$ allows limiting the complexity of policies, generating interpretable policies as highlighted in Figure 4.\n\nWEAKNESSES:\nThe experimental setup varies significantly throughout the different settings, leaving me with questions about the experimental methodology.\n\nQUESTIONS:\n- It is not immediately clear what the differences between the Differentiable Forward Reasoning module are with AlphaILP.\n- The object-centric states are similar to concept-based models (for instance, [2], which also performs rule learning). It might be worth discussing. \n- 3.2.1: NUDGE never removes state atoms?\n- 3.2.1: If $\\pi_\\theta$ rarely picks action $A$ in states $X$, is the eval rule still reliable?\n- Experiments:\n\t- If all networks get object-centric descriptions, how are those in $[0, 1]$? Or are all these binary?\n\t- What is the actor-critic/PPO model you use to train NUDGE?\n\t- What is the motivation behind adding basic rules? It might be worth highlighting this is an option to improve performance or mention that it does not matter.\n\t- Table 1 Left: Why not compare to PPO? Why did you use expert supervision here? That means you did not test your actual method. \n\t- Figure 3: The comparison by steps is somewhat unfair because NUDGE uses a pretrained model at initialisation. Why not train all models until convergence?\n\t- The conclusion that NUDGE adapts to environmental changes is misleading: Rather, it is easy to adapt by changing the logic, but NUDGE is not adaptive itself.\n\t- The explainability claims are only tested with a single example. It is unclear why NUDGE would be more explainable with LIME-style diagnostics than a neural network trained on object-centric states.\n- Table 2 claims NUDGE is FOL, but it is weaker: It does not support infinite domains or function symbols\n\nMinor comments:\n- Make sure to use \\citep consistently\n- The statement that humans use symbolic languages to compose their reasoning is something I'd like to see a citation for\n- Throughout the paper, you refer to the rules and their weights as 'probabilistic'. However, $\\alpha$ILP is not a probabilistic logic (in fact, the softOr is not fuzzy either, as it is not a t-conorm [3]). The invalid use of the word probability is confusing.\n- Typos:\n\t- 123 reads wrong to me\n\t- 188: a given neural policy\n\t- Equation 3 seems invalid. What does the = mean between the $\\prod$ and $\\mathbf{v}$?\n\t- Figure 3: Is the largest $k$ 5 or 10? The legend and caption don't agree.\n\t- 245: designed to\n\t- 255: Agents can 'open' keys?\n\t- 298: opening quotes\n\n[1]: Hikaru Shindo, Viktor Pfanschilling, Devendra Singh Dhami, Kristian Kersting: alphaILP: Thinking Visual Scenes as Differentiable Logic Programs.  Machine Learning (Special Issue on Learning and Reasoning 2022), vol. 112, pp. 1465–1497, 2023\n\n[2] Barbiero, P., Ciravegna, G., Giannini, F., Zarlenga, M. E., Magister, L. C., Tonda, A., ... & Marra, G. (2023). Interpretable Neural-Symbolic Concept Reasoning. arXiv preprint arXiv:2304.14068.\n\n[3] van Krieken, E., Acar, E., & van Harmelen, F. (2022). Analyzing differentiable fuzzy logic operators. Artificial Intelligence, 302, 103602.\n\nLIMITATIONS:\nThe method requires a (pretrained / hand-designed) perception module that can be hard to come by. The paper does not discuss this limitation thoroughly. Otherwise, limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents NUDGE, an RL technique that combines neural network and logic program learning to learn policies that are interpretable and explainable by humans. NUDGE produces agents that match or exceed the performance of agents trained using standard RL techniques like PPO.\n\nSTRENGTHS:\nThe paper is well-written, clear, and addresses the significant challenge of uninterpretable reinforcement learning policies. The paper proposes to solve this by learning policies that are interpretable and explainable by design. \n\nI am unqualified to assess the originality of this paper because I am unfamiliar with the logic program learning literature.\n\nWEAKNESSES:\nThis work can be improved with stronger definitions of interpretability and explainability. Currently, it’s unclear what criteria make a model interpretable or uninterpretable (line 43). What exactly makes a policy readable by humans? Is this person an expert or a layperson? I suspect that interpretability here means that an expert can assign meanings to the policy. Similarly, what constitutes an explanation (line 44)? From their definition, it seems that the explanation need not be causal? Do all methods from explainable AI count?\n\nThere are also existing definitions of both terms from the community. I would like to see the authors engage with the existing literature on definitions of interpretability and explainability. See Lipton 2016 for a discussion on interpretability. I think that having set definitions is important for this work in order to avoid moving goalposts.\n\nLine 141: Another weakness is that the relational perception module maps all entities to a particular index in the valuation vector. This method does not seem like it would scale to new objects. Similarly, the identity of entities seem to be fixed given a particular validation vector. It seems like the parallel version of this weakness (parallel in the sense of scaling issues) for the forward reasoning module is solved by neurally guided symbolic abstraction (line 193).\n\nI also found some parts on what parameters are and aren't learned to be confusing. See questions.\n\nQUESTIONS:\n- Line 3: What do you mean by “articulate the thinking” in this definition?\n- How do you envision this work translating to real world settings? This work makes a lot of sense in the rule-based testbed, where much of the non-rule-based information is extraneous. Can naturalistic inputs be represented in this way? I could see an argument for this work as automating the pipeline approaches in vision-based control settings. \n- Line 142: I’m curious whether there are analogues in cognitive science for these modules you’ve created.\n- Line 143: Is $\\Theta$ learned? \n- Line 204: What is $\\theta$? I don’t see $\\theta$ anywhere else.\n\nNits:\n- Line 1: “dominating” —> “dominant?\n\nLIMITATIONS:\nThe authors have adequately addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to learn more interpretable and explainable logical policies. It modifies NLRL, Neural Logical RL, with (1) an extra neural-guided heuristic, i.e., alignments with a pretrained neural policy, for filtering logical rules during search; (2) changing policy learning algorithm to actor-critic from policy gradients; (3) pre-defining / hardcoding the meanings of symbols for more interpretability; and (4) explanation of policies according to the gradients. It conducted experiments in five simple games.\n\nSTRENGTHS:\nThis paper uses pre-trained policy networks as neural guidance for faster search of logical rules (filtering according to their alignments). This could be an interesting direction to explore. However, experiments that evaluate/support its effectiveness are not adequate in the paper.\n\nWEAKNESSES:\nThis paper claims that their methods improve the interpretability and explainability of logical policy learning. However, as far as I understand, this is not true because \n* The proposed methods (compared with existing ones) are mainly about optimization and logical rule mining, instead of about interpretability and explainability.\n* Policies become more interpretable only because they change the problem setting to an easier and more interpretable one. They assume predefined/hardcoded meanings of all the predicates/symbols (e.g., `close_by(obj1, obj2)`), as well as a pretrained neural module to detect the objects and those meaningful attributes. \n  - Previous methods do not assume those and instead, jointly learn the symbols and their meanings with logical rules. \n  - Any model with access to those predefined meanings and those pretrained detection modules is more interpretable and explainable. For example, when given access to those symbolic inputs (i.e., memory states), even DQN can adapt to environmental changes (as defined in Q2, Sec. 4) and would be explainable by performing the gradient analysis. \n\nFrom the optimization perspective, the new problem setting is less interesting and challenging as well. With access to symbolic inputs (e.g., the memory states of Atari games), classic ILP methods may even work, especially when considering the relatively small search space in the five simple games. The authors should compare with those baselines as well. \n\nThere are other minor concerns such as \n* If the neural policies are much worse than the symbolic ones as reported, why and how would it be helpful to distill/use them as neural guidance?\n* The games are relatively simple and easy. \n* There should be experiments analyzing the effects of actor critics v.s. policy gradients.\n\nQUESTIONS:\n* Why are the policies more interpretable and explainable? How can the proposed methods contribute to them? \n* Are there results of classic symbolic methods? Can NUDGE perform better than them? \n* Are there results on more complex and difficult games?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces NUDGE, a framework for interpretable and explainable policy reasoning and learning in reinforcement learning. NUDGE employs differentiable forward reasoning during inference to derive a set of weighted rules that form an interpretable policy. During training, NUDGE leverages neurally-guided symbolic abstraction to efficiently distill rule-based symbolic representations from neural-based policies. It then employs gradient-based policy optimization using actor-critic methods to learn the weights of these rules. Empirical results demonstrate that NUDGE achieves comparable performance to neural-based policies while providing interpretable and explainable logical representations. Furthermore, the rule-based symbolic policies in NUDGE exhibit adaptability and flexibility in addressing environmental changes. Overall, NUDGE offers a promising approach to achieving interpretable and adaptive policies in reinforcement learning.\n\nSTRENGTHS:\n+ NUDGE introduces end-to-end reasoning architectures based on differentiable forward reasoning, allowing for comprehensive policy generation directly from raw input. \n+ Its neurally guided symbolic abstraction leverages a pre-trained neural policy to induce a set of candidate action rules, which are subsequently optimized using the critic component of an actor-critic agent. \n+ Notably, NUDGE policies are differentiable, enabling the utilization of action gradients with respect to input atoms. This unique feature facilitates the explanation of state features that influence the decision-making process, shedding light on how specific actions are chosen in a given state. \n\nThese aspects collectively contribute to NUDGE's interpretability and explainability, providing insights into the underlying mechanisms of the learned policies.\n\nWEAKNESSES:\nIt appears that the individual components of NUDGE are not inherently novel on their own. The paper's main contribution lies in the integration of existing components to form the NUDGE framework. By bringing together existing components such as differentiable forward reasoning, neurally guided symbolic abstraction, and policy optimization, NUDGE creates a framework for interpretable and explainable policy learning. Although the individual components are not novel on their own, the paper would have been acceptable if it can still demonstrate the novel insights that arise from their integration. However, the novelty of NUDGE in comparison to prior work such as PIRL [Verma et al., 2018] appears to be very limited. Both NUDGE and PIRL employ a pretrained neural policy as a guide for learning policies. Additionally, both NUDGE and PIRL utilize random search to update policy structures and employ optimization techniques to learn policy parameters. The primary distinction lies in the fact that NUDGE learns programmatic policies within logical programs, whereas PIRL learns policies within functional programs. While NUDGE can learn from raw image inputs, this aspect is not considered a core contribution, as it relies on off-the-shelf tools for object extraction from visual inputs.\n\nQUESTIONS:\nThere is confusion regarding the paper's claim on policy explainability in relation to NUDGE. The approach used for explainability appears to be fairly standard and not inherently tied to differentiable logical programs. In fact, it can be applied to explain a neural network as well e.g. annotating regions of an image that mostly influence policy output. Therefore, it remains unclear how NUDGE specifically enhances policy explainability compared to existing methods. Could the authors elaborate on that?\n\nLIMITATIONS:\nAlthough the paper is well executed and integrates existing components effectively, it does not present advancements that significantly differentiate it from prior work PIRL [Verma et al., 2018].", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose neurally guided differentiable logic policies (NUDGE), which learns differentiable logical policies that are represented by interpretable rules. It is capable of producing explanations for their decisions in complex environments. NUDGE utilizes neurally guided symbolic abstraction to boost the model's performance. Specifically, the candidate rules for the logic-based agents are obtained efficiently by being guided by neural-based agents. Experimental results show that NUDGE can achieve competitive performance compared with the neural baselines. Besides, the NUDGE agent can adapt to environmental changes and its learned policies are interpretable and explainable.\n\nSTRENGTHS:\n1. The authors propose using a novel mechanism called neurally guided symbolic abstraction, to boost the neural logic model's performance.\n2. The writing of the paper is well-organized and clear, and the evaluations are quite comprehensive.\n\nWEAKNESSES:\nIt is unclear how are the abstract head atoms being interpreted, for example, **right_go_to_door** used in the GetOut task and **right_to_eat** in the Fishes tasks. Are they automatically learned or interpreted by developers manually?\n\nQUESTIONS:\n1. What's the input representation for the neural baselines, namely, DQN agents and PPO agents? Is it the same with the neural logic methods?\n2. Wonder how does NUDGE perform under more logically challenging environments, for example, grid environments which are utilized in previous work [1][2].\n3. How are the head atoms interpreted, for example, **right_go_to_door** used in the GetOut task and **right_to_eat** in the Fishes tasks. Are they automatically learned or interpreted by developers manually?\n\n[1] Cao, Yushi, et al. \"GALOIS: boosting deep reinforcement learning via generalizable logic synthesis.\" Advances in Neural Information Processing Systems 35 (2022): 19930-19943.\n[2] Yichen Yang, Jeevana Priya Inala, Osbert Bastani, Yewen Pu, Armando Solar-Lezama, and Martin Rinard.\nProgram synthesis guided reinforcement learning for partially observed environments. Advances in Neural\nInformation Processing Systems, 34, 2021.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}