BlendRL: A Framework for Merging Symbolic and Neural Policy Learning
Authors: Hikaru Shindo
Quentin Delfosse
Devendra Singh Dhami
Kristian Kersting
Venue: ICLR 2025 Spotlight (2025)
Keywords: Neuro-Symbolic AI
Differentiable Reasoning
Reinforcement Learning
Interpretable AI
First-order logic
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: We propose a framework that jointly learns symbolic and neural policies for reinforcement learning.

Abstract: Humans can leverage both symbolic reasoning and intuitive responses. In contrast, reinforcement learning policies are typically encoded in either opaque systems like neural networks or symbolic systems that rely on predefined symbols and rules. This disjointed approach severely limits the agents’ capabilities, as they often lack either the flexible low-level reaction characteristic of neural agents or the interpretable reasoning of symbolic agents. 

To overcome this challenge, we introduce *BlendRL*, a neuro-symbolic RL framework that harmoniously integrates both paradigms. 
We empirically demonstrate that BlendRL agents outperform both neural and symbolic baselines in standard Atari environments, and showcase their robustness to environmental changes. Additionally, we analyze the interaction between neural and symbolic policies, illustrating how their hybrid use helps agents overcome each other's limitations.

Published as conference paper ICLR 2025
BLENDRL: A FRAMEWORK FOR MERGING SYMBOLIC AND NEURAL POLICY LEARNING
Hikaru Shindo' Quentin Delfosse' Devendra Singh Dhami Kristian Kersting 133,4,
Department of Computer Science, Technical University of Darmstadt, Germany
Deptt of Mathematics and Computer Science, Eindhoven University of Technology, Netherlands
Hessian Center for Artificial Intelligence (hessian.AD), Germany
GGermnn Research Center for Artificial Intelligence (DFKI), Germany
"Centre for Cognitive Science, Technical University of Darmstadt, Germany
(hi kark..shinna quent n del foase, kerst ng] ut www.aadd.taaa...........dd.....d de, d.hami @tue.
ABSTRACT
Humans can leverage both abstract reasoning and intuitive reactions. In contrast, rein- forcement learning policies typically encoded in either opaque systems like neural networks or symbolic systems that rely on predefined symbols and rules. This disjointed approach severely limits the agents' capabilities, they often lack either the flexible low- level reaction characteristic of neural agents or the interpretable reasoning of symbolic agents. To overcome this challenge, work that harmoniously integrates both paradigms within RL agents that use mixtures of both logic and neural policies. We empirically demonstrate that BlendRL agents outper- form both neural and symbolic baselines in standard Atari environments, and showcase their robustness tween neural and symbolic policies, illustrating how their hybrid use helps agents over- come each other's limitations.
introduce BlendRL, neuro-symbolic RL frame-
environmental changes. Additionally, we analyze the interaction be-
INTRODUCTION
Reach the jony
Gel oxygen and divers
To solve complex problems, humans employ two fundamental modes of thinking: (1) instinctive responses for immediate reac- tion and motor control and (2) abstract reasoning, using distinct identifiable concepts. These two facets of human intelligence are commonly referred as System and System (Kahneman) (001)) While our reasoning system requires symbols to build interpretable decision rules, instinctive reactions do rely on such inductive bias, but lack transparency. Both systems are required in challeng- ing RL environments, such as Kangaroo, where the agent's goal to reach its joey (at the top), captured by monkeys that need to be punched out of the way, or Seaquest, where the agent controls a submarine, and needs to collect swimming divers without running out oxygen (ef. Figure []. Developing agents that can effectively use both information processing systems has proven to be persis- tent challenge (Lake et al.] 2017 Mao et al. 2019 Kautz (002). The main difficulty not only lies in achieving high- level capabilities with both systems, but also in seamlessly integrating these systems interact synergistically, in order to maximize performances without sacrificing transparency.
High vea Auuoome
Punch the enemy
Shoot the enemy
Low level reaction
Figure 1: Most problems require both reasoning (top) and reacting (bottom), to master egg. the Kangaroo (left) and Seaquest (right) Atari games.

---

Published as a conference paper at ICLR 2025
Deep neural networks have demonstrated an ability to effectively learn policies across wide range of tasks without relying on any prior knowledge about the task (Mnih et al. 2015 Schulman et al.)
2017 Badia et al. 2020 Bhatt et al., 2024). However, these black box policies can leverage shortcuts that are imperceptible to human observers (Locatello et al. [020; Liu & Borisyuk] (2024). For instance, in the simple Atari Pong game, deep agents tend to rely on the enemy's position rather than the ball's one (Delfosse et al. (024d), demonstrating deep learning systems' tendency to rely on shortcut learning opportunities, that then fail to generalize to slightly modified environments.
To enhance reasoning capabilities. symbolic reasoning has been integrated into approaches, using e.g. as logic-based policies Viang & Luo| 2019, Kimura et al., 202T; Cao et al. 2022 Delfosse et al. 2023a) or program-based -ased frameworks (Sun al. [2200] Verma et al. 2018 Lyu et al. 2019: Cappart et al.) 202T Kohler et al.] (024). These methods offer transparency, revisability, better generalization, and the potential for curriculum learning. However, they often rely on the incorporation of specific human inductive biases, necessary for solving the tasks, thus requiring experts to provide essential concepts or potential logic rules. Moreover, actions involving low-level reactions with subtle movements are challenging, if not impossible, to encode within these frameworks. This limitation underscores the constraints of symbolic systems in terms of their learning capabilities. Consequently, an important question emerges: How can we build intelligent agents that leverage the strengths of both neural and symbolic modeling? The current approach to combining these systems typically uses a top-down (i.e. sequential) method: using deliberative systems (e.g. planners) to provide high-level reasoning to select reactive systems (e.g. Deep RL), that offer quick, low-level re- sponses (Kokel et al. (021). However, this top-down approach is not always suitable, e.g. a self-driving can recompute its plan on low-density highway, but must react quickly in heavy traffic, without replanning. Agents that can select for either neural or symbolic modeling based on the context are necessary.
We introduce BlendRL, framework that integrates neural and logic-based policy learning in parallel. BlendRL agents learn logic-based interpretableee reasoning as well as low-level control, combined through a blending function that utilizes hybrid state representation. They can utilize high-level reasoning (e.g. for path finding), which benefits from a symbolic (or object-centric) state representation, and low-level reac- tions, for finetuned control skills (e.g. shooting at enemies), using pixel-based state representations. Al- though its neural component is less interpretable, it assists the agent in adapting to situations where it lacks sufficient symbolic representations. BlendRL provides hybrid policies, by distinctly modeling these two types of information processing systems, and selects its actions using a mixture of deep neural networks and differentiable logic reasoners (Evans & Grefenstette 2018, Shindo et al., 2023, 2024b). Additionally, we propose an Advantage Actor-Critee Critic (A2C)-based learning algorithm for BlendRL. agents, that incorporates Proximal Policy Optimization (PPO) and policy regularization, supplemented by analysis on the interactions between neural and symbolic components of trained agents. Overall, we make the following contributions:
(i) We propose BlendRL. to jointly and simultaneously train symbolic and neural policies.
(ii) For efficient learning on the proposed framework, we adapt the PPO Actor Critic algorithm on the hybrid state representation. Moreover, we proposed a regularization method to balance neural and symbolic policies, providing agents that are both transparent reasoners and accurate reactors.
(iii) We empirically show that BlendRL agents outperform neural and the vate-of-the-an neuro- symbolyc baselines on environments where agents need to perform both high-level reasoning and low-level react- ing, Moreover, we demonstrate the robustness of the BlendRL agents to environmental changes.
(iv) We provide a deeper analysis of the interactions between neural and symbolic policies, revealing how hybrid representations and policies can help agents overcome each other's limitations.
We start off by providing the necessary background, then introduce our BlendRL method for policy rea- soning and learning. We experimentally evaluate BlendRL on three complex Atari games, comparing its performance to purely neural and logic baselines, Following this, we discuss related work before conclud- ing. Our code and resources are openly available
htttps://gitti www.m//ml-reeearrc/bbeeeed
2

---

Published as a conference paper ICLR 2025
2 BACKGROUND
Let us introduce the necessary background before formally introducing our BlendRL method.
Deep Reinforcement Learning. In RL, the task is modelled as a Markov decision process, M =< A, P.R. 7>, where, at every timestep t, agent in state 8 E S, takes action a E.A, receives reward Y: R(st, ae) and a transition to the next state $1+1, according to environment dynamics P( a.)) Deep agents attempt to learn parametric policy, To(ay]ss), in order to maximize the return (i.e. E.rrr, with E [0. 1). The desired input to output (i.e. state to action) distribution is not directly accessible, as RL agents only observe returns. The value VA, (8x) (resp. Q-value QT (t)) function provides the return of the state (resp. state/action pair) following the policy Te. We provide more details in App. A.13
tuple (P,D, V). where P is a set of predicates, D set of constants, F a set of function symbols (functors), and V a set of variables. A term is either a constant (e.g. obj1, agent), a variable (e.g. 01), or a term which consists of a function symbo { An atom a formula p(t,...., ta), where is predicate symbol (e.g. closeby) and ti..... are terms. A ground atom or simply a fact an atom with no variables (e.g. closeby(obj) (1b)),,ob,,,))))) obj2)). A literal an atom (A) or its negation (-A). A clause is a finite disjunction (V) of literals. A ground clause is a clause with no variables. A definite clause is a clause with exactly one positive literal. A, B1, Bi,... Bn are atoms, then A V -Bi V... VBB is definite clause. We write definite clauses in the form of A :- B1..... Bn. A is the rule head, and the set {B...... B is its body. We interchangeably use definite clauses and rules.
First-Order Logic (FOL). In FOL, a Language
Differentiable Forward Reasoning is a data-driven approach to reasoning in First-Order Logic (FOL) Ruus- sell & Norvig 2010). In forward reasoning, given a set of facts and rules, new facts are deduced by applying the rules to the facts. Differentiable forward reasoning is differentiable implementation of forward rea- soning. utilizing tensor-based differentiable operations (Evans & Grefenstette 2018 8 [Shindo et al. 2023) or graph-based approaches (Shindo et al., 2024b). This approach can be efficiently applied to reinforcement learning tasks by encoding actions in the form of rules, where the head defines an action and the body speci- fies its conditions. To learn the importance or truth value of each rule, they can be associated with learnable rule weights. Consequently, hypotheses can be formulated in terms of rules and learned from data.
3 BLENDRL
BlendRL integrates both abstract reasoning and instinctive reactions by combining symbolic and neural poli- cies. As illustrated in Figure 2 the neural policy processes sub-symbolic (i.e. pixel-based) representations compute distribution over the actions, while the reasoning module employs differentiable reasoning on symbolic states. These action distributions are then blended to obtain the final action distribution. Let us point out that the logic reasoner does not use planning. BlendRL produces model free RL agents. We begin by describing the inner workings of each policy type, and of the blending module. Next, we discuss how we leverage the common sense encapsulated in pretrained large language models (LLMs) to obtain symbolic concepts and their evaluation functions. Finally, we describe how we adapted the PPO actor-critic algorithm to perform end-to-end training of BlendRL modules. Let us first formally introduce the state representations.
3.1 HYBRID STATE REPRESENTATIONS
BlendRL agents employ two distinct state representations: (i) pixel-based representations and (ii) object- centric representations, that can be extracted by object discovery models (Redmon et al. 2016; Lin al.) 2020 Delfosse et al. 2023b Zhao et al. 2023). Pixel-based representations usually consist stack of
In experiments, opted to rules without function symbols they were not necessary (cf. Fig However, BlendRL agents capable of handling them. (cf. App.A. A.1.3
3

---

Published as conference paper ICLR 2025
Symbolic state
Logic policy " logne Reasoning module
Logic distribution
ooj
geoD M
48
16
60
-0.2
46
73
0
Reacting module
-0.8 Blending module B
Final action distribution
BHTH
Raw pixel state æ
Neural policy To
nenanal
Neural distribution
LLM Assisted Logic Rule Generation
Figure 2: Overview of the BlendRL. framework. BlendRL employs two policy types compute action probabilities. A deep neural policy (bottom) handles low-level reactions, operating on pixel-based states, while differentiable forward reasoners (Shindo et al.) (023) (top) manage high-level reasoning symbolic, object-centric states extracted from the pixel-based input. The blending module then merges the two action distributions by taking weighted sum based the current state. The logic policy and the blending module incorporate human inductive biases extracted using language models (LLMs) and the task context. All three components (the two policies and the blending module) can be trained jointly using gradients.
raw images, provided by the environment, fed to a deep convolutional network, as introduced by Mnih et al. (2015). Our considered symbolic (object-centric) representations consist list of objects, with attributes (e.g. position, orientation, color, etc.), allowing for logical reasoning on structured representations (Zada-
ianchuk al. 2021 Liu al. 2021 Yoon al. 2023 Wüst et 2024 Stammer et al. 2024).
Raw sub-symbolit states, E FxXxXHC consist of the last F observed frames, while symbolic states z E IRxx represents the corresponding symbolic (or object-centric) state, with n denoting the number of objects and of m extracted properties. As illustrated in Figure [ on the Seaquest Atari environment, symbolic state consists of objects such as the agent's submarine, the oxygen level, a diver, sharks, etc., with their and positional coordinates, well as orientation and value. If a property is not relevant to object (e.g. the orientation for the oxygen bar), is set to O.
3.2 THE BLENDED -OURO-SYMROLIC POLICIES
Using both state representations, BlendRL agents compute the action selection probabilities by aggregating the probabilities of its neural and its logic policies.
The neural policy, TT neeural FFx + [0, 1]4, consists of neural network, parameterized by 0, that computes action distributions based on pixel state representations (x). Convolutional neural networks based
policies MMiih al. 2015) Schulman et al. 2017 Hessel et al. 2018 are the most common deep policy
types (Huang et al. 2022). but visual transformers can also be used (Chen et al. 2021 Parisotto et al.
2020
The logic policy, L.logic R -> [0, 1]4, is differentiable forward reasoner (Shindo et al.
2023
parameterized by Φ, thht reasons over object-centric states, such as the one depicted in Figure rules are rules in first-order logic that encode conditions for an action be selected (Reiter) 2001 Delfosse
Action
et al. 2023a). An action rule defines action as its head atom (action atom) and encodes its preconditions in its body atoms (state atoms). We provide exemplary action rules for Seaquest in Listing
4

---

Published as a conference paper ICLR 2025
If enemy coconut is close to the agent, use the neural policy, but if nothing around the agent, logic.
W1
logic (X) :- sothing arcund arcund(Agen))
neural (X) PlosebyTagens Enemy)
W3
neural (X) :-
closeby (Dgen.. Coconul)
y
20
-1
0
20
8
-1
0
g
24
8
0
30
46
46
73
0
Figure 3: LLMs allow for generating transparent blending function, Blending exemplified Kanga- roo. Top: An LLM generates a transparent blending function that evaluates if (i) there exists an enemy close to the player and if (ii) nothing is around the player. Bottom: Two states are evaluated: the body predicates (i.e. nothing.ss werthingaaouur hnguaroudd and closeby) values are multiplied with the rules' associated learnable weight (displayed on their left), obtain the neural/logic importance weighting.
These rules are transparent, egg. [R1] can be inter-
[Ri] ] 0.73: up (X): is __eeptyyyyyo (Oxygen).
preted as "Select UP if the oxygen is empty". The body atom is Lmeepyyy is state predicate, whose
[R2] 0.42: up (X) :-above (Diver, (Diver,Agent)
[R3] 0.31: left (X) :-leftoor (Di (Diver, Agent).
truth level can be computed from an object-centric state. Each state predicate is associated with (differentiable) function, known as valuation function, to compute its truth value, or confidence. For example, is empty can be mapped to a function, such as sigmoid((* a)/r), which translates the actual oxygen value E [0, 100] (from the object-centric state) into truth score ranging from [0, 1]. The second rule [R2] represents the same action selection, UP, but is motivated by the aim of collecting divers. The third rule [R3] selects another action (LEFT), if diver is left of the player. After evaluat- ing the valuation functions for each state predicate, we perform differentiable forward reasoning (Shindo) et al. 2023) to deduce action atoms defined by the action rules based on the state atoms. Forward reasoning involves inferring all deducible knowledge (i.e. the head atoms for actions) from an observed state. This process allows obtain confidence levels (as probabilities) for all actions defined in the symbolic policy.
Listing I: Exemplary action rules for Seaquest.
Contrary to NUDGE policies (Delfosse et al., 2023a). BlendRL generates action rules and their necessary elements (predicates and their valuation functions) using Large Language Model (LLM), described in Section 3.3 We also integrate memory-efficient overcome potential memory bottlenecks associated with conventional symbolic policies. NUDGE and other common logic policies consumes memory quadratically with respect to the number of relations and entities, significantly limiting their scalability. In contrast, BlendRL's symbolic policy scales linearly, suitable more complex environments scalable training parallelization.
pesssae-passsing forward reasoners (Shindo et al.) (022b)) to
The blending module is differentiable function parameterized by It can be encoded as an explicit logic- based function (B) IR Xx xm + [0. 1)). as an implicit neural network-based (i.e. pixel) state evaluator (B, Fx x H -> [o, 1)), or a combination of both. While a logic-based policy provides the advantage of transparency, it relies inductive biases. If these are not available, a neural blender is necessary.
FigureB3dserribee the overall process of the blending module. It computes distributions over neural and logic policies given symbolic states, based on the blending rules generated by LLMs. The blending weighted rule set of an agent trained on Kangaroo depicted in the top right corner. It encodes the fact that the neural deadly thrown coconut is around (to allow for dodging
module should be selected when monkey the coconut or adjusting the position to optimally punch the monkey). When nothing is around the agent, it safely rely its logic policy (depicted just above it), which allows it to navigate the Kangaroo environment. Formally, the blending module provides the weight 3 E [0, 1] of the neural policy, 3 Bx or = (x)x), (and thus implicitly of the neural policy). We empirically show Kangaroo and
5

---

Published as conference paper ICLR 2025
$ Policy ruleset (Kangaroo)
0.73 up (X) on on_ladder (Player, Ladder) same_flood (Playyr,aa Ladder)
0.74 right (X) :- (Player, Ladder), (adder) same same_flooz floor (Player, Ladder).
0.72 left (X) :-right_ef ( (Player, Ladder), () same_ same_f foor layer, Ladder).
$ Blending ruleset (Kangaroo)
0.78 neural (X) :-c cseby (P layer, Monkey)
0.51 neural (x) :closeeby (Player, Coconat).
0.11 logic :-tttttttinngaroo
(Player).
Figure 4: BlendRI. provides interpretable policies and blending modules as sets of weighted first-order logic rules. A subset the policy's weighted action rules and the blending module's ' rules from a BlendRL agent trained Kangaroo. All the logic rule sets for each environment are provided in Appendix A.5
Seaquest BlendRL agents with symbolic blending modules outperform ones with neural modules (ef. Table Appendix A..
Finally, given a raw state æ, and its symbolic version, 2, the final action distribution is computed as:
(() = 3 و
neural (x) + (1 - 3) logic (z),
where s (x, v). Note that the blending module can also become a rigid selector, if replaced by 1
3.3 LLM GENERATED LOGIC POLICIES
BlendRL employs Language Models (LLMs) to generate symbolic programs for precise reasoning, follow- ing a chain of thought principle (Wei et al. 2022 Kojima et al. 2022):
uses the task context and detectable objects description and implementation create state predicates, (ii) formulates action rules, with conjunctions the generated predicates as body,
(iii) generates the predicates' valuations (i.e. their python implemented functions).
For steps (ii) and (iii), we used the few-shot prompting approach, providing the LLM with example logic ruleset obtained from NUDGE policies (Delfosse et al., 2023a). This circumvents the need for an expert to provide the set of logic rules, the used logic predicates, and their implementations, thus allowing users effectively introduce inductive biases in natural language (cf. Appendix [A.4] and A.5 for additional details), A subset of the logic module of a BlendRL trained on Kangaroo is provided Figure 4 The associated
weights of the LLM-generater rules have been adjusted to maximize performance. GPT4 was the chosen LLM that we consistently used in our experiments.
3.4 OPTIMIZATION
We use the PPO actor-critic to train BlendRL agents, and also employ hybrid value functions. We compose the hybrid crities by computing values using both visual and object-centric states.
Mixed Value Function. As the value function approximates the expected return, we did not use logic to encode it. However, BlendRL incorporates a hybrid value function, that uses both the subsymbolic (æ) and symbolic (or object-centric z) state representations. Given state (z, z), the value is defined as:
(s) B CNN (x) + (1 - -g. (z).
(2)
with CNNN CNN, that share its convolutional layers with the neural actor (Schulman et al. 2017) and " voc small MLP on the object-centric state, and (xx 2), the blending weight.
6

---

Published as a conference paper ICLR 2025
Environments
DQN
PPO
NLRL
NUDGE
SCoBots
INTERP.
INSIGHT
BlendRL
Kangaroo 98.4
26.1
111.2
112.1
149.7
64.4
474,4
Seaquest 61.3
14.8
-3.3
-3.6
52.2
20.8
58.2
94.7
DonkeyKong
3.5
28.4
0.4
1.7
5.8
25.1
48.4
Table 1: BlendRL surpasses deep and symbolic agents in our evaluation. Human normalized scores of BlendRL to different deep (DQN and PPO) and symbolic (NLRL, SCoBots, INTERPRETER and INSIGHT) baselines. Raw results for each agent type, as well as human scores are given in Appendix A.6 A.6
Regularization. To ensure the use of both neural and logic policies, we introduce a regularization term for BlendRL agents. To further enhance exploration, we penalize overly peaked blending distributions. This approach ensures that agents utilize both neural and logic policies without deactivating either entirely.
R -
3 log 3 - 3) log(1 B)
(3)
This helps agents avoid suboptimal policies typically caused by neural policies due to the sparse rewards.
4 EXPERIMENTS
We outline the benefits of BlendRL over purely neural or symbolic approaches, supported by additional in- vestigations into BlendRL's robustness to environmental changes. Furthermore, we examine the interactions between neural and symbolic components and demonstrate that BlendRL can generate faithful explanations.
We specifically aim answer the following research questions:
(Q1) Can BlendeRL. agents overcome both symbolic and neural agents' shortcomings?
(Q2) Can BlendRL. can produce both neural and symbolic explanations for its action selection?
(Q3) Are BlendRL agents robust to environmental changes?
(Q4) How do the neural and symbolic modules interact to maximize BlendRL's agents overall performances?
Let us now provide empirical evidence for BlendRL's 's ability to learn efficient and understandable policies, even without being provided with all the necessary priors to optimally solve the tasks.
4.1 EXPERIMENTAL SETUP
Environments. We evaluate BlendRL on Atari Learning Environments (Bellemare et al. 2013), the most popular benchmark for RL (particularly for relational reasoning tasks). For resource efficiency, we use the object centric extraction module of (Delfosse al., 2024b). Specifically, in Kangaroo, the agent needs reach and climb ladders, leading to the captive joey, while punching incoming monkey antagonists, that try stop it. In Seaquest, the agent has to rescue divers, while shooting sharks and enemy submarines. also needs to surface before the oxygen runs out. Finally, in Donkey Kong the agent has to reach the princess at the top, while avoiding incoming barrels thrown by Donkey Kong. For additional details, cf. Appendix AA. To further test BlendRL. abilities to overcome the potential lack of concept necessary to solve task, we omitted some part of the game in our prompt for the LLM that generates the policy rules. Specifically, we kept out the facts that agents can punch the monkeys in Kangarvo, can shoot the enemies in Seaquest and can jump over the barrels in DonkeyKong. To test robustness and isolate different abilities of the agents, we employ HackAtari (Delfosse et al.) 2024a). that allow to customize the environments (e.g. remove enemies). For these ablation studies, we provide details about the use modification at the relevant point of the manuscript.
Baselines. We compare BlendRL to purely neural PPO and DQN agents. Both agent types incorporate the classic CNN used for Atari environments. Additionally, we evaluate NUDGE, which uses pretrained neural PPO agent for searching viable policy rules (Delfosse et al. 2023a), and NLRL (Jiang & Luo 2019), which
7

---

Published as a conference paper ICLR 2025
the agent
next ladders
above
are on the left of
the agent
diver
the agent
the left of
the next larder
Figure 5: Qualitative examples of explanations produced by BlendRI, in the form of importance maps, computing with gradient-baser attributions for both neural and logic policies. The logic-based module also allows BlendRL to generate textual interpretations, enhancing its transparency and interoperability. Inspired
by (Luo et al. 2024), we generate textual explanations using LLMs, that we provide in App. A.15
generates rules based on templates Evans & Grefenstette (2018). We also provide the reported scores of the following Jsymbolic agents: SCoBots (Delfosse et al.J 2024d), INTERPRETER Kohler et al. (2024) and INSIGHT (Luo et al.) (024)) agents (or extend their evaluation if not available, e.g. in DonkeyKong' We train each agent types until all of them converge to stable episodic return (i.e. for 15K episodes for Kangaroo and DonkeyKong and 25K for Seaquest). For additional details, cf. Appendix
A.8 A.8
4.2 RESULTS AND ANALYSIS
Comparison to neural and neuro-symbolic agents (Q1). Table[]shows the final human normalized scores of each evaluated agent across our selected Atari environments. BlendRL surpasses all the logic-based baselines in all tested scenarios. In the Kangaroo environment, which requires fewer intuitive actions due the relatively small number of enemies, NUDGE shows fair performance, even if its are less able to punch the monkeys and avoid their thrown coconuts. However, in the other environments, populated with more incoming threats, where neural policy becomes critical for accurate controls, NUDGE lags behind. Additionally, the purely neural PPO agents often fall into suboptimal policies. For instance, in Seaquest, surfacing without collecting any divers lead to negative reward. Neural PPO agents thus concentrate on shooting sharks collect reward, but never refill their oxygen. In contrast, BlendRL effectively selects its logic module to collect divers and surface when needed and its neural module to efficiently align itself with the enemies and shoot them. Overall, BlendRL significantly outperforms both baselines across different environments, underscoring the efficacy of neuro-symbolic blending policies in efficiently harnessing both neural and symbolic approaches policy learning.
BlendRL agents are interpretable and explainable (Q2). BlendRL's symbolic policies can easily be in- terpreted as they consist of set of transparent symbolic weighted rules, as exemplified in Figure 4 for Kangaroo. The interpretable blending module prioritizes neural agents in situations where finetuned control is needed (i.e. for dodging an incoming deadly coconut or accurately placing oneself next to the monkey and punching it). Conversely, the logic module is in use when no immediate danger is present, egg. as path finding proxy. The logic rules for the other environments are provided Appendix A.55
BlendRL also produces gradient-based explanations, as each component is differentiable. Logic-based ex- planations computed using action gradients (i.e. Omfogii()) (z)/Oz) the state atoms, that underline the most important object properties for the decision. Further, the integrated gradients method (Sundararajan et al., 2017) on the neural module provides importance maps. We can visualize these two explanations, merging them with the blending weight 3, as shown in Figure The most important logic rules for the decision are also depicted. BlendRI. provides interpretable rules for it reasoning part and importance maps.
BlendRL is robust to environmental changes (Q3). Deep agents usually learn "by heart" spurious correla- tions during training, and thus fail to generalize to unseen environments, even on simple Atari games (Fare-
brother et al. 2018 Delfosse al. 2024a). We used HackAtari variations of the environment, to disable
8

---

Published as a conference paper ICLR 2025
Kangaroo
Seaquest
Seaquest
action confiderce
neural logic
selection ratio
1.0
0.5
0.5
0.5
0.0
0.0
0.0
episodic sleps
spisodic steps
training steps
Figure 6: BlendRL uses both its neural and logic modules. Through an episode, the highest action probability is depicted (left, top), with the selected module at this step (bottom). BlendRL makes use of both module types through an episode. It progressively modifies their selection ratio along training Seaquest (right), as its progress allows it to uncover parts of the game with more enemies.
the enemies in Kangaroo and Seaquest and to disable the barrels in DonkeyKong. We additionally used a modified Kangaroo environment with relocated ladders.
As expected, our BlendRL agent can still complete the tasks (and thus collect reward) on these safer ver- sions of the environments (ef. Table (2). BlendRL.
Return
Kangaroo
Seaquest
Donkey
Neural PPO 0.0
0.0
0.0.
BlendRL 187
113
263
agents indeed rely on the knowledge incorporated in their logic-based policies and blending modules, and only rely on their neural modules for control accurate
Table 2: Contrary to neural PPO, BlendRL is ro- bust to environmental changes. Average returns on the modified environments (over 10 seeds).
skills (e.g. aiming and shooting/punching as we fur- ther show hereafter.
Neural and symbolic policy interactions in BlendRL (Q4). We here investigate how the symbolic and neural policies interoperate. One concern of such hybrid systems is its potential reliance only one module. The system could indeed learn to e.g. only rely on its neural component (L.e. B ~ 1.0). Figure 6 (left and center) illustrates the outputs of BlendRL's neural and logic policies for 1k steps (i.e. 1-3 episodes). Specifically, the maximum value of the action distributions (max, T. Logie (als;) and max, TT apeural (al st)), each time step is depicted (at the top), underlined by the importance weighting of each module (blue logic is mainly selected (3 x 0), red if neural is (3 ~ 1)), indicating how much each module is used at each step on Kangaroo and Seaquest. Further, BlendRL's agents can adapt the amount to which they select each com- ponent through training, as depicted on the right of this figure. As Seaquest is progressive environment, in which the agent first faces few enemies (thus mainly relying on its logic module to collect the divers), before progressively accessing states in which many more enemies are spawning (cf. Figure 12). BlendRL agents first mainly relies on its logic module (blue), while progressively shifting this preference to its neural one (red) for accurately shooting enemies. These results demonstrate the efficacy of BlendRL's policy reasoning and learning on neuro-symbolic hybrid representations, thereby enhancing overall performance. We provide further analysis comparing neural and logic blending modules in Appendix A. highlighting that the logic-based blending module can utilize both policies effectively, resulting in better performance.
Overall, our experimental evaluation has shown BlendRL's agents ability to learn several Atari environ- ments that require both reasoning and reacting capabilities. We showed that they outperform the commonly used neural PPO baseline, well the state-of-the-art logic agents, NUDGE. We further demonstrated their ability to generalize to unseen scenarios, on slightly modified environments (compared to the training ones), and that they can efficiently alternate between the two module types obtain policies that can both produce interpretations, quantifying the impact of each symbolic properties and of each pixel region.
5 RELATED AND FUTURE WORK
Relational Reinforcement Learning (Relational RL) (Dzeroski et al., 2001] Kersting et al. 2004 Kersting
& Driessens 2008 Lang al. 2012 Hazra & Raedt 2023) has been developed to address RL. tasks

---

Published as a conference paper at ICLR 2025
relational domains by incorporating logical representations and probabilistic reasoning. BlendRL extends this approach by integrating differentiable logic programming with deep neural policies. The Neural Logic Reinforcement Learning (NLRL) framework (Jiang Luo 2019) was the first to incorporate Differentiable Inductive Logic Programming (öILP) (Evans & Grefenstette (2018) into the RL domain. ØILP learns gen- eralized logic rules from examples using gradient-based optimization. NUDGE (Delfosse et al.] (023a) extends this method by introducing neurally-guided symbolic abstraction, leveraging extensive work on @ILP (Shindo et al., a to learn complex programs. INSIGHT (Luo et al., 2024) is another neuro- symbolic framework that learns object-centric states and neural policies, distilled in EQL-based policies, then provided to an LLM,, that produces textual explanations about these policies. These approaches aim to represent policies using symbolic representations. In contrast, BlendRL integrates both neural and symbolic policies and leverages both during inference.
The integration of planners with RL has been explored achieve deliberate thinking in policy learning- For example, RePReL. (Kokel et al.) 2021) decomposes multi-agent planning tasks using a planner and then employs reinforcement learning for each agent solve subtasks. In these frameworks, planners for long- term (slow) reasoning are typically separate components. In contrast, BlendRL computes both symbolic and neural policies at the same level, allowing them be learned jointly, thereby enhancing overall performance. Additionally, planners are often used in model-based RL to generate hypothetical experiences that improve value estimates [Sutton|[199) [99T] Kaiser et al. 2019). contrast, BlendRL. incorporates the symbolic reasoning module directly into its policy, enabling joint learning with neural modules. Another notable approach is
to use Linear Temporal Logic (LTL) to instruct RL agents (León et al. 2022 Kuo et al. 2020 Vaezipoor ct al. 2021 Qiu al. 2023), where LTL formulas represent instructions to guide agents and the task is to train (deep) agents to follow them. In contrast, in BlendRL, logic expresses the policy itself and directly influences decision-making To this end, various interpretations of the fast and slow systems have been developed for RI. (Botvinick et al. 2019 Duan et al.] 2017: Tan & Motani 2023 Anthony et al.) 2017) While deep agents focused on object-centric and relational concepts have been explored (Shanahan al., 2020 Feng & Magliacane, 2023). BlendRL uniquely integrates these concepts with hybrid neuro-symbolic
policies, making it applicable to environments that require both abstract reasoning and reactive actions.
Additionally, BlendRL is related to prior work using LLMs for program generation. For example, LLMs have been applied to generate probabilistic programs (Wong et al.] (023). Answer Set Programs (Ishay ct al. [223]] Yang et al. (2023), differentiable logic programs (Shindo et al. 2024a)), and programs for visual reasoning (Surís et al. 2023] (Stani et al. 2024). Our symbolic policy representation inspired by situation calculus (Reiter 2001). which is an established framework to describe states and actions in logic.
6 CONCLUSION
In this study, we introduced BlendRL, a pioneering framework that integrates symbolic and neural policies for reinforcement learning. BlendRL employs neural networks for reactive actions and differentiable logic reasoners for high-level reasoning, seamlessly merging them through a blending module that manages distri- butions over both policy types. We also developed a learning algorithm for BlendRL. agents that hybridizes the state-value function on both pixel-based and object-centric states and includes a regularization approach to enhance the efficacy of both logic and neural policies.
Our empirical evaluations demonstrate that BlendRL agents significantly outperform purely neural agents and state-of-the oftte-of-hhe-rr neuro-symbolic baselines in popular Atari environments. Furthermore, these agents hibit robustness to environmental changes and are capable of generating clear, interpretable explanations across various types of reasoning, effectively addressing the limitations of purely neural policies. Our com- prehensive analysis of the interactions between symbolic and neural policy representations highlights their synergistic potential to enhance overall performance.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. The authors thank Sriraam Natarajan for his valuable feedback on the manuscript. This research work was partly funded by the German Federal Ministry of Education and Research, the Hes- sian Ministry of Higher Education, Research, Science and the Arts (HMWK) within their joint support of the National Research Center for Applied Cybersecurity ATHENE, via the "SenPai: XReLeaS" project. It also benefited from the HMWK Cluster Project 3AI. We gratefully acknowledge support by the EU ICT- 48 Network of AI Research Excellence Center "TAILOR" (EU Horizon 2020, GA No 952215), and the Collaboration Lab "AI in Construction" (AICO). The Eindhoven University of Technology authors received support from their Department of Mathematics and Computer Science and the Eindhoven Artificial Intelli- gence Systems Institute.
This research aims to advance neuro-symbolic reinforcement learning through the BlendRL framework, which combines neural and symbolic policies for enhancing performance and trans- parency. Our work includes no harmful examples or applications and is intended solely for positive con- tributions to Al and machine learning. We conducted our experiments using well-established benchmark environments, specifically popular Atari games, which are ethically recognized testbeds in the research community, Our methods and results aim to advance the field responsibly. Our code and resources are openly available to ensure transparency, reproducibility, and foster further research in this area.
Ethics Statement.
REFERENCES
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS), 2017.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhao- han Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In Proceed- ings of the International Conference on Machine Learning (ICML), 2020.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res., 2013.
A. Bhatt, D. Palenicek, B. Belousov, M. Argus, A. Amiranashvili, T. Brox, and J. Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Matthew Botvinick, Sam Ritter, Jane X. Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in Cognitive Sciences, 23(5):408 422,042222222 2019.
Yushi Cao, Zhiming Li, Tianpei Yang, Hao Zhang, Yan Zheng, Yi Li, Jianye Hao, and Yang Liu. GALOIS: boosting deep reinforcement learning via generalizable logic synthesis. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS), 2022.
Quentin Cappart, Thierry Moisan, Louis-Martin Rousseau, Isabeau Prémont-Schwarz, and Andre A Cire. Combining reinforcement learning and constraint programming for combinatorial optimization. In Pro- ceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Proceedings of the Conference Advances in Neural Information Processing Systems (NeurIPS), 2021.
Quentin Delfosse, Hikaru Shindo, Devendra Dhami, and Kristian Kersting. Interpretable and explainable logical policies via neurally guided symbolic abstraction. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS), 2023a.
11

---

Published as a conference paper at ICLR 2025
Quentin Delfosse, Wolfgang Stammer, Thomas Rothenbacher, Dwarak Vittal, and Kristian Kersting, Boost- ing object representation learning via motion and object continuity. In Machine Learning and Knowledge Discovery Databases: Research Track European Conference (ECML PKDD), 2023b.
Quentin Delfosse, Jannis Blüml, Bjarne Gregori, and Kristian Kersting. Hackatari: Atari learning environ- ments for robust and continual reinforcement learning. In Workshop on Interpretable Policies in Rein- forcement Learning @ @RLC-2024, 2024a.
Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sebastian Sztwiertnia, and Kristian Kersting. Ocatari: Object-centric atari 2600 reinforcement learning environments. In Proceedings of the First Conference on Reinforcement Learning (RLC), 2024b.
Quentin Delfosse, Patrick Schramowski, Martin Mundt, Alejandro Molina, and Kristian Kersting, Adaptive rational activations to boost deep reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024c.
Quentin Delfosse, Sebastian Sztwiertnia, Wolfgang Stammer, Mark Rothermel, and Kristian Kersting. In- terpretable concept bottlenecks to align reinforcement learning agents. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS), 2024d.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL'2: Fast reinforcement learning via slow reinforcement learning. In Proceedings of the International Conference on Learming Representations (ICLR), 2017.
Saso Dzeroski, Luc De Raedt, and Kurt Driessens. Relational reinforcement learning. Machine Learning (MLJ), 2001.
Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research (JAIR). 2018.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv Freprint:/810.00122: 2018
Fan Feng and Sara Magliacane. Learning dynamic attribute-factrre world models for efficient multi-object reinforcement learning. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS), 2023.
Rishi Hazra and Lue De Raedt. Deep explainable relational reinforcement learning: A neuro-symbolic approach. In Machine Learning and Knowledge Discovery in Databases: Research Track European Conference (ECML PKDD), 2023.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Hor- gan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araūjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research (JMLR), 2022.
Adam Ishay, Zhun Yang, and Joohyung Lee. Leveraging large language models to generate answer set programs. In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning (KR), 2023.
Zhengyao Jiang and Shan Luo. Neural logic reinforcement learning. In Proceedings of the International Conference Machine Learning (ICML), 2019.
12

---

Published as a conference paper ICLR 2025
Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, New York, 2011.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based re- inforcement learning for atari. arXiv Preprint: (aaaat::930003330 2019.
Henry A. Kautz. The third summer: Aaai robert s. engelmore memorial lecture. Al Magazine, 43(1): 105-125, 2022.
Kristian Kersting and Kurt Driessens. Non-parametric policy gradients: a unified treatment of propositional and relational domains. In Proceedings of the International Conference on Machine Learning (ICML), 2008.
Kristian Kersting, Martijn van Otterlo, and Lue DeRaedt. Bellman goes relational. In Proceedings of the International Conference on Machine Learning (ICML), 2004.
Daiki Kimura, Masaki Ono, Subhajit Chaudhury, Ryosuke Kohita, Akifumi Wachi, Don Joven Agravante, Michiaki Tatsubori, Asim Munawar, and Alexander Gray. Neuro-symbolic reinforcement learning with first-order logic. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Hector Kohler, Quentin Delfosse, Riad Akrour, Kristian Kersting, and Philippe Preux. Interpretable and editable programmatic tree policies for reinforcement learning, arXiv Preprint, 2024.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS), 2022.
Harsha Kokel, Arjun Manoharan, Sriraam Natarajan, Balaraman Ravindran, and Prasad Tadepalli. Reprel: Integrating relational planning and reinforcement learning for effective abstraction. Proceedings of the International Conference on Automated Planning and Scheduling (ICAPS), 2021.
Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. In Proceedings of the Conference Ad- vances in Neural Information Processing Systems (NeurIPS), 1999.
Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas. In Proceedings of the IEFE/RS International Conference on Intelligent Robots and Systems, IROS, 2020.
Brenden M. Lake, Tomer D. Ullmann Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 2017.
Tobias Lang, Mare Toussaint, and Kristian Kersting. Exploration relational domains for model-based reinforcement learning. Journal of Machine Learning Research (JMLR), 2012.
Borja G. León, Murray Shanahan, and Francesco Belardinelli. In nutshell, the human asked for this: Latent goals for following temporal specifications. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations 2020.
13

---

Published as conference paper ICLR 2025
Andrew Liu and Alla Borisyuk. A role of environmental complexity on representation learning in deep reinforcement learning agents. arXiv
2024.
Iou-Jen Liu, Zhongzheng Ren, Raymond A Yeh, and Alexander G Schwing. Semantic tracklets: An object- centric representation for visual multi-agent reinforcement learning. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021.
John W. Lloyd. Foundations of Logic Programming. Springer, Berlin, Heidelberg, 1984.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Proceed- ings of the Conference on Advances in Neural Information Processing Systems (NeurIPS), 2020.
Lirui Luo, Guoxi Zhang, Hongming Xu, Yaodong Yang. Cong Fang, and Qing Li. End-to-end neuro- symbolic reinforcement learning with textual explanations. In Proceedings of the International Confer- ence Machine Learning (ICML), 2024.
Daoming Lyu, Fangkai Yang, Bo Liu, and Steven Gustafson. SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2019.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 2016.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez. Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for rein- forcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
Wenjie Qiu, Wensen Mao, and He Zhu. Instructing goal-conditioned reinforcement learning agents with temporal logic objectives. In Proceedings of the Conference on Advances in Neural Information Process- ing Systems (NeurIPS), 2023.
Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern
Recognition (CVPR), 2016.
Raymond Reiter. Knowledge in Action: Logical Foundations for Specifying and Implementing Dynamical Systems. MIT Press, 2001.
Stuart Russell and Peter Norvig. Artificial intelligence modern approach. Prentice Hall, 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy opti- mization algorithms. arXiv Preprint: 170t:/m:77700033347 2017.
14

---

Published as conference paper ICLR 2025
Murray Shanahan, Kyriacos Nikiforou, Antonia Creswell, Christos Kaplanis, David G. T. Barrett, and Marta Gareeo.. An explicitly relational neural network architecture. In Proceedings of the International Con- ference on Machine Learning (ICML), 2020.
Hikaru Shindo, Devendra Singh Dhami, and Kristian Kersting. Neuro-symbolic forward reasoning. arXiv Preprim:2110.093883, 2021a.
Hikaru Shindo, Masaaki Nishino, and Akihiro Yamamoto. Differentiable inductive logic programming for structured examples. Proceedings the AAAI Conference on Artificial Intelligence (AAAI), 2021b.
Hikaru Shindo, Viktor Pfanschilling, Devendra Singh Dhami, and Kristian Kersting, ailp: thinking visual scenes as differentiable logic programs. Machine Learning (MLJ), 2023.
Hikaru Shindo, Manuel Brack, Gopika Sudhakaran, Devendra Singh Dhami, Patrick Schramowski, and Kristian Kersting, Deisam: Segment anything with deictic prompting. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS), 2024a.
Hikaru Shindo, Viktor Pfanschilling, Devendra Singh Dhami, and Kristian Kensting. Learning differentiable logic programs for abstract visual reasoning. Machine Learning (MLJ), 2024b.
Wolfgang Stammer, Antonia Wüst, David Steinmann, and Kristian Kersting. Neural concept binder. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS), 2024.
Aleksandar Stanić, Sergi Caelles, and Michael Tschannen. Towards truly zero-shot compositional visual reasoning with ilms as programmers. Trunsactions on Machine Learning Research (TMLR), 2024.
Shao-Hua Sun, Te-Lin Wu, and Joseph J. Lim. Program guided agent. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reason- ing. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), 2023.
Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 2 (4):160-163, 1991.
John Chong Min Tan and Mehul Motani. Learning, fast and slow: A goal-directed memory-based approach for dynamic environments. In IEEE International Conference on Development and Learning (ICDL), 2023.
Pashootan Vaezipoor, Andrew C. Li, Rodrigo Toro Icarte, and Sheila A. Mcllraith. Ld2action: Generalizing LTL instructions for multi-task RI. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Pro- grammatically interpretable reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 2018
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou. Chain-of-thouge prompting elicits reasoning in large language models. In Proceedings of the Conference Advances in Neural Information Processing Systems (NeurIPS), 2022.
15

---

Published as conference paper ICLR 2025
Lionel Wong, Gabriel Grand, Alexander K. Lew, Noah D. Goodman, Vikash Mansinghka, Jacob Andreas, and Joshua B. Tenenbaum. From word models to world models: Translating from natural language to the probabiliittic language of thought. arXiv Preprint:2306 12672, 2023.
Antonia Wüst, Wolfgang Stammer, Quentin Delfosse, Devendra Singh Dhami, and Kristian Kersting. Pix2code: Learning to compose neural visual concepts as programs. In Proceedings of Conference on Uncertainty in Artificial Intelligence (UAI), 2024.
Zhun Yang, Adam Ishay, and Joohyung Lee. Coupling large language models with logic programming for robust and general reasoning from text. In Findings of the Association for Computational Linguistics (ACL), 2023.
Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object-centric representations for reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICMI), 2023.
Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Self-supervised visual reinforcement learning with object-centric representations. In Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2021.
Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv 230print:2306.1222 12011 2023.
16

---

Published as conference paper ICLR 2025
A APPENDIX
A.1 DETAILS OF DIFFERENTIABLE FORWARD REASONING
In this section, we provide the details of differentiable forward reasoning.
A.1.1 WHAT IS FORWARD REASONING?
Forward Reasoning is a data-driven approach of reasoning in FOL. (Russell & Norvig 2010). Forward reasoning is performed by applying function called the Te operator, deducing new ground atoms using given clauses and ground atoms.
For of clauses C, Te operator (Lloyd 1984 atoms G. i.e.
function that applies clauses in C using given ground
Tc(9) = gu {
A B1, ....... Bn EC¹ ) ..... ({B, ...,, B,) g S
(4)
where C* is set of all ground clauses that can be produced from C.
Example. Let C be the following rules:
Policy ruleset (Kangaroo)
up (8) :-on_ladder (Player, Ladder), , same floor (Player, Ladder) right (X) =x)ff=eft_of (Player, Ladder), same_ floor (Player, Ladder). left (X) :-right_ of (Player, Laddez), hame_floor (Player, Ladder).
First, the rules are grounded by considering all possible substitutions the variables. For example, we have the following grounded rules:
Policy ruleset (Kangaroo)
up (img) :-on_lee con_ladder (player, ladder1), same_ floor (player, ladder1). right (img) =-left_of of (player, laddez1), same floor (player, ladderl). left (ing) --ri --right_of (player, ladderl). ) same Floor (player, ladderl).
For simplicity, we consider 3 constants: img. player, ladder1, and generate only grounding that satisfies type constraints, e.g. we do not generate on on.ladder (ladder1, ladder1) as it violates the type specification of the predicate, i.e. on ondder should take constants that represent the player and ladder arguments.
Now, following Eq. 4 the head atoms that represent actions are deduced by applying the Te operator:
BlendRL, the initial (()) be ground atoms that contain the following ground atoms:
ground atoms (()) is set of state atoms that encodes an observed state. Let
on on_ Ladder (player, ladder1), same floor (player, ladder1), left_ (player, ladderl) rrihht_o of (playez, ladder1).
Then (2) ())(()) is computed as follows:
up (img), eeft (img) right (ing),
on ladder (player, ladderl) same same_floor floor (player, ladderl), left_of (player, ladderl), right_of (playez, ladder1).
where the first ground atoms are newly deduced by 1-step forward reasoning. In this example, all rules are considered true (1.0 of weight for each), and different actions of up, left, right are deduced from an observed state. By using g(2), the agent can perform action decisions based on logical deductive reasoning.
Now let how perform forward reasoning in differentiable manner.
17

---

Published as conference paper ICLR 2025
A.1.2 WHAT IS DIFFERENTABLE FORWARD REASONING?
encodes the nawwardchaining reasoning process in FOL using differentiable tensor operations. To mitigate its memory- consuming bottleneck (cf. subsequent subsection 1.3 for details), using graph representations instead of tensors has been proposed (Shindo et al. 2024b). We adapt this approach the RL domain.
Disserentiable forward reasoning has been proposed by (Evans & Grefenstette 2018).
The key idea is to represent set rules as a bipartite graph and perform message passing to conduct forward reasoning. Let us introduce forward reasoning graph, core concept of this approach.
Definition A.1 A Forward Reasoning Graph is bipartite directed graph (Vg, Vg is a set of nodes representing ground atoms (atom nodes), V/. is set of nodes representing conjunctions (conjunction nodes), Eg-in is set of edges from atom to conjunction nodes and Ex-ig is a set of edges from conjunction to atom nodes.
where
Example (reasoning graph). 7ig depictss the forward reasoning graph for the Kangaroo policy rules in For simplicity, we visualize for the first 2 rules that define actions of up and right.
Fig. 4
up(img) right(img)
same_ haan_ffoorrppaarrrldddeer on [___adder(pplyyrr//dderr) left left_ of
Figure 7: Forward reasoning graph for policy rules for Kangaroo. A reasoning graph consists of atom nodes (left) and conjunction nodes (right). Only relevant nodes are shown (Best viewed color).
reasoning graph is represented as a bipartite graph. The left nodes represent ground atoms, and the right nodes represent conjunctions representing bodies of ground rules, i.e. conjunction node corresponds to a ground clause. It is obtained by grounding rules i.e. removing variables by, e.g. Player + player, Ladder +- ladder1. By performing bi-directional message passing the reasoning graph using soft-logic operations, BlendRL computes logical consequences in a differentiable manner.
Let move how to perform message passing to perform forward reasoning. Essentially, forward reasoning consists of two steps: (1) computing conjunctions of body atoms for each rule and (2) computing disjunctions for head atoms deduced by different rules. These two steps can be efficiently computed on bi-directional bessage-passing on the forward reasoning graph. We now describe each step in detail.
(Direction +) From Atom to Conjunction. First, messages are passed the conjunction nodes from atom nodes. For conjunction node v; E Vn. the node features are updated:
(++1 - VV((V
JEN(i) 0/3)
(5)
where A is a soft implementation of conjunction, and V is a soft implementation of disjunction. Intuitively, probabilistic truth values for bodies of all ground rules are computed softly by Eq.
(Direction +-) From Conjunction to Atom. Following the first message passing, the atom nodes are then updated using the messages from conjunction nodes. For atom node Vg. the node features are updated:
((+1) 0
= V
(t) 1
Wie 00)
(6)
JEN(()
18

---

Published as conference paper ICLR 2025
where W/i weight of edge ej-+- We assume that each rule C₁ E C' has its weight 0x. and Wyz 0x if edge Ej-ii on the reasoning graph is produced by rule Cj- Intuitively, in Eq. new atoms are deduced by gathering values from different ground rules and from the previous step.
We used product for conjunction, and log-sum-exy, exp function for disjunction:
( (x1,...,, ..)) log Σ
exp(<1/7),
where 7 > a smooth parameter. Eq. approximates the maximum value given input "i,.....00 Zn
Example (message passing). Fig. illustrates the message- saaaggge-pssinggggg process the Kangaroo reasoning
graph. We consider an observed object-centric state such that state atoms have the following distribution:
0.9: same_tloor (prrr(paaag (player, ladder1)
0.3: on on_iadder (player, ladderl)
0.8: right_of (player, ladderl)
(Direction -+) These values (weights of [0.9, 0.3, 0.8) are assigned to corresponding atom nodes, and these values are passed to the conjunction nodes following Eq, 5 (cf. the left-most figure in Fig. 88.
(Direction <-) A smooth operation amalgamates the incoming values to conjunction nodes. We simply employ the arithmetic multiplication here. Le for each conjunction node, we compute the multppicationn of all incoming messages. The resulting values correspond evaluation scores of bodies of ground rules. Then, the values on conjunction nodes are passed to the left for action atoms (cf. Fig.
Direction
Direction
Result
0.0
0.0
0.27 up(img)
0.0
0.72 0.72
A 0.27
0.9
0.9
0.9 same_ laar
0.3
0.72
0.3 on_ladder(playerrlldderr]
0.8
0.8
left (aaeoo((llayerrdddrr))
Figure 8: Message passing on the reasoning graph. (Best viewed in color).
Prediction. The probabilistic logical entailment is computed by the bi-directional pessage-passing. Let x (U) E [0, 1][9 be input node features, which map a fact to scalar value, RG be the reasoning graph, w be the rule weights, B be background knowledge, and T N be the infer step. For fact G, E G, BlendRL computes the probability as:
GG (0)) RG, w, W. B,T) = > [.]
(8)
where x (T)) atooms passing.
E
[0, 1] is the node features of atom nodes after T-steps of the bi-directional message-
Example (prediction). In the right-most figure in Fig. g BlendRL extracts values for action atoms, iee. [0.27, 0.72] for actions up and right. By taking softmax over these values, BlendRL's symbolic policy computes valid action distributions.
19

---

Published as a conference paper ICLR 2025
A.1.3 WHY GRAPH-BASED REASONING?
The original tensor-based reasoner (Evans & Grefenstette 2018). used NUDGE (Delfosse et al. 2023a), is memory- intensvv.. The memory consumption increases quadratically with the number of ground repre- sentations of rules and atoms when using tensors, while increases linearly with graph representations.
Let C' be the number of rules in the policy and G be the number of atoms representing actions and states. We can derive the number of ground representations these C* and G", respectively. Note that C* and G can be very large as they reflect all possible groundings (substitutions to remove variables in the rules and atoms). The tensor-based reasoner consumes quadratically O(C x G*). However, the graph-based reasoner consumes only linearly O(C* G*).
This greatly affects the scalability of the method. In practice, we consider parallelized (vectorized) envi- ronments train agents to get diverse experiences. Let Newy be the number of parallelized environments to train agents. Then the tensor-based consumes memory O(Nerv X C x G") but the graph-based one consumes only O(Nens (C* + G))). We empirically observed that memory efficiency is the key to BlendRL agents' successful training. In Seaquest, the rensor-based one scaled up to most 100 environments on NVIDIA A100-SXM= 44440B44-BBB GPU. However, the graph-based one scaled more than 500 parallelized environments, significantly improving performance.
has been proven that the graph-based reasoner can handle complex programs with function sym- bols (Shindo et al., 2024b). Consequently, BlendRL is capable of handling programs that contain function symbols due to its memory-efficient graph-based reasoner. Incorporating programs with structured terms using function symbols, such as lists and trees, would be significant enhancement e.g. meta interpreters
A.2 RULE GENERATION DETAILS
In this section, we present the complete set of prompts utilized for our rule generation process. We consis- tently employed GPT-4o for generating these rules.
Initially, we provided a general format instruction as a prompt, detailing the structure of rules that define actions. The prompt used was as follows:
Given instruction and available predicates, generate rules that dafine actions in the rule format. The rule formst 's
action ic
Each cond_i сan be either of:
(1) pxed(obeccts
(2) (Feaye))y
(3) (Elayee,,
where *action" and "pred" provide pradicates. predicatas provided svcslablo_predicatt
Show only the rule without any decoration.
Subsequently, we provide examples demonstrating rule generation. All the rules included here are derived from trained NUDGE agent on the GetOut environment (Delfosse et al., [023a). which non-Atari environment. We have added textual interpretations for each rule, pairing them with the corresponding output.
Examples:
Jump enemy getting by the player.
jump (X) :-closeby (F) (Flayer, #nemy).
left get the key if the player doesa't have the key and the player is the right of the key. rigght Key).
http:://geenaa..mm//nn/////llll
20

---

Published as conference paper ICLR 2025
left the door if the player has the key and the player is the right of the door.
left_to (X) key (Playar) onuright (Player, Door).
Go right to the key if the player doasn't have the key and the player is on the left of the key. @ighh_to_get_kee key (8) mnot_hawe__eyy (P) (Player) lon_left (P Ialayer, Key).
right to go to the docr if the player has the key and the player is en the left of the door. sight_to, tood keyave_key (ellayer) onnrrgght (Player, Door).
Finally, we offer environment--peciffe instructions, detailing the complete prompt of actions for each envi- ronment:
Kangoroo DankeyKong
Go if the player la the ladder and the player and the ladder are the aame level.
zight to the Ladder if the player is on the left of the laddez and the playez and the ladder are on the level.
Go left to the ladder if the player is on the right of the ladder and the player and the ladder are on the level.
Seaquest
to the diver if the player is deeper than the diver, the diver is visible, and the collected
divers are net Sull.
up
if full divers
collected.
Go left if the player is on the right of the diver, the diver ia visible, and the colledted divera are
Go right if the player is on the left of the diver, the diver is visible, and the collected divers are
not full.
to the diver if the player is deeper than the diver, the diver is visible, and the collected divers are net sull.
down the diver if the player is higher than the divar, the diver visible, and the collected divera are not full.
For the blending rules, we used the following prompts:
Kangaroo
If a monkey or coconut is close by, use the neural agert.
if nothing is around the player, use the logic agent.
$ seaquest
If a shark or miselle is close by, use the neural agent. playes, use logic agent.
nothing
i Lonkeykong
a barrel close by, use neural agent. nothing is around the player, use the logic agent.
A.3 PREDICATE GENERATION DETAILS
this section, we provide detailed explanations on the predicate (relation) generations for the relational state representations in BlendRL. We consistently employed GPT--4o for generating these rules.
Initially, we provided general task instructions of the predicate generation task as a prompt:
Given a task concept and an object description, generate a set of predicazes (relations) to represent the
the task. Use only simple concepta predicates.
Subsequently, we provide examples demonstrating predicate generation. All the environments included here (GetOut, 3Fish, and Heist) are environments used for NUDGE al. (2023a), which are Atari environments. We have added textual task descriptions for each environment, pairing them with the corresponding output.
21

---

Published as conference paper ICLR 2025
Example:
the Getout envirenment, the task is and need to dodge them when they get close by. left contrary, if the key is on the right of the player, they should left to get the key. The aame applies the door.
the door after getting key, You will find enemies
That means
key
player, they should go right gez key.
Predicates:
left_of, right_of, close_by.
example:
the 3Pish
the task la to aurvive and grow by eating other fish, You will Lind enemiea and need to dodge them when they get close That means if fish that Le smalles than the player is on the left of the player, they should go right to eat the fish, The same applies to other oriantations: go left, go above, and go down.
Predicates:
gge"_than, left_of, right_of, above, below
example:
the stist ronment,
by leating keys. Each has fireerent colors and can be opened only by key that has the same color, That means If the key is on the left of the player, they should go right get the key, on the contrary, the key on the right the player, they ahould left to get the key, Also, if the player has key, they should go to the box that has the same color.
la open
Predicates:
left_of, right_of, aoove, same_colos
Finally, we offer environmeent-specfft instructions, detailing the complete prompt of actions for each envi- ronment:
$ Kangaroo
the Kangaroo rovinnn dodge or Lire then. when they get close by, To resch the joey, the player needs to climb ladders the left
is go reach Joay. will find monkeys and need
to go the upper flocrs. That means right to get to the Izdder. on the contrary, if the player is on the right of the ladder, they should go left to get to ladder, To the ladder, the playe" shou"s located ladder the same floor.
player
ladder, they should go
Predicates:
s Seaquest
the Seaquest ervironment, the task is to rescue divers we managing oxygen levela, You find enemy summarines and need to shoot dodge them when they gez close That neans if player is on the left of the diver, they should go right to rescue the diver. On the contrary, i5 the right of the diver, they should left other directions: above and below. If divers are fully collected (5 divers), then go up to the surface to rescue them. Additionally, surface to replenish when low.
player
the diver. this applies
player
oxygen leveis
Predicates:
: LonkeyKong
is to reach top, the player can rescue character while avoiding obatacless You wt 11 find barrela rolling towards the player and reed to jung over them when they get close by, That means if the player is on the left of the ladder, the right of the
Conkey Kong envirenment,
they should go right reach the ladder. On the contrary, if the player laddar, they should go left to reach the ladder. if the player reaches a iadder, they should to higher lavals.
climb it to Predicates:
22

---

Published as conference paper ICLR 2025
A.4 BODY PREDICATES AND THEIR VALUATIONS
We here provide examples of valuation functions for evaluating state predicates (e.g. closeby, left_of, etc.) generated by LLMs (in Python).
subset of valuation functions for Kengaroo end BonkeyKong (generated by LIMS)
def left_ of th. Tensor, obji th.Tersor) -> Tensor:::
x playerl....
obi_x obil..... 1]
pbi cbjl;,
objectness
return (a < obi_x
pro_prob same level ladder ayla, obj)
def ociose_by(playeee th. Tensor, obj: Tensor)) > th. Tensor:
player_x playerl:, 11
player_y playerli, 2] obj x obil:, 11
obj_y obiti, 21
ebi_prob ob)l;, 0] objectness
dist (player_x cbj pow (2) y_dist (player_y obj_y).
dist (x_dist y_distt).sqrt return sigmoid(dist) ebi_prob
def on on_ladder th. Tensor, obj: th. -> th, Tensor: player_x player[..... 1J
obi x obji...,
return sigmoid (abs (player_x obj_x) gamma)
subset of valuation functions for Seaquest (generated by LIMs)
def fuil divediiee (cbjs: Tensor) >> Tenoor:: divers_vs = objsli, -6:] [em_collected__oo diff = nun_collected__ivee
- th. th..sumch (divers_vs 0,,, dimm1)
return sigmoid(1 diff)
def not_tul __divers (cbjjs th. Tensor) th, return full (eveerssbbb
def above (player: th. Tensor, obj: th. Tensor) -> th, Tensor:
player_y playerl.... 21
obj_y = obil..... 2]
obi_prob cbil:, 0]
return sigmoid( (player_y - obj_y) / gamma) cbj_prob
A.5 BLENDRL RULES
We here provide the blending and action rules obtained by BlendRL Seaquest and DonkeyKong.
slending rulset (Seaquest)
98 neural_agent (X) ==cllose_by_eneme E).
0. 74 neural_agent
htt.clooe_____niaaeeeeeee (F,
0. logic_agent (X) :1::visbbee_diver(())
02. logic_agent (X) ==0xygen_loww@11.
1.0 logic_agent (X) [f_l fxl
$ Policy rulset (Seaquest)
0.21
=xxygen___ (R)
0.22 up rescue (X) :-full divers
divess (X)
0.21 left_to_diver (X): Light_of_divee (2,0)
viieeelllellll.e.ii.... dive (D).
23

---

Published as conference paper ICLR 2025
0,24 Eight_te_divee : (2, visible (D))
0.23 up_to_df ver (X) --derper_thann_llveee (P, sisblee_ivee (P).
0.22 nown_to_divers t-iigher_thhhhtttceo diver viveble_diver (D).
* Aleiingg roleset (Kangaroe, DonkeyKong)
0.92 peural_agent (X) = barzel (2,
0.28 logic_agent (X) =nthhhinnnggg (X)
$ Policy roleset (Kangaroo, DonkeyKong)
0.88 up_ adder (X) tttoo_ (P, T.), same_floorg
0.47 right_ladder ladder (X) :-left_ooop (2, L2, same floor (2,
0.18
left_ledder (X) t-right_of (2, same_floor (2, L).
A.6 DETAILED RESULTS
We hereafter provide the final results of conducted experiments (L.e. DON, PPO, NLRL, NUDGE and
BlendRL), as well as of existing symbolic baselines, namely NLRL (Jiang & Luo 2019). NUDGE (Delfosse
al. 2023a). SCoBots (Delfosse al. 2024d), INTERPRETER (Kohler 2024 and INSIGHT (Luo
ct al. 2024) as well as our trained BlendRL agents.
Enviromments
DQN
PPO
NLRL*
Human
Random
Kangaroo
2696
790.0=280.8
3034+11
2739
100
Seaquest
2794
837.3246.7
7540
4425
215.50
Donkey Kong
253.3+45.1
2080=1032
29+0
7320
NUDGE
SCoBots
INTERPRETER
INSIGHT
BlendRL
Kangaroo
3058+25
4050.0+217.8
1800.0#0.0
12619=132
Seaquest 64+0
2411.3+377.0
1092.9+154.6 1837.74459.4
2665.74728.2
4204+10
DonkeyKong 122425
426.7+64.3
3541#43
Table 3: Raw scores for deep (DQN and PPO) as well as symbolic (NLRI, NUDGE, SCoBots, INTER- PRETER, INSIGHT and BlendRL agents).
We also provide the learning curves for the trained PPO. NUDGE, and BlendRL agents in Fig.
Kangaroo
Seaquest
DonkeyKong
4K
BlendRL NUDGE
episodic return
10K
3K
NeuraiPPO
3K
2K
5K
1K
5K
5K
20K
25K
10K 12K
15K
opisade
Figure 9: BlendRL surpasses both purely neural and logic-based policies environments requiring reasoning and reaction capabilities, shown by its superior episodic returns after training (averaged over 3 seeded reruns). BlendRI. outperforms the logic-based state-of-the-art agent (i.e. NUDGE) across all tested environments. The purely neural PPO agents frequently get trapped in suboptimal policies, BlendRI. lever- ages its integrated reasoning and reacting modules to tackle various aspects of the tasks.
24

---

Published as conference paper ICLR 2025
A.7 EXPERIMENTAL DETAILS
We then provide more details about our implementations.
Hardwares. All experiments were performed on one NVIDIA -0000-XM4440GBBB GPU with Xeon(R):8174 CPU@3.100M and 100 GB of RAM.
Value function update. We update the value function and the policy as follows: the following, we consider a (potentially pretrained) actor-critic neural agent, with Ve its differentiable state-value function parameterized by & (critic). Given a set of action rules C, let TTC.w) be a differentiable logic policy. BlendRL learns the weights of the action rules in the following steps. For each non non-terminal state of each episode, we store the actions sampled from the policy (a, ~ ((c) and the next states 81+1- We update the value function and the policy follows:
5 = + y70(61)))1)) (s()
(9)
Φ + (
(10)
W = W + VV In (s))
(11)
The logic policy W)).W) thus learns to maximize the expected return.
A.8 TRAINING DETAILS
We hereby provide further details about the training. Details regarding environments will be provided the next section A.9 We used the Adam optimizer (Kingma & Ba 2015) for all baselines.
BlendRL. We adopted an implementation of the PPO algorithm from the CleanRL project (Huang et al., 2022). Hyperparameters shown in Table The object-centric critic is described in Table 5 We provide a pseudocode of BlendRL policy reasoning in Algorithm 1
Parameter
Value
Explanation
0.99
Discount factor for future rewards
learning
0.00025
Learning rate for neural modules
legii_leaming_rate
0.00025
Learning rate for logic modules
blender_ blender.Jeamnnng.ne Jearning_rate
0.00025
Learning rate for blending module
blend ent cnef
0.01
Entropy coefficient for blending regularization (Eq.
clip_coef 0.1
Coefficient for clipping gradients
ent.coef 0.01
Entropy coefficient for policy optimization
max_grad_nom
Maximum norm for gradient clipping
num_envs 512
Number of parallel environments
num_steps
128
Number of steps per policy rollout
total, timestepss
20000000
Total number of training timesteps
Table 4: Hyperparameters for BlendRL training.
Layer
Configuration Linear( 120) ReIUU)
Fully Connected I Layer
Activation 1
Fully Connected Layer
Linear( 120, 60) ReLU() Linear(60. No..)
Activation 2
Fully Connected Layer
Table 5: Object-centric critic networks for BlendRL.
25

---

Published as conference paper ICLR 2025
Algorithm 1 BlendRL. Policy Reasoning
Input: TT Tgeural Logge VONN Voc, blending function B, state (a, 2)
1: B ((x, z) Compute blending weight
2: action B mg nuaural (a) + (1 3) T
Logic (2))(2)
(z) # Action is sampled from the mixed policy
3: value 3- 1r VONN(( (a) + (1 B) Vi0o (=) Compute the state value using /
4: return action, value
NUDGE. We used public code to perform experiments with the CleanRL training script. All hyperpa- rameters are shared with the BlendRL agents, as described in Table We used the same ruleset as BlendRL agents for NUDGE agents. The critic network on object-centric states is described in Table
Neural PPO. We used an implementation of the neural ppo algorithm from the CleanRL project. The agent consists of actor network and critic network, sharing their weights except for the last layer. The base network shared by the actor and the critic is shown in Table. 66 linear layer with non-shared weights follows for each actor and critic after on top of the base network.
Layer
Configuration
Convolutional Layer
Conv2d(4, 32, stride=4) ReLU()
Activation 1 Convolutional Layer
Conv2d(32, 64, 4, stride=2)
Activation 2
ReI.U()
Convolutional Layer
Conv2d(64, 64, 3, stride=1)
Activation 3
ReLU()
Flatten Layer
Flatten()
Fully Connected Layer
Linear(64 77*7,512
Activation 4
ReLUO)
Table 6: Layer configuration of the neural ppo agent.
A.9 ENVIRONMENT DETAILS
We hereby provide details of the environments used in experiments. We used HackAtarr) frame- work that offers modifications of Atari environments to simply them or change them for the robustness test. In our experiments, the modifications we used shown in Table 7
Environment
Option
Explanation
Kangaroo
disable disable_falling falling
falling_ccconut coconut
Disable the falling coconut
change_level0
The first stage is repeated
random position
Randomize the starting position
Seaquest
No option
DonkeyKong
change level0 random-position
The first stage is repeated Randomize the starting position
Table 7: Options and explanations for different environments
26

---

Published as conference paper ICLR 2025
A.10 ABLATION STUDY: NEURAL VS. LOGIC BLENDING MODULE
We here provide an ablation study on the logic blending module, reruning BlendRL. using a neural one Kangaroo and Seaquest. As shown in Table 8 the agents that encompass logic-based blending modules outperform the neural ones.
Figure I0 presents the entropies of the blending weights. An entropy value of 1.0 signifies that neural and logic policies are equally prioritized (with each receiving weight of 0.5), while an entropy value of 0,0 indicates that only one policy active, with the other being completely inactive. In both environments, the logic blending module consistently produced higher entropy values for the blending weights, indicating effective utilization of both neural and symbolic policies without overfitting to either one.
Entropies over Blending Weights
logic bl. neural bl.
Episodic Return
Kangaroo
Seaquest
Entropy
Neural Blending
91.6 = 43.6
17.8 0.55
Logic Blending
186_12
39.947.12
Table 8: Comparison: Neural v.s. Logic for policy blending. Average returns over 10 different random seeds are shown of trained agents. The logic blending module outperforms neural one consistently.
kangaroo seaquest Environment
Figure 10: Logic blending keep both policies effective. Entropies over blending weights are shown.
A.11 ABLATION STUDY: NOISE ON OBJECT-CENTRIC STATES
We conducted additional experiments by introducing noise the input object-centric states (only at time). Specifically, we made some objects in the input state invisible at varying noise levels, ranging from 0.1 to 0.5. For example, a noise rate O. indicates that detected objects are invisible with a 10% probability. We evaluated 10 episodes and reported the mean episodic returns and lengths. Fig. 11 shows episodic returns and lengths for three environments. We observed the following facts: (i) Noise significantly impacts the overall performance of BlendRL agents. This is because the blending module relies on object-centric states, and the introduced noise can lead to incorrect decisions. For example, agents may mistakenly use logic policies when enemies nearby. (ii) Noise affects episodic return more than episodic lengths overall. notable exception in the Seaquest environment, where episodic lengths decreased significantly due to noise. This suggests that high-quality object-centric states erucial when reactive actions frequently required, such as when there are many enemies. Obviously, training agents in noisy environments would increase the agents' robustness.
27

---

Published as conference paper ICLR 2025
kangarco
seaquesi
donkeykong
Episodlo Return
1.0
1.0
1.0
0.5
0.5
0.5
0.0
Episode Langth
1.0
1.0
1.0
0.0
0.0
0.0
Noise
Figure 11: Episodic return and length with different noise ratios on object-centric states.
A.12 PROGRESSIVE ENVIRONMENT ILLUSTRATION
As explained by Delfosse al. 2024c Seaquest is progressive environment in which the agent first needs to master easy tasks, before being provided with more complex ones in newly unlocked parts of the environment, reflected by the number of enemy be shot here.
100
8250
16280
a
4
4
SKYGIM
DKYSEN
BIYGIN
Alll/SiON
AETI-SION
AlT-/SiON
Figure 12:
Seaquest is progressive environment. The more an agent collects points/reward (depicted the top), the more enemies are spawning
A.13 DEITAILED BACKGROUND OF REINFORCEMENT LEARNING
We provide more detailed background for reinforcement learning. Policy-based methods directly optimize Ty using the noisy return signal, leading to potentially unstable learning. Value- methods learn to approximate the value functions V. or Qo. and implicitly encode the policy, egg. by selecting the actions with the highest Q-value with high probability (Mnih al. 2015). To reduce the variance of the estimated
Q-value function, one learn the advantage function Aş (St, a.) Qs(Or, az) Va(st). An estimate of
the advantage function can be computed as Ág a²) = Σ 7'r++s + + (8+++) Vo(ss) (Mnih et al.
2016). The Advantage Actor-critic (A2C) methods both encode the policy To (i.e. actor) and the advantage
28

---

Published as conference paper ICLR 2025
Input State
BlendRL Explanation
Logic Neural Explanation
Input State
BlendRL Explanation
Logic / Neural Explanation
Logic Mode
11
200
Neural Mode
Figure 13: We depict BlendRL explanations on states where symbolic policy is prioritized (top) and those where neural policy is activated (bottom) on Kangaroo and Seaquest.
function As (i.e. critic), and use the critic provide feedback the actor, as in (Konda & Tsitsiklis, 1999). To push my take actions that lead higher returns, gradient ascent be applied (0) Elogg (a s) A) Proximal Policy Optimization (PPO) algorithms ensure minor policy updates that avoid catastrophic drops (Schulman et al.) 2017). and can be applied to actor-critic methods. To do so, the main ==(a]s)
objective constraints the policy ratio r(0) Tea [a]s)* following T.PR (0) [[min(r(0) clip(r()))
1 c) where clip constrains the input within [1 c, + c. PPO actor-critic algorithm's global
objective is L(0, Φ) (() LVP (Ø)], F with = (st) - (s+))) being the value function
loss. An entropy term can also be added to this objective to encourage exploration.
A.14 BLENDRL EXPLANATION: CASE STUDY
We conducted an additional ablation study examining how symbolic and neural explanations are blended over time. Fig. 13 showcases BlendRL's explanations of different states on Kangaroo and Seaquest. The top-row represents explanations when the logic policy is active, and the bottom-row represents those where neural policy is prioritized.
The observations are summarized follows:
When the symbolic policy is active, the logic policy provides clear explanations using relevant objects in an object-centric representation. The neural policy, however, produces very noisy expla- nations that neither highlight important objects nor illustrate the reasoning behind them.
When the neural policy is active, the logic policy continues to produce solid explanations objects but may miss critical ones necessary for effective action, such enemy (monkey) close the agent. The neural policy, despite being noisy, provides informative explanations by highlighting relevant objects, indicating its ability capture objects vital for reactive actions.
29

---

Published as conference paper ICLR 2025
These observations suggest that the symbolic policy complements the neural policy, allowing the latter focus more effectively on reactive actions.
A.15 DETAILED EXPLANATIONS BY LLMS
Inspired by (uoo et al. 2024). we utilized LLMs to generate interpretable textual explanations on top of BlendRL's logical explanations. We instruct LLMs by specifying the task of explanation generation, pro- viding an environment description, observing facts (BlendRL's logic explanation), and an action taken. We show the generated explanations for each scene in Fig.
Kangaroo: "The agent observed that it was positioned to the left of ladderl, which is crucial for ascending the upper floors where the joey might be located. Since the player and ladder] are the same floor, the agent decided to move right towards ladderl. This action aligns with the task objective of reaching the joey, as navigating the ladder is necessary step to climb and advance higher levels in the environment""
Seaquest: "The agent moved downwards because the player is positioned above a diver and there are no immediate threats present around the player. Since the divers have not been fully collected yet, the agent prioritized descending to rescue the diver below. This decision aligns with the task of rescuing divers while managing the absence of immediate external threats."
"The agent observed that the player was on the right side of ladder9 and on the same floor. To progress towards reaching the top and rescuing Princess Peach, the player needed approach ladder9 to climb it and advance to higher level. Therefore, the action taken was moving left. which allowed the player to position themselves directly over ladder9, enabling them climb il and continue their ascent while avoiding obstacles."
DonkeyKong:
The generated textual explanations provide clearer insights into the action dationnmaking of the agents, enhancing their transparency.
The full prompts we used to generate the textual explanation are the following:
t Kangaroo
Generate explanation of the action taken by agent given task description abserved facta, taken action. Show only the generated anation.. Be concise.
Task Description: In the Kangareo environment, the task is to go to reach the jony. You will find tododge fire then when they close by. to climb laddersto go to the upper floors.
monkays
joey, player needs
Observed Facts: player ls left laddest, player and ladderl are the same floor
Taken Action: right
Textual Explanation:
3 Seaquest
Generate an explaration of the act taken by the agent given taak description observed facta, and Laken action. Show only the generated explanation.
the Seaquest arvironment, the
divers while managing exygen levels. You will find eremy submarines and need to ahoot or dodge them whan they get close by. "f divers are fully them. Additionally,
is
collected diversl, thea monitor oxygen levels and ascend to the surface to replenish when low.
player must
Observed Facts: player is above diver, nothing is sorund player, divers not fully collected
Taken Action: down
Textual axpiinatioo:
ConkeyKong
30

---

Published as conference paper ICLR 2025
Generate an explanation of the action taken by the agent given task description observed fects, and taken action. Show only the generated explanation.
Donkey Kong envirenment, cbstacles. You will find barrels rolling towards the player and need to jump over or dedge them when they get close by. the player reachea ladder, they ahould elima to advance to higher levels. additionally, the player mast moniter and avoid obstacies like Tammss that move across platforms.
top to reseue
while avoiding
Observad Facts: player 15 on right of ladders, ayer 1s an the same floor as laddar9 Action: left
Textual Exglanation:
A.16 ACTION DISTRIBUTION IN NEURAL AND LOGIC POLICIES
Figure 14 illustrates the distribution of actions taken by the logic and neural policies in trained BlendRL agent, based on 10k steps in the Seaquest environment. While the logic components primarily execute reactive actions fire, the neural agents predominantly handle complementary actions.
Seaquest Action Distribution fire
right
up
left
down
down
right left
Logic Agent
Neural Agent
Figure 14: Action distribution of a BlendRL agent in Seaquest. We depict the proportion for each logic and neural agent, respectively.
A.17 ABLATION: INDUCTIVE BIAS FOR LLM RULE GENERATION
To identify the significance of the few-shot examples in the mule-generation step, we provide ablation study using imperfect rule examples for LLMs. Instead using optimal rules obtained by NUDGE (Delrosse et al.. 2023a), we provide non-optimal but grammatically correct rules. We used the following suboptimal rules:
if enemy is getting close by player. sight (8) cLoseby (Player, Enemy).
Jump to get the key if the player has the key and the player is on the right of the key. jump_get__ke (X) i has_key (Playesl i ongright (Playez, Key).
Go right to go to the door if the player is close by an enamy and the player has the key. saght__o_tt_ooooo (X) :-closeby IPlayer, Bnemy), hay_key (Player).
We compare the rule- dileggeneration performance with (i) 5 optimal rule examples from NUDGE policies, (ii) 3 suboptimal rule examples, (iii) suboptimal rule example 10 and (iv) no rule example.
We used the first example with right and closeby.
31

---

Published as conference paper ICLR 2025
Thole the success rate of the rule generation 10 trials GPT-4o. a generated rule set represents the same policy as those in Fig. and Section A.5 with the original prompt, we count it as success. For all three environments, GPT4-o generated correct action rules with suboptimal rule examples. The performance decreased by reducing the number of examples, especially in Seaquest. Without examples, it failed to generate valid action rules.
Methods
Kangaroo
Seaguest KonkeyKong
NUDGE Rules (5 Examples)
100%
100%
100%
Suboptimal Rules (3 Examples)
100%
100%
100%
Suboptimal Rule (1 Example) No Examples
90%
10%
90%
0%
0%
0%
Table 9: Success rate of action rule generation with optimal NUDGE rules, sub-optimal rule examples, and without any rule example.
Without examples, incorrectly formatted rules are often generated, e.g. for Seaquest:
kailure cases in Seaquest
so_up_diver (X) :-deeper (X, diver), visible (diver), not (full (divers)
go_up_ rescuueeeeeeeeeeeeeeeeee (X) :-col i-colllecteeeeee lected (full (divers)). )
go_left right_of diver) visible (divez) (collected (divers) )
go_right (X) of (X, diver) vissible (diver) not (follected(full (divers) )
go_up_ (X) s-deeper (x, daver), visible (diver), not (collected (full (divers)
diver (X) :-higher (X, diver), visible (diver), [collected (divers))).
This result suggests that the provided few-shot rule examples inform LLMs about the general rule structure to define actions, but they do not convey the semantics or specific strategies required for the environments.
32


Reviews:
[{"review_id": "z2N1mIv3AX", "reviewer": "Reviewer_NFD8", "summary": "This paper integrates condition-based logic decisions with neural network-based reinforcement learning policies through an LLM-based hybrid module to address the shortcomings of both approaches. It achieved better results in three Atari games compared to standard PPO and logic-based method.", "questions": "see weakness", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "1. The paper is clearly written and easy to follow.\n2. The utilization of the language model shows a certain level of innovation.", "weaknesses": "1. The overall concept is not particularly novel, with numerous similar works, such as the well-known fast and slow systems, already existing.\n2. The paper and appendix lack crucial details on how the LLM generates rules and calculates hybrid probabilities, and to what extent this is based on the content provided in the prompts. This is essential for determining whether the method can generalize to more diverse tasks.\n3. The paper consistently emphasizes complex tasks, yet the experimental environment is not particularly sophisticated. Atari is a basic, outdated, and relatively simple benchmark that has largely been mastered. The three selected tasks are not among the recognized challenging ones in Atari (indeed, they are relatively simple). Truly difficult tasks, such as Montezuma’s Revenge, would be more appropriate as experimental environments.\n4. The experiments should compare against more advanced RL algorithms. For example, Agent57 has already achieved a score of 30k on Kangaroo (while blend RL scores less than 20k) and 1000k on Sequest (while blend RL scores less than 5k). Therefore, the current experimental results do not demonstrate the superiority of the method.\n5. Overall, the experiments lean towards simpler explorations and lack ablation studies that would reveal the characteristics of the method. For instance, I couldn't find which LLM was used, nor were there detailed experiments on the impact of object-centric representations on the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YmmcsuUqb1", "reviewer": "Reviewer_xzvw", "summary": "The paper presents BlendRL, a neuro-symbolic RL framework that allows agents to blend resoning from pixel and object level representations. Unlike previous  neuro-symbolic RL approach BlendRL learns concurrently it's low and haigh level policies. Specifically, a blending module, informed by rules generated by a language model, dynamically determines the optimal mix of neural and symbolic policies based on the task context. \n\nAuthors test BlendRL in Seaquest and Kangaroo (from Atari), where agents must alternate between quick responses and logical planning, results show that BlendRL consistently outperforms purely neural or symbolic agents.", "questions": "See weaknesses above\n\n--- Post rebuttal --- \n\nAfter reading Author's responses and the updated paper I have no more concerns. I believe this will be an interesting contribution for those working with neuro-symbolic approaches.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is very well written and easy to follow. The proposed framework is quite novel to the best of my knowdledge, since the level 2 and 1 systems are usually much more separated than in BlendRL, and the results against vanilla PPO or a symbolic approach (NUDGE) are promising.", "weaknesses": "My biggest concern with this work is the lack of empirical comparison with any other neuro-symbolic baselines, e.g. [1-4] \n\nSince BlendRL also weights heavily on object-based representations and relational learning it would have been good (although I don't consider this critical)  to include some contrast with deep learning approaches for such kind of learning, e.g. [1, 5-6].\n\nWhile the point above will raise my confidence on the paper, I still thinkt hat the novelty of the approach and the results included outweight the weakpoints.\n\n[1] Borja G. León, Murray Shanahan, and Francesco Belardinelli. In a nutshell, the human asked for this: Latent\ngoals for following temporal specifications. In International Conference on Learning Representations, 2022.\n\n[2] Kuo, Yen-Ling, Boris Katz, and Andrei Barbu. \"Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas.\" 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.\n\n[3] Vaezipoor, Pashootan, et al. \"Ltl2action: Generalizing ltl instructions for multi-task rl.\" International Conference on Machine Learning. PMLR, 2021.\n\n[4] Qiu, Wenjie, Wensen Mao, and He Zhu. \"Instructing goal-conditioned reinforcement learning agents with temporal logic objectives.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[5] Shanahan, Murray, et al. \"An explicitly relational neural network architecture.\" International Conference on Machine Learning. PMLR, 2020.\n\n[6] Feng, Fan, and Sara Magliacane. \"Learning dynamic attribute-factored world models for efficient multi-object reinforcement learning.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iCS8xMv2QC", "reviewer": "Reviewer_NRGe", "summary": "The paper proposed to jointly learn a mixture of neural policy and symbolic policy (represented in a differentiable way) via reinforcement learning. In the experiments, the neural policy is simply a CNN-based policy, the differentiable symbolic policy is based on a method called differentiable forward reasoner, and the weights of the mixture can themselves be seen as the output of a trainable policy. The proposed method is tested on three Atari games.", "questions": "1) If I wanted to apply it to a new game, what should be done to generate the action rules (Section 3.3)? \n\n2) Does using the NUDGE policies to generate examples for the LLM provide a strong inductive bias to BlendRL?\n\n3) How were the hyperparameters obtained?\n\n4) How did you choose the three Atari games? Why did you use different ones compared to those used in NUDGE?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of learning a mixture of neural policy and symbolic policy is novel. Given the presented experimental results, it seems to lead to nicely interpretable decision rules (at least in the symbolic part). Also, surprisingly to me, it seems that it automatically learns to make \"reflex\" decisions preferably using the neural policy. \n\nThe overall proposition nicely integrates several previous techniques to offer an end-to-end method, with little human inputs.", "weaknesses": "The paper should be more self-contained. For instance, this work builds on several existing propositions (notably by Shindo et al. and Delfosse et al.). The authors should recall and provide more details of those previous works to help the reader more easily understand BlendRL and appreciate its novelty. For instance, I think the current presentation of differentable forward reasoner is too light, e.g., how is the set \\mathcal C related to the forward reasoning graph? What's the advantages/disadvantages of this representation using a graph compared to other frameworks, e.g., dILP?\nAlso, Section 3.3,  which seems to me to be a key component that can have a huge impact on BlendRL, is lacking in details. For instance, I suggest the authors to provide more information about step (i). In addition, doesn't using the NUDGE policies to generate examples for the LLM provide a strong inductive bias to BlendRL?\n\nThe experimental validation is missing some details and maybe a bit light. The authors should explain how they chose their hyperparameters (e.g., Entropy coefficient for blending regularization). Since BlendRL requires a more complex policy model than Neural PPO or NUDGE), I believe it's possible via (potentially costly) hyperparameter tuning to obtain better results for BlendRL.\nThe authors only evaluates on three Atari games, which are moreover different from those used in NUDGE. Why is that so? Would BlendRL still outperform NUDGE on the tasks where it has demonstrated a good performance?\n\nAlthough I haven't followed closely the latest developments in neuro-symbolic approaches, I believe that the literature in this direction is quite rich. For instance, I think that there are other works trying to combine System 1 and System 2 capabilities in AI agents and there are other neuro-symbolic RL approaches. The paper could be improved by better situating BlendRL in the existing literature. \n\nMinor points:\nThere are a number of typos that should be corrected, e.g.,\n- line 107: I believe that function symbols are not used in this work. This should be clarified.\n- line 162: What's called \"state\" actually corresponds to \"observation\"\n- line 177: Regarding the dimension of the logic policy, should F appear?\n- lines 852-855: the learning rates are missing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5qTJwK5RSw", "reviewer": "Reviewer_pgu6", "summary": "The paper presents BlendRL, a neuro-symbolic reinforcement learning framework that integrates symbolic reasoning and neural policies, aiming to improve agent performance by mixing neural and symbolic policies. The authors demonstrate BlendRL’s efficacy in classic Atari games, highlighting that the framework outperforms purely neural baselines and neuro-symbolic baselines like NUDGE.", "questions": "1. Figure 6 references textual interpretation; could you kindly provide a more detailed explanation?\n2. In cases where the logic chain becomes quite long, would the policy still be considered explainable?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is clearly written, and the proposed framework is compelling in its attempt to bridge the gap between symbolic reasoning and neural policy learning. The dual use of symbolic logic for high-level reasoning and deep neural networks for low-level reaction is well-motivated and timely. The experiments convincingly show that BlendRL agents perform better than existing methods, particularly in environments that demand both reasoning and quick reflexes. For instance, in Seaquest, the neural policy was able to effectively manage shooting enemies, while the symbolic component handled pathfinding and resource collection, demonstrating how the hybrid model synergistically enhances performance.", "weaknesses": "Overall, I think this work will have a positive impact on the community, but I still have some concerns:\n1. The paper evaluates the proposed method in a limited set of experimental environments. It would strengthen the validation of the method if the authors could evaluate BlendRL in a wider variety of environments, especially those with different characteristics (such as the environments BeamRiderNoFrameskip-v4, EnduroNoFrameskip-v4, and QbertNoFrameskip-v4).\n2. While the paper demonstrates strong performance, it would be more impactful if the method were tested in more realistic environments such as IsaacGym[1] or MetaDrive[2]. These environments pose more challenging and realistic scenarios, which could further validate the generalizability and robustness of BlendRL. It would also be valuable for the authors to discuss any challenges or adjustments required to apply BlendRL to these more complex environments, and to comment on how they expect its performance and benefits to scale in such domains.\n3. The explanation of how symbolic policies contribute to the overall behavior of mixed policies could benefit from more clarity. It would be helpful if the authors provided a detailed analysis or specific examples showing how the symbolic explanations correspond to the mixed policy's actions. Including case studies that illustrate how explanations evolve as the blending between neural and symbolic components changes could enhance understanding of the interpretability and functionality of the method.\n\n[1] Makoviychuk, Viktor, et al. \"Isaac gym: High performance gpu-based physics simulation for robot learning.\" arXiv preprint arXiv:2108.10470 (2021).\n\n[2] Li, Quanyi, et al. \"Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning.\" IEEE transactions on pattern analysis and machine intelligence 45.3 (2022): 3461-3475.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)