BlendRL: A Framework for Merging Symbolic and Neural Policy Learning

## Abstract

Abstract Humans can leverage both symbolic reasoning and intuitive reactions. In contrast, reinforcement learning policies are typically encoded in either opaque systems like neural networks or symbolic systems that rely on predefined symbols and rules. This disjointed approach severely limits the agents’ capabilities, as they often lack either the flexible low-level reaction characteristic of neural agents or the interpretable reasoning of symbolic agents.
To overcome this challenge, we introduce BlendRL , a neuro-symbolic RL framework that harmoniously integrates both paradigms within RL agents that use mixtures of both logic and neural policies.
We empirically demonstrate that BlendRL agents outperform both neural and symbolic baselines in standard Atari environments, and showcase their robustness to environmental changes.
Additionally, we analyze the interaction between neural and symbolic policies, illustrating how their hybrid use helps agents overcome each other’s limitations.

## 1 Introduction

Figure: Figure 1: Most problems require both reasoning (top) and reacting (bottom), to master *e.g.* the Kangaroo (left) and Seaquest (right) Atari games.
Refer to caption: /html/2410.11689/assets/x1.png

To solve complex problems, humans employ two fundamental modes of thinking: (1) instinctive responses for immediate reaction and motor control and (2) abstract reasoning, using distinct identifiable concepts.
These two facets of human intelligence are commonly referred to as System 1 and System 2 (Kahneman, ).
While our reasoning system requires symbols to build interpretable decision rules, instinctive reactions do not rely on such inductive bias, but lack transparency.
Both systems are required in challenging RL environments, such as Kangaroo, where the agent’s goal is to reach its joey (at the top), captured by monkeys that need to be punched out of the way, or in Seaquest, where the agent controls a submarine, and needs to collect swimming divers without running out of oxygen (*cf.* Figure [1](#S1.F1)).
Developing agents that can effectively use both information processing systems has proven to be a persistent challenge (Lake et al., ; Mao et al., ; Kautz, ).
The main difficulty not only lies in achieving high-level capabilities with both systems, but also in seamlessly integrating these systems to interact synergistically, in order to maximize performances without sacrificing transparency.

Deep neural networks have demonstrated an ability to effectively learn policies across a wide range of tasks without relying on any prior knowledge about the task (Mnih et al., ; Schulman et al., ; Badia et al., ; Bhatt et al., ).
However, these black box policies can leverage shortcuts that are imperceptible to human observers (Locatello et al., ; Liu & Borisyuk, ).For instance, in the simple Atari Pong game, deep agents tend to rely on the enemy’s position rather than the ball’s one (Delfosse et al., ), demonstrating deep learning systems’ tendency to rely on shortcut learning opportunities, that then fail to generalize to slightly modified environments.

To enhance reasoning capabilities, symbolic reasoning has been integrated into approaches, using *e.g.* as logic-based policies (Jiang & Luo, ; Kimura et al., ; Cao et al., ; Delfosse et al., ) or program-based frameworks (Sun et al., ; Verma et al., ; Lyu et al., ; Cappart et al., ; Kohler et al., ). These methods offer transparency, revisability, better generalization, and the potential for curriculum learning.
However, they often rely on the incorporation of specific human inductive biases, necessary for solving the tasks, thus requiring experts to provide essential concepts or potential logic rules.
Moreover, actions involving low-level reactions with subtle movements are challenging, if not impossible, to encode within these frameworks. This limitation underscores the constraints of symbolic systems in terms of their learning capabilities.
Consequently, an important question emerges: *How can we build intelligent agents that leverage the strengths of both neural and symbolic modeling?*
The current approach to combining these systems typically uses a top-down (*i.e.* sequential) method: using deliberative systems (*e.g.* planners) to provide slow, high-level reasoning to select reactive systems (*e.g.* Deep RL), that offer quick, low-level responses (Kokel et al., ).
However, this sequential approach is not always suitable, *e.g.* , a self-driving can recompute its plan on a low-density highway, but must react quickly in heavy traffic, without replanning.
Agents that can select for either neural or symbolic modeling based on the context are necessary.

We introduce *BlendRL*, a framework that integrates neural and logic-based policy learning in parallel.
BlendRL agents learn logic-based interpretable reasoning as well as low-level control, combined through a blending function that utilizes a hybrid state representation.
They can utilize high-level reasoning (*e.g.* for path finding), which benefits from a symbolic (or object-centric) state representation, and low-level reactions, for finetuned control skills (*e.g.* shooting at enemies), using pixel-based state representations.
Although its neural component is less interpretable, it assists the agent in adapting to situations where it lacks sufficient symbolic representations.
BlendRL provides hybrid policies, by distinctly modeling these two types of information processing systems, and selects its actions using a mixture of deep neural networks and differentiable logic reasoners (Evans & Grefenstette, ; Shindo et al., ; ).
Additionally, we propose an Advantage Actor-Critic (A2C)-based learning algorithm for BlendRL agents, that incorporates Proximal Policy Optimization (PPO) and policy regularization, supplemented by analysis on the interactions between neural and symbolic components of trained agents.
Overall, we make the following contributions:

We propose BlendRL to jointly and simultaneously train symbolic and neural policies.

For efficient learning on the proposed framework, we adapt the PPO Actor Critic algorithm on the hybrid state representation.
Moreover, we proposed a regularization method to balance neural and symbolic policies, providing agents that are both transparent reasoners and accurate reactors.

We empirically show that BlendRL agents outperform neural and the state-of-the-art neuro-symbolic baselines on environments where agents need to perform both high-level reasoning and low-level reacting. Moreover, we demonstrate the robustness of the BlendRL agents to environmental changes.

We provide a deeper analysis of the interactions between neural and symbolic policies, revealing how hybrid representations and policies can help agents overcome each other’s limitations.

We start off by providing the necessary background, then introduce our BlendRL method for policy reasoning and learning. We experimentally evaluate BlendRL on three complex Atari games, comparing its performance to purely neural and logic baselines.
Following this, we discuss related work before concluding.
Our code and resources are openly available.(^1^11 [https://github.com/ml-research/blendrl](https://github.com/ml-research/blendrl))

## 2 Background

Let us introduce the necessary background before formally introducing our BlendRL method.

Deep Reinforcement Learning.
In RL, the task is modelled as a Markov decision process, $\mathcal{M}=<\!\!\mathcal{S},\mathcal{A},P,R,\gamma\!\!>$, where, at every timestep $t$, an agent in a state $s_{t}\in\mathcal{S}$, takes action $a_{t}\in\!\mathcal{A}$, receives a reward $r_{t}=R(s_{t},a_{t})$ and a transition to the next state $s_{t+1}$, according to environment dynamics $P(s_{t+1}|s_{t},a_{t})$.
Deep agents attempt to learn a parametric policy, $\pi_{\theta}(a_{t}|s_{t})$, in order to maximize the return (*i.e.* $\sum_{t}\gamma^{t}r_{t}$, with $\gamma\in[0,1]$).
The desired input to output (*i.e.* state to action) distribution is not directly accessible, as RL agents only observe returns.
The value $V_{\pi_{\theta}}(s_{t})$ (resp. Q-value $Q_{\pi_{\theta}}(s_{t},a_{t})$) function provides the return of the state (resp. state/action pair) following the policy $\pi_{\theta}$. We provide more details in App. [A.1](#A1.SS1).

First-Order Logic (FOL).
In FOL, a Language $\mathcal{L}$ is a tuple $(\mathcal{P},\mathcal{D},\mathcal{F},\mathcal{V})$,
where $\mathcal{P}$ is a set of predicates, $\mathcal{D}$ a set of constants,
$\mathcal{F}$ a set of function symbols (functors), and $\mathcal{V}$ a set of variables.
A term is either a constant (*e.g.* $\mathtt{obj1},\mathtt{agent}$), a variable (*e.g.* $\mathtt{O1}$), or a term which consists of a function symbol.
An atom is a formula ${\tt p(t_{1},\ldots,t_{n})}$, where ${\tt p}$ is a predicate symbol (*e.g.* $\mathtt{closeby}$) and ${\tt t_{1},\ldots,t_{n}}$ are terms.
A ground atom or simply a fact is an atom with no variables (*e.g.* $\mathtt{closeby(obj1,obj2)}$).
A literal is an atom ($A$) or its negation ($\lnot A$).
A clause is a finite disjunction ($\lor$) of literals.
A ground clause is a clause with no variables.
A definite clause is a clause with exactly one positive literal.
If $A,B_{1},\ldots,B_{n}$ are atoms, then $A\lor\lnot B_{1}\lor\ldots\lor\lnot B_{n}$ is a definite clause.
We write definite clauses in the form of $A~{}\mbox{:-}~{}B_{1},\ldots,B_{n}$.
$A$ is the rule head, and the set $\{B_{1},\ldots,B_{n}\}$ is its body.
We interchangeably use definite clauses and *rules*.

Differentiable Forward Reasoning is a data-driven approach to reasoning in First-Order Logic (FOL) (Russell & Norvig, ). In forward reasoning, given a set of facts and rules, new facts are deduced by applying the rules to the facts. Differentiable forward reasoning is a differentiable implementation of forward reasoning, utilizing tensor-based differentiable operations (Evans & Grefenstette, ; Shindo et al., ) or graph-based approaches (Shindo et al., ). This approach can be efficiently applied to reinforcement learning tasks by encoding actions in the form of rules, where the head defines an action and the body specifies its conditions. To learn the importance or truth value of each rule, they can be associated with learnable rule weights. Consequently, hypotheses can be formulated in terms of rules and learned from data.

## 3 BlendRL

BlendRL integrates both abstract reasoning and instinctive reactions by combining symbolic and neural policies.
As illustrated in Figure [2](#S3.F2), the neural policy processes sub-symbolic (*i.e.* pixel-based) representations to compute a distribution over the actions, while the reasoning module employs differentiable reasoning on symbolic states.
These action distributions are then blended to obtain the final action distribution.
We begin by describing the inner workings of each policy type, and of the blending module.
Next, we discuss how we leverage the common sense encapsulated in pretrained large language models (LLMs) to obtain symbolic concepts and their evaluation functions. Finally, we describe how we adapted the PPO actor-critic algorithm to perform end-to-end training of BlendRL modules.
Let us first formally introduce the state representations.

### 3.1 Hybrid State Representations

BlendRL agents employ *two* distinct state representations: (i) *pixel-based representations* and (ii) *object-centric representations*, that can be extracted by object discovery models (Redmon et al., ; Lin et al., ; Delfosse et al., ; Zhao et al., ).
Pixel-based representations usually consist of a stack of raw images, provided by the environment, fed to a deep convolutional network, as introduced by Mnih et al. ().
Our considered symbolic (object-centric) representations consist of a list of objects, with attributes (*e.g.* position, orientation, color, etc.), allowing for logical reasoning on structured representations (Zadaianchuk et al., ; Liu et al., ; Yoon et al., ; Wüst et al., ; Stammer et al., ).

Raw sub-symbolic states, $\bm{x}\in\mathbb{R}^{F\times W\times H\times C}$, consist of the last $F$ observed frames, while symbolic states $\bm{z}\in\mathbb{R}^{F\times n\times m}$ represents the corresponding symbolic (or object-centric) state, with $n$ denoting the number of objects and of $m$ extracted properties.
As illustrated in Figure [2](#S3.F2), on the Seaquest Atari environment, a symbolic state consists of objects such as the agent’s submarine, the oxygen level, a diver, sharks, etc., with their $x$ and $y$ positional coordinates, as well as orientation and value. If a property is not relevant to an object (*e.g.* the orientation for the oxygen bar), it is set to 0 0.

Figure: Figure 2: Overview of the BlendRL framework. BlendRL employs two policy types to compute action probabilities. A deep neural policy (bottom) handles low-level reactions, operating on pixel-based states, while differentiable forward reasoners (Shindo et al., ) (top) manage high-level reasoning on symbolic, object-centric states extracted from the pixel-based input. The blending module then merges the two action distributions by taking a weighted sum based on the current state. The logic policy and the blending module incorporate human inductive biases extracted using language models (LLMs) and the task context. All three components (the two policies and the blending module) can be trained jointly using gradients.
Refer to caption: /html/2410.11689/assets/x2.png

### 3.2 The Blended Neuro-Symbolic Policies

Using both state representations, BlendRL agents compute the action selection probabilities by aggregating the probabilities of its *neural* and its *logic* policies.

The neural policy, $\pi_{\theta}^{\mathit{neural}}:\mathbb{R}^{F\times W\times H\times C}\rightarrow[0,1]^{A}$, consists of a neural network, parameterized by $\theta$, that computes action distributions based on pixel state representations ($\bm{x}$).
Convolutional neural networks based policies (Mnih et al., ; Schulman et al., ; Hessel et al., ) are the most common deep policy types (Huang et al., ), but visual transformers can also be used (Chen et al., ; Parisotto et al., ).

The logic policy, $\pi_{\phi}^{\mathit{logic}}:\mathbb{R}^{n\times m}\rightarrow[0,1]^{A}$, is a differentiable forward reasoner (Shindo et al., ), parameterized by $\phi$, that reasons over object-centric states, such as the one depicted in Figure [2](#S3.F2).
Action rules are rules in first-order logic that encode conditions for an action to be selected (Reiter, ; Delfosse et al., ).
An action rule defines an action as its head atom (*action atom*) and encodes its preconditions in its body atoms (*state atoms*).
We provide exemplary action rules for Seaquest in Listing LABEL:lst:program1.

These rules are transparent, *e.g.* [R1] can be interpreted as ”Select UP if the oxygen is empty”.
The body atom is_empty is a *state predicate*, whose truth level can be computed from an object-centric state.
Each state predicate is associated with a (differentiable) function, known as a *valuation function*, to compute its truth value, or confidence. For example, is_empty can be mapped to a function, such as $\mathit{sigmoid}((x-\alpha)/\gamma)$, which translates the actual oxygen value $x\in[0,100]$ (from the object-centric state) into a truth score ranging from $[0,1]$. The second rule [R2] represents the same action selection, UP, but is motivated by the aim of collecting divers.
The third rule [R3] selects another action (LEFT), if a diver is left of the player.
After evaluating the valuation functions for each state predicate, we perform differentiable forward reasoning (Shindo et al., ) to deduce action atoms defined by the action rules based on the state atoms.
Forward reasoning involves inferring all deducible knowledge (*i.e.* the head atoms for actions) from an observed state.
This process allows us to obtain confidence levels (as probabilities) for all actions defined in the symbolic policy.

Contrary to NUDGE policies (Delfosse et al., ), BlendRL generates action rules and their necessary elements (predicates and their valuation functions) using a Large Language Model (LLM), as described in Section [3.3](#S3.SS3).
We also integrate memory-efficient message-passing forward reasoners (Shindo et al., ) to overcome potential memory bottlenecks associated with conventional symbolic policies.
NUDGE and other common logic policies consumes memory quadratically with respect to the number of relations and entities, significantly limiting their scalability.
In contrast, BlendRL’s symbolic policy scales linearly, suitable to more complex environments scalable training parallelization.

Figure: Figure 3: LLMs allow for generating transparent blending function, Blending exemplified on Kangaroo. Top: An LLM generates a transparent blending function that evaluates if (i) there exists an enemy close to the player and if (ii) nothing is around the player. Bottom: Two states are evaluated: the body predicates (*i.e.* nothing_around and closeby) values are multiplied with the rules’ associated learnable weight (displayed on their left), to obtain the neural/logic importance weighting.
Refer to caption: /html/2410.11689/assets/x3.png

The blending module is a differentiable function parameterized by $\lambda$.
It can be encoded as an explicit logic-based function ($B_{\lambda}:\mathbb{R}^{F\times n\times m}\rightarrow[0,1]$), as an implicit neural network-based (*i.e.* pixel) state evaluator ($B_{\lambda}:\mathbb{R}^{F\times W\times H\times C}\rightarrow[0,1]$), or a combination of both.
While a logic-based policy provides the advantage of transparency, it relies on inductive biases.
If these are not available, a neural blender is necessary.

Figure [3](#S3.F3) describes the overall process of the blending module.
It computes distributions over neural and logic policies given symbolic states, based on the blending rules generated by LLMs.
The blending weighted rule set of an agent trained on Kangaroo is depicted in the top right corner.
It encodes the fact that the neural module should be selected when a monkey or a deadly thrown coconut is around (to allow for dodging the coconut or adjusting the position to optimally punch the monkey). When nothing is around the agent, it can safely rely on its logic policy (depicted just above it), which allows it to navigate the Kangaroo environment.
Formally, the blending module provides the weight $\beta\in[0,1]$ of the neural policy, $\beta=B_{\lambda}(\bm{z})$ or $\beta=B_{\lambda}(\bm{x})$, (and thus implicitly of the neural policy).
We empirically show on Kangaroo and Seaquest BlendRL agents with symbolic blending modules outperform ones with neural modules (*cf.* Table LABEL:tab:blending_comparison in Appendix LABEL:app:ablation).

Finally, given a raw state $\bm{x}$, and its symbolic version, $\bm{z}$, the final action distribution is computed as:

$$ $\displaystyle\pi_{(\theta,\phi,\lambda)}(s)$ $\displaystyle=\beta\cdot\pi^{\mathit{neural}}_{\theta}(\bm{x})+(1-\beta)\cdot\pi^{\mathit{logic}}_{\phi}(\bm{z}),$ (1) $$

where $s=(\bm{x},\bm{z})$.
Note that the blending module can also become a rigid selector, if replaced by $\mathbbm{1}_{\beta>0.5}$.

Figure: Figure 4: BlendRL provides interpretable policies and blending modules as sets of weighted first-order logic rules. A subset of the policy’s weighted action rules and the blending module’s rules from a BlendRL agent trained on Kangaroo. All the logic rule sets for each environment are provided in Appendix [A.4](#A1.SS4).

### 3.3 LLM generated logic policies

BlendRL employs Language Models (LLMs) to generate symbolic programs for precise reasoning, following a chain of thought principle (Wei et al., ; Kojima et al., ):

it uses the task context and detectable objects description and implementation to create state predicates,

it formulates action rules, with conjunctions of the generated predicates as body,

it generates the predicates’ *valuations* (*i.e.* their python implemented functions).

For steps (ii) and (iii), we used the few-shot prompting approach,
providing the LLM with an example logic ruleset obtained from NUDGE policies (Delfosse et al., ).
This circumvents the need for an expert to provide the set of logic rules, the used logic predicates, and their implementations, thus allowing users to effectively introduce inductive biases in natural language (*cf.* Appendix [A.3](#A1.SS3) and [A.4](#A1.SS4) for additional details).
A subset of the logic module of a BlendRL trained on Kangaroo is provided in Figure [4](#S3.F4).
The associated weights of the LLM-generated rules have been adjusted to maximize performance.

### 3.4 Optimization

We use the PPO actor-critic to train BlendRL agents, and also employ hybrid value functions.
We compose the hybrid critics by computing values using both visual and object-centric states.

Mixed Value Function.
As the value function approximates the expected return, we did not use logic to encode it.
However, BlendRL incorporates a hybrid value function, that uses both the subsymbolic ($\bm{x}$) and symbolic (or object-centric $\bm{z}$) state representations.
Given state $s=(\bm{x},\bm{z})$, the value is defined as:

$$ $\displaystyle V_{(\mu,\omega)}(s)=\beta\cdot V^{\mathit{CNN}}_{\mu}(\bm{x})+(1-\beta)\cdot V^{\mathit{OC}}_{\omega}(\bm{z}).$ (2) $$

with $V^{\mathit{CNN}}_{\mu}$ a CNN, that share its convolutional layers with the neural actor (Schulman et al., ) and $V^{\mathit{OC}}_{\omega}$ a small MLP on the object-centric state,
and $\beta=B_{\lambda}(\bm{x},\bm{z})$, the blending weight.

Regularization.
To ensure the use of both neural and logic policies, we introduce a regularization term for BlendRL agents. To further enhance exploration, we penalize overly peaked blending distributions. This approach ensures that agents utilize both neural and logic policies without deactivating either entirely.

$$ $\displaystyle R=-\beta\log\beta-(1-\beta)\log(1-\beta)$ (3) $$

This helps agents avoid suboptimal policies typically caused by neural policies due to the sparse rewards.

Figure: Figure 5: BlendRL surpasses both purely neural and logic-based policies in environments requiring reasoning and reaction capabilities, shown by its superior episodic returns after training (averaged over $3$ seeded reruns). BlendRL outperforms the logic-based state-of-the-art agent (*i.e.* NUDGE) across all tested environments. The purely neural PPO agents frequently get trapped in suboptimal policies, BlendRL leverages its integrated reasoning and reacting modules to tackle every aspect of the tasks.
Refer to caption: /html/2410.11689/assets/x4.png

## 4 Experiments

We outline the benefits of BlendRL over purely neural or symbolic approaches, supported by additional investigations into BlendRL’s robustness to environmental changes. Furthermore, we examine the interactions between neural and symbolic components and demonstrate that BlendRL can generate faithful explanations.
We specifically aim to answer the following research questions:
(Q1) Can BlendeRL agents overcome both symbolic and neural agents’ shortcomings?
(Q2) Can BlendRL can produce both neural and symbolic explanations for its action selection?
(Q3) Are BlendRL agents robust to environmental changes?
(Q4) How do the neural and symbolic modules interact to maximize BlendRL’s agents overall performances?

Let us now provide empirical evidence for BlendRL’s ability to learn efficient and understandable policies, even without being provided with all the necessary priors to optimally solve the tasks.

### 4.1 Experimental Setup

Environments.
We evaluate BlendRL on Atari Learning Environments (Bellemare et al., ), the most popular benchmark for RL (particularly for relational reasoning tasks).
For resource efficiency, we use the object centric extraction module of (Delfosse et al., ).
Specifically, in *Kangaroo*, the agent needs to reach and climb ladders, leading to the captive joey, while punching incoming monkey antagonists, that try to stop it.
In *Seaquest*, the agent has to rescue divers, while shooting sharks and enemy submarines.
It also needs to surface before the oxygen runs out.
Finally, in *Donkey Kong* the agent has to reach the princess at the top, while avoiding incoming barrels thrown by Donkey Kong.
For additional details, *cf.* Appendix LABEL:app:env_details.
To further test BlendRL abilities to overcome the potential lack of concept necessary to solve task, we omitted some part of the game in our prompt for the LLM that generates the policy rules.
Specifically, we kept out the facts that agents can punch the monkeys in Kangaroo, can shoot the enemies in Seaquest and can jump over the barrels in DonkeyKong.
To test robustness and isolate different abilities of the agents, we employ HackAtari (Delfosse et al., ), that allow to customize the environments (*e.g.* remove enemies).
For these ablation studies, we provide details about the use modification at the relevant point of the manuscript.

Baselines.
We compare BlendRL to purely neural PPO agents. Both agent types incorporate the classic CNN used for Atari environments.
Additionally, we evaluate NUDGE, that uses a pretrained neural PPO agent for searching viable policy rules (Delfosse et al., ), SOTA among logic agents on Atari tasks.
We train each agent types until all of them converge to a stable episodic return (*i.e.* for $15$K episodes for Kangaroo and DonkeyKong and $25$K for Seaquest).
For additional details, *cf.* Appendix LABEL:app:training_details.

### 4.2 Results and Analysis

Comparison to neural and neuro-symbolic agents (Q1).
Figure [5](#S3.F5) illustrates the episodic returns of the BlendRL, NUDGE, and NeuralPPO agents across various Atari environments.
BlendRL surpasses the logic-based state-of-the-art NUDGE baseline (Delfosse et al., ) in all tested scenarios.
In the Kangaroo environment, which requires fewer intuitive actions due to the relatively small number of enemies, NUDGE shows fair performance, even if its are less able to punch the monkeys and avoid their thrown coconuts.
However, in the other environments, populated with more incoming threats, where neural policy becomes critical for accurate controls, NUDGE lags behind.
Additionally, the purely neural PPO agents often fall into suboptimal policies.
For instance, in Seaquest, surfacing without collecting any divers lead to negative reward. Neural PPO agents thus concentrate on shooting sharks to collect reward, but never refill their oxygen.
In contrast, BlendRL effectively selects its logic module to collect divers and surface when needed and its neural module to efficiently align itself with the enemies and shoot them.
Overall, BlendRL significantly outperforms both baselines across different environments, underscoring the efficacy of neuro-symbolic blending policies in efficiently harnessing both neural and symbolic approaches to policy learning.

BlendRL agents are interpretable and explainable (Q2).
BlendRL’s symbolic policies can easily be interpreted as they consist of a set of transparent symbolic weighted rules, as exemplified in Figure [4](#S3.F4) for Kangaroo.
The interpretable blending module prioritizes neural agents in situations where finetuned control is needed (*i.e.* for dodging an incoming deadly coconut or accurately placing oneself next to the monkey and punching it).
Conversely, the logic module is in use when no immediate danger is present, *e.g.* as a path finding proxy. The logic rules for the other environments are provided in Appendix [A.4](#A1.SS4).
BlendRL also produces gradient-based explanations, as each component is differentiable.
Logic-based explanations are computed using action gradients (*i.e.* $\partial\pi^{logic}_{\phi}(\mathbf{z})/\partial\mathbf{z}$) on the state atoms, that underline the most important object properties for the decision.
Further, the integrated gradients method (Sundararajan et al., ) on the neural module provides importance maps.
We can visualize these two explanations, merging them with the blending weight $\beta$,
as shown in Figure [6](#S4.F6). The most important logic rules for the decision are also depicted.
BlendRL provides interpretable rules for it reasoning part and importance maps.

Figure: Figure 6: Qualitative examples of explanations produced by BlendRL, in the form of importance maps, computing with gradient-based attributions for both neural and logic policies. The logic-based module also allows BlendRL to generate textual interpretations, enhancing its transparency and interpretability.
Refer to caption: /html/2410.11689/assets/x5.png

**Table 1: Contrary to neural PPO, BlendRL is robust to environmental changes. Average returns on the modified environments (over $10$ seeds).**
| Return | Kangaroo | Seaquest | DonkeyKong |
| --- | --- | --- | --- |
| Neural PPO | $0.0_{\pm 0.0}$ | $0.0_{\pm 0.0}$ | $0.0_{\pm 0.0}$ |
| BlendRL | $\mathbf{187_{\pm 120}}$ | $\mathbf{113_{\pm 106}}$ | $\mathbf{263_{\pm 124}}$ |

BlendRL is robust to environmental changes (Q3).
Deep agents usually learn “by heart” spurious correlations during training, and thus fail to generalize to unseen environments, even on simple Atari games (Farebrother et al., ; Delfosse et al., ).
We used HackAtari variations of the environment, to disable the enemies in Kangaroo and Seaquest and to disable the barrels in DonkeyKong.
We additionally used a modified Kangaroo environment with relocated ladders.

As expected, our BlendRL agent can still complete the tasks (and thus collect reward) on these safer versions of the environments (*cf.* Table [1](#S4.T1)).
BlendRL agents indeed rely on the knowledge incorporated in their logic-based policies and blending modules, and only rely on their neural modules for control accurate skills (*e.g.* aiming and shooting/punching), as we further show hereafter.

Figure: Figure 7: BlendRL uses both its neural and logic modules. Through an episode, the highest action probability is depicted (left, top), with the selected module at this step (bottom). BlendRL makes use of both module types through an episode. It progressively modifies their selection ratio along training on Seaquest (right), as its progress allows it to uncover parts of the game with more enemies.
Refer to caption: /html/2410.11689/assets/x6.png

Neural and symbolic policy interactions in BlendRL (Q4).
We here investigate how the symbolic and neural policies interoperate. One concern of such an hybrid systems is its potential reliance on only one module.
The system could indeed learn to *e.g.* only rely on its neural component (*i.e.* $\beta\approx 1.0$).
Figure [7](#S4.F7) (left and center) illustrates the outputs of BlendRL’s neural and logic policies for $1k$ steps (*i.e.* $1$-$3$ episodes).
Specifically, the maximum value of the action distributions ($\max_{a}\pi^{logic}_{\phi}(a|s_{t})$ and $\max_{a}\pi^{neural}_{\theta}(a|s_{t})$), at each time step $t$ is depicted (at the top), underlined by the importance weighting of each module (blue if logic is mainly selected ($\beta\approx 0$), red if neural is ($\beta\approx 1$)), indicating how much each module is used at each step on Kangaroo and Seaquest.
Further, BlendRL’s agents can adapt the amount to which they select each component through training, as depicted on the right of this figure. As Seaquest is a progressive environment, in which the agent first faces few enemies (thus mainly relying on its logic module to collect the divers), before progressively accessing states in which many more enemies are spawning (*cf.* Figure LABEL:fig:Seaquest_evolution), BlendRL agents first mainly relies on its logic module (blue), while progressively shifting this preference to its neural one (red) for accurately shooting enemies. These results demonstrate the efficacy of BlendRL’s policy reasoning and learning on neuro-symbolic hybrid representations, thereby enhancing overall performance. We provide a further analysis comparing neural and logic blending modules in Appendix LABEL:app:ablation, highlighting that the logic-based blending module can utilize both policies effectively, resulting in better performance.

Overall, our experimental evaluation has shown BlendRL’s agents ability to learn on several Atari environments that require both reasoning and reacting capabilities.
We showed that they outperform the commonly used neural PPO baseline, as well as the state-of-the-art logic agents, NUDGE.
We further demonstrated their ability to generalize to unseen scenarios, on slightly modified environments (compared to the training ones), and that they can efficiently alternate between the two module types to obtain policies that can both produce interpretations, quantifying the impact of each symbolic properties and of each pixel region.

## 5 Related Work

Relational Reinforcement Learning (Relational RL) (Dzeroski et al., ; Kersting et al., ; Kersting & Driessens, ; Lang et al., ; Hazra & Raedt, ) has been developed to address RL tasks in relational domains by incorporating logical representations and probabilistic reasoning. BlendRL extends this approach by integrating differentiable logic programming with deep neural policies. The Neural Logic Reinforcement Learning (NLRL) framework (Jiang & Luo, ) was the first to incorporate Differentiable Inductive Logic Programming ($\partial$ILP) (Evans & Grefenstette, ) into the RL domain. $\partial$ILP learns generalized logic rules from examples using gradient-based optimization. NUDGE (Delfosse et al., ) extends this method by introducing neurally-guided symbolic abstraction, leveraging extensive work on $\partial$ILP (Shindo et al., ; ) to learn complex programs. INSIGHT (Luo et al., ) is another neuro-symbolic framework that jointly learns structured states and symbolic policies, producing textual explanations. Unlike these approaches, which rely solely on symbolic policy representations, BlendRL integrates both neural and symbolic policies and trains them jointly.

The integration of planners with RL has been explored to achieve deliberate thinking in policy learning. For example, RePReL (Kokel et al., ) decomposes multi-agent planning tasks using a planner and then employs reinforcement learning for each agent to solve subtasks. In these frameworks, planners for long-term (slow) reasoning are typically separate components. In contrast, BlendRL computes both symbolic and neural policies at the same level, allowing them to be learned jointly, thereby enhancing overall performance. Additionally, planners are often used in model-based RL to generate hypothetical experiences that improve value estimates (Sutton, ; Kaiser et al., ). In contrast, BlendRL incorporates the symbolic reasoning module directly into its policy, enabling joint learning with neural modules.

Additionally, BlendRL is related to prior work using LLMs for program generation.
For example, LLMs have been applied to generate probabilistic programs (Wong et al., ), Answer Set Programs (Ishay et al., ; Yang et al., ), differentiable logic programs (Shindo et al., ), and programs for visual reasoning (Surís et al., ; Stanić et al., ).
Our symbolic policy representation is inspired by *situation calculus* (Reiter, ), which is an established framework to describe states and actions in logic.

## 6 Conclusion

In this study, we introduced BlendRL, a pioneering framework that integrates symbolic and neural policies for reinforcement learning. BlendRL employs neural networks for reactive actions and differentiable logic reasoners for high-level reasoning, seamlessly merging them through a blending module that manages distributions over both policy types.
We also developed a learning algorithm for BlendRL agents that hybridizes the state-value function on both pixel-based and object-centric states and includes a regularization approach to enhance the efficacy of both logic and neural policies.

Our empirical evaluations demonstrate that BlendRL agents significantly outperform purely neural agents and state-of-the-art neuro-symbolic baselines in popular Atari environments. Furthermore, these agents exhibit robustness to environmental changes and are capable of generating clear, interpretable explanations across various types of reasoning, effectively addressing the limitations of purely neural policies. Our comprehensive analysis of the interactions between symbolic and neural policy representations highlights their synergistic potential to enhance overall performance.

Acknowledgements.
The authors thank Sriraam Natarajan for his valuable feedback on the manuscript. This research work was partly funded by the German Federal Ministry of Education and Research, the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK)
within their joint support of the National Research Center for Applied Cybersecurity ATHENE, via the “SenPai: XReLeaS” project. It also benefited from the HMWK Cluster Project 3AI.
We gratefully acknowledge support by the EU ICT-48 Network of AI Research Excellence Center “TAILOR” (EU Horizon 2020, GA No 952215), and the Collaboration Lab “AI in Construction” (AICO).
The Eindhoven University of Technology authors received support from their Department of Mathematics and Computer Science and the Eindhoven Artificial Intelligence Systems Institute.

## Appendix A Appendix

### A.1 Deitailed Background of Reinforcement Learning

We provide more detailed background for reinforcement learning.
Policy-based methods directly optimize $\pi_{\theta}$ using the noisy return signal, leading to potentially unstable learning.
Value-based methods learn to approximate the value functions $\hat{V}_{\phi}$ or $\hat{Q}_{\phi}$, and implicitly encode the policy, *e.g.* by selecting the actions with the highest Q-value with a high probability (Mnih et al., ).
To reduce the variance of the estimated Q-value function, one can learn the advantage function $\hat{A}_{\phi}(s_{t},a_{t})=\hat{Q}_{\phi}(s_{t},a_{t})-\hat{V}_{\phi}(s_{t})$.
An estimate of the advantage function can be computed as
$\hat{A}_{\phi}(s_{t},a_{t})=\sum_{i=0}^{k-1}\gamma^{i}r_{t+i}+\gamma^{k}\hat{V}_{\phi}(s_{t+k})-\hat{V}_{\phi}(s_{t})$ (Mnih et al., ).
The Advantage Actor-critic (A2C) methods both encode the policy $\pi_{\theta}$ (*i.e.* actor) and the advantage function $\hat{A}_{\phi}$ (*i.e.* critic), and use the critic to provide feedback to the actor, as in (Konda & Tsitsiklis, ).
To push $\pi_{\theta}$ to take actions that lead to higher returns, gradient ascent can be applied to $L^{PG}(\theta)=\hat{\mathbb{E}}[\log\pi_{\theta}(a\mid s)\hat{A}_{\phi}]$.
Proximal Policy
Optimization (PPO) algorithms ensure minor policy updates that avoid catastrophic drops (Schulman et al., ), and can be applied to actor-critic methods.
To do so, the main objective constraints the policy ratio $r(\theta)=\frac{\pi_{\theta}(a\mid s)}{\pi_{\theta_{\text{old }}}(a\mid s)}$, following $L^{PR}(\theta)=\hat{\mathbb{E}}[\min(r(\theta)\hat{A}_{\phi},\operatorname{clip}(r(\theta),1-\epsilon,1+\epsilon)\hat{A}_{\phi})]$, where $\operatorname{clip}$ constrains the input within $[1-\epsilon,1+\epsilon]$.
PPO actor-critic algorithm’s global objective is $L(\theta,\phi)=\hat{\mathbb{E}}[L^{PR}(\theta)-c_{1}L^{VF}(\phi)]$,
with $L^{VF}(\phi)\!=\!(\hat{V}_{\phi}(s_{t})-V(s_{t}))^{2}$ being the value function loss.
An entropy term can also be added to this objective to encourage exploration.

### A.2 Details of Differentiable Forward Reasoning

We provide the details of differentiable forward reasoning.

###### Definition A.1

A Forward Reasoning Graph is a bipartite directed graph $(\mathcal{V}_{\mathcal{G}},\mathcal{V}_{\land},\mathcal{E}_{\mathcal{G}\rightarrow\land},\mathcal{E}_{\land\rightarrow\mathcal{G}})$, where $\mathcal{V}_{\mathcal{G}}$ is a set of nodes representing ground atoms (atom nodes), $\mathcal{V}_{\land}$ is set of nodes representing conjunctions (conjunction nodes), $\mathcal{E}_{\mathcal{G}\rightarrow\land}$ is set of edges from atom to conjunction nodes and $\mathcal{E}_{\land\rightarrow\mathcal{G}}$ is a set of edges from conjunction to atom nodes.

BlendRL performs forward-chaining reasoning by passing messages on the reasoning graph.
Essentially, forward reasoning consists of *two* steps: (1) computing conjunctions of body atoms for each rule and (2) computing disjunctions for head atoms deduced by different rules. These two steps can be efficiently computed on bi-directional message-passing on the forward reasoning graph.
We now describe each step in detail.

(Direction $\rightarrow$) From Atom to Conjunction.
First, messages are passed to the conjunction nodes from atom nodes.
For conjunction node $\mathsf{v_{i}}\in\mathcal{V}_{\land}$, the node features are updated:

$$ $\displaystyle v_{i}^{(t+1)}=\bigvee\left(v_{i}^{(t)},\bigwedge\nolimits_{j\in\mathcal{N}(i)}v_{j}^{(t)}\right),$ (4) $$

where $\bigwedge$ is a soft implementation of *conjunction*, and $\bigvee$ is a soft implementation of *disjunction*.
Intuitively, probabilistic truth values for bodies of all ground rules are computed softly by Eq. [4](#A1.E4).

(Direction $\leftarrow$) From Conjunction to Atom.
Following the first message passing, the atom nodes are then updated using the messages from conjunction nodes.
For atom node $\mathsf{v_{i}}\in\mathcal{V}_{\mathcal{G}}$, the node features are updated:

$$ $\displaystyle v_{i}^{(t+1)}=\bigvee\left(v_{i}^{(t)},\bigvee\nolimits_{j\in\mathcal{N}(i)}w_{ji}\cdot v_{j}^{(t)}\right),$ (5) $$

where $w_{ji}$ is a weight of edge $e_{j\rightarrow i}$.
We assume that each rule $C_{k}\in\mathcal{C}$ has its weight $\theta_{k}$, and $w_{ji}=\theta_{k}$ if edge $e_{j\rightarrow i}$ on the reasoning graph is produced by rule $C_{k}$.
Intuitively, in Eq. [5](#A1.E5), new atoms are deduced by gathering values from different ground rules and from the previous step.

We used product for conjunction, and *log-sum-exp* function for disjunction:

$$ $\displaystyle\mathit{softor}^{\gamma}(x_{1},\ldots,x_{n})=\gamma\log\sum_{1\leq i\leq n}\exp(x_{i}/\gamma),$ (6) $$

where $\gamma>0$ is a smooth parameter. Eq. [6](#A1.E6) approximates the maximum value given input $x_{1},\ldots,x_{n}$.

Prediction.
The probabilistic logical entailment is computed by the bi-directional message-passing.
Let $\mathbf{x}_{\mathit{atoms}}^{(0)}\in[0,1]^{|\mathcal{G}|}$ be input node features, which map a fact to a scalar value, $\mathsf{RG}$ be the reasoning graph, $\mathbf{w}$ be the rule weights, $\mathcal{B}$ be background knowledge, and $T\in\mathbb{N}$ be the infer step.
For fact $G_{i}\in\mathcal{G}$, BlendRL computes the probability as:

$$ $\displaystyle p(G_{i}~{}|~{}\mathbf{x}_{\mathit{atoms}}^{(0)},\mathsf{RG},\mathbf{w},\mathcal{B},T)=\mathbf{x}^{(T)}_{\mathit{atoms}}[i],$ (7) $$

where $\mathbf{x}_{\mathit{atoms}}^{(T)}\in[0,1]^{|\mathcal{G}|}$ is the node features of atom nodes after $T$-steps of the bi-directional message-passing.

By optimizing the cross-entropy loss, the differentiable forward reasoner can solve Inductive Logic Programming (ILP) problems with propositional encoding (Shindo et al., ), where the task is to find classification rules given positive and negative examples.
It has been successfully applied to complex visual scenes (Helff et al., ), meta-level reasoning and learning (Ye et al., ), predicate invention (Sha et al., ; ), and self-explanatory learning (Stammer et al., ).

Figure: Figure 8: Forward reasoning graph for rules in Listing LABEL:lst:program1. A reasoning graph consists of *atom nodes* and *conjunction nodes*, and is obtained by grounding rules *i.e.* , removing variables by, *e.g.* , $\mathtt{X}\leftarrow\mathtt{obj1}$, $\mathtt{Y}\leftarrow\mathtt{obj2}$. By performing bi-directional message passing on the reasoning graph using soft-logic operations, BlendRL computes logical consequences in a differentiable manner. Only relevant nodes are shown (Best viewed in color).
Refer to caption: /html/2410.11689/assets/x7.png

###### Definition A.1

### A.3 Body predicates and their valuations

We here provide examples of valuation functions for evaluating state predicates (*e.g.* closeby, left_of, etc.) generated by LLMs (in Python).

[
mathescape,
language=Python,
]
# A subset of valuation functions for Kangaroo and DonkeyKong (generated by LLMs)
def left_of(player: th.Tensor, obj: th.Tensor) -¿ th.Tensor:
x = player[…, 1]
obj_x = obj[…, 1]
obj_prob = obj[:, 0] # objectness
return sigmoid(alpha ¡ obj_x - x) * obj_prob * same_level_ladder(player, obj)

def _close_by(player: th.Tensor, obj: th.Tensor) -¿ th.Tensor:
player_x = player[:, 1]
player_y = player[:, 2]
obj_x = obj[:, 1]
obj_y = obj[:, 2]
obj_prob = obj[:, 0] # objectness
x_dist = (player_x - obj_x).pow(2)
y_dist = (player_y - obj_y).pow(2)
dist = (x_dist + y_dist).sqrt()
return sigmoid(dist) * obj_prob

def on_ladder(player: th.Tensor, obj: th.Tensor) -¿ th.Tensor:
player_x = player[…, 1]
obj_x = obj[…, 1]
return sigmoid(abs(player_x - obj_x) ¡ gamma)
…

# A subset of valuation functions for Seaquest (generated by LLMs)
def full_divers(objs: th.Tensor) -¿ th.Tensor:
divers_vs = objs[:, -6:]
num_collected_divers = th.sum(divers_vs[:,:,0], dim=1)
diff = 6 - num_collected_divers
return sigmoid(1 / diff)

def not_full_divers(objs: th.Tensor) -¿ th.Tensor:
return 1 - full_divers(objs)

def above(player: th.Tensor, obj: th.Tensor) -¿ th.Tensor:
player_y = player[…, 2]
obj_y = obj[…, 2]
obj_prob = obj[:, 0]
return sigmoid( (player_y - obj_y) / gamma) * obj_prob
…

### A.4 BlendRL Rules

We here provide the blending and action rules obtained by BlendRL on Seaquest and DonkeyKong.
{lstlisting}[
mathescape,
language=Prolog,
style=Prolog-pygsty,
]
0.98 neural_agent(X):-close_by_enemy(P,E).
0.74 neural_agent(X):-close_by_missile(P,M).
0.02 logic_agent(X):-visible_diver(D).
0.02 logic_agent(X):-oxygen_low(B).
1.0 logic_agent(X):-full_divers(X).
0.21 up_air(X):-oxygen_low(B).
0.22 up_rescue(X):-full_divers(X).
0.21 left_to_diver(X):-right_of_diver(P,D),visible_diver(D).
0.24 right_to_diver(X):-left_of_diver(P,D),visible_diver(D).
0.23 up_to_diver(X):-deeper_than_diver(P,D),visible_diver(D).
0.22 down_to_diver(X):-higher_than

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 5
presentation: 2
questions: see weakness
soundness: 1
strengths: 1. The paper is clearly written and easy to follow.
2. The utilization of the language model shows a certain level of innovation.
summary: This paper integrates condition-based logic decisions with neural network-based reinforcement learning policies through an LLM-based hybrid module to address the shortcomings of both approaches. It achieved better results in three Atari games compared to standard PPO and logic-based method.
weaknesses: 1. The overall concept is not particularly novel, with numerous similar works, such as the well-known fast and slow systems, already existing.
2. The paper and appendix lack crucial details on how the LLM generates rules and calculates hybrid probabilities, and to what extent this is based on the content provided in the prompts. This is essential for determining whether the method can generalize to more diverse tasks.
3. The paper consistently emphasizes complex tasks, yet the experimental environment is not particularly sophisticated. Atari is a basic, outdated, and relatively simple benchmark that has largely been mastered. The three selected tasks are not among the recognized challenging ones in Atari (indeed, they are relatively simple). Truly difficult tasks, such as Montezuma’s Revenge, would be more appropriate as experimental environments.
4. The experiments should compare against more advanced RL algorithms. For example, Agent57 has already achieved a score of 30k on Kangaroo (while blend RL scores less than 20k) and 1000k on Sequest (while blend RL scores less than 5k). Therefore, the current experimental results do not demonstrate the superiority of the method.
5. Overall, the experiments lean towards simpler explorations and lack ablation studies that would reveal the characteristics of the method. For instance, I couldn't find which LLM was used, nor were there detailed experiments on the impact of object-centric representations on the method.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses above

--- Post rebuttal --- 

After reading Author's responses and the updated paper I have no more concerns. I believe this will be an interesting contribution for those working with neuro-symbolic approaches.
soundness: 2
strengths: The paper is very well written and easy to follow. The proposed framework is quite novel to the best of my knowdledge, since the level 2 and 1 systems are usually much more separated than in BlendRL, and the results against vanilla PPO or a symbolic approach (NUDGE) are promising.
summary: The paper presents BlendRL, a neuro-symbolic RL framework that allows agents to blend resoning from pixel and object level representations. Unlike previous  neuro-symbolic RL approach BlendRL learns concurrently it's low and haigh level policies. Specifically, a blending module, informed by rules generated by a language model, dynamically determines the optimal mix of neural and symbolic policies based on the task context. 

Authors test BlendRL in Seaquest and Kangaroo (from Atari), where agents must alternate between quick responses and logical planning, results show that BlendRL consistently outperforms purely neural or symbolic agents.
weaknesses: My biggest concern with this work is the lack of empirical comparison with any other neuro-symbolic baselines, e.g. [1-4] 

Since BlendRL also weights heavily on object-based representations and relational learning it would have been good (although I don't consider this critical)  to include some contrast with deep learning approaches for such kind of learning, e.g. [1, 5-6].

While the point above will raise my confidence on the paper, I still thinkt hat the novelty of the approach and the results included outweight the weakpoints.

[1] Borja G. León, Murray Shanahan, and Francesco Belardinelli. In a nutshell, the human asked for this: Latent
goals for following temporal specifications. In International Conference on Learning Representations, 2022.

[2] Kuo, Yen-Ling, Boris Katz, and Andrei Barbu. "Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.

[3] Vaezipoor, Pashootan, et al. "Ltl2action: Generalizing ltl instructions for multi-task rl." International Conference on Machine Learning. PMLR, 2021.

[4] Qiu, Wenjie, Wensen Mao, and He Zhu. "Instructing goal-conditioned reinforcement learning agents with temporal logic objectives." Advances in Neural Information Processing Systems 36 (2024).

[5] Shanahan, Murray, et al. "An explicitly relational neural network architecture." International Conference on Machine Learning. PMLR, 2020.

[6] Feng, Fan, and Sara Magliacane. "Learning dynamic attribute-factored world models for efficient multi-object reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1) If I wanted to apply it to a new game, what should be done to generate the action rules (Section 3.3)? 

2) Does using the NUDGE policies to generate examples for the LLM provide a strong inductive bias to BlendRL?

3) How were the hyperparameters obtained?

4) How did you choose the three Atari games? Why did you use different ones compared to those used in NUDGE?
soundness: 3
strengths: The idea of learning a mixture of neural policy and symbolic policy is novel. Given the presented experimental results, it seems to lead to nicely interpretable decision rules (at least in the symbolic part). Also, surprisingly to me, it seems that it automatically learns to make "reflex" decisions preferably using the neural policy. 

The overall proposition nicely integrates several previous techniques to offer an end-to-end method, with little human inputs.
summary: The paper proposed to jointly learn a mixture of neural policy and symbolic policy (represented in a differentiable way) via reinforcement learning. In the experiments, the neural policy is simply a CNN-based policy, the differentiable symbolic policy is based on a method called differentiable forward reasoner, and the weights of the mixture can themselves be seen as the output of a trainable policy. The proposed method is tested on three Atari games.
weaknesses: The paper should be more self-contained. For instance, this work builds on several existing propositions (notably by Shindo et al. and Delfosse et al.). The authors should recall and provide more details of those previous works to help the reader more easily understand BlendRL and appreciate its novelty. For instance, I think the current presentation of differentable forward reasoner is too light, e.g., how is the set \mathcal C related to the forward reasoning graph? What's the advantages/disadvantages of this representation using a graph compared to other frameworks, e.g., dILP?
Also, Section 3.3,  which seems to me to be a key component that can have a huge impact on BlendRL, is lacking in details. For instance, I suggest the authors to provide more information about step (i). In addition, doesn't using the NUDGE policies to generate examples for the LLM provide a strong inductive bias to BlendRL?

The experimental validation is missing some details and maybe a bit light. The authors should explain how they chose their hyperparameters (e.g., Entropy coefficient for blending regularization). Since BlendRL requires a more complex policy model than Neural PPO or NUDGE), I believe it's possible via (potentially costly) hyperparameter tuning to obtain better results for BlendRL.
The authors only evaluates on three Atari games, which are moreover different from those used in NUDGE. Why is that so? Would BlendRL still outperform NUDGE on the tasks where it has demonstrated a good performance?

Although I haven't followed closely the latest developments in neuro-symbolic approaches, I believe that the literature in this direction is quite rich. For instance, I think that there are other works trying to combine System 1 and System 2 capabilities in AI agents and there are other neuro-symbolic RL approaches. The paper could be improved by better situating BlendRL in the existing literature. 

Minor points:
There are a number of typos that should be corrected, e.g.,
- line 107: I believe that function symbols are not used in this work. This should be clarified.
- line 162: What's called "state" actually corresponds to "observation"
- line 177: Regarding the dimension of the logic policy, should F appear?
- lines 852-855: the learning rates are missing?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Figure 6 references textual interpretation; could you kindly provide a more detailed explanation?
2. In cases where the logic chain becomes quite long, would the policy still be considered explainable?
soundness: 3
strengths: The paper is clearly written, and the proposed framework is compelling in its attempt to bridge the gap between symbolic reasoning and neural policy learning. The dual use of symbolic logic for high-level reasoning and deep neural networks for low-level reaction is well-motivated and timely. The experiments convincingly show that BlendRL agents perform better than existing methods, particularly in environments that demand both reasoning and quick reflexes. For instance, in Seaquest, the neural policy was able to effectively manage shooting enemies, while the symbolic component handled pathfinding and resource collection, demonstrating how the hybrid model synergistically enhances performance.
summary: The paper presents BlendRL, a neuro-symbolic reinforcement learning framework that integrates symbolic reasoning and neural policies, aiming to improve agent performance by mixing neural and symbolic policies. The authors demonstrate BlendRL’s efficacy in classic Atari games, highlighting that the framework outperforms purely neural baselines and neuro-symbolic baselines like NUDGE.
weaknesses: Overall, I think this work will have a positive impact on the community, but I still have some concerns:
1. The paper evaluates the proposed method in a limited set of experimental environments. It would strengthen the validation of the method if the authors could evaluate BlendRL in a wider variety of environments, especially those with different characteristics (such as the environments BeamRiderNoFrameskip-v4, EnduroNoFrameskip-v4, and QbertNoFrameskip-v4).
2. While the paper demonstrates strong performance, it would be more impactful if the method were tested in more realistic environments such as IsaacGym[1] or MetaDrive[2]. These environments pose more challenging and realistic scenarios, which could further validate the generalizability and robustness of BlendRL. It would also be valuable for the authors to discuss any challenges or adjustments required to apply BlendRL to these more complex environments, and to comment on how they expect its performance and benefits to scale in such domains.
3. The explanation of how symbolic policies contribute to the overall behavior of mixed policies could benefit from more clarity. It would be helpful if the authors provided a detailed analysis or specific examples showing how the symbolic explanations correspond to the mixed policy's actions. Including case studies that illustrate how explanations evolve as the blending between neural and symbolic components changes could enhance understanding of the interpretability and functionality of the method.

[1] Makoviychuk, Viktor, et al. "Isaac gym: High performance gpu-based physics simulation for robot learning." arXiv preprint arXiv:2108.10470 (2021).

[2] Li, Quanyi, et al. "Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning." IEEE transactions on pattern analysis and machine intelligence 45.3 (2022): 3461-3475.

Meta-review: This paper presents BlendRL, a framework that integrates neural and symbolic policies for reinforcement learning, demonstrating improved performance over both pure neural networks and symbolic baseline approaches. The majority of reviewers praised the paper's clear presentation, novel integration of neural-symbolic components, and comprehensive empirical validation in Atari game environments. Initial concerns about technical novelty and comparison with existing work were adequately addressed through additional experiments and detailed clarifications of BlendRL's unique hybrid policy learning approach. The authors were highly responsive during the discussion period, providing thorough analysis of the symbolic-neural policy interactions and making significant improvements to the manuscript. 

While there is room to expand evaluation to more complex environments in future work, I recommend acceptance based on the paper's strong technical contribution and the authors' comprehensive response to reviewer feedback.

Decision: Accept (Spotlight)