BlendRL: A Framework for Merging Symbolic and Neural Policy Learning

Authors: Hikaru Shindo, Quentin Delfosse, Devendra Singh Dhami, Kristian Kersting
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords: neuro-symbolic ai, differentiable reasoning, reinforcement learning, interpretable ai, first-order logic

Abstract:
Humans can leverage both symbolic reasoning and intuitive responses. In contrast, reinforcement learning policies are typically encoded in either opaque systems like neural networks or symbolic systems that rely on predefined symbols and rules. This disjointed approach severely limits the agents’ capabilities, as they often lack either the flexible low-level reaction characteristic of neural agents or the interpretable reasoning of symbolic agents. 

To overcome this challenge, we introduce *BlendRL*, a neuro-symbolic RL framework that harmoniously integrates both paradigms. 
We empirically demonstrate that BlendRL agents outperform both neural and symbolic baselines in standard Atari environments, and showcase their robustness to environmental changes. Additionally, we analyze the interaction between neural and symbolic policies, illustrating how their hybrid use helps agents overcome each other's limitations.

Decision: accept (spotlight)
Reviewer scores: 5, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1729150964066

thread_type:
official_review

<thread_object>
creator: reviewer_pgu6

<content>
summary:
The paper presents BlendRL, a neuro-symbolic reinforcement learning framework that integrates symbolic reasoning and neural policies, aiming to improve agent performance by mixing neural and symbolic policies. The authors demonstrate BlendRL’s efficacy in classic Atari games, highlighting that the framework outperforms purely neural baselines and neuro-symbolic baselines like NUDGE.

strengths:
The paper is clearly written, and the proposed framework is compelling in its attempt to bridge the gap between symbolic reasoning and neural policy learning. The dual use of symbolic logic for high-level reasoning and deep neural networks for low-level reaction is well-motivated and timely. The experiments convincingly show that BlendRL agents perform better than existing methods, particularly in environments that demand both reasoning and quick reflexes. For instance, in Seaquest, the neural policy was able to effectively manage shooting enemies, while the symbolic component handled pathfinding and resource collection, demonstrating how the hybrid model synergistically enhances performance.

weaknesses:
Overall, I think this work will have a positive impact on the community, but I still have some concerns:
1. The paper evaluates the proposed method in a limited set of experimental environments. It would strengthen the validation of the method if the authors could evaluate BlendRL in a wider variety of environments, especially those with different characteristics (such as the environments BeamRiderNoFrameskip-v4, EnduroNoFrameskip-v4, and QbertNoFrameskip-v4).
2. While the paper demonstrates strong performance, it would be more impactful if the method were tested in more realistic environments such as IsaacGym[1] or MetaDrive[2]. These environments pose more challenging and realistic scenarios, which could further validate the generalizability and robustness of BlendRL. It would also be valuable for the authors to discuss any challenges or adjustments required to apply BlendRL to these more complex environments, and to comment on how they expect its performance and benefits to scale in such domains.
3. The explanation of how symbolic policies contribute to the overall behavior of mixed policies could benefit from more clarity. It would be helpful if the authors provided a detailed analysis or specific examples showing how the symbolic explanations correspond to the mixed policy's actions. Including case studies that illustrate how explanations evolve as the blending between neural and symbolic components changes could enhance understanding of the interpretability and functionality of the method.

[1] Makoviychuk, Viktor, et al. "Isaac gym: High performance gpu-based physics simulation for robot learning." arXiv preprint arXiv:2108.10470 (2021).

[2] Li, Quanyi, et al. "Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning." IEEE transactions on pattern analysis and machine intelligence 45.3 (2022): 3461-3475.

questions:
1. Figure 6 references textual interpretation; could you kindly provide a more detailed explanation?
2. In cases where the logic chain becomes quite long, would the policy still be considered explainable?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730358589522

thread_type:
official_review

<thread_object>
creator: reviewer_nrge

<content>
summary:
The paper proposed to jointly learn a mixture of neural policy and symbolic policy (represented in a differentiable way) via reinforcement learning. In the experiments, the neural policy is simply a CNN-based policy, the differentiable symbolic policy is based on a method called differentiable forward reasoner, and the weights of the mixture can themselves be seen as the output of a trainable policy. The proposed method is tested on three Atari games.

strengths:
The idea of learning a mixture of neural policy and symbolic policy is novel. Given the presented experimental results, it seems to lead to nicely interpretable decision rules (at least in the symbolic part). Also, surprisingly to me, it seems that it automatically learns to make "reflex" decisions preferably using the neural policy. 

The overall proposition nicely integrates several previous techniques to offer an end-to-end method, with little human inputs.

weaknesses:
The paper should be more self-contained. For instance, this work builds on several existing propositions (notably by Shindo et al. and Delfosse et al.). The authors should recall and provide more details of those previous works to help the reader more easily understand BlendRL and appreciate its novelty. For instance, I think the current presentation of differentable forward reasoner is too light, e.g., how is the set \mathcal C related to the forward reasoning graph? What's the advantages/disadvantages of this representation using a graph compared to other frameworks, e.g., dILP?
Also, Section 3.3,  which seems to me to be a key component that can have a huge impact on BlendRL, is lacking in details. For instance, I suggest the authors to provide more information about step (i). In addition, doesn't using the NUDGE policies to generate examples for the LLM provide a strong inductive bias to BlendRL?

The experimental validation is missing some details and maybe a bit light. The authors should explain how they chose their hyperparameters (e.g., Entropy coefficient for blending regularization). Since BlendRL requires a more complex policy model than Neural PPO or NUDGE), I believe it's possible via (potentially costly) hyperparameter tuning to obtain better results for BlendRL.
The authors only evaluates on three Atari games, which are moreover different from those used in NUDGE. Why is that so? Would BlendRL still outperform NUDGE on the tasks where it has demonstrated a good performance?

Although I haven't followed closely the latest developments in neuro-symbolic approaches, I believe that the literature in this direction is quite rich. For instance, I think that there are other works trying to combine System 1 and System 2 capabilities in AI agents and there are other neuro-symbolic RL approaches. The paper could be improved by better situating BlendRL in the existing literature. 

Minor points:
There are a number of typos that should be corrected, e.g.,
- line 107: I believe that function symbols are not used in this work. This should be clarified.
- line 162: What's called "state" actually corresponds to "observation"
- line 177: Regarding the dimension of the logic policy, should F appear?
- lines 852-855: the learning rates are missing?

questions:
1) If I wanted to apply it to a new game, what should be done to generate the action rules (Section 3.3)? 

2) Does using the NUDGE policies to generate examples for the LLM provide a strong inductive bias to BlendRL?

3) How were the hyperparameters obtained?

4) How did you choose the three Atari games? Why did you use different ones compared to those used in NUDGE?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659427011

thread_type:
official_review

<thread_object>
creator: reviewer_xzvw

<content>
summary:
The paper presents BlendRL, a neuro-symbolic RL framework that allows agents to blend resoning from pixel and object level representations. Unlike previous  neuro-symbolic RL approach BlendRL learns concurrently it's low and haigh level policies. Specifically, a blending module, informed by rules generated by a language model, dynamically determines the optimal mix of neural and symbolic policies based on the task context. 

Authors test BlendRL in Seaquest and Kangaroo (from Atari), where agents must alternate between quick responses and logical planning, results show that BlendRL consistently outperforms purely neural or symbolic agents.

strengths:
The paper is very well written and easy to follow. The proposed framework is quite novel to the best of my knowdledge, since the level 2 and 1 systems are usually much more separated than in BlendRL, and the results against vanilla PPO or a symbolic approach (NUDGE) are promising.

weaknesses:
My biggest concern with this work is the lack of empirical comparison with any other neuro-symbolic baselines, e.g. [1-4] 

Since BlendRL also weights heavily on object-based representations and relational learning it would have been good (although I don't consider this critical)  to include some contrast with deep learning approaches for such kind of learning, e.g. [1, 5-6].

While the point above will raise my confidence on the paper, I still thinkt hat the novelty of the approach and the results included outweight the weakpoints.

[1] Borja G. León, Murray Shanahan, and Francesco Belardinelli. In a nutshell, the human asked for this: Latent
goals for following temporal specifications. In International Conference on Learning Representations, 2022.

[2] Kuo, Yen-Ling, Boris Katz, and Andrei Barbu. "Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas." 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.

[3] Vaezipoor, Pashootan, et al. "Ltl2action: Generalizing ltl instructions for multi-task rl." International Conference on Machine Learning. PMLR, 2021.

[4] Qiu, Wenjie, Wensen Mao, and He Zhu. "Instructing goal-conditioned reinforcement learning agents with temporal logic objectives." Advances in Neural Information Processing Systems 36 (2024).

[5] Shanahan, Murray, et al. "An explicitly relational neural network architecture." International Conference on Machine Learning. PMLR, 2020.

[6] Feng, Fan, and Sara Magliacane. "Learning dynamic attribute-factored world models for efficient multi-object reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

questions:
See weaknesses above

--- Post rebuttal --- 

After reading Author's responses and the updated paper I have no more concerns. I believe this will be an interesting contribution for those working with neuro-symbolic approaches.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730859660091

thread_type:
official_review

<thread_object>
creator: reviewer_nfd8

<content>
summary:
This paper integrates condition-based logic decisions with neural network-based reinforcement learning policies through an LLM-based hybrid module to address the shortcomings of both approaches. It achieved better results in three Atari games compared to standard PPO and logic-based method.

strengths:
1. The paper is clearly written and easy to follow.
2. The utilization of the language model shows a certain level of innovation.

weaknesses:
1. The overall concept is not particularly novel, with numerous similar works, such as the well-known fast and slow systems, already existing.
2. The paper and appendix lack crucial details on how the LLM generates rules and calculates hybrid probabilities, and to what extent this is based on the content provided in the prompts. This is essential for determining whether the method can generalize to more diverse tasks.
3. The paper consistently emphasizes complex tasks, yet the experimental environment is not particularly sophisticated. Atari is a basic, outdated, and relatively simple benchmark that has largely been mastered. The three selected tasks are not among the recognized challenging ones in Atari (indeed, they are relatively simple). Truly difficult tasks, such as Montezuma’s Revenge, would be more appropriate as experimental environments.
4. The experiments should compare against more advanced RL algorithms. For example, Agent57 has already achieved a score of 30k on Kangaroo (while blend RL scores less than 20k) and 1000k on Sequest (while blend RL scores less than 5k). Therefore, the current experimental results do not demonstrate the superiority of the method.
5. Overall, the experiments lean towards simpler explorations and lack ablation studies that would reveal the characteristics of the method. For instance, I couldn't find which LLM was used, nor were there detailed experiments on the impact of object-centric representations on the method.

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732223446859

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Remark

comment:
Dear Reviewers,

Thank you for your fruitful and valuable feedback. We sincerely appreciate the time and effort you have devoted to helping us refine our paper. Let us clarify our contribution and provide more comparison to deep and symbolic agents.

## BlendRL's contribution and chosen environments
Our research builds on the well-established logic-based neuro-symbolic RL literature. Notably, NLRL (ICML 2019) [1] and NUDGE (NeurIPS 2023) [2] are pivotal works in integrating differentiable logic into RL agents. These studies have primarily been assessed in synthetic or simple environments (e.g. blocks world), which do not require a solid combination of System 1 and System 2. Despite this limitation, they have been published in top-tier conferences, as their transparent yet learnable policy representations offer significant insights to the community. However, applying these approaches to more complex tasks is challenging and remains a significant bottleneck, mainly due to their reliance on expert-provided inductive bias (*i.e.* concepts such as *closeby*) necessary to solve the tasks. BlendRL agents address this shortcoming in two by:
* (i) utilizing LLMs to obtain these relevant symbolic concepts and their associated logic rules.
* (ii) integrating neural policies that can compensate for a potential lack of these necessary concepts. 

The chosen Atari environments challenge the agents' as they incorporate multiple objectives (e.g., shooting enemies, managing oxygen, and collecting divers in *Seaquest* [3]). They allow us to demonstrate that BlendRL agents can meaningful logic policies (as we showcase their rules) *and* even beyond if not provided with all the necessary context. While we are currently extending the set of testing environments (e.g. to *QBert* as suggested by Reviewer pgu6), we believe that the existing results already validate our paper’s claim.

Our contribution focuses on allowing logic-based RL policies to work harmoniously with neural ones, aligning them more closely with Kahneman's System 1 and System 2 concepts [4], and not on developing the SOTA RL algorithm. To this end, BlendRL aims to create transparent and robust agents that combine the strengths of both neural and logic information processing methods.

## More comparison to deep and neuro-symbolic baselines
We agree that comparing BlendRL to additional deep learning and neuro-symbolic baselines could enhance the validation of our proposed method. Consequently, we present comparisons with **6 additional baselines** in the table below. Our evaluation includes 2 deep learning, 5 neuro-symbolic, and 1 human baselines against BlendRL.  We trained NLRL [1] agents in the selected environments. For other baselines, we report the performance as described in their respective papers if available, or extend the evaluation based on their provided code. We are also currently working on obtaining results of INSIGHT agents in *Kangaroo* and *DonkeyKong* [7], that will be included in the camera ready version.
|Environments|DQN[8]    |PPO        |NLRL*[1]|NUDGE[2]|SCoBots[5]  |Interpreter[6]|INSIGHT[7]  |BlendRL  |Human[8]|
|------------|----------|-----------|--------|--------|------------|--------------|------------|---------|---------|
|Kangaroo    |2696      |790.0±280.8|3034±11 |3058±25 |4050±218|1800±0    |-           |**12619**±132| 2739    |
|Seaquest    |2794      |837.3±46.7 |75±0    |64±0    |2411±377|1093±155  |2666±728.2|**4204**±10  |4425    |
|DonkeyKong  |253.3±45.1|2080±1032  |29±0    |122±25  |426.7±64.3  |1838±459  |-           |**3541**±43  |7320[9]    |


*References*

-------

[1] Jiang, Z., & Luo, S. (2019). Neural logic reinforcement learning. ICML.

[2] Delfosse, et al. (2023). Interpretable and explainable logical policies via neurally guided symbolic abstraction. NeurIPS

[3] Bacon, et al. (2017). The oqption-critic architecture. AAAI

[4] Daniel, K. (2017). Thinking, fast and slow.

[5] Delfosse, et al. (2024). Interpretable concept bottlenecks to align reinforcement learning agents. NeurIPS.

[6] Kohler, et al. (2024)  "Interpretable and Editable Programmatic Tree Policies for Reinforcement Learning." Workshop on Interpretable Policies in Reinforcement Learning@ RLC.

[7] Luo, et al. (2024) End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanations. ICML.

[8] Nair, et al. (2015). Massively parallel methods for deep reinforcement learning. ICML Workshop

-------

Thank you once again for your insightful comments. We have addressed the concerns raised by each reviewer in our responses. The corresponding revisions in the updated manuscript are highlighted in *blue*.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents BlendRL, a framework that integrates neural and symbolic policies for reinforcement learning, demonstrating improved performance over both pure neural networks and symbolic baseline approaches. The majority of reviewers praised the paper's clear presentation, novel integration of neural-symbolic components, and comprehensive empirical validation in Atari game environments. Initial concerns about technical novelty and comparison with existing work were adequately addressed through additional experiments and detailed clarifications of BlendRL's unique hybrid policy learning approach. The authors were highly responsive during the discussion period, providing thorough analysis of the symbolic-neural policy interactions and making significant improvements to the manuscript. 

While there is room to expand evaluation to more complex environments in future work, I recommend acceptance based on the paper's strong technical contribution and the authors' comprehensive response to reviewer feedback.