RLSF: Reinforcement Learning via Symbolic Feedback

Authors: Piyush Jha, Prithwish Jana, Pranavkrishna Suresh, Arnav Arora, Vijay Ganesh
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords: Symbolic Feedback, Reinforcement Learning, Large Language Models, Program Synthesis

Abstract:
Reinforcement Learning with Human Feedback (RLHF) is considered a standard approach to fine-tuning Large Language Models (LLMs) but faces challenges such as unsound reward models, sparse rewards, and difficulties in collecting human preference data, limiting its effectiveness for complex, domain-specific tasks.

We propose Reinforcement Learning via Symbolic Feedback (RLSF), a novel fine-tuning paradigm where reasoning tools (e.g., solvers, provers, algebra systems) serve as the RL environment, providing fine-grained feedback via \textit{poly-sized certificates} (e.g., proofs) that characterize errors in the LLM-generated object with respect to specific correctness criteria. RLSF aims to improve the domain-specific understanding of LLMs more effectively than traditional reward signals. By enabling token-level corrections without requiring differentiable reasoning systems, RLSF addresses key limitations of traditional reward models.

Via extensive evaluations, we show that our RLSF-based fine-tuning of LLMs outperforms traditional approaches on five different applications (that have some associated logical or domain constraints), namely, program synthesis from natural language pseudo-code to programming language (+31.43\% in functional correctness for Google's CodeGemma-2b compared to supervised fine-tuning, +17.01\% in functional correctness compared to GPT-3.5 -- 100$\boldsymbol\times$ larger), three chemistry tasks (+5.5\% exact match for molecule generation, +19.4\% exact match for forward synthesis, +33.7\% exact match for retrosynthesis, using Meta's Galactica-1.3b, compared to GPT-4 -- 1000$\boldsymbol\times$ larger), and solving the Game of 24 (+25\% success rate using Meta's Llama2-7b compared to traditional methods, and +7\% success rate compared to GPT-3.5 -- 25$\boldsymbol\times$ larger). A takeaway is that fine-tuning via RLSF enables relatively smaller LLMs to significantly outperform closed-source models that are orders of magnitude larger (e.g., GPT-4).

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730409821299

thread_type:
official_review

<thread_object>
creator: reviewer_5fx7

<content>
summary:
This paper presents a paradigm for finetuning large language models. In tasks where outputs are expected to belong to a certain formal language, symbolic programs can be used to check the validity / correctness and can be used as rewards to finetune an LLM. The authors demonstrate this approach on three problems - code generation, molecular generation / rethrosynthesis, and the so called game of 24.

strengths:
1) The paper is written with good clarity. I am unsure about how original this work is because I am not aware of the literature in this area. But, I think the approach is fairly straightforward. 
2) The results, especially in code generation tasks seem better than baselines like GPT-3.5.

weaknesses:
1) Although there seems to be no glaring weakness to me, I am unsure about the significance of the chemistry problems that are used. I do not know how significant is Fingerprint Tanimoto Similarity or ~35%.
2) I am also not sure how novel this approach is. Can the authors point towards similar related work? Right now there is only one citation in the, "Neurosymbolic Reinforcement Learning (NRL)"  paragraph. 

But both points are speculations rather than weaknesses, I would just appreciate an answer.

questions:
Please look at the weakness section for questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696036413

thread_type:
official_review

<thread_object>
creator: reviewer_sfgh

<content>
summary:
This paper proposes RLSF or RL with symbolic feedback. RLSF's main difference with RLHF are two components. 1) A symbolic feedback generator that produces poly-sized certificates which basically means the feedback is not too unreasonably large compared to the input, .i.e is polynomial in size of the input. An examples of these symbolic feedback generator is the output of a compiler when compiling a program. 2) A function that takes in this feedback, the policy-sized certificate, and translates it to per-token rewards, e.g. assigning negative rewards to the lines of code that had syntax problems. Learning from this symbolic feedback improves RL post-training of LLMs on various tasks.

strengths:
It is not clear how to integrate symbolic feedbacks into a RL post-trainng of LLMs. Therefore, it seems RL post-training is missing out those feedbacks and finding a way to integrate them into the training loop should have advantages. This work is a first attempt at doing these which is valuable by trying to translate this feedbacks to environment rewards. It is different from other works that want to give the LLM a way to use symbolic tools.  Also, the work tests the idea on diverse tasks: Translation of Pseudo-Code to Code, Modelling chemical reactions, and game of 24 as a toy example. The work shows token-level rewards given by this system improves performance especially in terms of lexical or sytax errors.

weaknesses:
In terms of novelty, the fact that fine-grained feedback works better is well-known in RL community. For examples, There is a plethora of works on reward shaping. However, I know that in NLP, people know RLHF with its boolean feedback. Also in RL for reasoning we have papers that try to incorporate process based rewards instead of scalar rewards. Therefore, showing fine-grained signal works better is not considered very novel. However, I have not seen another work that tries to somehow extract and translate token-level feedback from these symbolic feedbacks. That is the the novelty of the paper. I think this paper has an interesting message for people who are doing RL post-training to try to incorporate rewards of previously symbolic systems which I think is important and is a strength. 

In terms of the experimentation, I think there is a big problem in the coding experiments. It may be that I misunderstood but I want to double check. After the SFT is done on the models, it seems the actual comparison is between training with the binary feedback, and the then the symbolic feedback. The main problem is that the symbolic feedback also contains the actual rewrad of whether the test cases are passed or not while the binary feedback is just whether the program compiles or not. That is not the correct comparison as the second option has the access to the actual success reward on the tasks in top of the integration of the symbolic feedbacks. The comparison that I think shows the significance of the inclusion of the symbolic feedback is to compare A) training with the success reward which can include the compilability reward as well  B) training with the success reward augmented with symbolic rewards.

I don’t have an expertise in the tasks involving chemical reactions. I have asked a few questions about them to make sure the signficant of the contribution.

In terms of writing, I think the paper is not written in terms with its actual message. I think the paper is written from the angle that its message is that “fine-grained signal is better”. However, this is a well-known message and not surprising. The actual hard task is how to transform a symbolic feedback to a fine-grained signal. If the paper was written about the details and challenges of this conversion I think its contribution would be much more apparent. I think the authors have actually done the hard work of the details and nitty-gritty of this conversion but decided to put the focus on the RLSF rather than on those details. There are many questions about how-to of this extraction of signal from symbolic feedback to fine-grained signal that I think it deserves a well written paper to discuss them. I have asked them in the questions section and I would like a lot if the authors include a section about that in their paper.

questions:
1-In the pseudo-code to code translation experiments, is there a reason that you did not have a run with the success reward (maybe alongside compilability reward)? What happens in that experiments and is there still benefit to integrate the symbolic feedback from the compiler?

2-What are the challenges and details of the choices for converting the symbolic feedback to per-token level rewards? I understand when a line does not compile, we can put negative rewards on that line. But is it always correct? If I am not wrong, there are cases in which a line does not compile because of a line before is not written. For example, a variable is not defined or a space or newline is forgotten. Isn’t it possible that with this the conversion actually misses the actual source of the problem and keep it untouched? I think there is some degree of credit assignment going on and the assignment is based on best guesses. While I am not familiar with the chemistry tasks, I am wondering what the output of the symbolic feedback generators look like and how easy it is to transform to per-token rewards? Also, I did not follow the explanation of the paper on how they integrated the algebraic system to the game of 24. Can you explain this in detail? I think the details of how to cover these signal is the actual contribution of the paper. 

3-For modelling the chemical reactions, you take out a pre-trained language model and then fine-tune it. Is that correct? I am just wondering if this is standard practice because these reactions are very far from written human texts and I was wondering if the standard practice is to just train from scratch? Can you cite other papers that also employ the same strategy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721843654

thread_type:
official_review

<thread_object>
creator: reviewer_t2ku

<content>
summary:
This paper propose a new fine-tuning paradigm for LLM post-training called RLSF, where the reasoning tools can provide feedback to the LLMs via poly-sized certificates characterizing errors in the LLM-generated object with respect to some correctness specification.
RLSF is evaluated across five different tasks, demonstrating superior performance compared to traditional fine-tuning methods.

strengths:
* Innovativeness: RLSF is the first reinforcement learning paradigm that utilizes multi-dimensional certificates generated by symbolic reasoning tools to provide fine-grained feedback, addressing the limitations of traditional reward models.

* Efficiency: RLSF significantly enhances the performance of LLMs across multiple tasks, particularly excelling in the conversion of natural language pseudocode to C++ and in chemical tasks.

* Practicality: RLSF does not require a differentiable symbolic reasoning system, which increases its flexibility and applicability in real-world scenarios.

* Experimental Validation: The article validates the effectiveness and superiority of RLSF through experiments on five different tasks, showcasing its potential in specific domain tasks.

weaknesses:
* The proposed RLSF relies on symbolic reasoning tools to generate feedback, and the performance and availability of these tools may influence the effectiveness of RLSF. While the article mentions that these tools perform well in practice, it does not provide a detailed discussion of their limitations and possible alternatives.
* The abstract section is overly lengthy and could benefit from a more concise phrasing.

questions:
* The RLSF method demonstrates exceptional performance on specific tasks, However, its applicability to a broader range of reasoning tasks or general tasks remains unclear. Is there theoretical or empirical evidence supporting its effectiveness across different tasks?
* Does the performance of symbolic tools decline when handling complex or large-scale problems? Are there alternative solutions or improvements that can be considered?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731219015258

thread_type:
official_review

<thread_object>
creator: reviewer_bsi8

<content>
summary:
This paper proposes RLSF, a fine-tuning approach that uses symbolic tools to provide token-level feedback for LLMs. The authors evaluate RLSF on five tasks across three domains: code generation, chemistry, and math problem solving (Game of 24).

strengths:
# Implementation Details
- Clear description of the RLSF approach
- Good reproducibility through detailed experimental setup
- Comprehensive evaluation across multiple domains
# Results
- Shows consistent improvements over baselines across different tasks
- Provides detailed metrics and comparisons
- Demonstrates potential practical utility

weaknesses:
# Limited Technical Novelty
- The core idea of using symbolic tools for feedback is not new - the paper acknowledges similar approaches in code generation and verification
- The main contribution appears to be applying token-level feedback in RL fine-tuning, which is an incremental advance
- The approach is essentially a straightforward combination of existing techniques (RL, symbolic verification, token-level feedback)
# Experimental Focus
- The paper is primarily focused on empirical results across three domains
- Limited theoretical analysis or justification for why this approach works better
- No significant algorithmic innovations beyond combining known components

questions:
1. How does the approach scale to more complex tasks requiring deeper reasoning?
2. Why focus on these particular domains/tasks? How generalizable is the approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732489435193

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised PDF addressing reviewer feedback

comment:
We thank the reviewers for their valuable feedback, which has helped us improve the clarity and presentation of our work. Based on the suggestions, we have made the following revisions:

1. We have updated Sections 5.1 and 6.1 and added Appendix C.2 to provide clearer explanations, along with detailed examples, of how symbolic feedback is converted into fine-grained signals for the chemistry tasks and the Game of 24.

2. We have updated Section 4.4 and added Appendix A.2 to include experiments comparing the use of binary success reward and binary compilability reward in pseudo-code to code translation task.

3. Following reviewer feedback, we have made the abstract more concise while retaining all critical elements of our contributions.

We believe these changes address the reviewers' concerns and further strengthen the manuscript. Thank you for your insightful comments and suggestions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Authors present a novel fine-tuning paradigm for large language models through symbolic feedback, by providing feedback in the form of poly-sized certificates at the token level. They evaluate on five tasks across three domains: code generation, chemistry, and math problem solving (Game of 24).

Reviewers note several technical strengths of the work, including that it does not need a differentiable symbolic reasoning system, exhibits enhanced performance across multiple tasks, and the writing is clear. However, there are concerns about the novelty of the contribution. It is already widely known that fine-grained feedback outperforms sparse, response-level feedback, and reviewers note that the paper is structured as if that is the main contribution, rather than the incorporation of symbolic language for feedback.

After reading through Section 3 of the paper, I agree with reviewers that the way the paper is currently structured is not sufficient to highlight how symbolic feedback can be turned into dense reward in a general way that characterizes a meaningful contribution. I encourage the authors to revise and resubmit their manuscript to highlight this symbolic feedback as reward framework.