On Designing Effective RL Reward at Training Time for LLM Reasoning

Authors: Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, Yi Wu
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, RLHF, PPO, LLM for Reasoning, Reward Design

Abstract:
Reward models have been increasingly critical for improving the reasoning capability of LLMs. Existing research has shown that a well-trained reward model can substantially improve model performances *at inference time* via search or best-of-N votes. 
However, the potential of reward models during *RL training time* still remains largely under-explored. 
It is currently unclear whether these reward models can provide additional training signals to RL training that uses sparse success rewards, which verify the correctness of solutions.
In this work, we evaluate popular reward models for RL training, including the Outcome-supervised Reward Model (ORM) and the Process-supervised Reward Model (PRM), and train a collection of LLMs for math problems using RL by combining these learned rewards with success rewards. Surprisingly, even though these learned reward models have strong inference-time performances, they may only bring marginal improvements or even hurt RL *training*, producing worse performances than LLMs trained with the success reward only. 
We find that *training collapse* easily occurs in RL training when PRM simply serves as reward shaping in addition to the success rewards. Our further analysis reveals two issues that may lead to the sub-optimal performance.
Therefore, we introduce two novel reward refinement techniques, including the **Clip** and the **Delta** mechanisms, to tackle the identified issues.
We evaluate our techniques with multiple reward models over a set of 1.5B and 7B LLMs on MATH and GSM8K benchmarks, where both **Clip** and **Delta** consistently enhance RL training. 
Finally, we also demonstrate that with a carefully designed reward function, pure RL training without any additional supervised tuning can further improve all the evaluated LLMs, including the state-of-the-art 7B LLM Qwen2.5-Math-7B-Instruct on MATH and GSM8K benchmarks.

Decision: reject
Reviewer scores: 6, 6, 8, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730435973014

thread_type:
official_review

<thread_object>
creator: reviewer_ffha

<content>
summary:
The paper explores how to design effective RL rewards models to improve LLMs in mathematical tasks.
It evaluates two popular reward models - ORM and PRM - during RL training.
The authors argue that using these reward models do not improve and even degrade performance during RL training; due to "reward hacking".
The authors propose two ad-hoc reward refinement techniques, Clipping and Delta, and demonstrated their performance improvement.

strengths:
- The paper has positioned itself well as an NLP paper by referring a number of recent papers on LLM and reward models.
- The solution suggested by the paper effectively improves from previous baseline.

weaknesses:
- After indroducing ORM, the paper does nothing about it; since it is introduced and evaluated, we need at least an analysis on why it does not help. 

- While the paper is suspecting the observed phenomenon as "reward hacking", I would say this is closer to the wrong usage of reward functions. As we have binary correctness label, and ORM and PRM try to get estimation of it with partial generation, all rewards will be nonnegative. In episodic settings without any discount, this naturally lead to the agents that favor long trajectory, i.e., long solution generation, regardless of whether its correct or not. The paper does not tell us about the hyperparameter choice of $\alpha$, but with some large enough $\alpha$, this may lead to agent preferring long wrong generations over short correct generations.

- The proposed method is, in my perspective, not novel. To solve sparse reward problems, the first go-to should be reward-shaping methods. the proposed delta mechanism can be understood as a potential-based reward shaping method where PRM is the potential function. The proposed clip-delta mechanism can be understood as a potential-based reward shaping method where PRM-clip is the potemtial function. The overall paper can actually be understood as using a reward shaping methods to handle sparse rewards of RLHF, and I believe the paper should position itself well against reward-shaping methods that have been extensively studied in the past.

- The paper argues that PR-Normed approach overfits and the performance degradation is severe. To see SR+proposed method algorithms do not suffer from the same problem, we need more information, e.g., learning curves of all algorithms.

- While in this paper the Delta (giving agent no preference on length of generation) or the Clip (giving agent preference on shorter generation) worked, it depends on the task; on the tasks where longer generations are advantageous, both modifications might not be beneficial. In that sense, the paper only evaluates on single task, and it would not be a robust assessment on the algorithm's performance.

questions:
- Why does ORM not suffer from the same problem?
- What's the difference between PRM and ORM: why PRM does help when ORM does not help?
- If we use PR-Clip itself, then it becomes an algorithm that prefers shorter generation as we increase $\eta$. Do you have results on the performance on varying $\eta$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730442699173

thread_type:
official_review

<thread_object>
creator: reviewer_smof

<content>
summary:
This paper introduces two methods to prevent reward hacking while training LLM with reinforcement learning algorithms, Clipping and Delta. The paper shows superior results when adding these techniques on top of process rewards, compared to baselines without these techniques.

strengths:
The paper is written clearly, and well-motivated. Extensive experiments are done, ablating different parts of the algorithm to show that clipping and delta mechanisms are actually helping.

weaknesses:
Reward clipping is not a novel idea and has been studied before. Reward hacking, even more so. IMO, a valid baseline for this paper needs to be a well-written PRM. However, looking at the baselines used in this paper, whenever PR is used the result is worse than SR alone. Other papers have already shown PRM is better than ORM alone. Clearly, the baseline is too weak and isn't setup correctly.

Basically, the authors are showing evidence that SR + PR + reward-hacking-prevention is better than SR alone, or SR + plain PR. I don't find this to be a new contribution.

The work also lacks direct examination of how these two methods work under the hood. For example, how often is the reward non-zero after clipping? i.e. Is this truly a dense reward? When delta is applied on top of clipping, the argument that the reward is bound no longer holds, because r_process(q, p_k+1) can be 0. So what's really the mechanism in this case?

In Table 2, the experimental results show pretty marginal (or, non-existent) improvements on greedy decoding, on the only valid RL baseline, Qwen2.5-Math-7B-Instruct. The other comparisons are pretty meaningless since we all know RL works & it's not the contribution of this paper.

questions:
How often is the reward non-zero after clipping? i.e. Is this truly a dense reward?

Why does none of the PRM method in baselines work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646235695

thread_type:
official_review

<thread_object>
creator: reviewer_6vif

<content>
summary:
This paper investigates how to effectively use reward models during reinforcement learning (RL) training to improve LLMs' mathematical reasoning abilities. The authors discover that traditional reward models can either be ineffective (in the case of Outcome-supervised Reward Models) or lead to reward hacking through unnecessary repetition of steps (in the case of Process-supervised Reward Models). To address these issues, they introduce two novel techniques - "Clipping" and "Delta" mechanisms - that help prevent reward exploitation while maintaining the benefits of process rewards. Using these refined reward techniques, they demonstrate consistent improvements across various LLMs, including enhancing the performance of state-of-the-art models like Qwen2.5-Math-7B-Instruct on the MATH and GSM8K benchmarks.

The key innovation is showing that while reward models are useful for inference-time improvements, they need careful refinement to be effective during RL training. Their solutions help stabilize training and prevent the model from gaming the reward system through repetitive or unnecessary steps.

strengths:
Strengths:
1. Identifying the issues with using rewards: The authors systematically analyze both Outcome-supervised Reward Models (ORM) and Process-supervised Reward Models (PRM), revealing important limitations of each approach. They demonstrate that ORMs, despite working well at inference time, don't provide additional benefits beyond success rewards during training. More significantly, they uncover a serious reward hacking issue with PRMs, where models learn to game the system by repeating simple or unnecessary steps to achieve high rewards. This analysis is particularly valuable because previous work primarily focused on using rewards at inference time, making this identification a novel contribution to the field.
2. Experiments: The authors conduct comprehensive evaluations across multiple model sizes (1.5B and 7B parameters) and variants (including Qwen2, Qwen2.5, and both general and math-specific models) on standard benchmarks like MATH and GSM8K. Their experimental design includes thorough ablation studies showing the impact of different components, careful comparisons of various reward mechanisms, and detailed analysis using synthetic examples to demonstrate reward hacking.

weaknesses:
Limited Novelty:
- Clipping is indeed a well-established technique in the RL community, commonly used to stabilize training
- The concept of bounded rewards is a fundamental principle in RL, not a new innovation
- The Delta mechanism, while presented as novel, essentially implements reward shaping - another well-known concept in RL

The paper's contribution seems more incremental than innovative: It primarily applies known RL techniques to the specific context of LLM reasoning. The main finding that reward models can be exploited during training is somewhat expected given the general challenges of reward design in RL. The solutions proposed (Clipping and Delta mechanisms) are straightforward applications of existing RL principles.

Missing Reward Shaping related work:
1. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping, Ng et.al 
2. Hindsight credit assignment,  Harutyunyan et.al
3. Rudder: return decomposition for delayed rewards, Arjona et.al
4. Align-RUDDER: Learning from few demonstrations, Patil et.al
5. Modern Hopfield networks for return decomposition for delayed rewards, Widirich et.al

questions:
My current criticism is regarding the novelty of the paper. I am willing to increase my score if the authors convince me that the clipping and delta are novel enough.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659292986

thread_type:
official_review

<thread_object>
creator: reviewer_gcaq

<content>
summary:
This work discusses the role of reward models in enhancing the reasoning capability of LLM models during RL training, which is under-explored compared to inference. It shows the impact of popular reward models, the Outcome-supervised Reward Model (ORM) and the Processsupervised Reward Model (PRM), on the performance of LLMs after being combined with the success sparse reward signals on math problems. It was observed that such reward models may not help or even hurt the performance of the LLM due to the reward hacking issue. This work proposes two reward refinement methods to tackle this issue, named Clipping and Delta. Those techniques have shown potential in stabilizing the RL training of a collection of LLMs when evaluated on the MATH and GSM8K benchmarks. In addition, performance improvement across all evaluated LLMs can be obtained by carefully designing a reward function for pure RL training.

strengths:
- The paper is well-written, and the problem is nicely motivated.
- The empirical results are enough to assess the potential of the reward model, with some techniques for mitigating reward hacking during RL training to enhance the LLM reasoning.
- I appreciate the case study shown in Fig. 2 and the others added in the appendix.

weaknesses:
## Major Comments:
- We saw in the empirical results that the improvement in the small models was higher than in the larger ones. I disagree with the authors about the importance of evaluating the study or the proposed techniques on larger models.

## Minor Comments:
- Typo in Line 057: "on the reward models, it remains **un**clear whether the reward models can provide additional training".

questions:
- As far as I understand, the outcome reward (OR) gives a likelihood that the solution is correct. That means a solution can have a success reward of zero but still be close to the correct solution; hence, the likelihood could be relatively high. It can also be high because of the uncertainty of the reward model. **Why do you still think that did not help when combined with the success reward?** I believe a probability of success in the form of reward, instead of ones and zeros, should contribute to the learning. If I understood the outcome reward wrong, then the explanation of the reward models was unclear.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681130640

thread_type:
official_review

<thread_object>
creator: reviewer_mafz

<content>
summary:
The paper examines the use of learned reward models, such as Outcome-supervised (ORM) and Process-supervised Reward Model (PRM), to enhance reasoning in LLMs during RL training. They show that ORM does not improve and PRM can even hinder RL training due to reward hacking. The authors propose “Clip” and “Delta” to address this.

strengths:
- The proposed idea is new, interesting, and well-motivated. 
- The paper is easy to read and follow. 
- The addressed problem is of significance.

weaknesses:
- More details on the experimental setup could be provided for reproducibility, including the reward thresholds for the Clip and Delta mechanisms and hyperparameters for PPO. 
- Smaller LLMs may offer a larger scope of improvement, and so the proposed methods may seem to have been successful. However, to confirm the advantage, experiments on larger LLMs may be necessary; e.g., the paper reports GPT-4o-2024-08-06’s performance to be 92.9 on GSM8K, which is higher than almost all other models and variations, and offers less scope of improvement.

questions:
- Any discussion on the computational overhead or ease of integration of Clip and Delta into existing workflows would be beneficial. 
- Although mathematical reasoning is a good testbed, any comments on the potential applicability of these techniques in non-mathematical or multi-modal reasoning tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698970186

thread_type:
official_review

<thread_object>
creator: reviewer_7bhh

<content>
summary:
This work has two important messages for the field of LLM Reasoning: 1) the rewards models, especially the PRM can be hackable. Therefore, integrating them into LLM RL training may lead to hacking this reward performing worse than just removing it. 2) The paper argues that by clipping and then delta-ing the PRM rewards one can actually use these rewards models to gain a boost in their performance.

strengths:
I think the paper has important messages for LLM reasoning community:

1) The message that PRMs are hackable is important and valuable. Also, the paper digs into showing what goes wrong which provides insight into what actually goes wrong when using them. That the LLM trained with these PRMs leans towards some steps that are correct but does not move us closer to a solution. I think this contribution is also important. 

 2) Also, the paper shows limiting these rewards is not obvious. It is only by mixing the clipping and the delta method that they can boost the performance. This is interesting that it shows the problem is serious, although I don't agree that the clip-delta completely solved it.

weaknesses:
I think the paper focuses a lot on the boost it gets from mixing the clip and delta. However, I have some concerns if the clip and delta is a generalizable approach. First, the delta mechanism seems unmotivated. There is not nothing wrong with being unmotivated if it works super well. But, I think the gains are modest. The delta mechanism rewards action `a_t` if the reward of action `a_{t+1}` is less which is a very strong change to the RL environment. I understand that some interesting properties like the summation and small change to the returns arise from this, but I believe this is still an unmotivated change to the environment. However, maybe I don’t have the correct intuition on this. It is quite surprising that none of them work well on their own which makes me wonder if there is any intuition behind why this mixture of both works better? I am just afraid that this is not that well motivated.

questions:
1- What is the motivation behind the delta mechansim? Why should we assign credit to a_t if the reward of a_{t+1} is lower?

2- Can we consider the gains by mixing the clip and delta mechanism are strong and not modest? 

3- Your PRM is trained on automatic data. Is this possible that this hackability issue is because of the noise caused by the automatic procedure and would be avoided when trained on human-data or better PRM data generation (I understand better PRM data generation is itself a research question)?

4- Is there any intuition on why both clip-and-delta should be applied to get a boost and none of them work well in isolation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732208059778

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely thank all reviewers for their insightful and constructive feedback. In response, we have substantially revised and enhanced our paper to address the concerns raised and to further strengthen our contributions. Below, we outline the key updates and clarifications:

## 1. New Case Studies and Theoretical Analysis
We highlight key aspects of our revision as follows:
- **New Case Studies on Issues of PR (Fig. 3)**: We provide new in-depth case studies in Fig. 3 (also [available here](https://i.postimg.cc/Nj894gSL/case-study.jpg)) showing issues when the PRM simply serves as reward shaping in addition to the success reward. Specifically: 
    -  **Intrinsic biases of PRM** can be largely exploited by the training LLM to generate sub-optimal behavior patterns.
    - **Reward Misspecification Issue** could mistakenly promote incorrect steps through RL training.
- **Additional Mechanism Analysis (Sec 4.2)**: Our detailed analysis of the mechanisms highlights how the proposed mechanisms address specific issues,
    - **Clip mechanism mitigates the intrinsic biases of PRM** by bounding the rewards to an upper threshold, preventing the LLM from obtaining high rewards through undesired patterns.
    - **The Delta mechanism tackles the reward misspecification issue** by optimizing single-step PRM rewards, ensuring the steps promoted by RL training are aligned with the PRM.
- **Theoretical Insights for the Delta Mechanism (Appendix E)**: We provide theoretical analysis showing how the Delta mechanism makes RL training optimizates single-step PRM rewards alongside success rewards.
- **Additional Experiments & Training Curves (Appendix A)**: Additional experiments and training curves help better understand the limitations of baseline methods.

## 2. Novelty of Our Work
We highlight the **novelty of our work** as follows:
- **The First Study of PRM Issues in RL Training for LLM Reasoning:** Our work presents the first thorough study of the issues when simply adopting PRM as reward shaping in RL training for LLM reasoning. We find the intrinsice biases of PRM and the reward misspecification issue may lead to sub-optimal performance. The impact of these issues on RL training for LLM reasoning tasks has not been systematically analyzed in prior literature [1,2,3,4]. 
- **Simple Techniques and Practical Insights for Effective RL Training:** Although the proposed mechanisms are simple in design, they are supported by thorough analysis and experiments that show their ability to utilize PRM effectively, providing valuable rewards in RL training for LLM reasoning. Our studies and analysis present valuable insights to the community on how to design effective RL rewards for LLM reasoning.

## 3. Summary of Major Revisions
- **Fig.3**: Updated with new case studies and insights into mechanism effects.
- **Sec 4.1**: Discussion on the training results of PR \& OR. Additional case studies on the issues of PR. 
- **Sec 4.2**: The motivation and detailed explanation of the Clip and the Delta mechanism.
- **Sec 5**: Updated PPO training results with success rewards only in Table 2.
- **Appendix A**: Training curve and additional experiments.
- **Appendix D**: Hyperparameters and training setup.
- **Appendix E**: Theoretical analysis of the Delta mechanism and PRM.


[1] Shao, Zhihong, et al. "Deepseekmath: Pushing the limits of mathematical reasoning in open language models." arXiv preprint arXiv:2402.03300 (2024).

[2] Yang, An, et al. "Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement." arXiv preprint arXiv:2409.12122 (2024).

[3] Wang, Peiyi, et al. "Math-shepherd: Verify and reinforce llms step-by-step without human annotations." arXiv preprint arXiv:2312.08935 (2023).

[4] Havrilla, Alex, et al. "Teaching large language models to reason with reinforcement learning." ICML 2024 Workshop AI4MATH.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732549410892

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Motivation and Novelty of the Delta Mechanism

comment:
To address the common questions regarding the Delta mechanism, we would like to clarify the motivation behind the Delta mechanism and its core novelty.

The key novelty of the Delta mechanism lies in its ability to **guide RL training to optimize the single-step PRM reward for each individual reasoning step**. By contrast, PR encourages longer generations, even when the PRM rewards of individual steps are sub-optimal, as shown in our case studies (Fig. 2 and Fig. 3).

The contrast becomes clearer through an analysis of the policy gradient for PR and PR-Delta. Following [1], the policy gradient of RL training can be given in a step-wise manner,
    $$
        \nabla_\theta J_{r}(\pi_\theta)=\mathbb E_{q\sim \mathcal D,s\sim \pi_\theta(\cdot|q)}\large[\nabla_\theta \log\pi_\theta(s|q)\cdot \text{Correct}(q,s)+ \alpha\cdot\sum_{k=1}^{K}\nabla_\theta \log\pi_\theta(s^{(k)}|q,p^{(k-1)})\cdot \text{Return}(q,p^{(k)})\large] +\text{KL term}
    $$
where $\text{Return}(q,p^{(k)})$ denotes the return starting from step $k$. 


1. **RL Training with SR+PR.**
    The return is given by,
    $$
        \text{Return}(q,p^{(k)})=\underbrace{\sum_{k'\ge k}r_{process}(q, p^{(k')})}_{\text{Sum of PRM rewards}}
    $$

    This formulation encourages producing a larger number of reasoning steps, even when the PRM rewards are low. For example, an incorrect solution with 10 steps and an average PRM reward of 0.3 could be preferred over a partially correct solution with only 3 steps and a higher average PRM reward of 0.8. The RL training thus prioritizes the longer solution with lower average rewards.

2. **RL Training with SR+PR-Delta.**
    Introducing the Delta mechanism, the return of PR-Delta starting from step $k$ is adjusted by,
    $$
    \text{Return}(q,p^{(k)})=\left[\sum_{k'=k}^{K-2}r_{process}(q,p^{k'})-r_{process}(q,p^{(k'+1)})\right]+r_{process}(q,p^{(K-1)})=\underbrace{r_{process}(q,p^{(k)})}_\text{PRM reward at step k}
    $$
    Here, the training optimizes **the PRM reward for each intermediate step** rather than focusing on the aggregated reward. Since the PRM is trained to predict the quality of one step, this approach could enhance the reasoning process step by step.

**For further details, please refer to the revised paper, particularly Section 4.**

[1] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a way to utilize process reward models (PRMs) for RL training in LLM reasoning. The paper studies some issues with PRM rewards, followed by a discussion of how adjusting process rewards via clipping and delta mechanisms can help improve performance. The paper presents interesting takeaways -- including positive biases of the PRM, reward hacking of the PRM, and shows performance with their suggested changes to PRM rewards (though some ablations are still not clear; see discussion with Reviewer 7bHH).

While I enjoyed reading the paper, there're some weaknesses: (1) it is unclear if these findings hold true for any PRM, (2) in my opinion and in the opinion of some reviewers as well, the analysis section does not decouple issues with using process rewards vs errors in the process reward model itself, (3) connections to RL literature are not made explicit, and (4) the reviewers are not totally convinced by the ablations for the method proposed (although to the authors' credit, the proposed method is simple). 

Unfortunately while the paper has the potential to be quite impactful, due to the above issues we are not able to accept the paper at this moment.