VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment

Authors: Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, Nicolas Le Roux
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, Reasoning, Credit Assignment, RLHF, Post-Training

Abstract:
Large language models (LLMs) are increasingly applied to complex reasoning tasks that require executing several complex steps before receiving any reward. Properly assigning credit to these steps is essential for enhancing model performance. Proximal Policy Optimization (PPO), a state-of-the-art reinforcement learning (RL) algorithm used for LLM finetuning, employs value networks to tackle credit assignment. However, value networks face challenges in predicting the expected cumulative rewards accurately in complex reasoning tasks, often leading to high-variance updates and suboptimal performance. In this work, we systematically evaluate the efficacy of value networks and reveal their significant shortcomings in reasoning-heavy LLM tasks, showing that they barely outperform a random baseline when comparing alternative steps. To address this, we propose VinePPO, a straightforward approach that leverages the flexibility of language environments to compute unbiased Monte Carlo-based estimates, bypassing the need for large value networks. Our method consistently outperforms PPO and other RL-free baselines across MATH and GSM8K datasets with fewer gradient updates (up to 9x), less wall-clock time (up to 3.0x). These results emphasize the importance of accurate credit assignment in  RL finetuning of LLM and demonstrate VinePPO’s potential as a superior alternative.

Decision: reject
Reviewer scores: 5, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730642749510

thread_type:
official_review

<thread_object>
creator: reviewer_3mkl

<content>
summary:
The key motivation of this manuscript is to locate and solve the problem, while PPO is finetuning LLM，the value network is inaccurate and has high variances. It finds that in heavy and complex reasoning tasks, PPO barely outperform a random baseline due to this issue. Thus, this paper proposes a simple and straightforward approach, so called VinePPO, which computes the value using unbiased Monte Carlo estimation and improve the credit assignment. Many experiments on MATH and GSM8K datasets with RhoMath 1.1B and DeepSeekMath 7B, show that the proposed VinePPO can consistently outperforms PPO and other RL-free baselines.

strengths:
1.	The authors found the problem via systematical evaluation that, inaccurate value estimation can limit PPO’s ability to finetune LLMs in complex reasoning tasks. It can’t reflect the real reward and importance. This problem results in the barely fair performance compared with a random baseline.
2.	This paper proposes the VinePPO via utilizing MC samples to compute value in the PPO pipeline, and the value of a state can be estimated by the average return of K sampled trajectories from the state. 
3.	The experiments and analytics are convincing. The results of VinePPO is better than PPO, via improved credit assignment.

weaknesses:
1.	The proposed VinePPO is a straightforward method to use MC estimation. However, MC has been studied for a long time. It has zero bias, but also has high variance and computational efficiency problem.
2.	This paper adopts the math reasoning problem. The state is the concatenation of input prompt and generated tokens, so the following trajectories can be sampled from any state s, and then MC computation can work. But, if the problem is more complex, not simple math problem, MC might not work, because long trajectory or low efficiency.
3.	In VinePPO, K is very important, because accurate MC estimation needs K be large enough, which also would cause low efficient issue.
4.	Based on the discussion above, the generalizability of VinePPO is not analysed and  solved in the paper.

questions:
1.	The influence of K needs to be discussed, from both performance and efficiency.
2.	As MC is a method with high variance, does VinePPO outperform PPO when MC estimation is not inaccurate?
3.	In Fig 9, the ground truth is chosen as results via 256 MC samples. Is this reasonable?
4.	It might be more convincing to provide the resulting of credit assignment. For example, are critical steps detected by VinePPO?
5.	Some equations are not clear, for example:$S_{t+1} =  s_t;[a_t]$
</content>
</thread_object>

</thread>
<thread>
creation date:
1730773418387

thread_type:
official_review

<thread_object>
creator: reviewer_1twu

<content>
summary:
Large Language Models (LLMs) are increasingly being applied to complex reasoning tasks, ranging from solving math problems to developing code. The most common method for training these models is Proximal Policy Optimization (PPO), which addresses the credit assignment problem using a value network. However, this value network can be significantly biased, which may impact performance. The authors propose VinePPO, inspired by VineTRPO, to learn a value function from Monte Carlo (MC) samples. They demonstrate that the value function from PPO performs poorly, while the MC sample estimates of the value function show strong performance and leverage compute-efficient inference techniques.

strengths:
- The author's observation that the issue with PPO was the value function estimates is very insightful, given that there has been a lot of work to replace PPO with new techniques.
- The paper was well written.
- The paper's experimental results provide a lot of interesting insights regarding issues around PPO's value function.
- The authors performed experiments across several tasks, model sizes, and model types.
- The authors' ablations studies show interesting pitfalls of the value function from PPO. Additionally, the authors clarify the tradeoff between VinePPO and PPO.

weaknesses:
- I understand that the paper focuses on addressing the pitfalls of PPO; however, comparing it with RLOO [1] would provide practitioners with valuable context on which algorithm they might want to use in practice.
- The paper lacks details on how the inference engines were utilized to accelerate data gathering.

[1] Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs by Ahmadian et al. 2024

questions:
- Missing citations
   - RL + LLM: [1, 4]
   - RL: [2, 3, 5]
- How does VinePPO compare to the RLOO baseline as the value of K increases in RLOO?
-Did you do large-batch PPO updates? (Refer to [6] for the large-batch updates.) If you didn’t use the large-batch setting, essentially what you do is compute all the data statistics offline. This approach allows you to avoid loading the reward model onto the GPU, enabling you to increase your batch size much higher than if the reward model were loaded onto the GPU.
- Why is PPO more deterministic in early steps, while VinePPO is more deterministic in later steps, as mentioned in the "Error per reasoning step" section?
- Could you share a plot showing the "explained variance" of the value function you learn with normal PPO? (see [7])

[1] Learning to Generate Better Than Your LLM by Chang et. al 2023
[2] Exploring restart distributions by Tavakoli et al. 2018
[3] Data-efficient deep reinforcement learning for dexterous manipulation by Popov et al. 2017
[4] Dataset Reset Policy Optimization for RLHF by Chang et al 2024
[5] Mastering the game of Go with deep neural networks and tree search by Huang 2016
[6] SimPO: Simple Preference Optimization with a Reference-Free Reward by Meng et al. 2024
[7] http://joschu.net/docs/nuts-and-bolts.pdf
</content>
</thread_object>

</thread>
<thread>
creation date:
1731051760128

thread_type:
official_review

<thread_object>
creator: reviewer_s7we

<content>
summary:
The paper proposes vine-PPO, which uses Monte Carlo-based estimates to replace the value function. This approach is far more accurate and therefore performs better than the parameterized value function. Although the cost could be a concern, the authors argue that inference or generation is much faster due to many inference-optimized modules. Additionally, because of the rapid increase in performance, it may even be more efficient.

strengths:
- The idea of Monte Carlo estimates, although it has been used in traditional RL tasks, is novel for PPO in the context of LLMs. I find it quite interesting that it can achieve superior results even with K=1.
- The applicability stemming from the fact that it only replaces the value function, allowing it to be used in many PPO-like methods, is highly beneficial.
- The analysis of the value function helps clarify the motivation.
- The proposed method is simple and easy to follow, and the paper is well-written.

weaknesses:
- Fundamentally, I think the difference between your approach and GRPO [1] and RLOO [2] is that you have fine-grained value estimations by generating multiple responses from each intermediate group state. However, since this involves more computation, I wonder about the trade-offs compared to GRPO.
- This question arises because you do not compare your method with GRPO and RLOO. As these methods also employ similar ideas, why only compare with the original PPO? The authors should clearly explain the selection of baselines, and efficiency comparisons should also include this line of research.
- Furthermore, I wonder why you do not report baselines that use finer credit assignment for the DPO objective. Since you report that PPO performs better in terms of credit assignment, I am curious how it still shows superiority even when DPO is combined with finer credit assignment.
- Additionally, in practical situations, if one needs to find an optimal K for training configuration, it’s unclear whether we can say that Vine-PPO is more efficient in general, as it might require more hand-engineering. However, training the value network also requires engineering, so I wonder about the complexity comparison between these methods.

References

[1] Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

[2] Ahmadian et al. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

questions:
- How does the method's dependency on K differ by model? I am also curious about the K ablation.
- Additionally, I think creating a graph to show the trade-off between larger K values and efficiency would be interesting.
- Very minor, but there is a missing period on line 264 (or 265).
</content>
</thread_object>

</thread>
<thread>
creation date:
1731254624405

thread_type:
official_review

<thread_object>
creator: reviewer_wh2b

<content>
summary:
VinePPO uses Monte Carlo-based credit assignment, reducing reliance on large value networks and enhancing accuracy and efficiency. It outperforms PPO and other baselines on complex math tasks, particularly with challenging datasets. Performance improves with more Monte Carlo samples, demonstrating strong scalability potential.

strengths:
VinePPO uses Monte Carlo-based credit assignment, reducing reliance on large value networks and enhancing accuracy and efficiency. It outperforms PPO and other baselines on complex math tasks, particularly with challenging datasets. Performance improves with more Monte Carlo samples, demonstrating strong scalability potential.

weaknesses:
1. Lack of baselines. I suggest the author adding value-network-free methods as baselines, particularly GRPO [1] which also uses a PPO-like objective with the average reward of multiple rollouts as the baseline for the policy gradient.
2. Misuse of terminology. According to the hyperparameter setting for PPO provided in the Appendix where $\lambda = 1$ and $\gamma = 1$, PPO should produce an unbiased estimate for the value function. So it is better not to use "bias" in Line 467 and 475 but to use "inaccuracy".

questions:
Questions:
1. The results show that VinePPO is quite promising for LLM reasoning, but can we extend it to the more general alignment task?
2. Is there any intuitive or theoretical explanation for why value networks fail to provide accurate estimates?

[1] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. CoRR, abs/2402.03300.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732020345633

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary Response + RLOO and GRPO Baselines

comment:
## Summary Response 

We thank the reviewers for their feedback and helpful comments.

We are heartened to hear that Reviewer S7We found VinePPO to be a “novel method in context of RL finetuning of LLMs” where “far more accurate” Monte Carlo-based estimates replace the value function achieving superior result while being “simple, easy to follow and applicable to many PPO-like conditions” — and as highlighted by Reviewer 1TWU a “very insightful” finding that the value function is an important pitfall of the PPO. We are further pleased to hear that the reviewer w2hB finds VinePPO demonstrating “strong scalability potential” while reviewer 1TWU, 3mkL, and wh2B found our experiments to be thorough “ across several tasks, model sizes, and model types” “consistently outperforming PPO and other RL-free baselines” on challenging math datasets. 

We now clarify the main shared concern regarding “including GRPO and RLOO baselines” among the reviewers below and address reviewer specific questions in the individual responses.

## RLOO and GRPO Baselines

Please scroll to page 31 of the updated draft where we put all new contents for reviewers’ convenience. We will update the main content for camera-ready. The results of the RhoMath1.1B model on GSM8K and MATH are presented in appendix E of the paper (See Figure E.20, E.21). Our 7B models need more compute and powerful hardware. We are actively working on these runs and will update the paper if progress is made (*Update 11/21/2024: Added RLOO with DeepSeekMath 7B on GSM8K*).


**Result** As shown in Figure E.20, RLOO and GRPO perform worse than VinePPO. On GSM8K, RLOO and GRPO achieve 44.5% and 44.6%, respectively, compared to PPO's 50.1% and VinePPO's 53.4%. On MATH, they score 17.3% and 17.8%, while PPO and VinePPO reach 18.1% and 23.0%. These findings align with recent studies [1][2], where RLOO was found to be at best competitive with PPO (performance-wise). 

**Discussion** We agree with the reviewers that RLOO and GRPO baselines are useful for practitioners and we thank the reviewers for suggesting them. Meanwhile, we think it is important to note that VinePPO is in the opposite direction of RLOO and GRPO. RLOO and GRPO remove the fine-grained credit assignment machinery of PPO, the value network, and they basically assign every token in a response the same value. On the other hand, VinePPO doubles down on fixing the fine-grained credit assignment machinery, estimating accurate values via MC samples starting from each step. (see detailed analysis below).

**Implementation and Training Details of RLOO and GRPO** Due to character limit we've put it in the first reply to this comment.

## Value Prediction Analysis of RLOO and GRPO
We follow the same protocol in Section 7 of the original draft to analyze the accuracy of value prediction for RLOO and GRPO. 

**Scatter Plot + Mean Absolute Error** Figures G.24 and G.25 illustrate the distribution of value predictions across reasoning steps. RLOO and GRPO estimates show significant bias, frequently assigning high values to states with a low probability of success and low values to states with high probability. As demonstrated in Fig. G.26, although RLOO and GRPO have marginally lower MAE than PPO, their errors are still substantially higher compared to VinePPO.

**Explained Variance** Based on 1TWU suggestion, we additionally include the explained variance of value estimation in these methods. As shown in Figure G.26, VinePPO achieves higher explained variance than RLOO, GRPO, and PPO across both datasets.
Additionally, PPO’s value predictions show non-negative explained variance values close to one, indicating stable and effective training. 

## Compute Efficiency Analysis of RLOO and GRPO
Following the approach in Sec 6.4 of the original paper, we plot test set accuracy against wall-clock time under the same hardware configuration to evaluate the computational efficiency of RLOO and GRPO compared to PPO and VinePPO. As shown in Figure F.22 of the updated draft, on the MATH dataset, VinePPO reaches the peak performance of RLOO and GRPO 2.7x and 2.2x faster, respectively. Notably, on GSM8K, we see the same pattern and even PPO—despite the overhead of training an additional network—surpasses RLOO and GRPO in efficiency.



- [1] Noukhovitch, Michael, et al. "Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models." arXiv preprint arXiv:2410.18252 (2024).
- [2] https://huggingface.co/blog/putting_rl_back_in_rlhf_with_rloo
</content>
</thread_object>

</thread>
<thread>
creation date:
1732314651568

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated Paper Revision

comment:
Dear Reviewers and Area Chairs,


We summarize the updates to the paper during rebuttal, requested by the reviewers. Note that they are at pages 31 to 38 for reviewers’ convenience and the title of the sections are in blue. Specifically:

**Reviewer wh2b**:
1. Added RLOO and GRPO Baselines (Appendix E) 
2. Add more details regarding terminology "bias" (Appendix I)



**Reviewer S7We**:
1. Added RLOO and GRPO Baselines (Appendix E: Figures E20, E20.1, and E21) 
2. Added RLOO and GRPO Efficiency Analysis (Appendix F: Figure F.22)
3. Fixed typo (Line 264)


**Reviewer 1TWU**:
1. Added RLOO and GRPO Baselines (Appendix E: Figures E20, E20.1, and E21) 
2. Added visualization of “Explained Variance” metric throughout training for all methods (Appendix G: Figure G26)
3. Added technical details of inference engines in our software stack (Appendix J)


**Reviewer 3mkL**:
1. Added More Examples of Advantages in VinePPO (Appendix H: Figures H.27, H.28, and H.29)
2. Analyzed effect of K in VinePPO’s Efficiency (Appendix F: Figure F.23)


**Additional Analysis**:
1. Added Value Prediction Analysis of RLOO and GRPO shedding light into their inner working mechanism (Appendix G: Figures G.24 and G.25)

We are eager to engage in further discussion and would greatly appreciate any additional feedback or insights, especially as none of the reviewers have not yet had the opportunity to share their thoughts at the time of posting this.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732463209540

thread_type:
official_comment

<thread_object>
creator: area_chair_xvkp

<content>
title:
Reminder: Author-Reviewer Discussion Period Closing Soon

comment:
This is a reminder that the author-reviewer discussion period will end on Nov 26 AoE.

Your engagement during this phase is critical for providing valuable feedback and clarifications. If you have any remaining questions or comments, please take a moment to participate before the deadline.

Thank you for your contributions to this important process.

AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732925654491

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Updates to the Paper - Nov 27th

comment:
Dear Reviewers,

The results of our 7B models trained with RLOO are now finalized, and the updated paper includes RLOO results for all models and datasets. Please refer to Figure E.20.1 for the 7B results and Figure E.20 for the 1B results. These results should be particularly relevant to reviewers S7We and 1TWU, who requested the RLOO baseline.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733290157163

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Response

comment:
We thank all reviewers for their time and feedback.


## Summary of Discussion Period


The major concern raised by reviewers was the inclusion of RLOO and GRPO baselines. In response, **we updated the paper (added 7 pages + 11 figures), thoroughly implementing and analyzing these baselines in terms of final performance, efficiency, and value estimation accuracy.** Despite all our hyperparameter tuning (see General Response), RLOO and GRPO underperform VinePPO, even when controlling for compute. This is not a surprise and aligns well with our primary message on the importance of credit assignment. As shown in our additional analysis (Figs. G.24-G.26), RLOO and GRPO assign biased value estimates to intermediate steps, adversely affecting performance. These results are in line with recent studies on RLOO and GRPO [4,5]. Finally, besides our active participation, reviewers raised no further concerns during the discussion period.


## Recap of VinePPO


As noted by Reviewer 1TWU, many recent works [1,2,3, inter alia] attempt to simplify PPO in the context of RLHF by removing critical components, including credit assignment mechanisms, often with little to no drop in performance (reasoning tasks are an exception [3]). VinePPO is, to our knowledge, the first to address this contradiction by demonstrating that the credit assignment machinery of standard PPO (i.e., value networks) is underperforming. **VinePPO goes in the opposite direction** and attempts to fix this mechanism, demonstrating profound impacts across various axes: higher accuracy, faster convergence, better efficiency, and lower KL divergence. Here’s brief overview the credit assignment mechanisms of such methods:


| **Method**       | **Fine-grained Credit Assignment** | **Notes**  |
|-------------------|------------------------------------|------------|
| PPO (2022, [6])               | Yes                                | Trains a value network to predict the value each step during      |
| DPO (2023, [1])             | No                               | N/A       |
| RLOO & GRPO (2024, [2])      | No                               |  By design assign the same value to all steps.      |
| VinePPO (Ours)          | Yes                                |  Uses MC estimation to compute the value of each step.       |


**Primary Message of our Work**


Credit assignment, despite its importance in DeepRL, has become an overlooked aspect of RL methods for LLMs, with newer approaches often removing it entirely. Our work highlights the critical importance of this component, and we hope it encourages further research into this aspect of RL training for LLMs.


While VinePPO represents the initial attempt to principally address credit assignment in this context, it is **simple** (Reviewer S7We), **scalable** (Reviewer wh2B), and **generalizable** (Reviewer S7We).




---
1. Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” by Rafailov et al, 2023
2. Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs, Ahmadian et al, 2024
3. SimPO: Simple Preference Optimization with a Reference-Free Reward by Meng et al. 2024
4. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models by Noukhovitch et al, 2024
5. https://huggingface.co/blog/putting_rl_back_in_rlhf_with_rloo
6. Training language models to follow instructions with human feedback by Ouyang et al, 2022
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary of Scientific Claims and Findings

The paper introduces VinePPO, a reinforcement learning algorithm leveraging Monte Carlo-based methods to enhance credit assignment for fine-tuning large language models (LLMs) on reasoning tasks. VinePPO overcomes the limitations of Proximal Policy Optimization (PPO), particularly addressing issues with high variance and suboptimal performance of PPO’s value network in complex reasoning scenarios.

(b) Strengths of the Paper

1. The authors propose Monte Carlo-based credit assignment as a novel approach in the context of LLM fine-tuning.

2. Extensive experiments on challenging reasoning datasets (MATH, GSM8K) validate VinePPO’s superior performance and efficiency.

(c) Weaknesses of the Paper and Missing Elements

1. The initial absence of comparisons with key baselines (e.g., GRPO, RLOO) was a major concern, and while the rebuttals addressed these issues, some uncertainties about the experimental setup persist.

2. There is limited discussion on applying VinePPO to tasks beyond reasoning-intensive problems, particularly those with longer trajectories or greater computational complexity.

3. The parameter K, central to Monte Carlo sampling, requires further investigation regarding its impact on computational cost and overall efficiency.

(d) Decision and Rationale

Reviewers recognized VinePPO’s contributions to advancing reinforcement learning with human feedback (RLHF) for LLMs and its demonstrated empirical advantages. However, concerns about incomplete baseline comparisons, generalizability, and the lack of deeper theoretical insights resulted in mixed evaluations.