VinePPO: Refining Credit Assignment in RL Training of LLMs
Authors: Amirhossein Kazemnejad
Milad Aghajohari
Eva Portelance
Alessandro Sordoni
Siva Reddy
Aaron Courville
Nicolas Le Roux
Venue: ICML 2025 poster (2025)
Keywords: RL for LLM
Verifiable Rewards
Credit Assignment
Primary area: deep_learning->large_language_models

Abstract: Large language models (LLMs) are increasingly applied to complex reasoning tasks that require executing several complex steps before receiving any reward. Properly assigning credit to these steps is essential for enhancing model performance. Proximal Policy Optimization (PPO), a common reinforcement learning (RL) algorithm used for LLM finetuning, employs value networks to tackle credit assignment. However, recent approaches achieve strong results without it, raising questions about the efficacy of value networks in practice. In this work, we systematically evaluate the efficacy of value networks and reveal their significant shortcomings in reasoning-heavy LLM tasks, showing that they often produce poor estimate of expected return and barely outperform a random baseline when comparing alternative steps. This motivates our key question: Can improved credit assignment enhance RL training for LLMs? To address this, we propose VinePPO, a straightforward approach that leverages the flexibility of language environments to compute unbiased Monte Carlo-based estimates. Our method consistently outperforms PPO and other baselines across MATH and GSM8K datasets in less wall-clock time (up to 3.0x).  Crucially, it achieves higher test accuracy for a given training accuracy, capturing more generalization signal per sample. These results emphasize the importance of accurate credit assignment in RL training of LLM.

VinePPO: Refining Credit Assignment in RL Training of LLMs
Amirhossein Kazemnejad
Milad Aghajohari
Eva Portelance Alessandro Sordoni 12 Siva Reddy 134
Aaron Courville 145
Nicolas Le Roux 14
Abstract
1. Introduction
Reinforcement learning (RL) has become instrumental in training large language models (LLMs) to solve complex reasoning tasks such as mathematical problem solving (DeepSeek-Al et al., 2025), web navigation (Putta al., 2024), or code generation (OpenAI, 2024). In these settings, LLMs often engage in extended reasoning steps, executing multiple actions to arrive at solution. However, not all steps are equally impactful-some contribute significantly, while others are irrelevant or detrimental. For example, in Figure La, only step $2 provides key insight. Indeed, most reasoning steps generated by model do not affect the chance of it solving the problem (Figure 1.b). Identify- ing the contribution of each action is crucial for improving model performance. However, this is inherently difficult due to the significant delay between actions and their eventual effect. This issue, known as the Credit Assignment (CA) problem, is a core challenge in RL (Sutton & Barto, 1998).
Large language models (LLMs) are increasingly applied to complex reasoning tasks that require executing several complex steps before receiving any reward. Properly assigning credit to these steps is essential for enhancing model perfor- mance. Proximal Policy Optimization (PPO), a common reinforcement learning (RL) algorithm used for LLM finetuning, employs value networks to tackle credit assignment. However, recent ap- proaches achieve strong results without it, raising questions about the efficacy of value networks practice. In this work, we systematically evaluate the efficacy of value networks and reveal their sig- nificant shortcomings in reasoning heavy LLM tasks, showing that they often produce poor es- timate of expected return and barely outperform a random baseline when comparing alternative steps. This motivates our key question: Can im- proved credit assignment enhance RL training for LLMs? To address this, we propose VinePPO, a straightforward approach that leverages the flexi- bility of language environments to compute unbi- ased Monte Carlo-based estimates. Our method consistently outperforms PPO and other baselines across MATH and GSMSK datasets in less wall- clock time (up to 3.0x). Crucially, achieves higher test accuracy for given training accuracy, capturing more generalization signal per sample. These results emphasize the importance of accu- rate credit assignment in RL training of LLM.
Proximal Policy Optimization (PPO: Schulman al. 2017) addresses credit assignment through a value network (or critic), mechanism retained in its application to RL-based finetuning of LLMs (Ouyang et al., 2022). This network, typically separate model initialized from a pretrained checkpoint, is trained during PPO finetuning to estimate the expected cumulative rewards (or value) of an interme- diate action. In Figure 1.b, an ideal value network would assign high value to step $2 and subsequent steps, where the model has high chance of successfully solving the problem. PPO uses these value estimates to measure the advantage of each action and update the model accordingly.
However, recent approaches such as DPO (Rafailov et al., 2023) or GRPO (Shao et al., 2024) simplify PPO's design by discarding fine- tne-ggrined credit assignment and treating all tokens equally. Despite such simplifications, they often demonstrate strong performance (Xu et al., 2024; Chang et al., 2023). This challenges classic RL principles, where accurate CA is considered critical for optimal performance (Sutton & Barto, 1998; Greensmith et al., 2001), especially in tasks with delayed rewards. In this work, we address this apparent discrepancy by showing that PPO's credit as- signment mechanism, the value network, performs poorly in practice. Our systematic evaluation (Section 7) tasks
Code available at https: httt://c..ttht..oo//oo McGill-NLP/A -LVVii11MPP/VinePP /VinePPO
'Equal contribution 'Equal advising 'Mila Microssoft Research MMeGill University 'Canada CIFAR AI Chair Uniiversité de Montréal "HEC Montréal. Correspondence to: Amirhossein Kazemnejad Cammrrossein.kazemeee kazemnejad @illa.uubeee>.... Milad Aghajohari <aghajohm qullee.e
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
Prompt (8g)
p[correat (.))
Advantage
MascSSokhMathh
Let and & be nonzem real numbers such that (2 (a + bi) is pare imaginary. Find g
40%
Chain-of-Thought Response
20%
We expand the left-hand side get
0%
(a + bx) (2a 76) (- 7a 26)s.
This pare imaginary and only the real part O,
60%
200Maan 1.18
2a=
Then 36 NO
1.0
0.0
20%
0%
-1.0
-0.5
0.5
(a) A Sumple Rexponse
(b) Advantage
Figure 1: (Left) A response generated by the model. The notation p(correct(s.. represents the estimated probability of successfully solving the problem at step f. Here, only step 82 is critical; after this, the model always completes the solution correctly. (Right) The distribution of advantages, defined as p[correct) 1a+1) - p(correct|s; s.t), collected over a subset of MATH dataset (Hendrycks et al., 2021). Most steps show little or no advantage over the preceding step.
requiring seain-of-thoughhe reasoning reveals that the value network often provides imprecise estimates and fails to dif- ferentiate between promising and unproductive steps, which could explain why simplified approaches achieve compara- ble results without explicit fine-grained CA.
We analyze PPO's value network in reasoning tasks and find it often misestimates intermediate values, barely out- performing a random chance in ranking candidate steps.
We propose VinePPO, leveraging the flexibility of lan- guage environments compute unbiased, MC-based value estimates without relying separate critic.
These findings motivate central question: If we improve credit assignment in PPO rather than discarding it, how much enhance the RL training of LLMs? To explore this, propose VinePPO (Figure 2). which computes unbi- ased value estimates of the intermediate states with Monte Carlo (MC) estimation instead of employing value networks. Our key insight is that language- environments allow us to reset directly to any intermediate state simply by re- feeding the partial context, enabling efficient MC rollouts without the massive overhead usually seen in generic RL environments. VinePPO preserves PPO's overall framework but addresses the CA challenge fundamentally.
We empirically highlights the benefits of refined CA. VinePPO achieves the peak performance of baselines with less wall-clock time (up to 3.0x). better KL-divergence trade-off while exhibiting better generalization slope.
2. Related Work
Credit Assignment in Post-Training of LLM PPO, as applied in RL from Human Feedback (RLHF, Ouyang et al. 2022), pioneered RL finetuning of LLMs. However, its computational overhead and hyperparameter sensitivity led to the development of simpler alternatives. RL-free meth- ods such as DPO (Rafailov et al., 2023) operate in a bandit setting, treating the entire response single action. Simi- larly, rejection sampling methods like RestEM (Singh et al., 2024) finetune on full high-reward responses. RLOO (Ah- madian et al., 2024) and GRPO (Shao et al., 2024) abandon PPO's value network, instead using average reward from multiple samples as a policy gradient baseline. Recent work has emphasized finer credit assignment, with Hwang et al. (2024) and Setlur et al. (2024) introducing MC-based meth- ods to detect key errors in reasoning chains for use as ad-hoo hoc mechanisms in DPO. Our work, by contrast, fully embraces the RL training, with the target of fixing CA in principle. Parallel efforts have also focused on building better veri- fiers and reward models for per-step feedback, with recent attempts to automate their data collection using MC rollouts
We empirically evaluate the effectiveness and computational efficiency of MC value estimation in VinePPO. Across mul- tiple mathematical reasoning tasks, VinePPO consistently outperforms PPO and other credit assignment-free baselines. While its per-iteration runtime is generally slower due to MC sampling, VinePPO surpasses the peak performance of baselines with fewer gradient steps and ultimately less wall-clock time. Importantly, VinePPO achieves higher test accuracy for a given training accuracy, capturing more generalization signal per fitted training sample. This criti- cal, as gemuinely challenging verifiable reasoning tasks are scarce. These results underscore the importance of CA in RL-training of LLMs and highlight VinePPO straight- forward alternative to value network- approaches.
Our contributions are summarized as follows:
2

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
RLOO GRPO
PPO
VinePPO
a:
RLOO: (x) - d Σ R(n)
a
Va(+e) = ValNet(se
x
Exc(()) * R(
GRPO: Va(m) 1. R(a))
81
(x) 5 1)
-
b1= (s1)
81
Vxcc(e))
0.5
0.1934
0.66
82
(x) (00+1)
82 12= (52)
82
bam Vssc(#2)
0.3
0.3733
%
1.00
r3 0
0
0
0
Figure 2: Comparison of credit assignment mechanisms applied on training trajectories Tz's, depicted for states and sg. (Left) RLOO and GRPO both treat all intermediate states equally and use the average return of trajectory group T; ~ x (-12z) for the policy-gradien baselines b and b2. GRPO additionally normalize these returns to have a unit variance. In the case of RLOO, the computed baseline could be viewed as MC estimate of value but solely for the initial state. (Middle) PPO trains separate model to predict values for each state 8t- (Right) VinePPO generate auxiliary rollouts 77 -T(sss)) to obtain MC estimate of state 's value. Note that 77 's are only used for value estimation not update the policy directly.
(Ma et al., 2023; Uesato et al., 2022; Luo et al., 2024; Wang et al., 2024). Our method is orthogonal to these methods, operating within PPO-based training to optimize a given reward, instead of designing new ones.
VinePPO
PPO
MAE: 0.03
MAE: 0.11
1.00
1.00
0.75
0.75
0.50
0.50
0
0.25
0.25
Value Estimation in RL and Monte Carlo Tree Search (MCTS) Deep RL algorithms are typically categorized into value-based and policy-based methods. Policy-based methods like PPO usually employ critic networks for value prediction. An exception is the "Vine" variant of TRPO (Schulman et al., 2015), which uses MC samples for state value estimation. The authors, however, note that the Vine variant is limited to environments that allow intermediate state resets, rare in typical RL settings!. However, language generation when formulated RL environment enables such intermediate reset capabilities. In domains with similar reset capabilities, such as Go and Chess, MC-hased methods like AlphaGo (Silver et al., 2016) and AlphaZero (Silver et al., 2017) have emerged. AlphaGo's architecture includes a policy, trained using expert moves and self-play, and a value network that predicts game outcomes. At inference, employs tree search guided by MC rollouts and value network to select optimal moves. AlphaZero advances this approach by distilling MCTS outcomes into the policy. Re- cent works have adapted AlphaZero's 's principles to LLMs, employing similar search techniques for inference and tra- jectory distillation (Xie et al., 2024; Chen et al., 2024; Wan et al., 2024; Zhang et al., 2024; Hao et al., 2023). While this is a promising direction, our method is not an MCTS approach; it uses MC samples solely for value estimation during PPO training to improve credit assignment.
Predicted Value
0.00
0.00
0.00
0.25
0.50
0.75 1.00
0.00
0.25
0.50
0.75 1.00
RLOO
GRPO
MAE: 0.16
MAE: 0.15
1.00
1.00
0.75
0.75
St0p
2m
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75 1.00
Ground Truth Value
Figure 3: Distribution of predicted values for each state vs. ground truth (see Section 7 for details) for DeepSeekMath on the MATH dataset, highlighting the nature of errors: PPO exhibits biased value predictions, whereas VinePPO remains unbiased. Note that RLOO/GRPO do not predict values; plot their computed baselines against the ground truth value solely for demonstration.
3. Background
We focus on the RL tuning phase, following Ouyang et al. (2022); Shao et al. (2024). In this section, we provide an overview of actor-critic finetuning as implemented in the standard PPO framework.
This reflected in the design of Gym (Towers al., 2024), which only allows resets to the initial state.
3

---

VinePPO: Refining Credit Assignment in RL Training of LLMS
RL Finetuning In this setup, the policy Te repre- sents language model that generates response y [y0. (yo..... -T- autoregressively given an input x = .M......MMMII The goal of RL finetuning is to maximize the expected undiscounted (7 = 1) finite-horizon return, while incorporating a KL-divergence constraint to regular- ize the policy and prevent it from deviating too far from a reference policy Tref (typically the initial supervised fine- tuned, SFT, model). The objective can be written as:
8, under the policy. Formally, it is defined as:
A(sr; uz) = =0( az) - V(sz) = + (V(5++)) - V(s)).
(3)
where Q(+r, oz) is the state-action value and V((+) is the per-state value function?. The value function, V(s:) S + R. offers long-term assessment of how desir- able particular state is under the current policy. For- mally, it represents the expected cumulative reward ob- tained from starting in state St and following the policy thereafter': V(s)) = Er-a, [R(T] 60 st] PPO uses the same advantage-weige tion 2, but constrains policy updates through clipping to ensure stable training. For full details, see Appendix A.
(0) = Ee- [R y) , KL (To [ee]] (1)
policy gradient as in Equa-
where D is the dataset of prompts, R(x; y) is the sequence- level reward function, and A controls the strength of the KL penalty. Note that the policy To is initialized from Trof-
Estimating Advantage via Value Networks In practice, the advantage A(s,, az) is not known beforehand and is typically estimated by first using a value network VA to approximate the true value function V (st), then substituting the learned values into Equation 3 or alternative methods like GAE (Schulman et al., 2016). The value network parameterized by 6 and trained alongside the policy network TT. The training objective for the value network minimizes the mean squared error between the predicted value and the empirical return:
Language Environment as an MDP Language gener- ation typically modeled as a token-level Markov De- cision Process (MDP) in an actor-critic setting, where each response y is an episode. The state at time step 1, 8, E S, is the concatenation of the input prompt and the tokens generated up to that point: st = x; y<t = .M..... 3/o. ...., 1I) At each time step, the action a, corresponds to generating the next token yt from fixed vocabulary. The process begins with the initial state 8p = x, and after each action, the environment transitions the next state, 1 = s;; [ar]. by appending the action a, to the current state st. In this case, since states are always con- structed by concatenating tokens, the environment dynamics are known and the transition function is deterministic, i.e., ((5441/52 ot) = 1. During the generation process, the reward It is set to zero for all intermediate actions az s, with the sequence-level reward R(=; y) only applied at the final step when the model stops generating. A trajectory (50. do, $1, 01,...) is therefore sequence of state- action pairs, starting from the input prompt until the ter- minal state. Finally, we define the cumulative return of a trajectory J as R(T)= = EI
Lv(()) = ET 1 ト
E - -G)² (4)
where G. = Et=l TV is the empirical return from state St. PPO uses the same objective for Ve but applies clipping for training stability (see Appendix A. for details). In RL-tuning of LLMs, the value network is often initialized using the initial SFT policy To (or the reward model when available). with the language modeling head swapped out for a scalar head to predict values (Zheng et al., 2023). This setup leverages the prior knowledge of the pretrained model.
-I-1 ri= = rT-1 = R(x; y)
4. Accurate Credit Assignment with VinePPO
Policy Gradient Given this MDP formulation, policy gra- dient methods like PPO maximize Equation by repeatedly sampling trajectories and taking a step in the direction of the gradient 9pg = ((f( update the parameters. Policy gradient frz takes the following form:
As outlined in Section 3, a step in the PPO gradient update aims to increase the probability of better-than better-than-avveage ac- tions while decreasing the probability of those that perform worse process quantified by the advantage A(s,, at). However, the true advantage is generally unknown and must be estimated, typically by substituting estimates from value network into Equation 3. As we will elaborate in Section 7, value networks are often inaccurate and result in biased value computation. Fortunately, the language envi- ronment as an MDP (Section 3) offers a useful property that
9pg- = EYNNT Σ Vo lolog (o (o((a))a))))
(2)
where 8; = F; yKtt, ax Vt. and A(st, ar) is the advantage function. If A(s,, a.) > O. 9og will increase the probability of action a, in state st, and decrease it when A(s,, ax) < O. Intuitively, the advantage function quantifies how much better action ot is compared to average actions taken in state
Such derivation is possible as the language environment is deterministic.
"We drop the dependency on mo for brevity.

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
allows for unbiased estimation of (st). Since states are simply concatenated tokens, we can prompt the language model Toe to generate continuations from any intermediate state. This flexibility allows us to explore alternative future paths from arbitrary points in a generation.
5. Experimental Setup
Datasets and Pretrained LLMs We conduct experiments on publicly available LLMs and datasets to ensure repro- ducibility. We use base versions of DeepSeekMath 7B (Shao et al., 2024) and RhoMath 1.1B (Lin et al., 2024) which are pretrained on mathematical and natural language cor- pora, We chose mathematical reasoning datasets MATH (Hendrycks et al., 2021), competition-level mathematical problems, and GSM8K (Cobbe et al., 2021), simpler grade- school level math word problems. Both datasets are well- established and present range of difficulty levels. For each dataset, we finetune the base LLM on its respective train- ing set to obtain the initial SFT policy (Tzet). Throughout the paper, model names refer ones initialized from these SFT checkpoints. We employ full-parameter finetuming to leverage the models' full capacity (Biderman et al., 2024).
Specifically, computing the advantage requires access to (s²)) = [R(7) so = =t] VinePPO obtain an MC es- timation of this expectation by randomly sampling con- tinuations and averaging their returns. That is, for each state 8, in training trajectory т, we utilize the resetting property and re-feed the partial context corresponding to 8t to the current policy to sample K auxiliary rollouts m..... . ~ TT(- st). The empirical mean of returns across these rollouts serves as the value estimate:
(MC(s:)=
Σ R(mx). K
1
Baselines Our main baseline is the standard PPO frame- work (Ouyang et al., 2022; Huang et al., 2024), which VinePPO builds on and improves through better credit as- signment. We also compare against PPO variants that forego the credit assignment: RLOO (Ahmadian al., 2024) and GRPO (Shao et al., 2024). For RL-frre alternatives, we include RestEM (Singh et al., 2024), a form of iterative re- jection finetuning (Yuan et al., 2023; Anthony et al., 201 7), and DPO+ (Pal et al., 2024), a working variant of DPO with strong performance on reasoning. Except VinePPO and standard PPO, all other baselines omit explicit credit assignment by design: i.e. they assign the same weight to all the tokens of a response. All methods use the same SFT checkpoint to ensure fair comparison. For each experiment, we choose the best checkpoint based on a held-out valida- tion set. We compare all methods by accuracy (Pass @ 1) on test sets, measuring the correctness of final answers.
Critically, 7h's 's are used exclusively for value estimation and do not contribute directly to policy gradient updates as we lack CA on them. Once the value (M))(s) is computed, we estimate the advantages of each action using Equation 3:
Asc ox) = r(st. 0L) + (V4c(5141) - VMc((s)) (6)
For any K > 1, the policy gradient computed using the advantage estimator Amc is an unbiased estimate of the gradient of expected return 9pg- PPO framework then uses Ame to update the policy on trajectory
Variance and computational efficiency represent core trade- offs in every Monte Carlo estimation. Here, the sampling parameter K control such tradeoff- increasing K reduces estimator variance at the expense of increased sampling demands. In Section 6, we rigorously characterize these properties for VinePPO.
Training Details and Hyperparameters We adopt a bi- nary task reward R that evaluates final answer correctness against ground truth, following previous work (Pal et al., 2024; Singh et al., 2024). To ensure fair comparison, all methods consume the same number of episodes during train- ing: for each question, we sample eight episodes and go over the dataset 8 times, yielding 64 episodes per question across all methods. For PPO, we first conduct an exten- sive hyperparameter search (such as KL penalty coefficient, batch size, minibatch size, GAE X, number of epochs per iteration) and rigorously implement all established best prac- tices and well-known techniques (Huang et al., 2024; Ivison et al., 2024) (Refer to Appendix C.2 for the full list). This ensures our evaluation reflects PPO's state-of-the-ar con- figuration and its full potential. VinePPO inherits PPO's exact hyperparameters and only modifies the advantage es- timation, keeping the rest unchanged. This design allows us to isolate the effect of refined credit assignment. For PPO
To enhance the efficiency of AMC, we group states within a reasoning step and compute a single advantage, which is assigned to all tokens in that step (cxamples in Appendix B). This trades off granularity for efficiency, allowing finer reso- lution with more compute, or coarser estimates with limited resources. Furthermore, modern L.M inference engines (Kwon et al., 2023; Zheng et al., 2024) enable rapid on-the- fly generation", making our MC-based approach computa- tionally practical at scale.
By restricting modifications only to the advantage compu- tation stage of PPO, our approach also isolates the effects of improved credit assignment, revealing how unbiased ad- vantage estimation fundamentally alters policy optimization dynamics compared to value-network baselines.
Achieving up 5K tokens/second single Nvidia 100 GPU for 7B L.LMM in 16
5

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
GSMBK
MATH
RhoMath 1.1B
RhoMath 1.1B
25.0
55
23.0
50.1
22.5
46.4
19.2
45
42.8
44.5
44,6
17.3
17.3
17.8
18.1
17.5
40.3
Accuracy (+)
40-
15.0
Init.SFT
RestEM
RLOO
GRPO
DPO
PPO
VinePPO
Init.SFT
RestEM
RLOO
GRPO
DPO*
PPO
VinePPO
Matp
DeepSeekMath
80.1
46.0
80
78.9
42.8
75.3
74.8
74.4
72.0
37.6
36.4
69.6
34.9
70-
35
32.8
Ini RestEM
RLOO
GRPO
DPO
PPO
VinePPO
Init.SFT
RestEM
RLOO
GRPO
DPO*
PPO
VinePPO
Figure 4: VinePPO outperforms standard PPO, GRPO, RLOO, and other RL-free baselines on Pass @ performance on MATH and GSM8K datasets, while also exhibiting scalability across different model sizes.
RhoMath GSMBK
RhoMath 1.18
DeepSeekMath 7B GSMBK
DeepSeekMath 7B
0.81
Test Accuracy
0.225
0.78
0.44
0.200
0.75
0.72
0.36
0.40
0.150
0.69
0.32
0.6
0.7
0.8
0.9
0.2
0.3
0.4
0.5
0.6
0.8
0.9
0.4
0.5
0.6
0.7
Train Accuracy
Method
VinaPPO
PPO
RLOO
GRPO
DPO
RestEM
Figure 5: Generalization slope improves with improved credit assignment. VinePPO has steepest generalization: making the highest generalization gains than baselines when fitting the same amount of training data. On the other end of CA spectrum, RestEM overfits its training data.
variants (RLOO, GRPO), we closely follow their Hugging- Face implementations. For these, initialize with PPO's hyperparameters but perform additional tuning to stabilize training while maintaining the same episode budget. For RL- free baselines (RestEM, DPO+), we strictly adhere to their original implementations (Singh et al., 2024; Pal et al., 2024) and match their sample consumption to other RL methods. For VMC in VinePPO, we conduct full ablation study on K in Section 6.1, with K 9 used as the default setting unless otherwise specified. To ensure fair comparison of compute efficiency, we conduct controlled experiments in Section 6.2, where all methods are evaluated under identical
hardware and parallelization protocols. Full implementation details, including hyperparameters and training procedures, are documented in Appendix C.6 to ensure reproducibility.
6. Results
In this section, we evaluate the effect better CA on task performance, efficiency, and generalization dynamics.
6.1. Task Performance
VinePPO consistently outperforms standard PPO throughout training (Figure C.4) and other baselines (Figure 4) achiev-
6

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
RhoMath 1.1B
DeepSeckMath
24%
44%
1 115x Faster
Method
VinePPO
Accuracy
20%
3. 3.0k Faster
40%
PPO
RLOO
36%
16%
GRPO
32%
5
10
15
20
25
25
50
75
Wall Clock (Hours)
Figure 6: Accuracy vs. Wall Clock Time for both methods measured on the same hardware (shown only up to PPO's final performance). Despite VinePPO taking longer per iteration (up to 2x for 7B and 5x for 1.1B models), it passes PPO's peak performance in fewer iterations and less overall time.
(a) Effect of on Accuracy
(b) Effect of K on Efficiency
112GB for a 7B LLM, considering both model and its opti- mizer. Additionally, PPO requires a forward pass for value prediction and a forward-basd wwwkdddb pass for value network training. VinePPO replaces the value network with MC sam- ples. Since generation is expensive, each step of VinePPO is slower (up to 5x for RhoMath 1.1B and 2x for DeepSeek- Math 7B compared to PPO). VinePPO compensates for slower iterations by making each one more effective through better CA. Under the same hardware, it achieves higher test accuracy faster than baselines (Figure 6). Specifically, VinePPO matches PPO's peak accuracy in fewer gradient steps and less wall-clock time. Figure 6 shows RhoMath 1.IB and DeepSeek.atth 7B require about 3.0x and 1.51x less time and 9x and 2.8x fewer steps compared to PPO. This improvement occurs despite all hyperparameters be- ing tuned for PPO. Therefore, switching to VinePPO could enhance the performance within the same compute budget.
Increasing Compute
24%
20%
Accuracy
21%
18%-
10%
15%
V.P.
V.P.
V.P.
20
(K-1
1)
(K-3)
(K
Wall Clock (Hours)
V.P.KK (K) (a.lv.P.(K3)) V. 1)
PPO
PPO
Figure 7: (a) Effect of the number of auxiliary rollouts K for estimating (s)(s:) on RhoMath 1.IB and MATH (see Fig- ure D.11 for GSM8K). Increasing K consistently improves accuracy. (b) Wall-clock time for the same experiments. While increased sampling makes each iteration slower, the reduced variance leads to faster overall convergence.
6.3. Generalization Slope
ing the highest test accuracy both models and datasets. Notably, the performance gap widens in MATH which is more challenging than GSM8K. To confirm that PPO's lim- itations are not due to undertrained value networks, we measured their explained variance, a standard metric for value function quality, which ranged between 0.7-0.9 across tasks (Figure D.5), indicating well-trained critic. Because the PPO and VinePPO runs only differ in their value esti- mation, comparing these two isolates the effect of CA. As shown in Figure D.8, VinePPO reaches higher test accuracy given a limited KL budget. Additionally, VinePPO is more robust to higher sampling temperatures (Figure D.10).
High-quality and challenging reasoning tasks are scarce, making generalization key challenge. Once training instance is fitted, it provides no further signal for gener- alization. Thus, algorithms that maximize generalization efficiency are superior-achieving higher test accuracy for given train accuracy. As shown in Figure 5, VinePPO demonstrates the strongest generalization gains compared to other baselines. Notably, RestEM overfits near the end. This aligns with recent findings that RL generalizes while SFT primarily memorizes (Chu al., 2025). Overall, allo- cating more compute to refining credit assignment, rather than brute-force data fitting, leads to stronger generalization.
6.2. Computational Efficiency
6.4. Effect of K
Training on a single trajectory in GRPO, RLOO, RestEM and D involves forward and backward pass. PPO and VinePPO have extra computations of different types. PPO uses double GPU memory - the value network needs
We assess the impact of K, the number of MC samples, by running ablation on RhoMath 1.1B, varying K from I to 3 and 9, As shown in Figure 7, VinePPO improves with

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
(a) Estimation Error along the Reasoning Chain
(b) Top Action Identification
PPO
VinePPO
100%
0.03
0.10
75%
Method
Accuracy
0.09
0.02
VinePPO
MAE
50%
0.08
0.01
PPO
0.07
0.06
Random Chance
0.5
0.0
0.5
0%
300
400
500
600
700
Reasoning Progress
Step
Figure 8: (a) Visualizing the Mean Absolute Error (MAE) of the value predictions at different point of the reasoning chain. Value Network in PPO fails to generalize as the reasoning chain progresses, while VinePPO's value estimates become more accurate as the model become more deterministic. (b) Accuracy of identifying the top action in a set of five possible next states. VinePPO consistently outperforms the value network.
higher K since more MC samples reduce the variance of Amc. While high variance of MC estimation could theoret- ically hinder training, our results show that even small K values work well in this setting. Interestingly, increasing K also improves compute efficiency. Although each iteration takes longer, it becomes more effective. This suggests that increasing K provides a practical way to leverage additional computational resources for better performance.
mates. While PPO, policy-based method, depends heavily accurate value estimates, it raises interesting ques- tion: Can PPO's value network still rank actions correctly? We tested this by sampling five possible next steps from a shared initial state and measuring whether the method pre- dicted the next step with the highest ground truth value by assigning the highest predicted value. As shown in Figure 8.b, PPO's value network performs near chance levels for most of the training, improving only slightly over time. In contrast, VinePPO consistently identifies the top action with high accuracy throughout.
7. Why and How Value Networks Fail
In this section, we analyze the performance gap between PPO and VinePPO by focusing on their value predic- tions-theirr only difference. First, We establish "ground truth" " value at each reasoning step within trajectories by running 256 MC samples and averaging the returns. Next, We compare the value predictions against this ground truth.. We present the results for DeepSeekMath 7B, our biggest model, on the MATH dataset (all results in Appendix D.5).
Error Per Reasoning Step To understand value estima- tion dynamics, we plot value estimation error against rea- soning step position (normalized; 3rd of 10 steps = 0.3). As shown in Figure 8.a, PPO performs worse as reason- ing progresses. We hypothesize this is because early steps resemble training data, allowing the value network to rely memorization. Later steps are more diverse and value network struggles to generalize. VinePPO's prediction error decreases with reasoning progression. We attribute this to greater determinism in later steps, as the model conditions on longer context. This stability improves value estimation from the same number of MC samples.
Accuracy Figure 3 presents the distribution of value pre- dictions at each reasoning step. VinePPO's estimates are unbiased, with variance peaking at 0.5 and dropping to zero and 1. PPO's value network shows high bias and often misclassifies bad states (ground truth near 0) good and vice versa. We define a prediction "correct" if it is within 0.05 of the ground truth. As shown in Figure D. PPO's s value network starts with low accuracy, gradually improving to 65%. In contrast, VinePPO consistently achieves 70-90% accuracy throughout training.
8. Discussion
We showed that better credit assignment improves RL train- ing of LLMs. VinePPO a stepping stone to identify and fix PPO's broken credit assignment. It also opens two future research directions. VinePPO is the first RL post-training algorithm that scales generalization slope by scaling post- training compute. Algorithms that have better generalization trends are valuable given the limited resource of truly chal- lenging and verifiable reasoning tasks. Second, VinePPO highlights the value of reconsidering the implicit assump- tions behind default algorithm implementations borrowed
Top Action Identification In value-based RL, accurately ranking actions is more important than accurate value esti-
'The return of sampled trajectory starting from a step is
bernoulli random variable. Let 0 denote the true success probabil-
ity. Our estimator, = 256 2 X,, has variance #(1-0)
At
266
= 0.5 (maximal variance), this becomes 0.25 Rs 0.001.
256
8

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
from Deep RL. In Deep RL, we typically start with a random policy, making it crucial to quickly improve the model's per- formance. In this context, it's more effective to allocate compute toward gathering additional environment samples rather than perfecting each gradient update. However, with an already capable LLM, it is better to spend more com- pute to make sure we steer weights carefully. Overall, we hope VinePPO inspires the community develop more effective RL training algorithms for LLMs.
Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA,, pp- 5360-5370, USA, 2017.
URL https::/proceeeeeee bttps://prooeeddnssseeerrppppp paper/2017/hash.
Abstract.html.
Biderman, D., Ortiz, J. G., Portes, J., Paul, M., Greengard, P.. Jemnings, C., King, D., Havens, S., Chiley, V., Frankle, J., Blakeney, C., and Cunningham, J. LoRA Learns Less and Forgets Less. CoRR, URL. https:// doi: 10. 000488500ARXXI.2205...0 099673................ ao..org/10048355///ziii222
2024.
Impact Statement
Our work aims to improve the ability of large language models to perform complex reasoning tasks, potentially contributing to advances in fields such as education, scien- tific research, and software development. At the same time, more capable reasoning systems could be used irresponsibly, for instance, by automating sophisticated misinformation or other harmful applications. We therefore encourage re- searchers and practitioners to employ appropriate safeguards when applying our methods. Overall, this paper advances fundamental techniques machine learning: its societal im- pact will depend on responsible deployment and continued ethical considerations by the community.
Chang, J. D., Brantley, K., Ramamurthy, R., Misra, D., and Sun, W. Learning to generate better than your Ilm. arXiv preprint arXiv:2306. 1816, 2023.
Chen, G., Liao, M., Li, C., and Fan, K. AL- phaMath Almost Zero: process Supervision without pro- cess. CoRR, abs/2405.03553 2024. doi: 10.48550/ ARXIV TRXIV.2405.03555. 10.4 10.48550/arxiv.2405..
URL https://Goi.orr
Chu, T., Zhai, Y., Yang, J., Tong, S., Xie, S., Schuur- mans, D., Le, Q. V., Levine, S., and Ma, Y. Sft mem- orizes, generalizes: A comparative study of foun- dation model post-training, 2025. URL https:// wwrt.orr//abs/2501.777
Acknowledgements
We thank Matheus Pereira for his efforts on facilitating experimentation. AC and NR are supported by CIFAR Al Chair. SR is supported by a Facebook CIFAR AI Chair and NSERC Discovery Grant program. We thank Mila IDT team and Digital Research Alliance of Canada for the compute provided for experimentation.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Train- ing Verifiers to Solve Math Word Problems. CoRR, abs/2110.1416 141668 2021. URL https: /ttp:///xxxvvor/o////
References
Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J.. Pictquin, O., Üstün, A., and Hooker, S. Back to Basics: Revisiting REINFORCE-style Optimization for Learning from Human Feedback in LLMs. Ku, L., Martins, A., and Srikumar, (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), ACL 2024, pp. 12248-12267, Bangkok, Thailand, 2024. As- sociation for Computational Linguistics. doi: 18.13/53/ V1/2024.ACL-LONG..66 URL https:/ /oi.org/ 10.1 10.18553/01/2024..2
DeepSeck-AL, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang. P., et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URI. https:// arxiv.org/ wtt.oo///aa//51.1119999
Greensmith, E., Bartlett, P. L., and Baxter, J. Vari- ance Reduction Techniques for Gradient Estimates in Reinforcement Learning. In Dietterich, G., Becker, S., and Ghahramani, Z. (eds.), Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, pp. 1507-1514, Vancouver, British Columbia, Canada, 2001. MIT Press. URL https: paper/2001/haste $4b9auc2ddd55ee22f1lllaaaaa Abstract.htmm..
Anthony, T. Tian, Z., and Barber, D. Thinking Fast and Slow with Deep Learning and Tree Search. In Guyon, I, von Luxburg, U. Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
Hao, S., Gu, Y., Ma, H., Hong, J. J., Wang. Z., Wang. D. Z., and Hu, Z. Reasoning with Language Model is Planning with World Model. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pp. 8154-8173, Singapore, 2023. Associ- ation for Computational Linguistics. doi: 18.53/1// 2023.EMNLP--MIN.C 507. URL https: /ti.oogg//g///////////oo///// 10.18653/w1 htt.//6553//1222..mmmmmm 2ttp:///ww1222..mmmmmmaaa.....00
Problems with Language Models. In Koyejo, S., Mo- hamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrlPS 2022, New Orleans, peper..flles/ppagrr/222//// http:::/pppee..npps.ooo Laabbeeffffe2205f55033333----f Abstract-Conffrenceelte
LA. USA, 2022. URL hwtp::ttp///tc
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring Mathematical Problem Solving With the MATH Dataset. In Vanschoren, J. and Yeung, S. (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, 2021. URL https: catasets- benchma srancmmmrrk-srrrrrciiinnnnnerrrr
Lightman, H., Kosaraju, V., Burda, Y. Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's Verify Step by Step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, 2024. OpenReview.net. URL https: : /pppnreviee.net/ forum? Korum?iid-v8LCNNGE0:.
Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang. Y., Jiao, J., Duan, N., and Chen, W. Rho-1: Not All Tokens Are What You Need. CoRR, abs/2404.0796
paper/2021/hasht be93ab3erddcbb73e222ddlloo Abstract-rrund2..1n
2024. doi: 10.48550ARRXIV 200..XXXII.2444077555555 URL https: / //doi.org/ /eeg:/1..4888880
Huang, S., Noukhovitch, M., Hosseini, A., Rasul, K., Wang, W., and Tunstall, I.. The N+ Implementa- tion Details of RLHF with PPO: A Case Study on TL;DR Summarization. CoRR, abs/2403. 170311, 2024. doi: 10.48550AAKXIV.2003.. doi.org/ (tl.org/10.485555 /11.//////////.....1...
Luo, L., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng. L., Sun, J., and Rastogi. A. Improve Mathematical Reasoning in Language Models by Au- tomated Process Supervision. CORR, (tss22406.06522, 2024. doi: 0ww.://0/ARXX.22240..0. / //bb..org/1044555//az3333
URL https://
URL https:
Hwang, H., Kim, D., Kim, S., Ye, S., and Seo, M. Self-explore to Avoid the Pit: Improving the Rea- soning Capabilities of Language Models with Fine- grained Rewards. CoRR, abs/2404. 10346, 2024. doi: 00.48550/ARXIV.2404..0 dot.org/10.4855//arrra http://55000XX1..42400.
Ma. Q. Zhou, H., Liu, T., Yuan, J., Liu, P., You, Y., and Yang, H. Let's reward step by step: Step- level reward model as the Navigators for Reason- ing. CoRR, abs/2310.10080, 2023. doi: 10.48550/ ARXIV.2310.10080 URL https://coi.org/ 10.45500aarx...21100000
URL https://
Ivison, H., Wang. Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., and Hajishirzi, H. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. CoRR, 00s22406609299, 2024. doi: 10.48550/ARXIV.2444 220.X00V.2400092299 URL https:// doi.org/10.4480
OpenAL. OpenAl ol System Card, 2024. URL https:
//api.semanti www.cssaa..oro/oorrrre..eee. 272684752.
Ouyang. L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L.,, Mishkin, P. Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F,, Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K. and Oh, A. (eds.), Advances in Neural Information Processing Systems 35; An- nual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, 2022. URL http: http::/papprss.nigg
Kwon, W., Li, 7.,, Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient Memory Management for Large Language Model Serv- ing with PagedAttention.
Flinn, J., Seltzer, M. I., Druschel, P., Kaufmann, A., and Mace, (eds.), Proceed- ings of the 29th Symposium on Operating Systems Princi- ples, SOSP 2023, pp. 611-626, Koblenz, Germany, 2023. ACM. doi: 10. 00.1145//60000066366316666 URL https:
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V. V., Slone, A., Anil, C., Schlag, I., Gutman-Soll T., Wu, Y., Neyshabur, B., Gur- Ari, G., and Misra, V. Solving Quantitative Reasoning
Abstraatt-Confeeencehhaa
10

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing Fail- ure Modes of Preference Optimisation with DPO- positive. CoRR, 112,2222.112288, 2024. doi: 10.48550/ ARXIV.2402.1322
Setlur, A., Garg, S., Geng, X., Garg, N., Smith, V., and Kumar, A. RL Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight- fold. CoRR, abs/2406. 14532, 2024. doi: 10.48550/ ARXIV.2406.1O 14532. URL https:///oi.co
URL https: http://oo.orrgg/oo
Putta, P., Mills, E., Garg, N., Motwani, S., Finn, C., Garg, D., and Rafailov, R. Agent Advanced reasoning and learning for autonomous ai agents, 2024. URL https: //arxiv.org/ wwwp://bb//44.... /tt.//408.0070 77999.
Shao, Z., Wang, P. Zhu, Q. Xu, R., Song, J., Zhang, M. Li, Y. K., Wu, Y., and Guo, D. DeepSeek- Math: Pushing the Limits of Mathematical Reasoning in Open Language Models. CoRR, abs/2402.03300 03300, 2024. doi: 11448500AXXX..22222005555 URL https: http:///11.00555000022220000
Qwen. Qwen2.5-Math:
The world's leading open-sourced mathematical LLMs. https:
Silver, D., Huang, A., Maddison, J., Guez, A., Sifre, L., den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbremner, N., Sutskever, I., Lilli- crap, T. P., Leach, M., Kavukcuoglu, K., Graepel, T. and Hassabis, D. Mastering the game of Go with deep neural networks and tree search. Nat., 529(7587): 484-489, 2016. doi: 10. 10.1038/NAUREI166 URL
///g/enim..ithu...o//bbbbb wttp:////we2222......... 2024. Accessed: 2024-09-23.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct Preference Optimiza- tion: Your Language Model is Secretly Reward Model. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: An- nual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA. 2023. URL http: http:://papersnips.co faper..flees/paper//00 a8b-0DSee5ccc7774ee332256666 Abstract-Conference.t
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., Lillicrap. T. P., Simonyan, K., and Hass- abis, D. Mastering Chess and Shogi by Self-play with General Reinforcement Learning Algorithm. CoRR, abs/1712.018// 2017. URL http: http:///axxivorg/ abs 11/11112001111
Schulman, J. Notes on the KL-divergence Ap- proximation. http:///jschu.net www.nne//blo//e kk1 approx.html, 2020. Accessed: 2024-09-23.
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A. T., Kumar, A., Alemi, A. A., Rizkowsky, A., Nova, A., Adlam, B., Bohnet, B., Elsayed, G. F,, Sedghi, H., Mordatch, I., Simpson, I,, Gur, I., Snoek, J., Pen- nington, J., Hron, J., Kenealy, K., Swersky, K., Mahajan, K., Culp, L., Xiao, L., Bileschi, M. L., Constant, N., Novak, R., Liu, R., Warkentin, T., Qian, Y., Bansal, Y., Dyer, E., Neyshabur, B., Sohl-Dickstein, J., and Fiedel, N. Beyond Human Data: Scaling Self-training for Problem- solving with Language Models. Transactions on Ma- chine Learming Research, 2024, 2024. URL https: //tpenneview.net////eeeeeee for.mmtee/oooummmm 1d-1NAyUngGFK.
Schulman, J., Levine, S., Abbeel, P., Jordan, M. I., and Moritz, P. Trust Region Policy Optimization. In Bach, F. R. and Blei, D. M. (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, volume 37 of JMLR Workshop and Conference Pro- ceedings, pp. 1889-1897, Lille, France, 2015. JML..org. URL http: /ttt::/prrccedeee http:/ppcceeinnssll..pp.p schulman155 1..mml..
Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P. High-dimensional Continuous Control Using Generalized Advantage Estimation. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Leam- ing Representations, ICLR 201 6016Procedsingss San Juan, Puerto Rico, 2016. URL http: /ttp://aaxxvvoo/oo// abs/ 1506.02438.
Sutton, R. S. and Barto, A. G. Introduction to Reinforcement Learning. In Introduction to
Reinforcement Learning, 1998. URL https: //ppp..smmtti.ccoool.a..ooo/ggpp 261579713.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal Policy Optimization Algo- rithms. CoRR, abs/1707.06347,, 2017. URL http: //arxinorr/abs///77....33
Sutton, R. S., McAllester, D. A., Singh, S., and Mansour, Y. Policy Gradient Methods for Rein- forcement Learning with Function Approximation. Solla, S. A., Leen, T. K., and Müller, K. (eds.),

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
Advances in Neural Information Processing Systems 12, [NIPS Conference, pp- 1057-1063, Denver, Col- orado, USA, 1999. The MIT Press. URL http:
Search Boosts Reasoning via Iterative Preference Learn- ing. CoRR, abs/2405.00051,, 2024. doi: 10.48550/ REXIVV2405.00551
URL https://doi.org: 10.4 10.8550/42/1.245500000
//spppesciioscccpppee////-----Lp eradienn-matthoos-------eeee Learning- wwnctiinntnnnne approauainiiia-apiiiiiiiiaaaaaii
Xu, S., Fu, w., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C.,, and Wu, Y. Is DPO Superior PPO for LLM Alignment? A Comprehensive Study. In Forty-first Inter- national Conference on Machine Learning, JCML 2024, Vienna, Austria, 2024, OpenReview.net. URL https: / //apenrevie.nne Sttpeeeeteeeooumnnddeddddeeee 6xH8R
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, I.., Canton-Ferrer, C.,
Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J.. Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., et al. Llama 2: Open Foundation and Fine-tuned Chat Models. CoRR, abs/2307.09285. 2023. doi: 10.48550/ARXIV.2307..000 092888. URL https: // (aa.orgg/...4555/arz///..
Yuan, Z., Yuan, H., Li, C., Dong, G., Tan, C., and Zhou, C. Scaling Relationship on Learning Mathematical Reason- ing with Large Language Models. CoRR, abs/2308.0 01825, 2023. doi: 00.48550/RRXIV.230000
URL https:
Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Delcu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032 2024.
Zhang. D., Zhoubian, S., Yue, Y., Dong, Y., and Tang, J. ReST-MCTS*: LLM Self-training via Process Re- ward Guided Tree Search. CoRR, abs/2406.03811 2024. doi: 10. 10.48550/ARXI.22400 www.R/XII..220..... 03816. URL https:// // doi.org dol.org/10.48550//200 1tw.:/088555//a..........0
Trung, L. Q., Zhang, X., Jie, Z., Sun, P., Jin, X., and Li, H. ReFT: Reasoning with Reinforced Fine-tuning. In Ku, L., Martins, A., and Srikumar, V. (eds.), Pro- ceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), ACL. 2024, pp. 7601-7614, Bangkok, Thai- land, 2024. Association for Computational Linguistics. URL ht http
Zheng, L., Yin, L.. Xie, Z., Sun, C., Huang, J. Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Sglang: Efficient ex- ecution of structured language model programs. CoRR, abs/2312.071044 2024. URL http::s//axxiv.orgp/
doi: 10. 10.111865W11222444----00OOO 1//60...rg/.1..1853///////2
Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., Liu, Y., Jin, S., Liu, Q., Zhou, Y., Xiong, I.., Chen, L.,, Xi, Z., Xu, N., Lai, W., Zhu, M., Chang, C., Yin, Z. Weng, R. Cheng, W., Huang, H., Sun, T., Yan, H., Gui, T., Zhang, Q. Qiu, X., and Huang, X. Secrets of RLHF in Large Language Models Part I: PPO. CORR, abs/2307.00494, 2023. doi: 10.48550/ARXI.23077.000 //tbi..ogg/144455/aax//..//
Uesato, J., Kushman, N., Kumar, R., Song. H. F., Siegel, N. Y., Wang, L., Creswell, A., Irving, G., and Hig- gins, I. Solving math word problems with process- and outcome-based feedback. CoRR, abs/2211.11277 2022. doi: 10.48550/ARXIV 2211...4....1...2.... URL https: / ///pb//.rgg.../555/////a////////
URL https:
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Rad- ford, A., Amodei, D., Christiano, P. F., and Irving, G. Fine-tuning Language Models from Human Pref- erences. CoRR, abs/1909.08599 2019. URL http: // //arxiv.org/abs/10000000
Wan, Z., Feng, X., Wen, M., McAleer, S. M., Wen, Y., Zhang, W. and Wang, J. AiphaZero-like Tree-search can Guide Large Language Model Decoding and Train- ing. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna, Austria, 2024. OpenReview.net. URL https: U /ppenreview.net//
Wang, P., Li, L., Shao, Z., Xu, R. X., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Ver- ify and reinforce lims step-by-step without human an- notations. CoRR, abs/2406.06592, 2024. URL https: //axxiv.orggabs//21222222
Xie, Y., Goyal, A., Zheng, W., Kan, M., Lillicrap. T. P., Kawaguchi, K., and Shieh, M. Monte Carlo Tree
12

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
A. Reviewing PPO
PPO, as used in RL tuning of LLMs, formulates language generation as token-level MDP (Section 3), where each response y an episode. The state at time step t, 8, E S, is the concatenation of the prompt and the tokens generated so far: The action aj corresponds to generating the next token yt from the model's vocabulary. Given prompt æ, an episode of this MDP starts from the initial state 8o z. and with each action taken, the environment moves to a subsequent state, 8t+11 81; [er]. by adding the action a, to the existing state 8,, In the language environment, because states are always formed by concatenating tokens, the environment dynamics are fully known, and the transition function is deterministic, meaning P(*++1 (st, ae) = 1. Throughout the generation process, the reward Tr is set to zero for all intermediate actions ae, with the sequence-lever reward R(:: y) applied only at the final step when the model stops the generation. That is:
s, 2: y = [zo...... :MM1 yo..... 1t- 90
= r(st R( if = T - 1, where 81411 y is terminal,
otherwise.
A trajectory == (5o, do, 51, a1....) )) thus represents a sequence of state-action pairs that begins at the input prompt and continues until reaching the terminal state. Finally, the cumulative return of a trajectory T is defined as R(T) = Ei Tr-1 = R(x:
The goal of RL tuning is to maximize the expected return of the model's responses to prompts in the dataset, as defined by the reward function (Equation 1). PPO, similar to other policy gradient methods, achieves this goal by repeatedly sampling trajectories for batch of prompt sampled from D and taking multiple optimization steps in the direction of the gradient Oppo to update the parameters. PPO gradient Ippo is defined as the gradient of the following loss:
aa(a se) Tez (az 8t)
(0) Brawxo
Σ min (=0
A clip(0)4{** - 3 KL[TT
|| Tret]
(8)
where To, is the policy at the previous iteration, e is the clipping parameter, is the KL penalty coefficient, A7+ A A at) is the advantage estimate for policy TTg and the clip(9) function is:
To (a (x)) To, (ut. 8t)
e, 1e).
clip(0) clip
Note that the KL penalty could be also added the reward function R. We follow the more recent implementations (Shao et al., 2024; Qwen, 2024), where it is added to the loss function. The KL term can be computed using the following unbiased estimator (Schulman, 2020):
Tref( 18)) To (T(aa st) No (at 8t)
log Tret(ae 88) 1,
KīT(())
(10)
where Feet denotes the reference model (initial SFT).
A.1. Value Network
In addition to the policy To. PPO also trains a separate value network V. to obtain an estimate the true values (ss.) of states 8t. Parameterized by Φ. the value network is trained alongside the policy network va using the following loss:
T Σ max (116((s)) Gil% [dip(()-G:)) c.11)]
EVaINGt (Ø) _ B
(11)
where Vg. is the value network at the previous iteration, G. = T-- gp -tpr r is the empirical return from state St, c' is a
value clipping parameter, and the clip(0) is defined as:
@ clip (Vo(s)), (st) e', VoL (st) (s)))()))))) +e).
(st) In RL-tuning of LLMs, the value network is typically initialized from the initial policy Neर (or the reward model, available), replacing the language modeling head with a scalar output head to predict values (Zheng et al., 2023) This approach takes advantage of the base model's prior knowledge for value estimation.
(12)
13

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
Advantage Estimation Once the estimated values (5() are obtained, the advantages A(s ae) are computed using the GAE (Schulman et al., 2016):
A(st, ar) 2 AGAE (4)
(13)
= (1- (4 + +AA + +
(14)
20
Σ E622)6604
(15)
EG2j (reHt + (ren+
(V((s44411) - -inn))-VVVosn (o((ct1)
(16)
where 6; = = (Vo(c4)) - Vo(ss) is the temporal difference error, À is the GAE parameter, and y is the discount factor.
Also, we have:
1
7 2'0e-1 = rx + ---11 + .... T(tk-1 + (8(4+))) V (82).
(17)
1=0
Adjusting the GAE parameter À allows for trade-off between bias and variance in the advantage estimates. However, as we discuss in Appendix C.6, we found that X 1 works best in our experiments (similar to the findings of Trung et al. (2024) and Ahmadian et al. (2024)). In this case, the GAE simplifies to the following form (assuming y 1):
Å GGAB (t,, ax) = Ei= Tr (st).
B. Reasoning Step Separation Examples
In this section, we outline the methodology used segment solutions into discrete reasoning steps for the MATH and GSM8K datasets, illustrated in Figures B.1 and B.2.
For the MATH dataset, we begin by splitting solutions based on clear natural boundaries such as newline characters or punctuation marks (c.g., periods or commas). Care is taken to avoid splitting within mathematical expressions. ensuring that mathematical formulas remain intact. After this initial segmentation, if any resulting step exceeds 100 characters, we further try to divide it by identifying logical breakpoints, such as equal signs (=) within math mode.
For the GSM8K dataset, we take a simpler approach. segmenting the reasoning steps by newlines alone as with this task newlines already serve natural delimiters.
C. Experimental Details
C.1. Datasets
We focus on mathematical reasoning datasets that require step-by-step solutions and are widely used to evaluate the reasoning capabilities of LLMs. Below a brief overview the datasets used in our experiments:
MATH (Hendrycks et al., 2021) The MATH dataset contains problems from high school math competitions, covering wide range of topics such as algebra, geometry, and probability. For experiments, Lightman et al. (2024), which consists of 500 problems for testing and 12,500 problems for training. We further divide the training set into 11,500 problems for training and 500 problems for validation. Each problem includes a step-by-ster solution, ending in a final answer marked by (boxed() in the solution (e.g..". "..so the smallest possible value of c is π "). This marking allows for verification of the correctness model-generated responses by comparing the final answer to the ground truth. We use the scripts provided by Lewkowyez al. (2022), Lightman et al. (2024), and Shao al. (2024) to extract and compare the final answers to the ground truth.
the OpenAl split provided by
GSM8K (Cobbe et al., 2021) The GSM8K dataset comprises high-quality grade-school math problems, requiring basic arithmetic or elementary algebra to solve. Although simpler than the MATH dataset, GSM8K is still widely used to assess the reasoning capabilities of LLMs. It contains 1,319 problems for testing and 7,473 for training. To create a validation set, we further split the training set into 7,100 problems for training and 373 for validation. Verifying the correctness of model responses is straightforward, as the final answer is typically an integer, marked by #### in the solution.
14

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
Prompt
Prompi
we express 2 + ar
form a( h)? + &. then what is
Suppose the roots polynomial ar? maz positive prime integers
(not necessarily distinct). Given that m. < 20, how many possible values of are there?
Response
We complete the square.
First, factor out of the
Respouse
lerms 3z" x to get (x²2 + $)
We can square x
Let be the prime rooks. Then, know that
get 2
E so..s" so
+ and pq. 1 Since
20, the primes
and y must both be less than 20.!
3 x2" 2
36
TTe peimes less than 20 are 2, 3, 5, 7, 11, 13, 17, 19. Now we list all possible pairs (p. q) such that + g < 20," remembering to also include the cases which = q:
3
3
6
36
1
3
x
12'
(2, 2), (2, 3), (2. 6) (2, 7), (2. (2, (2.
(3,3), (3. 5), (3, 7), (3. 11), (3, 13)
and
(5. (5, 7), (5, 11), (5, 13)7
2
3
1
(7. 7), (7, 11)
3
There + 18 pairs Fach pair produces value and furthermore, these values distinct. bocause every positive integer bas unique prime
49
We see that A
12
factorization.
Thus, there are 18 possible values for n..
Figure B.I: Examples of solutions separated into its reasoning steps on the MATH dataset. Steps are highlighted using distinct colors. denotes the reasoning step boundary.
Prompt
Prompt
Tobias is buying a new pair of shoes that costs $95. He has been saving up his money each month the past months. gets $5 allowance He also mows lawns und shovels driveways. He charges S15 mow lawn and shovel. After buying the shaes, he has change. mows lawns, how many driveways did he shovel?
Tim rides his bike back and forth to work for each of bis 5 workdays. His work of 200 miles.
20 miles away. He also goes weekend bike at 25 mph how much time does he spend biking week?
Response
Response
He saved up $110 total because + 15 110
He hikes 20 40 miles each for work.
allowance
151
So he bikes 40 5 200 miles for work"
He camed 560 mowing lawns because 4 15 = 601
meens
total 200+200 4400 work?
So bikes total 400 25m hours
He eamedd S35 shoveling driveways because 110-60-15 60-6600-5:33 3
ANeN 16!
He shoveled driveways because 35/7 35/ 5.
Figure B.2: Examples solutions separated into its reasoning steps on the GSMSK dataset. Steps are highlighted using
distinct colors. denotes the reasoning step boundary.
C.2. PPO Implementation
To ensure PPO implementation is robust, and evaluation reflects its full potential, have applied set of well- established techniques and best practices from the literature (Huang et al., 2024; Ivison et al., 2024; Zheng et al., 2023). Below, we outline the key implementation details that were most effective in our experiments:
Advantage Normalization: After calculating the advantages, we normalize them to have zero mean and unit variance,
15

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
Table 1: Summary of PPO hyperparamters used in the experiments.
Parameter
Value
TRAINING
Optimizer
Adam
Adam Parameters (81,
(0.9, 0.999)
Learning rate
x 10-6
Weight Decay
0.0
Max Global Gradient Norm for Clipping
1.0
Learning Rate Scheduler
Polynomial
Warm Up
3% of training steps
# Train Steps For MATH dataset
1000 steps (around 8 dataset epochs)
# Train Steps For GSM8K dataset
650 steps (around 8 dataset epochs)
GENERAL
Maximum Response Length
1024 tokens
Maximum Sequence Length for RhoMath 1.1B Maximum Sequence Length MeepSeekMath
2048 tokens
2500 tokens
PPO
# Responses Prompt Episodes per PPO Step
8
Scarch Space: (8. 16.33
512
Scarch Space: {256, 512]
# Prompts per PPO Step
512/8 8
Mini-batch Size
64
# Inner epochs per PPO Step
2
Search Space: (1, Scarch Space: {0.6, 0.8, 1.0)
Sampling Temperature
0.6
Discount Factor ?
1.0
GAE Parameter
1.0
Search Space: [0.95 - 1.0] Search Space: (le-1, le-2, 3e-3, le-4)
KL Penalty Coefficient |
le-4
Policy Clipping Parameter
0.2
Value Clipping Parameter e'
0.2
not only across the batch but also across data parallel ranks. This normalization step applied consistently in both our PPO and VinePPOimplementations..
Reward Normalization: We follow Ivison et al. (2024) and do not normalize the rewards, as the reward structure in our task is already well-defined within the range of [0, 1]. Specifically, correct responses are assigned a reward of 1, while incorrect responses receive
End-of-Sequence (EOS) Trick: As detailed in Appendix A, rewards are only applied at the final token of a response, which corresponds to the EOS token when the response complete. For responses that exceed the maximum length, we truncate response the maximum length and apply the reward to the last token of the truncated sequence. We also experimented with penalizing truncated responses by assigning a negative reward (-1), but this did not lead performance improvements.
Dropout Disabling:
During the RL tuning phase, we disable dropout across all models. This ensures that the log probabilities remain consistent between different forward passes. thereby avoiding stochastic effects that could hurt training stability.
Fixed KL Coefficient We use a constant coefficient for the KL penalty. Although the original PPO implementation for finetining language models (Ziegler al., 2019) utilized an adaptive KI, controller, more recent implementations typically do not use this approach (Ouyang al., 2022; Touvron et al., 2023; Xu al., 2024).
C.3. SFT Models
To ensure a systematic and reproducible evaluation, we create our SFT models गee by finetuning the base pretrained LLMs (as opposed to their "Instruct" version) on the training splits of the respective datasets. Specifically, we produce four distinct SFT models: two base LLM (DeepSeekMath 7B and RhoMath 1.IB across two datasets (MATH and GSM8K). The base models are finetuned using the Adam optimizer without weight decay. We employ a learning rate warm-up over 6% of
16

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
Table 2: Summary of RLOO and GRPO hyperparamters used in the experiments.
Parameter
Value
TRAINING
Optimizer
A
Adam Parameters (21
(0.9, 0.999)
Learning rate
x 10-6
Weight Decay
0.0
Max Global Gradient Norm for Clipping
1.0
Learning Rate Scheduler
Polynomial
Warm Up
3% of training steps
# Train Steps For MATH dataset
1000 steps (around 8 dataset epochs)
# Train Steps For GSM8K dataset
650 steps (around 8 dataset epochs)
GENERAL
Maximum Response Length
1024 tokens
Maximum Sequence Length for RhoMath 1.1B Maximum Sequence Length MeepSeekMath
2048 tokens
2500 tokens
RL ALGORITHM
# Responses Prompt
8
Episodes per PPO Step
512
# Prompts per PPO Step
512/8
Mini-batch Size
64
# Inner epochs per PPO Step Sampling Temperature Discount Factor ?
2
0.6
1.0
KL Penalty Coefficient |
3e-3
Search Space: {1e-2, 3e-3, 1e-3, 3e-4, le-4)
Policy Clipping Parameter
0.2
the total training steps. Each model is trained for three epochs with a batch size of 64, and the best checkpoint is selected based validation accuracy. For each SFT model, conduct hyperparameter sweep over learning rates in the range {1 10-7,3 10-7, x 10- 10 6,1x 10 10-5,0 x 10 8 10 00-5, 10 10-43 ensure optimal performance. We then use these SFT models as the initial checkpoint for training the methods mentioned in our paper.
C.4. Evaluation
We evaluate each method's performance the test sets of each dataset. For example, when report that PPO achieves 42.8% accuracy on the MATH dataset for the DeepSeekMath 7B model, this means the PPO training was initialized with the SFT model specific to DeepSeekMath 7B on the MATH dataset, and accuracy was measured on the MATH test set. Our primary evaluation metric is accuracy, specifically Pass@1, which reflects the percentage of correctly answered problems on the first attempt. This metric crucial because it represents a realistic user interaction, where the model is expected to deliver a correct answer without the need for multiple tries. For each evaluation, we sample a response from the model for given prompt, using maximum token length of 1024 and a temperature of 0.35. A response is considered correct if its final answer matches the ground truth final answer, detailed in Appendix C. Furthermore, each accuracy score is averaged over 16 evaluation rounds, each conducted with different random seeds. This will ensure a robust and low variance assessment of model performance.
C.5. Other Baselines
GRPO (Shao et al., 2024) and RLOO (Ahmadian et al., 2024) GRPO replaces PPO's value network with policy gradient baseline computed from the average return of group of responses to the same input. For each training question x, all algorithms generates G responses, yielding training trajectories Tı, T2, ..., To ~ x(| with corresponding returns R1, B2..... Rg. Note that in the case GRPO, we need to have G > Then, GRPO computes the empirical mean
Ni о Σi= Rd and standard deviation on of these returns. For each trajectory 7;, the advantage A(s a) for all state-action
17

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
Table 3: Summary of RestEM hyperparamters used in the experiments.
Parameter
Value
TRAINING
Optimizer
AdamW
Adam Parameters (21,
(0.9, 0.999)
Learning rate
1 10
Weight Decay
0.0
Max Global Gradient Norm for Clipping
1.0 Polynomial
Learning Rate Scheduler
Warm Up
3% of training steps
RESTEM
iterations
10
# Sampled Responses per Prompt
8
Search Space: {8, (8.32)
Sampling Temperature Checkpoints every iteration Checkpoint Selection
0.6
Search Space: (0.6, 0.8, 1.0)
500 step
until validation improves
Search Space: (until validation improves, best validation]
Table 4: Summary of OPO-Positive hyperparameters used in the experiments.
Parameter
Value
TRAINING
Optimizer
AdamW
Adam Parameters (81,
(0.9, 0.999)
Learning
1 x 10-6
Weight Decay
0.0
Max Global Gradient Norm for Clipping
1.0 Polynomial
Learning Rate Scheduler
Warm Up
3% of training steps
IPO-POSITIVE
# DPO-8
0.1 for MATH, 0.3 for GSM8K 50.
# OPO-Positive-A
Epochs
3
Search Space: (3,8)
Sampled Responses per Prompt
64
Search Space: {8, 64
# Pairs per prompt
64
Search Space: {8, 64)
Sampling Temperature
0.6
pairs (s, a) E Ty is defined as:
R. Pa
A(8, a)
Notably, this introduces bias in policy gradient estimation because the return RG of the current trajectory used in computing its own baseline. RLOO addresses this bias by employing a leave-one-out strategy for baseline computation. Specifically, for each trajectory T,, the baseline computed using the returns of all other trajectories in the group, excluding R,, Let pa denote the empirical mean of (Ry 1izt- The advantage for all acte-nction pairs in тą then computed as:
A(s,a) = R3 p
This modification ensures that the baseline for each trajectory is independent of its return, yielding unbiased policy gradient estimate.
+ (DPO-Positive) (Pal et al., 2024) The original DPO method has a failure mode when the edit distance between positive (correct) and negative (incorrect) responses is small. In these cases, the probability of both responses tends to decrease. This issue is especially common reasoning and mathematical tasks, where multiple solution paths may involve similar equations or steps. Although DPO achieves its goal by reducing the probability of the incorrect response more
18

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
than the correct one, it ultimately still lowers the likelihood of generating the correct response. This undermines model performance, making it a failure mode despite partially fulfilling the DPO objective. (Pal et al., 2024; Hwang et al., 2024). While previous methods mitigated this issue by maintaining high edit distance between positive and negative response pairs, PrO-Positive (Pal et al., 2024) addresses it more effectively. It introduces an additional term to the DPO objective, penalizing any reduction in the probability of the correct response below its probability under the reference model. This ensures that the correct response is not overly suppressed, even when the edit distance small. The final objective of DPO-Positive is:
3 log TO (V(v22z) log
To To (x) Tiat (3) x
LDPO-Posinve Tiet) (w
log a
TTr (x)
DPO Original temm
Tret Mee (x)) A max 1,
(18)
To (V(yuol)) (x)
DPO-Positive additional lerm
where À a hyperparameter controlling the weight of the additional term keeping the probabilities of correct responses high. We chose DPO-Positive baseline due to its strong performance in (Setlur al., 2024).
RestEM (Singh et al., 2024) RestEM is iterative method where, in each iteration, the base model is trained correct, gelf-gennrated responses from the chosen checkpoint of the previous iteration. RestEM takes gradient steps to maximize this objective until the fine-tuned model's accuracy drops on a validation split. The objective of the fine-tuning process is to maximize the log- eigllikelihood of correct responses. Training the model with a maximum likelihood objective on correct responses is mathematically equivalent to training the model with REINFORCE (Sutton et al., 1999), without a baseline, where the entire response treated as a single action. The reward when the response is correct, and otherwise. Specifically, we have:
F Fege (e) Vo log Po(y/aa)) (ratt) Vo log Ro
(19)
max likelikel on correct responses
REINFORCE
Therefore, maximizing log-likelihood training on correct responses is equivalent to train with policy gradient without precise credit assignment, such as without advantages for specific actions. In our experiments, we observe the impact of this limitation in both Figure C.3 and Figure where RestEM overfits on the training data.
C.6. Hyperparameters
In this section, we present a comprehensive overview of the hyperparameters used in our experiments. It's important to note that the number of training episodes was carefully selected to ensure that the amount of training data remained consistent across all methods.
PPO Finetuning LLMs using PPO is known to be sensitive to hyperparameter selection, and finding the optimal settings is critical for achieving strong performance. To ensure the robustness of our study, explored hyperparameter values reported in recent studies (Shao et al., 2024; Zheng al., 2023; Ivison et al., 2024; Huang et al., 2024) and conducted various sweeps across wide range of values to identify the best configuration for our tasks and models. Specifically, we find the set of hyperparameters that perform best across both MATH and GSM8K using RhoMath I.1B model. Then, we employ the optimal set of parameters for the rest of our experiments. The full set of hyperparameters, along with their respective search spaces, is detailed Table
VinePPO We utilized the same hyperparameter setup as in the PPO implementation (Table 1) for VinePPO.
RLOO and GRPO Since policy optimization in RLOO and GRPO is similar to PPO, we initialze their hyperparameters from PPO. This not only ensure we start from a strong set of values, but also allows for a systematic comparison among these algorithms. We further tune their KL coefficient for stable training. Note that lack of credit assignment mechanism could lead to high variance policy gradient update, resulting in unstable training (Greensmith et al., 2001). See Table 2 for the full list.
19

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
Table 5: Average time spent per each training step for different methods and models measured for MATH dataset
Method
Model
Hardware
Average Training Step Time (s)
PPO
RhoMath 1.IB
4 Nvidia A.00 80GB
80
VinePPO
RhoMath 1.1B
4 x Nvidia A100 80GB
380
PPO
DeepSeekMath 7B
8 Nvidia H1100 80GB
312
VinePPO
DeepStekMath 7B
x Nvidia H100 80GB
583
RestEM To ensure fair comparison equalize the number of sampled responses for training between our RestEM and our PPO runs. Therefore, in each RestEM iteration we sample 8 responses per prompt and train for 8 epochs on the correct responses. To enhance RestEM's performance, we also conducted a sweep of other reasonable parameters(Table 3), This included increasing the number of samples to expand the training data and reducing the number of correct responses per question to minimize seerftting.Howweee we observed no significant improvement.
DPO+ (DPO-Positive) We adopted the same hyperparameters as those used by Setlur et al. (2024). In addition, we conducted search for the optimal value of 3 to see if using the same 3 as in our PPO experiments would yield better performance than the values they recommended. To maintain fair comparison, we ensured that the number of training samples in our DPO* runs matched those in our PPO run where we trained for eight epochs, with each epoch consisting of training on eight responses per question. To match this, we generated 64 pairs of positive and negative responses given 64 self- gelf-generated responses from the base model. (Table 4)
C.7. Compute
All experiments were conducted using multi-GPU training to efficiently handle the computational demands large-scale models. For the RhoMath 1.1B model, we utilized node with 4 x Nvidia A100 80GB GPUs train both PPO and VinePPO. For the larger DeepSeekMath 7B model, employed a more powerful setup, using node with Nvidia H100 80GB GPUs. Additionally, for training DeepSeekMath 7B models with the RestEM approach, we used a node with 4 x Nvidia A100 80GB GPUs. The average training step time for each method on the MATH dataset is presented in Table 5.
C.8. Software Stack
Both PPO and VinePPOrequire a robust and efficient implementation. For model implementation, we utilize the Huggingface library. Training is carried out using the DeepSpeed distributed training library, which offers efficient multi-GPU support. Specifically, we employ DeepSpeed ZeRO stage 0 (vanilla data parallclism) for RhoMath I.IB and ZeRO stage 2 (shared optimizer states and gradients across GPUs) for MeepSeekMath 7B For trajectory sampling during RL training, we rely on the vLLM library (Kwon et al., 2023), which provides optimized inference for LLMs. Additionally, VinePPOleverages vLLM to generate Monte Carlo samples for value estimation. Specifically, after each RL training iteration, the current policy's checkpoint is loaded into vLLM. Then, we use vLLM's serving API sample new trajectories and also Monte Carlo Samples for VinePPO's value estimation. In our setup, spawn a separate vLLM engine on each GPU rank. This would allow for data parallelism during both sample generation and training. This software stack ensures that our experiments are both efficient and reproducible. For instance, during VinePPO training, we achieve an inference speed of up to 30K tokens per second using 8 x Nvidia H100 GPUs with the DeepSeekMath 7B model.
C.9. Reproducibility
In this study, all experiments were conducted using open-source libraries, publicly available datasets, and open-weight LLMs. To ensure full reproducibility, we will release both Singularity and Docker containers, equipped with all dependencies and libraries, enabling our experiments to be run on any machine equipped with NVIDIA GPUs, now or in the future. Additionally, we will make our codebase publicly available on GitHub https: : / /
11 N1-NLP/ VinePPO
20

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
D. Full Results
D.I. Training Plots
In this section, we present additional training plots for both PPO and VinePPO on the GSM8K dataset, as shown in Figure D.6. Figure D.7 further illustrates the trade-off between accuracy and KL divergence, while Figure D.9 highlights the computational efficiency of the models
We observe consistent patterns with the results reported in Section 6. Although the performance gap for the DeepSeekMath 7B model is narrower GSM8K, VinePPO still higher accuracy with significantly lower KL divergence and faster per- iteration time (this happens because responses to GSM8K problems are typically shorter, making MC estimation quite fast).
D.2. Explained Variance and Mean Absolute Error (MAE) of Value Prediction During Training
To ensure healthy training runs, we assess value prediction accuracy using explained variance and mean absolute error (MAF). Explained variance quantifies how much of the variance in ground-truth values captured by the estimator:
2g g (Ug 0g)²
Explained Variance
Σg=1 (1g 8)2
where 1g are ground-truth values, 6g are predictions, and 1 E g=1 Ug is the mean the ground-truth values. The mean absolute error (MAB) is given by:
MAE
As shown in Figure D.5, PPO shows improving explained variance and decreasing MAE, indicating stable training. VinePPO achieves the highest explained variance and lowest MAE. RLOO and GRPO are included solely for demonstration, illustrating the deviation of their baselines from ground truth value estimates.
D.3. KL Divergence
The RL objective (Equation 1) balances maximizing task performance while constraining deviations from the initial policy Tref, measured by KL divergence. We analyze how VinePPO and PPO navigate this trade-off by plotting task accuracy against KL divergence KL[Tell [re] throughout training (Figure D.8). Results show VinePPO consistently achieves higher accuracy at same KI. divergence, indicating more efficient use of the "KL budget." This efficiency stems from VinePPO's more precise credit assignment. As shown in Figure 1, many advantages are zero, and VinePPO excludes these steps from the loss. By avoiding unnecessary updates non-contributing tokens, VinePPO reduces non-essential parameter adjustments that would inflate KL.
D.4. Temperature Tolerance
Sampling temperature is critical hyperparameter controlling the randomness of sampled trajectories. At higher temperatures models generates more diverse trajectories, accelerating early training through increased exploration. However, this diversity challenges PPO's value network, requiring generalization over wider range of states. We compared VinePPO and PPO using temperatures 7' E {0.6, 0.8, 1.0} over the initial third of training steps. Figure D. shows VinePPO consistently benefits from higher temperatures, achieving faster convergence. Conversely, PPO fails to benefit from increased exploration and even diverges
1.0, where trajectories are most diverse.
D.5. Value Prediction Analysis
In this section, we provide additional plots for value analysis. Specifically, Figures D.13 to D.16 demonstrates these plots for on the MATH dataset, and Figures D.17 D.20 on the GSM8K dataset.
Furthermore, we present the prediction error per step in Figures D.21 D.24.
For GSM8K runs of RhoMath 1.IB, different hardware was used, making direct comparison wall-clock time not feasible.
21

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
E. More Examples of Advantages in VinePPO
In addition to Figure I, we provide more examples of effective advantage computation of VinePPO in Figures E.25 to E.27.
22

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
RhoMath 1.1B GSM8K RestEM
frain
tost
Accuracy
0.8
0.7
0.8
20000
30000
40000
Global Step
RhoMath 1.1B MATH RestEM
0.6
0.18
Accuracy
My
0.17
0.4
0.3
0.16
0.2
10000
20000
30000
40000
10000
20000
30000
40000
Global Step
DeepSeekMath 7B GSM8K RestEM
frahh
lest
0.90
M Mu
Accuracy
0.72
0.85
0.71
0.80
20000
40000
20000
40000
Global Step
DeepSeekMath 7B MATH RestEM
test
0.37
Accuracy
0.36
0.36
0.34
20000
40000
20000
40000
Global Step
Figure C.3: Performance comparisons across different models and datasets: (a) RhoMath 1.IB GSM8K, (b) RhoMath 1.1B on MATH, (c) Metp 7B on GSM8K, and (d) DeepSeekMath 7B on MATH. The yellow points are chosen checkpoints based on the RestEM rule. Within each iteration, we train on the generated data of the chosen checkpoint for eight epochs and then we choose the first place where performance on a validation split drops following Singh et al. (2024)
23

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
RhoMath 1.1B
DeepSeekMath
44%
Accuracy (+)
Method
20%
40%
VinePPO
PPO
16%
36%
12%
32%
200
400
600
800
1000
0
200
400
600
800
1000
Training Step
Figure C.4: Comparison of the training behavior between VinePPO and PPO. VinePPO demonstrates consistently higher accuracy (as measured the test set of MATH dataset) throughout the training. Refer to Appendix D for more detailed plots.
RhoMath 1.1B GSM8K
Math 1.1B MATH
DeepSeekMath 7B GSM8K
DeepSeekMath 7B MATH
1.0
explained variance
0.9
0.7
0.6
0.5
200
250
200
600
250
500
1000
iteration
method VinePPO
PPO
RLOO
GRPO
RhoMath 1.1B GSM8K
RhoMath 1.1B MATH
DeepSeekMath 7B GSM8K DeepSeekMath 7B
MATH
0.2
mae
0.1
0.0
200
400
600
0
250
500
750
200
400
600
250
500
750
1000
iteration
method VinePPO
RLOO
GRPO
Figure D.5: Explained Variance and Mean Absolute Error of values. VinePPO demonstrates higher explained variance in value predictions and lower mean absolute enror compared to RLOO, GRPO, and PPO across both datasets. Additionally, PPO's value predictions show non-negative explained variance values close to one, indicating stable and effective training. Note that RLOO and GRPO are included solely for demonstration, illustrating the deviation of their baselines from ground truth value estimates.
24

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
RhoMath 1.1B
Matp 7B
80%
52%
Accuracy (+)
Method
48%
76%
VinePPO
44%
PPO
72%
40%
200
400
600
0
200
400
600
Training Step
Figure D.6: Comparison of the training behavior between VinePPO and PPO. VinePPO demonstrates consistently higher accuracy throughout the training the GSM8K dataset. Refer Figure C.4 for MATH dataset.
RhoMath 1.1B
DeepSeekMatt kMtth 7B
55%
80%
Accuracy (-)
50%
Method
VinePPO
75%
45%
PPO
40%
70%
35%
10
20
10
20
[[[rr]
Figure D.7: Task accuracy as function of KL divergence during training the GSMSK dataset. VinePPO significantly higher accuracy per KL. Refer to Figure D.8 for MATH dataset.
RhoMath 1.1B
DeepSeekMath 7B
22.5%
45%
Accuracy (
Method
20.0%
40%
VinePPO
17.5%
PPO
35%
1
15.0%
30%
10
20
30
10
15
20
[e(r
Figure D.8: Task accuracy as a function of KL divergence during training on the MATH dataset. VinePPO achieves higher accuracy, reflecting more efficient credit assignment and focused updates.
25

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
MeepSeek 7B
2.2x Faster
80%
Accuracy
Method
76%
VinePPO
PPO
72%
68%
10
20
30
40
Wall Clock (Hours)
Figure D.9: Accuracy vs. Wall Clock Time for both methods measured on the same hardware throughout the entire training. Since the responses to GSM8K problems short, VinePPO is even faster per-iteration in setup and it reaches PPO's peak performance in fewer iterations and less overall time.
45%
Method
VinePPO
Accuracy (+)
40%
PPO
Te Temparature
0.6
35%
SFT
0.8
1.0
30%
0
100
200
300
Training Step
Figure D.10: Test accuracy during training with higher temperature presented for DeepSeekMath 7B and MATH dataset. VinePPO can tolerate higher temperatures.
(a) Effect K on Task Performance
56%
Increasing Compute
54%
53.4
Accuracy
51.9
50.1
50.1
50%
48%
PPO
VinePPO (K=1)
VinePRO (K=3)
VinePPO (K=9) 9)
Figure D.11: Ablating the number of auxiliary trajectories K for estimating VMC(s)) on RhoMath 1.1B and GSM8K.
Increasing K consistently improves task performance. (see Figure 7 for MATH dataset)
26

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
(a) Value Prediction
100%
Accuracy
75%
50%
25%
0%
250
500
750
1000
Step
Figure D.12: Value prediction accuracy formulated as classification problem, where prediction is considered correct if it falls within 0.05 of the ground truth.
MATH DeepSeekMath 7B PPO
Training Step:
Training Step: 120
Training Step:
Training Step: 420
MAE. 0.27
MAE
MAE 0.16
MAE 0.17
1.00
1.00
1.00
1.00
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
Predicted Value
0.00
0.00
0.00
0.25
0.50
0.75 1.00
0.00
0.25
0.50
0.75 1.00
0.00
0.25
0.50
0.75 1.00
0.00
0.25
0.50
0.75 1.00
Training
Training Step: 660
Training Step:
Training Step: 960
0.16
0.11
MAE:
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25 0.50 0.75 1.00
0.00
0.25
0.50 0.75 1.00
0.00
0.25
0.50
0.75
1.00
Ground Truth Value
Figure D.13: Distribution of predicted values for each state ground truth (computed using 256 MC samples) during training. MAE denotes the Mean Absolute Error (MAE).
27

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
MATH DeepSeekMath 7B VinePPO
Training Step:
Training Step: 120
Training Step: 360
Training Step: 420
MAE: 0.04
MAE 0.05
MAE: 0.04
MAE: 0.03
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0,50
0.50
3000000000
00
0.50
0.50
0.25
0.25
0.25
0.25
Predicted Value
0.00
0.00
0.00
0.00
0.00
0.25
0.50
0.75 1.00
0.00
0.25
0.50
0.75 1.00
0.25
Training Step:
Training Step: 660
Training Step:
Training Step: 960
MAE: 0.04
MAE 0.03
MAE: 0.03
MAE: 0.01
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25 0.50 0.75 1.00
0.00
0.25
0.75
Ground Truth Value
Figure D.14: Distribution predicted values for each state vs. ground truth (computed using 256 MC samples) during training. MAE denotes the Mean Absolute Error (MAE).
MATH RhoMath 1.1B PPO
Training Step:
Training Step: 120
Training Step:
Training Step: 400
MAE. 0.15
MAE 0,16
MAE. 0.14
MAE. 0.16
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
Predicted Value
0.00
0.00
0.00
0.00
0.00
0.25
0.50
0.75 1.00
0.00
0.25
0.50
0.75 1.00
0.25
0.50
0.75 1.00
Training Step:
Training Step: 680
Training Step:
Training Step: 880
MAE: 0,16
MAE 0.15
MAE: 0.13
MAE 0.14
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
0.2b
0.50
0./5
1.00
0.00
0.00
0.25 0.50 0.75 1.00
0.00
0.2b
0.50
0.15
1.00
Ground Truth Value
Figure D.15: Distribution predicted values for each state vs. ground truth (computed using 256 MC samples) during training. MAE denotes the Mean Absolute Error (MAE).
28

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
MATH RhoMath 1.1B
VinePPO
Training Step:
Training Step: 120
Training Step: 320
Training Step: 440
MAE: 0.03
MAE 0.04
MAE: 0.04
MAE: 0.03
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
000
0,50
0.50
10:0
0.50
0.50
8000
0.25
0.25
0.25
0.25
Predicted Value
0.00
0.00
0.00
0.00
0.00
0.25
0.50
0.75 1.00
0.00
0.25
0.50
0.75 1.00
0.25
Training Step:
Training Step: 640
Training Step:
Training Step: 920
MAE: 0.03
MAE 0.04
MAE: 0.03
MAE: 0.02
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
0.25 0.50
0.75
1.00
0.00
0.25 0.50 0.75 1.00
0.00
0.25
Ground Truth Value
Figure D.16: Distribution predicted values for each state vs. ground truth (computed using 256 MC samples) during training. MAE denotes the Mean Absolute Error (MAE).
GSM8K DeepSeekMath 7B PPO
Training Step:
Training Step: 180
Training Step:
Training Step: 300
MAE. 0.20
MAE 0.14
MAE 0,12
MAE. 0,12
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
Predicted Value
0.00
0.00
0.00
0.00
0.00
0.25
0.50
0.75 1.00
0.75
0.00
0.25
0.50
0.75 1.00
0.25
0.50
0.75 1.00
Training Step:
Training Step: 480
Training Step:
Training Step: 600
MAE: 0.08
MAE 0.06
MAE 0.08
MAE: 0.05
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
0.2b 0.50
0./5
1.00
0.00
0.50 0./5
0.00
0.25 0.50 0.75 1.00
0.25
0.50 0.15 1.00
Ground Truth Value
Figure D.17: Distribution predicted values for each state vs. ground truth (computed using 256 MC samples) during training. MAE denotes the Mean Absolute Error (MAE).
29

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
GSMBK DeepSeekMath 7B
VinePPO
Training Step:
Training Step: 180
Training Step: 240
Training Step: 300
MAE: 0.03
MAE 0.02
MAE 0.02
MAE: 0.01
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0,50
0.50
0.50
0.50
0000
0.25
0.25
0.25
0.25
Predicted Value
0.00
0.00
0.00
0.00
0.00
0.25
0.50 0.75 1.00
1.00
0.00
0.25
0.50
0.75 1.00
0.25
Training Step:
Training Step: 480
Training Step:
Training Step: 600
MAE:
0.01
MAE 0.01
MAE 0.01
MAE: 0.01
1.00
1.00
1.00
0.75
0.75
0.75
0.75
64
0.50
0.50
00
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25 0.50 0.75 1.00
0.00
0.25
0.75
Ground Truth Value
Figure D.18: Distribution predicted values for each state vs. ground truth (computed using 256 MC samples) during training. MAE denotes the Mean Absolute Error (MAE).
GSM8K RhoMath 1.1B PPO
Training Step:
Training Step: 160
Training Step:
Training Step: 280
MAE. 0.28
MAE 0.22
MAE. 0.23
MAE. 0.16
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
Predicted Value
0.00
0.00
0.00
0.00
0.00
0.25
0.50
0.75 1.00
0.75 1.00
0.00
0.25
0.50
0.75 1.00
0.25
0.75 1.00
Training Step:
Training Step: 400
Training Step:
Training Step:
MAE: 0.14
MAE 0.18
MAE 0.16
MAE 0.13
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00 0.2b 0.50
0./5
1.00
0.00
0.50
0.00
0.2b 0.50 0.75 1.00
0.00 0.25
0.50
0./5 1.00
Ground Truth Value
Figure D.19: Distribution predicted values for each state vs. ground truth (computed using 256 MC samples) during training. MAE denotes the Mean Absolute Error (MAE).
30

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
GSM8K RhoMath 1.1B VinePPO
Training Step: 40
Training Step: 160
Training Step: 240
Training Step: 280
MAE: 0.05
MAE 0.05
MAE: 0.04
MAE: 0.04
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0,50
0.50
0.50
0.50
100000
0.25
0.25
0.25
Predictad Value
0.00
0.00
0.00
0.00
0.00
0.25
0.50 0.75 1.00
0.00
0.25
0.50
0.75 1.00
0.00
0.75
Training Step. 360
Training Slep. 440
Training Step. 480
Training Step. 640
MAE: 0.04
MAE 0.04
MAE: 0.03
MAE: 0.03
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
1000
0000
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.00
0.25 0.50
0.75
1.00
0.00
0.25 0.50 0.75 1.00
0.00
0.50 0.75
Ground Truth Value
Figure D.20: Distribution predicted values for each state vs. ground truth (computed using 256 MC samples) during training. MAE denotes the Mean Absolute Error (MAE).
PPO
PPO Step
PPO
PPO 840
PPO Step
0.25
0.30-
0.34
0.20
0.16
0.12
0.08-
0.15
0.12
0.12
0.20
0.06
0.07
0.15
0.00
MAE
VinePOO Scop 60
VinaPPO Step 120
VinePPO @ Step 360
VinePPO @ Step 420
VinaPPO & Stop 840
VinaPPO Step 960
0.08
6.08
0.00
0.00
0.02
0.05
0.04
0.04
0.38
0.02
0.04
0.04
0.04
0.02
0.01
0.00
0.00
0.00
0.00
0.0
0.5
1.0
0.0
0.5
100
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
10
Reasoning Progress
Figure D.21: Visualizing the Mean Absolute Error (MAE) the value predictions in different point of reasoning chain, plotted for DeepSeekMath 7B MATH dataset.
31

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
PPO
PPO Step 180
PPO
PPO
PPO Step 600
0.25
0.18
0.00
0.30
0.08
0.25
0.20
0.15
0.28
0.07
0.00
0.09
0.00
0.25
0.10
0.10
0.04
MAE
VinePPO Step 60
VinePPO Step 180
VinePPO @ Step 240
VinePPO @ Step 300
VinePPO Step 480
VirePPO Step 600
0.05
0.04
0.04
0.02
0.00
0.02
0.02
0.02
0.02
0.02
0.01
0.01
0.00
0.00
0.0
0.5
1.0
0.0
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Reasoning Progress
Figure D.22: Visualizing the Mean Absolute Error (MAE) of the value predictions in different point of reasoning chain, plotted for Math 7B GSM8K dataset.
PPO Step
PPO Step 120
PPO Step
PPO A
PPO 800
PPO Step 880
0.20
0.20
0.20
0.18
0.15
0.15
0.15
0.28-
0
0.12
0.10
0.30
0.12
0.06
0.08
MAE
VinePPO Step 40
VinePPO Step 120
VinePPO Step 320
VinePPO @ Step 520
VinePPO Step 840
VirePPO Step 920
0.07
0.08
0.05
0.0
0.06
0.04
0.04
0.00
0.04
0.00
0.02
0.03
0.03
0.01
0.02
0.02
0.00
0.0
0.5
0.0
0.
0.0
055
1.0
0.5
1.0
0.0
0.5
1.0
0.0
1.0
Reasoning Progress
Figure D.23: Visualizing the Mean Absolute Error (MAE) of the value predictions in different point of reasoning chain, plotted for RhoMath I.IB on MATH dataset.
32

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
PPO Step 40
PPO @ Step 80
PPO Step 160
PPO @ Step 280
PPO Step 360
PPO Step 480
0.23
0.25
0.20
0.26
0.20
0.20
0.20
0.15
MAE
VinePPO Step
VinePPO Step 80
VinePPO Step 160
VinePPO Step 280
VinePPO Step 360
VinePPO Step 480
0.08
0.06
0.08
0.05
0.03
0.24
0.04
0.03
0.02
0.00
0.00
0.00
Reasoning Progress
Figure D.24: Visualizing the Mean Absolute Error (MAE) of the value predictions in different point of reasoning chain, plotted for RhoMath I.IB on GSM8K dataset.
Prompt (5o)
Advantage
The vertex the parabola deseribed by the equation y -3x² 300 81 (17,, ns). What nt?
Response
$1 We can re-write the given equation as y -3( 2 + 10x) 81.
0.0
complete the square, 10//2)²
parentheses:
-32² 10x 25) 81 (-3)(25)
0.5
3a + 6) +
85 (3 5)2 6.
0.0
vertex of a parabola the form a(x h)2 at point (h, k))
so the vertex of the given parabola is at the point (- 66).
0.0
8% Therefore,
-6
0.0
Example of Advantages Computed VinePPO
Figure E.25: Steps 82 and $3 provide a critical insight for advancing the solution since they provide way organize the equation.
33

---

VinePPO: Refining Credit Assignment in RL Training of LLMs
Prompt
Advantage
Let f(x) 2 3 and ((f())) = 4x.. Find g(4).
0.4
Response
We know that of (x)) 4z.
can ligure
83 we can plug that value into gf (xx) lind s(4).
0.0
64 Since f(z) = 2x 3, we have f(4) = 2(4) 3 = 5.
0.0
R5 Therefore, (f(4 ((5) 4(5)
20 15
0.0
Example of Advantages Computed VinePPO
Figure F.26: This example demonstrates an error in step 82 where the correct step would what when f(x) 4. This mistake prevents model successfully solve the question after $2.
Prompi (5o)
Advantage
expression 4+3e
single
Response
We common denominator of 7:
0.0
4
3e
2-7
0.0
00.3
3+3r-14
0.4
0.0
7
Example of Advantages Computed by VinePPO
Figure F.27: Steps and 84, despite impacting the probability of solving the question, they are not meaningfully incorrect or insightful.
34


Reviews:
[{"review_id": "V3LVTG6Jpx", "reviewer": "Reviewer_p3iT", "summary": "1. Problem addressed: Credit assignment (CA) in RL training of LLMs, specifically for reasoning-heavy tasks.\n2. Method proposed: VinePPO, which replaces value networks with Monte Carlo (MC)-based value estimation for improved CA.\n3. Main results:\n   1. VinePPO outperforms PPO and other baselines (GRPO, RLOO, RestEM, DPO+) across MATH and GSM8K datasets.\n   2. VinePPO achieves better accuracy with fewer iterations and less wall-clock time.\n   3. VinePPO shows higher test accuracy for a given training accuracy.\n\n## update after rebuttal\n\nThe rebuttal provides few new things to resolve my concern but I can understand the authors' explanation. So I would keep my score as 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s attention to detail and positive feedback. Below, we address the questions asked:\n\n**Questions**\n> What about evaluation on more OOD benchmark datasets like AIME\n\nWe considered benchmarks like AIME too difficult for the base LLMs used in our experiments. However, this is an interesting question that could be explored with newer and stronger models.\n\n> Is VinePPO still effective for larger-scale models?\n\nWhile this question is ultimately empirical (and unfortunately beyond our current resources), we hypothesize that in larger models, which often produce longer responses, the credit assignment problem could be even more critical. In very long chains-of-thoughts, where reward is only received at the end, attributing credit to individual actions becomes increasingly challenging. Therefore, we believe methods like VinePPO, which offer better credit assignment, should be more relevant in such settings.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.\n\nThe only minor issue is that they haven't been examined on larger-scale models over 7B parameters. But I fully understand that this is reasonable considering the resource constraints.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "There is no proof nor theoretical analysis in the paper.", "experimental_designs_or_analyses": "The experimental design is sound and the analyses are valid.", "relation_to_broader_scientific_literature": "The paper is well-grounded in the broader scientific literature for the credit assignment problem in RL training of LLMs.\n\nFor novelty, previous works\n\n1. either focus on biased, low-variance credit assignment based on the value network, e.g., PPO,\n2. or focus on unbiased, high-variance credit assignment, e.g., GRPO, RLOO, RestEM, DPO+,\n\nVinePPO takes the advantage of unbiasedness of Monte Carlo sampling and resolves the problem of high variance by increasing the sampling number, leveraging the reverting flexibility of the language environment.\n\nFor performance, VinePPO outperforms PPO and other baselines (GRPO, RLOO, RestEM, DPO+) across the commonly used MATH and GSM8K datasets.", "essential_references_not_discussed": "Most related works are discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is insightful to leverage the reverting flexibility of the language environment and paralleled efficiency of modern inference engines.\n2. The paper obtains significant improvements over existing methods.\n3. The paper is well-written and easy to follow.\n\nWeaknesses: No significant weaknesses.", "other_comments_or_suggestions": "I have no other comments or suggestions.", "questions_for_authors": "1. What about evaluation on more OOD benchmark datasets like AIME?\n2. Is VinePPO still effective for larger-scale models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7hL1RDXnel", "reviewer": "Reviewer_6B57", "summary": "The paper highlights a problem with PPO applied to post-training LLMs: an inaccurate value function leads to poor credit assignment. To fix this, the paper proposes to replace the learned value function with an MC estimation. The experiments show that his approach can outperform classical PPO and other relevant baselines for post training LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and their focus on details in their feedback. We now address the raised concerns:\n\n> The proposed method is not new, albeit it's not commonly seen for solving CA in RL training LLMs\n\nWhile estimating intermediate-states values via auxiliary MC was known to be theoretically possible, it was never deemed practical in DeepRL [1]. This is because auxiliary rollouts require resetting the environment to intermediate states of an already sampled trajectory: a rare assumption in DeepRL. *In fact, even today, none of standard DeepRL frameworks implement or offer any feature resembling the \"Vine\" approach*. However, reset to intermediate states come naturally in language environments. VinePPO is, to our knowledge and as noted by Reviewer p3iT, the first to exploit this property, turning an idea once considered impractical in DeepRL into an effective method in RL for LLMs. Furthermore, this work goes well beyond demonstrating improved performance and provides thorough analysis to explain the underlying reasons of such improvement.\n\n[1] Schulman, John et al. “Trust Region Policy Optimization.” ArXiv abs/1502.05477 (2015)\n\n**Questions**\n> In figure 3, the plots for GRPO and RLOO, are they depicting the values for the initial state only or for intermediates too?\n\nFigure 3 includes all steps including the intermediate steps to assess the fine-grained credit assignment throughout trajectories. \n\n> Why does GRPO and RLOO perform so much worse? Is it a lack of hyperparameter tuning or samples per prompt?\n\nThis is an interesting, yet not surprising result. GRPO/RLOO lack any fine-grained credit assignments. In other words, they assign the same advantage to all the tokens in a response. This is known to be problematic in RL: actions with negative ground-truth advantage may be boosted if their sampled trajectory succeeds, while highly advantageous tokens may be penalized if their trajectory fails.\nAll PPO-based variants, such as VinePPO, GRPO, and RLOO inherit PPO’s hyperparameters. However, the lack of fine-grained CA in GRPO/RLOO (see Fig. 3 for their MAE) led to less stable training, requiring even an additional HP tuning compared to VinePPO (see Section C.6). Please note that the same sample per prompt is used for all experiments. In other words, the number of training trajectories, responses, is equal between all algorithms.\n\nWe hope we have adequately addressed all of your concerns and would be happy to clarify further if needed. With this in mind, we hope the reviewer increases their rating of our paper.", "claims_and_evidence": "The claims around the flaws of PPO in the context of post training LLMs and the advantages of MC estimates appear sufficiently justified by the experiments.", "methods_and_evaluation_criteria": "The proposed method is sensible for checking whether PPO is held back by an inaccurate value function and benchmark datasets are reasonable for comparing LLM RL post-training methods. The experiments could be even stronger, if the models were also compared on AIME.", "theoretical_claims": "the paper introduces no new theory", "experimental_designs_or_analyses": "I've checked all the experiments in the main paper and they appear sound.", "relation_to_broader_scientific_literature": "the paper highlights a shortcoming of RL post-training LLMs with PPO, which is a common setting in the literature. the method of using monte carlo samples to estimate values is not new, and has been for example proposed for TRPO. the primary contribution of the paper is that it provides many empirical results that demonstrate that learning a value function is hard in this context and that MC estimates are a viable alternative", "essential_references_not_discussed": "none that I'm aware of", "other_strengths_and_weaknesses": "strengths:\n- the paper highlights an important problem\n- the experiments are convincing and the analysis informative\n\nweaknesses:\n- the proposed method is not new, albeit it's not commonly seen for solving CA in RL training LLMs", "other_comments_or_suggestions": "the paper is well written", "questions_for_authors": "1. In figure 3, the plots for GRPO and RLOO, are they depicting the values for the initial state only or for intermediates too?\n2. Why does GRPO and RLOO perform so much worse? Is it a lack of hyperparameter tuning or samples per prompt?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ryPDF9YhBE", "reviewer": "Reviewer_WHfG", "summary": "This paper first analyzes why recent RL methods for LLM often ignore the critic component by showing the poor quality of the learned critic. To improve the critic component, the paper proposes a particularly simple but effective approach by running additional Monte Carlo samples to refine the value estimation, which is called the VinePPO algorithm. The algorithm works strongly on math reasoning benchmarks.\n\n## update after rebuttal\nnothing changed. Good paper and I vote for acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thrilled that the reviewer liked our work and happy to hear that our results were successfully replicated. We now address the question.\n\n**Questions**\n> How would vinePPO perform when training a reasoning model (e.g., o1/R1) with extremely long COT? How can we decide when to perform MC samples when the COT is really long?\n\nThis is a great question. While it is best answered empirically, we believe that credit assignment becomes even more critical in long-CoT models (such as O1/R1). As trajectories grow longer, the distance between the final reward and each individual action increases, making credit assignment more challenging. We hypothesize that in such settings, methods like VinePPO, which enable fine-grained credit assignment, could be more impactful. Moreover, VinePPO could be further optimized by adaptively deciding when to perform additional MC rollouts, for example, based on computed confidence intervals (also known as stopping criteria), which we leave for future work.", "claims_and_evidence": "Yes. The paper is very easy to follow with clear statements and solid experiments.", "methods_and_evaluation_criteria": "The method is intuitive and effective. In fact, our group has reproduced this method and it really works. The math reasoning benchmark is a popular testbed and the evaluation process in this paper is promising.", "theoretical_claims": "The derivations are all straightforward.", "experimental_designs_or_analyses": "The experiment follows the standard evaluation protocol of math reasoning tasks and RL training. They have also conducted sufficient ablation studies on the algorithmic components.", "relation_to_broader_scientific_literature": "This work directly applies to researchers on LLM. Although the method is specifically for RL training, the main point of the work, i.e., using Monte Carlo samples to enhance the value estimates, should be interesting to an even wide audience.", "essential_references_not_discussed": "The citations are sound.", "other_strengths_and_weaknesses": "I like this work. In fact, this work has already made an impact in the RL-for-LLM community.", "other_comments_or_suggestions": "For this draft, I don't have any specific comments. I just have a very separate question for future research that the authors may consider: how would vinePPO perform when training a reasoning model (e.g., o1/R1) with extremely long COT? How can we decide when to perform MC samples when the COT is really long?", "questions_for_authors": "No question. Good work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BH9t6Eo6xG", "reviewer": "Reviewer_3r7L", "summary": "This paper presents to use MC estimate (a well-established method in classical RL) to improve credit assignment in RL training for large language models (LLMs). The authors highlight the limitations of PPO’s value network, particularly in reasoning-intensive tasks, and propose VinePPO, which replaces traditional GAE estimation with unbiased MC estimation, leading to improved performance in mathematical reasoning benchmarks, including MATH and GSM8K.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort in reviewing the paper. We now address the concerns raised in the review.\n\n**Novelty**\n> MC estimation is one of the classic methods for value function estimation in RL. This paper did not introduce additional novel contributions. In fact, in PPO, GAE with $\\lambda =1$\n is also a standard form of MC estimation. Given that VinePPO primarily replaces GAE with its special case, the novelty is limited.\n\n*VinePPO is fundamentally different from setting the λ = 1 in GAE within PPO and does not replace GAE.* In fact, both PPO and VinePPO use GAE. Notably, GAE takes value estimates as inputs to compute advantages. The key difference lies in how these value estimates are computed: VinePPO use auxiliary rollouts while PPO trains a critic network. In both methods, GAE’s λ is still a hyperparameter. Our hyperparameter search (see section C.6.) found λ = 1 for PPO and  λ = 0 for VinePPO. \n\nEstimating the values of intermediate states via auxiliary MC was known to be theoretically possible but never deemed practical in DeepRL [1]. This is because auxiliary rollouts require resetting the environment to intermediate states: an uncommon assumption in Deep RL. However, language environments inherently support this capability. VinePPO is, to our knowledge and as noted by Reviewer p3iT, the first to exploit this property, turning an idea once considered impractical in DeepRL into an effective method in RL for LLMs. Moreover, this work goes beyond merely demonstrating better performance; it also provides thorough experiments to explain why.\n\n[1] Schulman, John et al. “Trust Region Policy Optimization.” ArXiv abs/1502.05477 (2015)\n\n\n**Variance and efficiency of MC estimation**\n\nWhile we agree with the reviewer that in theory variance of MC estimation might necessitate higher values of K, *we empirically establish that even values as small as K=9 demonstrate superior performance (Fig 4), efficiency (Figs 6-7), and stability (Fig D.10) over all baselines*. Our extensive study of compute efficiency in Sec. 6.2 and 6.4 shows that while each iteration takes longer with higher K, the improved credit assignment leads to less wall-clock time for reaching a target test accuracy.\n\n> Thus it is intuitive to use more samples (e.g., k=256 in Section 7) to validate the effectiveness of MC estimation. However, it cannot support the effectiveness of VinePPO (k=9) in the main results.\n\nPlease note that in Fig 3, VinePPO values estimates are computed with K=9 and they’re 3x more accurate than the next best baseline in terms of MAE (VinePPO(K=9): 0.03, PPO: 0.11). \n\n**Questions**\n> Given that estimating the expectation of E(R) from a single episode is intuitively unreliable, why does VinePPO (K=1) still outperform PPO?\n\nThis is a great question. First, we clarify that K refers to the auxiliary rollouts for value estimation. Regarding the question, both variance and bias can hinder performance. The value network in PPO exhibits high bias in its value estimates (see Fig 3 and Fig D.15). VinePPO even with K=1 provides an unbiased estimate of the value. Although the variance with K=1 might be high, the high bias of the PPO’s value estimates empirically result in worse performance. Note that such subpar performance of value network has been reported in previous work [2].\n\n[2] Ahmadian et al, \"Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs\" (2024)\n\nEfficiency was a key focus of our work, and we conducted a thorough analysis of VinePPO’s efficiency in the paper. Across all experiments VinePPO consistently demonstrated superior efficiency and performance compared to baselines. We hope this clarification address your concerns and encourage a fresh evaluation of our work.", "claims_and_evidence": "Good", "methods_and_evaluation_criteria": "Fair. See comments in Strengths And Weaknesses.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "I have checked experimental designs or analyses. See comments in Strengths And Weaknesses.", "relation_to_broader_scientific_literature": "This paper refines credit assignment via MC estimation in RL-based LLM training, contrasting with prior work like PPO, which relies on value networks, and DPO/GRPO, which discard fine-grained credit assignment.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n- The paper is well-written and easy to follow.\n- Credit assignment is a crucial challenge, not only in RL but also in reward model training, thus making the motivation of this work strong and relevant.\n\nWeaknesses\n- MC estimation is one of the classic methods for value function estimation in RL. This paper did not introduce additional novel contributions. In fact, in PPO, GAE with $\\lambda=1$ is also a standard form of MC estimation. Given that VinePPO primarily replaces GAE with its special case, the novelty is limited.\n- The effectiveness of MC estimation relies on having a sufficiently large number of samples to mitigate variance. However, in the main experiments, VinePPO only uses k=9 samples. With such a limited sample size, MC could suffer from high variance and instability.\n- One way to stabilize MC estimation is to massively increase the number of rollout episodes (potentially in the hundreds). But for LLMs, that’s super expensive, making practical use of VinePPO a great challenge. \n- Thus it is intuitive to use more samples (e.g., k=256 in Section 7) to validate the effectiveness of MC estimation. However, it cannot support the effectiveness of VinePPO (k=9) in the main results.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- Given that estimating the expectation of $E(R)$ from a single episode is intuitively unreliable, why does VinePPO (K=1) still outperform PPO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces VinePPO, a method that enhances credit assignment in RL training of LLMs via Monte Carlo-based value estimation. It is a well-executed empirical study that addresses a key challenge in RLHF with strong results on MATH and GSM8K. I recommend acceptance for publication on ICML 2025.
Decision: Accept (poster)