Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
Authors: Charles Arnal
Gaëtan Narozniak
Vivien Cabannes
Yunhao Tang
Julia Kempe
Remi Munos
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement learning
off-policy RL
LLM finetuning
bandits
Primary area: reinforcement_learning
TL;DR: In the context of off-policy RL, we give a theoretical analysis of the role of an additive reward correction in improving performance, accompanied by experiments on bandits and LLM posttraining.

Abstract: Reinforcement learning (RL) is increasingly used to align large language models (LLMs). Off-policy methods offer greater implementation simplicity and data efficiency than on-policy techniques, but often result in suboptimal performance. In this work, we study the intermediate range of algorithms between off-policy RL and supervised fine-tuning by analyzing a simple off-policy REINFORCE algorithm, where the advantage is defined as $A=r-V$, with $r$ a reward and $V$ some tunable baseline. Intuitively, lowering $V$ emphasizes high-reward samples, while raising it penalizes low-reward ones more heavily.
We first provide a theoretical analysis of this off-policy REINFORCE algorithm, showing that when the baseline  $V$ lower-bounds the expected reward, the algorithm enjoys a policy improvement guarantee. Our analysis reveals that while on-policy updates can safely leverage both positive and negative signals, off-policy updates benefit from focusing more on positive rewards than on negative ones.
We validate our findings experimentally in a controlled stochastic bandit setting and through fine-tuning state-of-the-art LLMs on reasoning tasks.

Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
Charles Arnal FAIR, MSL, Meta
Gaëtan Narozniak FAIR, MSL, Meta
Vivien Cabannes FAIR, MSL, Meta
Yunhao Tang Anthropic (ex-Meta)
Julia Kempe FAIR, MSL., Meta NYU Courant Institute and CDS
Rémi Munos FAIR, MSL, Meta
Abstract
Reinforcement learning (RL) is increasingly used align large language models (LLMs). Off-policy methods offer greater implementation simplicity and data efficiency than on on-policy techniques. but often result in suboptimal performance. this work, we study the intermediate range of algorithms between off-policy RL and supervised fine-tuning by analyzing a simple off-policy REINFORCE algorithm, where the advantage is defined as A = - V. with a reward and V some tunable baseline. Intuitively, lowering V emphasizes high-reward samples, while raising it penalizes low loward ones more heavily. We first provide theoretical analysis of this off-policy REINFORCE algorithm, showing that when the baseline V lower-bounds the expected reward, the algorithm enjoys policy improvement guarantee. Our analysis reveals that while on-policy updates can safely leverage both positive and negative signals, off- policy updates benefit from focusing more on positive rewards than on negative ones. We validate our findings experimentally in controlled stochastic bandit setting and through fine-tuning state-of-the-art L. on reasoning tasks.
Introduction
Reinforcement Learning (RL) has long been applied to align Large Language Models (LLMs) to users' preferences using human feedback [Christiano et al.) 2017 Ouyang et al. [022] Dubey et al.)
2024] more recently, it has been used to augment models in more general ways. and in particular to
develop their reasoning, coding and tool use capacities [Shao et al. 2024 Guo et al.
2025 Meta 2025 OpenAT [025). As RL theoretically allows models to learn beyond the limits of existing training data (see e.g. Silver al.) [2016), (AlphaEvolve-ter team [2025 ), we can expect to play an increasingly crucial role in years to come as models begin pushing scientific boundaries. So far, on-policy techniques, in which models are trained on samples that they generated, have been preferred to off-policy methods, in which models are trained using samples generated from another source, which can e.g. be an outdated version of themselves [Tang et al. 2024]. However, strictly on-policy cases; it also suffers from sample
RL can be difficult or even impossible to implement in many inefficiency, as each trajectory generated cannot be used for more than a single gradient update. This makes some degree of off- off-policyness unavoidable, and in fact desirable.
As such, we are interested in the study of off-policy RL techniques, with LLMs finetuning as our motivating example. Various sophisticated methods exist that deal with off off-policyness; while classical solutions based on Q-learning or value functions often struggle in the context of language modeling
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

[Zheng et al., 2023], losses based on importance sampling correction have yielded good results. though the variacce of the importance ratio can be problematic (Precup al. 2001 Schulman et al.) 2017]. In this paper, we consider simple alternative and study the behavior of gradient ascent on the expected objective
( = Eyna [log (v)(((v)) - -V]
as a function of the baseline V E R. where μ and π are the behavior (sampling) and current policies respectively and r(y) is the reward of trajectory y. We call this algorithm Asymmetrie REINFORCE (AsymRE). is asymmetric in the following sense: higher values for V put more emphasis on pushing down the probability of trajectories with low rewards (i.e. "failures") while mainly ignoring high rewards (i.e., "successes"), whereas a small V results in a more positive approach, where the model mostly increases the probability of successful trajectories while mainly ignoring failures. Our core intuition is that while a model can learn from both its successes and failures while training on-policy, it has less to learn from the failures of another model (i.e., when off- off-policy), which it might not be likely to produce in the first place, and as such should focus more heavily on the positive examples while training off-policy.
Our key contributions are as follows.
In Theorem we show that the AsymRE algorithm applied in tabular setting converges to a limit policy Th.v which we characterize. We show that when the baseline V is smaller than the expected reward V of the behavior policy 14, the limit policy T improves upon the behavior policy μs, while maintaining wide support. When V becomes larger than Va, a phase transition occurs and the support of " shrinks dramatically.
In Theorem scheme when V < VP, In combination with Theorem 4.2, we see that letting V 2 Ve results in premature convergence to a potentially suboptimal policy.
we study the iterative application of AsymRE in a policy improvement
We then verify our findings in a controlled yet rich setting, that of multi-armed bandits.
Finally, we validate our results on a larger scale by training Llama 8B and Qwen 3B models real-world data with AsymRE.
These results confirm our initial intuition that in an off-policy setting, conservatively picking a small baseline is the correct strategy.
2 Related works
Reinforcement learning for LLMs. Reinforcement learning methods are rapidly becoming the dominant paradigm for fine-tuning LLMS on complex tasks, such as mathematical reasoning and coding. It benefits from certain key advantages over supervised training methods: e.g., models can generate their own training samples in the absence of preexisting data of high enough quality. Reinforcement Learning from Human Feedback (RLHF) has emerged as cornerstone methodology for aligning large language models with human values and preferences (Achiam et al., 2023]. Early
systems [uyang et al. 2022 turn human preference data into reward modeling to optimize model behavior accordingly. DPO Rafailov et al., 2023) has been proposed as a more efficient approach that directly trains LLMs on preference data. However, as LLMs evolve during training, continuing training on pre-generated preference data becomes suboptimal due to the distribution shift from off-policy data. Thus, the need arises for additional data to be collected during mid-training- key phase in the iterative fine-tuning of LLMs (Touvron et al.) [2033 Bai et al. 2022 Xiong et al. [2024]
Guo et al., 2024). One line of works aims to mitigate by merging on-policy and off-policy data to
achieve improved performance alignment [Gu et al. 2017 Feng ct al. 2025
In verifiable domains, recent methods like GRPO (Shao al. 2024 Guo et al. 2025 have shown strong performance by leveraging binary reward signals. GRPO is REINFORCE
algorithm [Williams) 1992] that incorporates negative examples, increasingly recognized as an important ingredient for efficient learning [Ahmadian et al. [024]] At its core, REINFORCE is an on on-policy algorithm, with theoretical guarantees only when the reference model matches the trained model, limiting the user's ss ability to reuse past data. However, training is rarely truly on-policy: data is generated a parallel, asynchronous way. Moreover, given the computational cost of rollouts, reusing of trajectories is often desirable.
2

---

Off-policy approaches. While off- off-pooicinees is well-studied in RL Degris et al. 2012], it is not the case in the context of LLMs. Algorithms like REINFORCE tend to become unstable with off off-policy data [Pang and He 2021 One way to mitigate these issues is to introduce KI -e.uuaullaiizttion towards the data-generating policy or early stopping, which effectively diminish the influence of negative trajectory and the amount of off- learning. Q-learning (Watkins and Dayan 1992] is also able to handle off-policy data but is less suitable for deployment in LLMs. Perhaps the most common technique to address off-policy distribution shift is importance sampling in conjunction with clipping in SEINFORCE-SIY
algorithms (Schulman et al.] 2017] Munos et al. 2016 [Epeholt et al. 2018). In practice, though, it is well- that importance sampling is plagued with excessive variance [rrcup et al. 2001]. Other approaches to oof-policy RL use a consistency condition derived from the explicit solution of the KL-regularized policy optimization problem, which could be enforced any
data, off- or on-policy, see e.g., Rafailov et al. 2023], RRchemoon et al. (2024), Tang et al. (2025).
Cohen et al. (2025). Concurrent work to ours [Roux al. 2025] introduces
asymmetric variant of importance sampling to speed up learning, noting in passing that the baseline V in REINFORCE
plays role in connection with negative samples in off-policy data. (Zhu et al.) (2025) also investigates
the importance of negative and positive samples, though in an on-policy setting.
3 Setting
In reinforcement learning (RL), the task is to maximize the expected reward
ya def
Eysa [r(y)]
(2)
of trainable current policy æ given some reward function This is typically done by iteratively updating π using training samples (y. r(y)). where the trajectories are sampled from some behavior policy μú; here, each trajectory y potentially represents a whole sequence of states (or observations) and actions taken according to the policy, with partial rewards reflected in rly). If the training samples y are drawn from the current policy, i.e. if p kept equal to the evolving policy x, it is called on-policy RL; otherwise, it is off-policy RL.
In the absence of superior behavior policy μ. from which to sample, algorithms that on-policy as possible, i.e. in which jr is kept as close to the trained policy π as possible by frequently updating it, is usually preferable for L.MM training [Tang al. 2024]. However, on-policy methods suffer from various limitations. Though conceptually simple, they pose delicate engineering problems due to the need for asynchronous cooperation among multiple GPUs, often leading to suboptimal resource utilization. Furthermore, for to remain equal to т, the algorithm must wait for the current batch of trajectories be generated and evaluated using the reward function r, and for (and μ) be updated, before new batch can be produced. In many common settings, such as code generation, agentic interaction with the web or human feedback, rewards can take minutes or even hours to be produced; this is even more true for embedded agents interacting with the physical world. Fully on-policy methods can become extremely inefficient or even infeasible due this. Finally, a strictly on-policy algorithm can only use the trajectories it generates for a single policy update before having to discard them, whereas one might want to reuse samples generated earlier during training, or even coming from the training of another model. This makes them very sample-inefficient, and consequently compute-inefficient, as trajectories are often costly to generate.
This makes off-policy algorithms an attractive alternative [Rafallov et al. 2023 Richemond et al. 2024 Tang et al. 2025 Cohen et al. 2025]; we are in particular interested in oll-policy RL with delayed updates, in which the actor policy μ is simply an outdated version of the current policy æ, and updated (by setting it equal to т) every N training iterations of æ. Common off off-policy algorithms typically apply importance sampling correction to obtain unbiased estimates of the objective function [Schulman et al. 2017 EEspeollt et al. 2018 Tang et al.) 2025). Both approaches have drawbacks; in this paper, we investigate third method for handling ofo-po data.
Context-dependent tasks and LLMs. Context-dependent tasks, such as the ones for which LLMs are designed, add another layer of subtlety to the study of RL algorithms. In the setting above, we considered a single objective function, namely the expected reward from Equation 2 Conversely, the behavior policy of an LLM typically has to generate responses y T-|aa) for various prompts I ~ P sampled from some task distribution p, and its goal is to maximize the expected context- dependent reward max, Ea-parva (r v)). If each prompt defined entirely distinct problem,
3

---

then whatever RL algorithm we choose to apply would only see handful of trajectories associated to this problem, and learning would be essentially impossible. This is not so, as the regularities of language and of L.MMs' architectures make it so that similar prompts, such as "Please solve 2 + 3 ?" and "Compute 2 + = ?", yield similar problems and conditional distributions, and training on one question helps with the other. This additional structure on the space of conditioning prompts plays key role in enabling LLM training, yet is rarely discussed in RL literature.
4 Asymmetrie REINFORCE
We set ourselves in the general RL setting described earlier in which the goal is to maximize the expected reward V* Byyz [r(y]] of our current policy n. For that purpose, one may simple on-policy Policy Gradient (PG) algorithm, REINFORCE Williams 1992]. which updates the
parameters of the policy in the direction
Eysr V log t (v) (r(y) v)].
where V E is a value baseline and the gradient is taken with respect to some differentiable parameters. In practice, V is often set to be some learnt approximation V = V*8 of the value function of the current policy, or to be the empirical average V -->- [_((())))))))))) the rewards obtained when generating several responses y; (see e.g., Ramesh et al. [2024] Shao al. (2024)..
In the on-policy setting, the role of the value baseline is simply to offer possible variance reduction of the PG update: the expected behavior of the algorithm during the learning process, as well as its asymptotic performance, are not affected by the choice of this baseline. On the contrary, in the off-policy case, the choice of the baseline impacts the expected behavior of both the algorithm and the asymptotic policy. To analyze this behavior, we consider the simplest policy gradient algorithm, namely REINFORCE, applied to the off-policy setting. We call this algorithm Asymmetric REINFORCE (or AsymRE in short):
Definition 4.1 (Expected AsymRE and AsymRE), Given a behavior policy μ and a baseline E RR the expected Asymmetric REINFORCE is gradient ascent in the direction
Ey V log (y) (r(y) V)]
The Asymmetric REINFORCE is its stochastic counterpart using samples drawn from p.
This corresponds to stochastic gradient ascent on the expected objective:
J(x) = Ey-g [log (T) (r(v) V)].
Note that it does not coincide with the expected reward of æ, which is our true objective. As it does not involve any off-policy correction, such as importance sampling, we do not expect this algorithm to converge to an optimal policy in general. However will see that even off off-policy this algorithm offers policy improvement guarantees (under some condition on the choice of the baseline V).
Intuition The baseline V can be understood as a way control the emphasis put on the good trajectories (those with high rewards) versus the bad ones (low rewards). If V is large, greater (absolute) weight is given to gradient updates from bad trajectories, which are pushed down, and vice versa, if V is low, more weight is given to push-up the good ones. This matters not in an on-policy setting, since the baseline does not introduce bias to the expected dynamics. However, in the off-policy setting, the choice of the baseline in AsymRE changes the expected behavior of the policy gradient and provides us with way to introduce some asymmetry in our treatment of good versus bad trajectories. In the simplest of settings, in which rewards are binary r(y) E {0. 1}. letting
V = is equivalent to applying peppervssdd-earnninn on good trajectories while ignoring the bad ones. Conversely, setting V means learning from mistakes only, while ignoring good rewards. Intuitively, we expect a comparatively lower baseline to be more beneficial in an off- off-policy setting, as the failures of another policy are less informative for the current policy; explore this intuition in the remainder of this article.
Dynamics of AsymRE tabular setting. Our first result characterizes the dynamics and limits of the AsymRE algorithm in the case of tabular softmax policy representation.

---

Theorem 4.2. [Analysis of expected AsymRE for tabular softmax policies] Let Y be a finite set, f be some behavior policy whose support is Y, and consider softmax policy representation mly) def EEy e'aa') on Y, where the logits hyy are the policy parameters. We consider the expected AsymRE algorithm with respect to the logits initialized at some No with successive iterates T and learning rate n as per Definition 4.1 Then the AsymRE algorithm converges limit distribution T.y. The baseline parameter V controls the nature of the limit distribution:
IfV < VV, then af myy is defined by
TH (v) (ulv)(r(u)-V)) Tu.V)*
VH-V
where Tu,V is uniquely characterized by the constraint Ever(u(u)(r()) - -ma))+
VP V, and x= = -max(x,0) 0).
Vf VP, then Thy is defined by its support
supp(Taa) = arg max ((())((() - V), yEY
and by J..y (1) T (z) Toly)/ no (2) for any y, 2 supp(Tia.v)
1fV VP, then F can charge any of the elements in the set
{ul min (ru) V) ((=)(r(=)- vV V" V²> 0},
Rey
depending on the initial condition To-
As a consequence, supp(Ti.v)) S supp( v.)) when 1/2 S V1 < VH.
The proof of this theorem can be found in Appendix A
The result in Theorem is somewhat surprising in the fact that its behavior differs greatly from the on-policy REINFORCE on-policy REINFORCE algorithm are the optimal policies, i.e. those whose support is included in arg max, ra)) and mentioned above, are independent of the choice of the baseline V. However, in the off-policy REINFORCE algorithm, the role played by the baseline V is crucial. Not only does affect training dynamics, but also the final policies. Indeed, the size of the support of the limit policies T. v of AsymRE decreases as V increases, while for V = miny r(y) the support is Y itself. Thus, the higher V. the more deterministic the resulting policies. Moreover, there is an abrupt change of behavior as V becomes equal to or larger than Vи: at this tipping point, the support suddenly shrinks, and generically becomes singleton. This phase transition has important consequences which we detail below.
Comparison to on-policy REINFORCE:
The stable points of the
Policy improvement scheme with AsymRE: Let V < VP, and let define Tv the operator that takes as input behavior policy and returns the limit policy of the AsymRE algorithm (as defined in Theorem 4.2 2, i.e.:
(TvP) = =i.v()) & (n())((v)) - -v,vv for all y EEY.
Note that this limit policy does not depend on the choice of initial policy Tu- We now consider the repeated application of Tv in policy improvement scheme, in which the behavior policy Pe+1 of each iteration is the limit policy Trj4 of the previous iteration. The expected dynamics this process are detailed in the following theorem, whose proof can be found in Appendix A
Theorem 4.3. [Policy improvement dynamics] Let p. be any policy with support Y, and let V VP.
1. Each application of the AsymRE algorithm increases the expected reward: VTVP VM.
2. The sequence of expected rewards (TTV converges to some limit expected reward V
Let x def {y r(y) = V*). Then the mass of (Tv) (Tv)"H concentrates exponentially fast on 0 i.e.
Σ
((Tv)" c
for some 1.
5

---

3. There exists V such that the corresponding limit reward is optimal (i.e. v maX r(y)) if and only if Vo.g-
In practice, the number of training steps in each iteration of a policy improvement scheme is finite; the more steps there are, the more off policy the training. As the number of steps within an iteration increases, the current policy gets closer to the limit policy, whose support is always smaller than that of the initial policy, and smaller the larger V is. This can lead to the premature elimination of points y with high reward. Hence the more off-policy the training, the smaller the baseline needs to to prevent premature convergence to suboptimal solution. Interestingly, it can be shown that one can be over- conserrvvvv, letting V be very small results in much slower convergence rates. Thus there is trade-off between the speed of convergence and the risk of reaching suboptimal limit policy.
Note that if 2 VH, then the first point of the theorem does not hold any more and Eng-n [r()) can be smaller than Ey-u [r(y]] Furthermore, the support of the policy shrinks at the first policy improvement iteration (as described in Theorem 4.2). which makes all subsequent iterations useless.
As explained at the end of Section 3 LLMs are applied to variety of conditioning contexts x (the prompts), each of which corresponding to a distinet RL problem. The straightforward adaption of the AsymRE method to this setting is to train an LLM's policy π using the context-averaged version of our loss
From the tabular setting to LLMs
BgwD.vvvatla a)) [log (()l>) (r(y, V)],
where D is some distribution of prompts. Translating our theoretical findings to the case of LLMs requires some cautiousness. Note first that the expected reward V #) = (yy=u(-l [r 2)], whose value relative to V conditions the behavior of the AsymRE algorithm, differs for each prompt. A natural solution to consider instead the context-corrected loss
EgwD.vvtttla (u)) log (ulor) (r(u: x) - vu(1=) (-((=)-(( -vv]
so that the critical value becomes oV = 0 uniformly across the rs. Furthermore, though our theorems apply to each of the problems corresponding to the various prompts. the averaged behavior of the model over the distribution of problems is more subtle. At last, our results focus on the training loss of the trained policy, whereas we might be interested in the LLM's test loss on contexts on which they were not trained.
Nonetheless, we can hope that the regularities of LLMs with respect to prompts can allow us to partially predict the effects of training on certain prompts on the LLM's policy conditioned by other prompts. In particular, our results suggest that letting V be equal to or larger than 0 in the context- corrected loss in Equation can lead to dramatic decrease in the diversity of answers, corresponding to the collapse of the policy's 's support predicted by theorems. While this may not necessarily penalize the expected reward of a tabular policy, it is detrimental to a language model; in particular, poses a risk of severe overfitting of the training set, and consequently of poor test results.
5 Experiments
We validate our findings first controlled setting, then with LLMs on real-worle data,
5.1 Bandits
We illustrate our findings in a simple bandit setting. There are 100 arms. The expected reward r(v) of each arm y is chosen uniformly randomly in [0, 1]. We sample from a non-uniform behavior policy y defined softmax of the logits I(y) = 1/10, where y is indexed from 0 to 99. The current policy me is a softmax of its logits 4yy), initialized as To p. and its logits are updated according to the expected AsymRE algorithm with learning rate 1:
141 = 4 + ViEyay V log (v) (r(v) V)].

---

Expected reward differort the
Suppor: and expected rewart of the final policy noy for different valuas of
0.000
6:55
V: U.123
V 0.250
0.50
V. 0.373
Dipecced mraer
VI 0.500
6.75
V. 0.635
V
0.6
6
10000
2.123
C.55
Suppert XC
0.90
support of
0.55
smpectac roword (ag.3))
100
10
10a
302
0.
M.mu
Ittrabon numbert
00
02
04
0.0
De
10
Valoe of bundine
(b) The support of the final policy T is shown by the yellow dols; each dot corresponds to an arm (sorted by increasing reward values). The support suddenly drops to single atom when V crosses V* z 0.54.
(a) Expected reward Ey [r()] of the AsymRE al- gorithm as function of the training iteration and baseline V.
Figure I: Expected rewards and supports of the policies in the bandits experiments
Expected reward and support with AsymRE In Figures and lb report the expected reward Ey-s, [y)), of the expected AsymRE algorithm. We observe that the performance of the algorithm tends improve as the value of the baseline V gets closer some Vo., < VP 0.5405, However, the performance of all policies is upper bounded by sub-optimal value x 0.89, which is lower than the optimal expected reward (max, r(y) = 0.999 for this run).
Although these experiments may seem to support the choice of selecting a high value of the baseline, function of V, shows a complete loss of diversity in the resulting policies for high values of V, making future improvements in a policy iteration scheme impossible. This is also supported by the drastic decrease in entropy for high values of V reported in Figure 7 in the Appendix. We observe a phase transition for V around V". For V < УP, the size of the support decreases as V inereases,
Figure lb where we plot the support of the final policy mhy
predicted in the Theory Section 4 The policy stays supported by a large number of atoms even when V x V", as long as V VF But as soon as V 2 V", the support of the optimal policy "E.V becomes a singleton: arg max, ((y)(ry)) V)).
Thus, when V 2 VP, there is a risk that our off-policy REINFORCE algorithm might produce policies that lose their diversity. In addition, notice that when V is slightly larger than V the reward of the corresponding limit policy not as large as the reward of the best arm in the support of the limit policy for slightly smaller than VH, Thus the potential of finding this superior arm lost forever when V crosses Va.
Policy improvement with AsymRE We study a policy improvement scheme over 40 iterations, where at each iteration we apply the AsymRE algorithm for 500 steps using the policy obtained at the end of the previous iteration both our behavior policy and starting current policy. In Figure 2 we show the expected reward of the corresponding policies. We notice that when V < VP, for every policy iteration step, the performance of the improved policies Tv are better than the corresponding behavior policies (4, as predicted by Theorem 4.3 However, for V > V", we have already seen deterministic and suboptimal policy. Here
in Figure La that the policy converges iterating this algorithm does not permit to recover from this premature suboptimal convergence, regardless of the value of V > Va. We also observe that while they do not plateau as dramatically, trajectories corresponding to V only slightly smaller than V (such as 0.525 and 0.54) converge to a less favorable limit policy than for V < 0.5.
that
5.2 Large Language Models
We validate our findings by training LLMs reasoning tasks.
7

---

Expected reward for different valuees of the baseline V
V:-0.125
V: 0.000
V: 0.125
0.9
V: 0.250
V: 0.375
Expected reward
V: 0.500
0.8
V: 0.540
V: 0.600
0.7
V: 0.625
V:
V: 0.875
1.000
0.6
V: 1.125
100
102
10?
10
policy iterations ieeations
Figure 2: Performance of the current policies for 40 iterations of policy improvement with AsymRE. Each iteration 500 steps. The curve corresponding V z V red.
Experimental setup
We train Llama-3.1-88 I-Stu-nnttruct (Dubey al., 2024] (which we refer as Llama 8B), Llama-3.2-3BB Instrrcct (which we refer to as Llama 3B) and Qwen2.5--BBIInttuus Yang al. 2024 (which we refer to as Qwen 3B) with the AsymRE objective
J(m) = (. ra
Σ a)) (V (V)) (og(((((())))))
where as are prompts from the dataset D, each y/ is full sequence of tokens auto-regressively sampled from the L.M μ. and V à/6 E.e r(yt. is the empirical estimate of the average reward vu(-la) This loss is the empirical approximation of the context-corrected loss in Equation (4). We place ourselves in the delayed updates setting, i.e. the behavior policy μ is simply outdated version of the current policy π,, which we update every N training steps; this corresponds to policy improvement scheme where each iteration lasts N steps. We set the number G of samples
per prompt to 8. We train and test on the MATH dataset (Hendrycks et al. (2021]. 12.5k high school-level problems), as well as on subset of size 142k of the NuminaMath dataset (Ll al.
(2024), competition level problems). The reward of trajectory is if the answer is correct, and otherwise. Additional details regarding the experimental setup are given Appendix
Impact of the baseline öV The train and test losses for various baselines V + 8V are shown in Figures Bndd (Llama 8B and Qwen 3B respectively). Similarly what we observe with the bandits experiment (Figure 2) while the baseline 8V is smaller than the critical value 0, the model's training accuracy tends to increase with 5V. This is reflected in the test accuracy as well, though less strongly. When the baseline áV reaches, then passes 0, a phase transition occurs and both the training loss and the test loss suffer a catastrophic collapse. The larger 8V 2 O, the faster this collapse occurs (for 8V O, the collapse occurs at the very end of our training run). This corresponds in the theory (and in the bandit setting) to the regime where the limit policy becomes deterministic: while this can result in good expected reward in the accuracy in the LLM setting, where the model must retain diversity to answer the range of problems with which it is tasked. This loss of diversity can be observed in the entropy of the models, in Figure in the Appendix.
bandit setting, it leads drastic decrease in
The left side of training stability: we compare 7 independent training runs with 8V 0 and 7 training runs with 8V = -.. and find that all runs with 8V 0 collapse.
our claim that letting 8V be slightly smaller than 0 results in greater
Comparison to GRPO Though further testing would be needed to reach definitive conclusions, we also observe that our loss compares favorably GRPO both in terms of training speed and final accuracy in the off- off-policy setting, as shown the right side of Figure 5

---

Training accuracy MATH
Test accuracy on MATH
0.7
0v.
0.8
0.49
0.40
Accuracy
Accuracy
0.3
0.46
0.45
0.2
84 -35
NV 84-3a
0.1
0.43
0.42
1000
1400
1800
Steps
Steps
Figure 3: Training dynamics of Llama 8B on the MATH dataset (results are averaged over 3 seeds, and moving average with window of size is applicd). The behavior policy is updated every N = 250 training steps.
Training accuracy MATH
Test accuracy MATH
0.6
5y
0.6
0.5
0.4
Accuracy
Accuracy
0.3
0.3
0.2
0.1
Steps
Steps
Figure 4: Training dynamics of Qwen 3B on the MATH dataset (results are averaged over 3 seeds, and moving average with window of size 3 is applied). The behavior policy is updated every N 250 training steps.
Our additional results in Appendix Cn the larger NuminaMath dataset, as well on Llama 3B, support the same conclusions.
5.3 Discussion
These experiments corroborate our theoretical findings and can be summarized as follows:
Letting the baseline V be strictly smaller than the expected reward V/ the behavior policy μ makes the training stable, even when off-policy, and offers monotonic improvement of the policies.
Performance tends improve as V grows closer VP while remaining strictly smaller.
As soon V passes V", the policies become more deterministic. In the context of bandits, this leads to a suboptimal convergence of the policy improvement scheme. In the context of LLMs, this leads to collapse of the training and test loss.

---

Training accuracy MATH
0.7
GRPO vs AsymRE
0.8
Accuracy
0.5
0.4
Acturacy
0.3
Llara Axyrtttz
Lare aitoo
Ownn 181 Asundi
W11
6RPO
500
1000 1500 2000
1000 2000
3000
4000
5000
6300
Steps
Steps
Figure 5: Left: Training dynamics Llama 8B on the MATH dataset for two values of the baseline, 8V
and 8v = -0.1, and independent runs for each value. The behavior policy is updated every N 250 steps. We observe systematic collapse when 8V = 0. Right: Test accuracy of Llama 8B and Qwen 3B trained on the MATH dataset with GRPO and AsymRE (with V 0..1). The behavior policy is updated every N 250 steps. Asymmetric REINFORCE, leads faster convergence and better than GRPO.
While further large scale experiments needed to draw definitive conclusions regarding the training of LLMs, our findings suggest that adding a small conservative correction oV = -0.1 in the advantage term of the training objective
Σ
(r(ui, (V 8V)) log(+((s(z))
G
might result in greater training stability, and help prevent the collapses often observed in RL training.
Conclusion
We have thoroughly studied the behavior of our simple off- oof-cooicy algorithm AsymRE with respect to the baseline V, both theoretically and experimentally. We have confirmed our claims that in an off off- off-policy setting, selecting baseline slightly smaller than the expected reward of the behavior policy results in superior asymptotic performances, and can help alleviate the risk of training collapse, Such choice of baseline corresponds putting more emphasis on positive training examples, and less negative ones, which is intuitively reasonable.
Limitations and possible improvements While have analyzed in depth the role of the baseline V in our simple AsymRE method, would be interesting to extend our study to more sophisticated losses, which might e.g. include importance ratio correction and KL-divergence regularization. Finally, the possibility to leverage off- off-policyness by training over the same samples for multiple epochs was one of our initial motivations: quantifying the potential run-time and compute gains and the impact model performance of such scheme is an exciting future research direction.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in 1lms. arXiv preprint arXiv:2402.14770, 2024.
AlphaEvolve-team. signing advanced algorithms.
Alphaevolve:
gemini-powered
coding agent for de-
10

---

2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05062, 2022.
Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 8(3 8(3-4):231-357, 2015.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane I Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
Taco Cohen, David W. Zhang, Kunhao Zheng. Yunhao Tang, Remi Munos, and Gabriel Synnaeve. Soft policy optimization: Online off-policy for sequence models, 2025. URL https://aaxii org/abs/2500 05453
Ross Cressman and Yi Tao. The replicator equation and other game dynamics. Proceedings of the National Academy of Sciences, 111 (1 1011010877,, 2014. doi: 10. 10.1073/pnas. 1400823111. URL https::s::///wt pnas /rgg/oo//abs/1..10333.
1400823111
Thomas Degris, Martha White, and Richard arXiv: 4839, 2012.
Sutton. Off-policy actor-critic. arXiv preprint
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2xXiv 2024.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv: (80.001561 2018.
Yunzhen Feng, Ariel Kwiatkowski, Kunhao Zheng, Yaqi Duan, and Julia Kempe. PILAF: Optimal human preference sampling for reward modeling. In Foroy-second International Conference Machine Learning, 2025. URL https:://pperrevieett
Shixiang Shane Gu, Timothy Lillicrap, Richard E Turner, Zoubin Ghahramani, Bernhard Schölkopf, and Sergey Levine. Interpolated policy gradient: Merging on-policy and off-policy gradient estimation for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04779 2024.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song. and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi. Shengyi Costa Huang. Kashif Rasul, Longhui Yu, Albert Jiang. Ziju Shen, Zihan Qin. Bin Dong, Li Zhou, Yann Fleureau. Guillaume Lample, and Stanislas Polu. Numinamath. http:////wggnnnga.eco.o/.//////a ott.a//..////aiaaaaaaaaaaaaaaaa. project- projectt-umina/abtiee aimo- progress www.prewss...taaaaaaaaaaaa///tta dataset..aaa...aa 2024.
com/
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International
Conference on Learning Representations, 2017.
www.ntisscoola........... org/
CorpusID:535522200

---

Meta. The Ilama 4 herd: The beginning of a new era of natively multimodal ai innovation. https //ai meta. wwt.cccobbbb/a hammbbog//lamm
2025.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforeement learning. In Advances in Neural Information Processing Systems, pages 1054-1062, 2016.
OpenAL Introducing openai o3 and o4-mini. introducing-o33 o3- and 144minni/ 2025.
https:///ppeaa..com/
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In International Conference on Learning Representations, 2021. RovX uuuHau
Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policc temporal-difference learning with function approximation. In ICML, pages 417-424, 2001.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2023.
Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. Group robust preference optimization in reward-free rlhf, 2024. URL
https://arxiv org/ abs 20304
Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, et al. Offline regularised reinforcement learning for large language models alignment. arXiv preprint arXiv:2405. 19107, 2024.
Nicolas Le Roux, Mare G. Bellemare, Jonathan Lehensold, Arnaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-Laufer Sándor Toth, and Sam Work. Tapered off-policy reinforce: Stable and efficient reinforcement learning for Ilms, 2025. URL https
//arxiv. www./iivorrg/ee aba/2503.... 14286
Ernest Ryu and Stephen Boyd. A primer on monotone operator methods. Applied and Computational Mathematics, 2016
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1007.06344 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifree George Van Den Driessche, Julian Schritwieserr, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7787):484 489, 2016.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the perfor- mance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.0884 2024.
Yunhao Tang, Taco Cohen, David W. Zhang, Michal Valko, and Rémi Munos. Rl-finetuning llms from on- and off-policy data with a single algorithm, 2025. URL https::s:: //arxii org/orgga/... /bs/ 2503. 19612
12

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL htttt://///: org/ attaa/bb////2/2 2302. 13971
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279 292, 1992.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Reinforcement Learning, pages 5-32. Springer, 1992.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning, 2024.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115. 2024. doi: 48550/ arxiv. 2412.15112
10.4850/ARXX..2442.55555
URLhttts:://dd
org/10. 20g//0.48550//////
Rui Zheng, Shihan Dou, Songyang Gao, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Limao Xiong, Lu Chen, Zhiheng Xi, Yuhao Zhou, Nuo Xu, Wenbin Lai, Minghao Zhu, Rongxiang Weng, Wensen Cheng, Cheng Chang, Zhangyue Yin, and Xuanjing Huang. Secrets of rlhf in large language models part i: Ppo, 07 2023.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng, The surprising effectiveness of negative reinforcement in Ilm reasoning. arXiv preprint arXiv:2506.0/333 06 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the central claim that lowering the baseline off- off--ooliy policy gradient updates emphasizes successful trajectories and leads to policy improvement. These claims are substantiated with theoretical analysis (Theorems 4.2 and 4.3) and validated empirically in both tabular bandits and LLM fine-tuning tasks (Sections 5.1 and 5.2).
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the work are discussed, particularly regarding the need further explore importance sampling. KL divergence, and multiple passes over data to leverage off-policy training more effectively.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Full assumptions and detailed proofs provided for the theoretical results in Appendix, supporting Theorems of the theory section.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The experimental setup is described in detail in the experimental section, with hyperparameters, baselines, and update strategies specified. Additional training details in Appendix.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: While the paper provides detailed methodological descriptions and experimen- tal setup, do not publicly release the code or models for legal reasons.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and evaluation details, including learning rates, behavior policy updates, and dataset usage, are provided in empirical section and Appendix.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
14

---

Justification:
The paper does not report error bars or statistical significance tests, However, multiple seeds and smoothing are used in LLM experiments (Figure 3), which provides some robustness.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The paper provides hardware setup details (c.g., separation of GPU roles,
rollout/training workers) and discusses the update frequency controlling off-policyness (Appendix).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /sttss////orrrs..cc/p/l////////
Answer: [Yes]
Justification: The research adheres the NeurIPS Code of Ethics. It focuses on improving learning algorithms with no human subject data or potential ethical violations.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper is related the theory of reinforcement learning, which has many potential societal impacts, yet are too uncertain for us to discuss them precisely.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The models trained are not released and pose no known high-risk misuse potential. Thus, safeguards
Justification:
applicable.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper credits and cites existing datasets (e.g.. MATH) and pretrained models (e.g., 1--AMA-----8B-Instuuee with appropriate references.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new datasets models released with this paper.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The work does not involve human participants crowdsourcing.
15

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not applicable, as no human subject research conducted.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: The paper makes of LLMs core part of the experimental setup, clearly describing how these models are used and evaluated.
16

---

A Proofs of Theorems 4.2 and 4.3
A.1 Proof of Theorem 4.2
We first prove Theorem 4.2 which we restate for the reader's convenience:
Theorem 4.2. [Analysis of expected AsymRE for tabular softmax policies] Let Y be a finite set, be some behavior policy whose support is Y, and consider softmax policy representation (y) def c(1))/ Ey e'aa') on Y, where the logits ( are the policy parameters. We consider the expected AsymRE algorithm with respect the logits initialized some Ty with successive iterates T and learning rate per Definition 4.1 Then the AsymRE algorithm converges to limit distribution T T... The baseline parameter V controls the nature of the limit distribution:
V < VP, then T.y is defined by
(,vV()) = (r(v)(r(v) -) -u.V)
Vr-V
where Tu,V is uniquely characterized by the constraint Ever(u()(((()) - VK V. and x- = max(x,0) 0).
Vf Vr, then J is defined by its support
supp(T = arg max (u()(r(() - V), yEY
and by Tg.y (1) f., (2) To(y)/ no (2) for any y, 2 supp(Tra.)
Vf VP, then T can charge any of the elements in the set
{yl min (())(v)) v) -(z)(r(2) v >0,
zey
depending the initial condition To-
As a consequence, supp( v. S supp(T)),v)) when V22 $ V12 < V*.
Proof. For simplicity, omit the indices when the context makes them unnecessary; in particular, we write n, for nh v. We are optimizing
E E (r(y) - V) log (ty)
(S)
with the logits parameterization क (yy exp(((y)), hence gradient flow ascent corresponds to
a,I= -v=(( = nnaa - bT(U)), where F(I) def Σ
ayl(v) blog Σ exp(I(2))
(6)
y
and ay def ((g)
V). def Σ ay VH and 7 > 0 is some speed parameter. This follows
Vr from the definition of π (य) exp ())) Σ. exp((.), and of V" Ey (())(()).
Eventually, one may also consider gradient ascent with fixed stepsize 0, leading to the evolution
h+1 4 + VV))
(7)
We will see that, as long as v) not too large, dynamics (6) and 7 converge to the same limit.
A.1.1 Case V V
When Va V, which is equivalent to b 0, maximizing a linear function, which leads to
4 to + rtta, where lo is the initial condition of the logits and def (ay)vEY. This implies, assuming
I without loss of generality,
exp(a,! lo(y))
n. (v) lim
Hy E arg mag axpp(loo)) or Tly E arg max az moyy).
+ Σ, exp(axt + lu(2)
17

---

Speed of convergence.
Note that in this case, we can also characterize the speed of convergence: if
def
ay + max U2
Tv( explayt folayt+lol0n [o())) eexplaz(+1(x))) S exp((ay - a.). lo(y) min (o)),
which goes to zero exponentially fast as exp((ay - a.)(). af = max dz then
expla,t + lo(y))
Te (v) Σ expl(az++ +o(=))
explayt lo()))
exp(ay()
E
EEarggaax exp(lo(=)) + Σ E maxa, exp((a= ay)l 10(2))
=1, 2 maxay exp((a, a.
Sexurmaxay exp(lo(=))
Series expansion leads to
T (v) Y
|| me T.ll
S
expl zgarg max a
max
(a= - (1a=-yy))
zEarg a, exp(lo(2))
Hence the convergence speed is A exp(-ctt for c = min(a, ay) and some constant A. Note that this formula holds for both gradient flow and gradient ascent.
A.1.2 Case * VP
Convergence of the continuous dynamics. introduce the following function
To show the gradient flow dynamics converge, let us
6 - Σ mr()).
+ Σ (v)
The dynamics the logits can be cast as dynamics on the probabilities.
exp(l(())
exp(h(v))@0M())
exp(4(()) Ey exp(((()'))al Σ. exp(l(()))
0,)((y) = a E. exp(((E)) E. exp(l((=))
E. exp(l(()))
(v)0li - ne(y) Σ (2))01(((z)
m,()(aa (T(1)) m Σ ((=)((a-bb(()))
= (,) - bbr(() - Σ (=) + Σ m=(2)²) = mr(3) - -bTe(y) Tz),
where
= Σ (2) - Σ (z)²
Casting this dynamics on 4 we have
0. = Σ (ay - br((3))0 = Ea, (a, bTe( -bt Σ -m(()((a-bm(()))
- bne(2)))
Σ Te(w)(a,-brr(u))) -
Σ -
- bur(l))) Σ Te(=)(as - bbr
Σ - - bTT(
Σ Te(y)(a, Tele)(a, bTe(())
Vary wai (ay - bm, (Y)) 2 0,
18

---

with equality when ay ( is constant on the support of Because V is continuous and bounded above on the simplex, which is compact, D, converges and 0, goes zero. As V * V (which means that # 0), this shows that T. (1) either goes to zero, or converges to (ny T. //b for some constant T.. Thus, n, converges to some limit distribution n...
Since T. (y) is non-negative, we deduce that, with Y, S Y the support of Tw
T (y) = Lyy & Y.) (ay T..)*
A.1.3
Case V > V
In the VN V, i.e. b > will now show that Y, is actually defined by
Y. = { 0g
T. > 0}.
Let us proceed by contradiction. Assume the existence of such that ay T. and for which n. (y) = O. Then, by continuity of Ty, there exists an E and a such that for any > T, we have
(v) $ e/b,
ay > 22,
which implies
(z) (a) bT( -1,) ST()).
Due to the logits parameterization, we have (by) > O, which leads to =v 2 Axxp(ct) for some A > O, which is contradiction.
Finally, the fact that Σ n. (g) = uniquely characterizes T. Indeed, consider the function
G:TH Σ (ay +)+
Then
= Σ (ay r.)+
G(r.) G(r.==
Σ (V) =
The function G is continuous and strictly decreasing on [min ay. max ag]. with G(max a.) 0 and G(O) 2 1. If min Oy < 0, then G(0) 1, which implies the existence of unique т, (0, max ay) such that G(T.) 1. min ay 2 0, then the only solution of G((T)) is T, 0.
Inclusion of the supports. From the previous results, we already have the inclusion of the support of Va. in that of Tu.V for any V < V" Let respectively Tu be the limit 7, introduced above for V1, respectively V2. We would like to show that supp Ju, supp Tu, V. which is equivalent showing that
consider V/2 s V1 < VP, and let Ju, Vi
(r))( - Vo2) TT.V3 S0
p(y)(((y) - V/1) TA.V, S
This is consequence of the fact that T..V is increasing with V, which will prove Recall that we have
Σ (a, + =1,
hence
(((O))((o) -Tav)+== VH V.
For > O small enough, we have
(u(o)(((o) (V +e)) +uv)+ > E(u())((()) - V + -
=V- (V +6) ((m())((()) - (V + +)) +,
hence Ty,V+s 2 TW.V TH E(u())((v)) (V +s)) +)+ is decreasing in
19

---

From gradient flow to gradient ascent. We notice that F is the sum of linear term and log-sum-exp term. Since the log-sum-exp term is strictly concave on 1 (i.e. on the hyperplane E4y = E (0)( due to the sign of b, we are maximizing smooth strictly concave function. Gradient ascent on a concave objective can be seen as a discretization of the gradient flow, and known to follow the same dynamics as long as the step-size is smaller than twice the inverse of the largest absolute eigenvalue of the Hessian [Ryu and Boyd 2016) page 17]. In our case, the Hessian of F is equal to -b times the Hessian of the log-sum-exp function, which leads to
F1) - (diag(T)) πτ) xT))
simple calculation leads to the following bound on the operator norm:
F 6( [i)gg()]] + ||, ||) ((z(1)) + 1 26.
As a consequence, as long as 17 < 1/b = 1/(V" - V). the gradient ascent converges to the same
point as the gradient flow.
Remarks on the speed of convergence. When V min min r(y). In this case r. and since ay > O, this implies that ı has full support. The conservation of 2244)) (v)), which is a consequence of the fact that
Σ (()) aa T())= ( b-b=)
implies that if some logits go to plus infinity, others should go to minus infinity, which is not possible Tly) # for all y. As a consequence, the logits are bounded, and the maximizer of F is achieved on its domain. Moreover, because I' strictly concave on 1 and C it strongly concave and smooth on any compact set, which implies exponential convergence of both gradient flow and gradient ascent [Bubeck 2015 Theorem 3.10].
When VV > V > min r(y). To quantify the speed of convergence, we can use
lim 0,1(y) = ay - bu. = Byy E Y. + 1[y & Y.Ja,
Because V > min r(y) and (v() > 0 for all y (since has full support), this implies min ay < 0, which implies T. >> 0 as have shown above. For any y E Y., we have lim a,1(y) = T, > 0, which means that the logit I(y) will go linearly to plus infinity. For any y E Y., by definition dy S T. and lim 0,((2) = ay. which means that the logit ((y) behaves as ayt + o(t). As a consequence, for ay < Tw, we have
Te( exp(l(u)) Eexp(h(2)) exp(r,t) Σ exp(( - Tx) + o(t)) S exp((a, - r.)t + o(t)).
explatt+of)
In other words, the mass of such points decreases exponentially fast. The examples below show that similar conclusions cannot be drawn regarding other points.
Example of slow convergence when ay T. > O. In the event ay = T., the convergence can be slower. For example, one may consider a situation where I(y) = exp(r.t) and 1(z) = exp(T.t tt- log(t)). leading to convergence in 1/t. This is the case when
Y (1,2,3} μ (1/3, 1/3, 1/3), r= (9,3, -6), V = 0, a= (3, 1, 2), b= 2.
Then, one can check that 1 and (1,0,0). The dynamics on (2) can be written as
(2) x (2) - bT(2) Σ d (v) - Tv))))
=2( (x - (3m(1) + m() - (3) - 2 22 2 2 =2()(1 (x - (3(1- -())) + m() - 2(1 x2))² (2)² + (3T))))
-24 + 42)) + o(3())
From the previous derivation, know that x33 will go exponentially fast; hence the dynamics will be dominated by (2) ~ (2) which leads to T(2) ~ 1/4f, and the slow convergence.
Example of slow convergence when ay = T. = 0. Similarly to the previous example of slow convergence, considering
Y = [1,2}, == 11112,1 1/2). r= =2,,)), ==0 0
leads to 0, b= = 1, (1, and (2) 1/2t.
20

---

A.1.4 Case V < V
When V V, which is equivalent to function on 1+, which can lead to various limits depending on initial conditions (see after the end of the proof for short discussion).
maximizing (not minimizing) strictly convex
Let us first characterize the union of the supports of the potential limits of x. Using the previous characterization of Oui, get
(y) ( (zT(2)
0, log
π(2) ay bT(()) + - - as + bT(z) T
π(z) π (v)
By a₂ b(T(2)) =T))).
We deduce that for any (y, 2), since (yy) - T(2) S and -b b > 0.
Tv(
(v
exp((ay - 0= b)t).
ne (2) (2)
This means that min; Gy a2 b < 0 implies (y) = 0. Hence y cannot belong to the support of any potential limit. It is easy to show that situation is the same min, 02y 02
0, as Tu(y) Tu(2) 1, hence Gy a, (y) - no (2)) < and 0y a, ((x(T) x()) remains smaller than some constant c < 0 for all t. Reciprocally, if min; ay - > O, an initial condition To close to Dirac y will lead n. by. hence y belongs to the support of some potential limits.
Showing the convergence the case of the discrete gradient ascent follows from arguments similar to the the concave case treated above.
As mentioned in the proof, the dynamics the case V < V are harder to describe; the gradient flow equation is case of replicator equation [Cressman and Tao, 2014)), and mapping the initial condition mo to the final limit t, is not easy. As shown in Figure 6 the regions (To x. = σy) are not e.g. polytopes.
1.0
Limit (1,0,0)
Limit (0,1,0)
Limit (0,0,1)
0.8
0.6
Figure 6: Each point (z, y) in the triangle corresponds to an initial distribution To = (a, 1 on set of cardinality 3. The color point Mo indicates the limit of gradient ascent starting with as its initial condition: the three colors correspond to n. (1, 0, 0). = (0, 1, 0) and π. = (0, 1) respectively. Note that some initial conditions converge to other limits, but they are of mass in the simplex and do not appear in the figure.
21

---

However, a simple case is when arg max dy n arg max Gy (TT(()) is non-empty, in which case the dynamics converges toward the uniform distribution on this set. This is notably the case when To uniform.
A.2 Proof of Theorem 4.3
We now prove Theorem
Theorem 4.3. [Policy improvement dynamics] Let μu be any policy with support Y, and let V < VR.
1. Each application of the AsymRE algorithm increases the expected reward: VTvp VR.
2. The sequence of expected rewards V(TvY"H converges to some limit expected reward v
Let y def ty : r(y) = }. Then the mass of (Tv) }a concentrates exponentially fast Y i.e.
Σ SEY
((Tv)"))()) c
for some 1.
3. There exists Vo,u such that the corresponding limit reward is optimal (i.e. V maXzeY r(y)) if and only if V < Vo..-
Proof. Let us first prove that the value increases. Let us denote ā = Tvp, Y, its support, and V (defined as in Theorem 4.2b. We have
(V" - V)r(y) = (u(v)(((v) - V) - -
This implies that for any y Y..
r(y) (Vr V)T(y) +T + V.
M(v)
The value of the policy π can be computed
(VF (V=- V)T(v) +T + v)
v Σ F(w)r()) -=((2) x(y)
m(y)
T(y)
(v)
V - v) Σ
+ Σ
m(y)
k(x)
Using Cauchy-Schwarz, we have
Σ (()² mly)
T(2)²
T(y)
Σ
Σ 2(2)
Σ
v (uy) uuly)
=1
muv)
As consequence, we deduce
T(y) m(v)
V" 2 Va
Σ
VM.
Where have used the fact that 0.
After one application of Tv. we only keep in the support of Tvp the arms for which r(y) >
V + T/M(y) 2 V (and they do not reappear in the support of T for 2 1). This implies that TTZ..V 0 for n according to the characterization of r provided in the proof of Theorem 4.2 This, in turn, implies
Tvp(y) ((v) r(v) V Tvu(=) (==) r(z) V
for all 2 in the support of Tvp. Let denote
Y arg
max r(y)
yEsupp Tv
22

---

By recursion, we deduce, for & Y and E y both in the support of Tv p. that
Tuu(y) ply) r(y) V " TVu(2) pa(2) r(2)
which implies that TV)(y) goes exponentially fast to as n increases.
The last statement of the theorem is direct consequence of the characterization of y the monotonic- ity of the support of T as V increases, and the fact that for V minr(y), supp(TvH) -Y. O
B Details regarding our LLMs experiments
We provide additional details regarding our experimental setup from Subsection 5.2
We trained Llama-3.
with the AdamW optimizer Loshchilov and Hutter 2017 with learning rate x 10-$, Each time prompt is sampled from the dataset, trajectories are generated from the behavior policy and used compute the empirical batch average V. 128 trajectories are included in each gradient step. The maximum trajectories length is set to 2048 tokens (the generation stopped after this number of tokens is reached).
The GPUs were split into two categories: workers that generated the responses to the prompt sampled from the dataset, and trainers that fine-tuned the model on the generated trajectories. In order to control the degree of off- off--poicyness of the model, we varied the update interval parameter, which the number of gradient steps between updates of the weights of the workers. The larger this parameter, the more off-policy the training is.
Inference parameters set to temperature 1.0 (respectively 0.1) and top-p 1.0 (respectively 0.95) for the training (respectively for the evaluation).
C Additional experiments
We report additional experiments.
Entropies of the policies in the bandits setting In complement to Figure Ib we report in Figure 7 the evolution of the entropy of the policy T. under the AsymRE algorithm in our bandits experiment.
Entropy of the policies for differant values of the baseline
2.5-
Estrppy d'the polcydt
2.0
0.125
0.000
0.250
V: 0.375
1.0
0.500
0.625
V: 0.750
0.5
0.875
1.125
180
161
107
Iteration numberc
Figure 7: Entropy of the policy wa function of the iteration number for different values of the baseline parameter 8V. For the larger values 8V. entropy of the policies drops close 0 policy becomes deterministic.
Entropies of the policies the LLMs setting We report in Figure & the evolution of the entropy of the policy म. under the AsymRE algorithm in the LLMs setting.
Additional experiments with Llama 3B Figure 9 we represent the training dynamics (as in Figure 3 for Llama 3B when training the MATH dataset.
23

---

Policy entropy
1000
1600
Stops
Figure 8: Evolution the policy entropy during the training of Llama 8B on the MATH dataset (averaged over 3 seeds). The larger the baseline V is, the faster the entropy decreases.
Llama 3B, training accuracy MATH
Liama 3B, test accuracy on MATH
0.8
0.48
0.44
Accuracy
Accuracy
0.42
0.5
Ny 45-45
0.4
Nr. N5-45
0.45
BV
3V 0.9
0.38
0V 0.
09
BV:
96.
0.4
We.3
0.38
81:11
89
0.36
0.34
500
1000
1500
0
500
1000
1500
Steps
Steps
Figure 9: Training dynamics of Llama 3B on the MATH dataset (a moving average with window of size 3 is applied to the training curve). The behavior policy updated every N 250 training steps.
we represent the training dynamics for Llama 8B, Qwen 3B and Llama 3B respectively when training and evaluating on subsets of size 142k and 2k respectively of the NuminaMath dataset (rather than on the MATH dataset).
Additional experiments with NuminaMath In Figure 10 Figure and Figure 12
Llama 8B, training accuracy Numinaaath
Liama 8B, test accuracy NuminaMath
0.24
0.24
0.2
Accuracy
Accuracy
0.2
0.16
0.18
8V.
8V. 0.2
BV: 4.1
BV: 6.1
0.12
50.0
09:01
0.1
09.02
0.14
09
8v.23
06.03
1500
Steps
Steps
Figure 10: Training dynamics of Llama 8B on the NuminaMath sub-dataset (a moving average with a window of size 3 is applied to the training curve). The behavior policy is updated every N 250 training steps.
Additional comparisons GRPO We report in Figure a comparison between GRPO and AsymRE where the models trained and tested subsets of the NuminaMath dataset.

---

Qwen training accuracy NuminaMath
Qwen 3B, test accuracy NuminaMath
0.4
0.35
0.3
Accuracy
0.25
Acouracy
0.2
5ir 4.5
0.2
BV.
No0 4.2
0.15
3v 02
0.15
09: 4.2
09.
0.1
99.3
0.1
890
0.08
0.05
500
1000
1500
0
500
1000
1500
Steps
Steps
Figure 11: Training dynamics of Qwen 3B on the NuminaMath sub-datases (a moving average with a window of size 3 is applicd to the training curve). The behavior policy is updated every N 250 training steps.
Llama 3B, training accuracy on NuminaMath
Liama 3B, test accuracy on NuminaMath
0.24
0.22
0.2
0.18
0.2
Accuracy
Accuracy
0.18
0.18
0.14
0.12
0.1
80:11
86:91
09:22
0V.92
5v:33
9v:33
1000
Steps
Steps
Figure 12: Training dynamics of Llama NuminaMath sub-dataset moving average with a window of size is applied to the training curve). The behavior policy is updated every 250 training steps.
GFIPO vs AnynEit:
0.15
0.4
0.3
0.15
0.1
Unme onho
58, G2F0
1000
2000
3000
4200
5000
0000
Figure 13: Test accuracy of Llama 8B and Qwen 3B trained on subset of the NuminaMath dataset with GRPO
and AsymRE (with V -0.1). The behavior policy updated every N 250 steps, and moving average with window size applied to training curve.
25


Reviews:
[{"review_id": "I6j86bMZ2A", "reviewer": "Reviewer_Wxir", "summary": "RL has been very successful in aligning LLMs for various downstream tasks. However, the most popular RL methods are on-policy and have several drawbacks. This paper proposes a new off-policy method to address the limitations of on-policy RL methods when aligning LLMs. The paper proves two theorems: one regarding the limits of the algorithm in relation to various baselines, and the other concerning policy improvement. Additionally, the paper conducts several experiments using LLaMA on mathematical tasks. The results of the experiment show that the various baselines, as shown in the proof, empirically affect the models' downstream performance.", "questions": "- How does this paper compare to [1], PPO [2] in terms of the importance ratio enabling off-policy updates, and GRPO?\n- The original motivation of the paper was efficiency, but there are no comparisons to on-policy algorithms to demonstrate efficiency in any way.\n- For equation (1), how are you avoiding the importance sampling issue?\n- How do you choose the N to use for delayed updates? How sensitive is the proposed algorithm with respect to N?\n- Are the bounds in the bandit RL setting or the multi-step RL setting?\n- Could you provide more motivation for why you have the additional delta V term? The fact that the value of V varies for each prompt is expected, so I'm not sure why adding the correction term is necessary.\n- How is delta V trained in practice because it seems to be different than V^\\mu, which is computed using multiple rollouts.\n- For most of the bounds in the appendix, there are no assumptions made regarding the ratio of \\pi(y)/\\mu(y), which could be bad if the ratio is not assumed to be bounded, right?\n\n[1] Fine-Tuning Language Models with Advantage-Induced Policy Alignment by Banghua Zhu et al. 2023\n\n[2] Batch size-invariance for policy optimization by Jacob Hilton et al 2022", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe ability to train off-policy for LLMs alignment is a very important problem. The paper's proposal is scalable and has the potential to impact training models in an asynchronous manner. The paper comes with guarantees that hold in practice, which makes it a promising direction.\n\nWeaknesses:\n\nThe paper has several shortcomings. First, it omits some key baseline algorithms necessary for a thorough comparison. The evaluation is also limited to a single model and task, which raises concerns about its robustness and applicability to other tasks. Moreover, without including baseline algorithms, it’s unclear how the proposed method compares to existing approaches in the literature, such as [1]. Finally, some of the theorems in the appendix seem underexplained. \n\n[1] Fine-Tuning Language Models with Advantage-Induced Policy Alignment by Banghua Zhu et al. 2023", "was_revised": false, "final_justification": "", "rebuttal": "We are very thankful to the reviewers for their time and for their insightful comments.\n\nWhile the reviewers seem to agree that the papers' contributions are impactful, two shared concerns appear:\n* a lack of diversity in the experimental setup (not enough models, not enough datasets), and\n* a lack of comparison to popular RL algorithms.\n\nWe want to address these shortcomings in this general response (which we include in all individual responses), before answering each reviewer's questions separately.\n\nOur initial intent was for the article to be equally balanced between a theoretical section, a bandits section, and a section on real-world applications, which is why we chose to keep the experimental section concise. However, the reviewers' comments have helped us realize that the paper would be strengthened by the inclusion of additional experiments, and we thank them for that.\n\nWe have run a series of new experiments, which we hope will fully answer their concerns. In particular, we add two additional models (Qwen 2.5 3B and Llama 3.2 3B), a much larger dataset (NuminaMath) and a comparison to GRPO. The resulting tables can be found below (sadly, a recent change in the conference's rules does not allow us to share graphs anymore):\n\n**Table 1: Training Dynamics of the off-VPG Algorithm Across Different Baselines.**\nThis extends our Figure 3 with additional models and datasets (Same hyperparameters. Reported values are the maximum pass@1 accuracy (in %) on the test set observed during training (2000 steps, with evaluation every 200 steps). Some runs crash during training, as in Figure 3; these are marked in bold.)\n\n|Dataset|Model|δV=-0.5|δV=-0.3|δV=-0.2|δV=-0.1|δV=0|δV=0.1|δV=0.2|δV=0.3|\n|-------|-----|--------|--------|--------|--------|-----|-----|-----|-----|\n|MATH|Llama8B|48.7|49.1|49.3|49.3|**48.9**|**47.5**|**46.1**|**43.5**|\n||Llama3B|45.9|46.7|46.4|46.4|**45.4**|**44.5**|**43.9**|**45.0**|\n||Qwen3B|62.2|63.5|65.1|65.6|64.9|**37.8**|**14.7**|**13.0**|\n|NuminaMath|Llama8B|20.5|21.5|22.2|23.6|**23.1**|**17.8**|**18.0**|**17.2**|\n||Llama3B|20.5|21.8|21.8|21.7|22.7|**20.7**|**19.8**|**19.6**|\n||Qwen3B|20.3|37.6|40.6|42.0|40.8|**23.7**|**21.9**|**19.7**|\n\nIn our article, Figure 3 provides empirical evidence of how the choice of baseline δV affects the performance of our off-VPG algorithm. \nWe extend the same experiment to two additional models (Llama3B and Qwen3B) and an additional dataset (a 144k-token subset of NuminaMath)\n\nWe consistently observe the same two phenomena highlighted in Figure 3:\n\n- When the baseline is large (δV ≳ 0), training becomes unstable and often crashes (bold entries in **Table 1**).\n- As the baseline increases, the training becomes faster—reflected in the average left-to-right progression of higher scores within each row (for runs that did not crash).\n\nThe general appearance of the training and test accuracies are also coherent with our earlier experiments. These results reinforce the robustness of our recommendation: to use a slightly negative baseline, such as δV = -0.1.\n\n**Table 2: Comparison of off-VPG and GRPO, off-policy** \n(10000 steps, with evaluation every 200 steps)\n\n|Dataset (Train & Test)|Model|off-VPG (off-policy)|GRPO (off-policy)|\n|------------------|------|--------------------------|----------------|\n|MATH|Llama8B|52.0|50.9|\n||Llama3B|49.0|49.1|\n||Qwen3B|64.8|61.7|\n|NuminaMath|Llama8B|26.1|24.8|\n||Llama3B|25.5|25.1|\n||Qwen3B|44.3|40.8|\n\nWe conduct in **Table 2** a performance comparison between GRPO and off-VPG.\nWe use the GRPO variant from (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Shao et al), with KL regularization coefficient β = 0.001 and symmetric clipping at 0.2.\nFor off-VPG, in line with our previous conclusions, we use a fixed baseline of δV = -0.1.\n\nUnder these conditions, off-VPG appears to perform slightly better on average than GRPO, despite being more simple. All training runs are stable. We also observe that off-VPG achieves faster score improvements than GRPO, especially when training Qwen3B.\n\nNote that we do not yet claim that off-VPG is universally superior to GRPO--a fully rigorous comparison is beyond the scope of this work. Rather, we argue that its performance is competitive and reliable across diverse setups (three models and two datasets), and we plan on pursuing a more systematic investigation of this comparison in future work.\n\nIf the article is accepted, we plan on including two additional graphs in the main text (a variant of Figure 3 with Qwen3B, and a comparison between off-VPG and GRPO with Llama 8B), and to put the other results in the Appendix, as they do not change the article's overall narrative (though they make our conclusions more robust).\n\nNote that we are very willing to run additional experiments in the remaining discussion period, should the reviewers deem it necessary.\n\n--- End of general response ---\n\n* First, it omits some  [...] off-policy updates, and GRPO?\n* The evaluation is also [...] in the literature, such as [1].\n\nRegarding the diversity of datasets and models, as well as comparisons to other algorithms (GRPO), we refer to Tables 1 and 2 of the general response above, where it is shown that our method is competitive and that our results are robust.\nThank you for pointing out the APA paper, which we will reference in our revised manuscript. This paper shares goals with ours. However, the approaches differ significantly in terms of methodology, theory, and experiments.\n\n* Finally, some of the theorems in the appendix seem underexplained.\n\n*Do you have particular examples in mind? We are happy to give you additional explanations, and potentially add them to the appendix.*\n\n* The original motivation [...] efficiency in any way.\n\n*In this context, efficiency should not be understood purely in terms of \"final score at the end of training\". Some of the practical advantages of off-policy RL, such as the capacity to handle asynchronicity between the worker Gpus (in charge of generating trajectories) and trainer Gpus (in charge of updating the weights) or the greater simplicity in implementation (no constant communication between worker Gpus and trainer Gpus), are already known to the community. Others, such as the fact that it allows several passes over the same data points (resulting in more data efficiency), have not yet been analyzed in depth in the context of LLMs, but we plan on rigorously studying these in future work.\n(For a pure score comparison, consider also Tables 1 and 2 from our general response and Table 3 in our response to reviewer oGYm: off-policy and on-policy final accuracies seem comparable, but these are preliminary experiments)*\n\n* For equation (1), how are you avoiding the importance sampling issue?\n\n*We do not face this issue, because the objective function that we optimize is NOT the expected reward $\\mathrm{E}_{y \\sim \\pi}[r(y)]$ of the trained policy $\\pi$ (though it is not far from it, as the actor policy $\\mu$ is kept close to $\\pi$ in the setups that we consider).\nThis is what makes theorems 4.2 and 4.3 interesting, as they describe the limit policy of a gradient descent on this unusual off-policy objective function.*\n\n* How do you choose the N to [...] respect to N?\n\n*The larger the number $N$ of steps between two policy updates, the closer we are to the limit distribution $\\pi_{\\mu,V}^\\star$ (using the notations of Thm 4.2), hence to the situation that our theory describes. In our bandits experiments, we have observed that the training curves in the policy improvement setting remain similarly shaped when we let $N$ vary, showing the robustness of our conclusions. In real world applications, the choice of $N$ is often dictated by practical concerns, such as the ratio between the number of trainer Gpus and worker Gpus available.*\n\n* Are the bounds in the bandit RL setting or the multi-step RL setting?\n\n*Theorem 4.2 applies in the case where the actor policy $\\mu$ is fixed, and a gradient descent is performed on the trained policy $\\pi$'s logits. Theorem 4.3 describes the policy improvement case, where the policy $\\pi$ is trained on a given actor policy $\\mu$, then the actor policy is updated $\\mu \\leftarrow \\pi$ and the process is repeated. '*\n\n* Could you provide more motivation [...] is necessary.\n* How is delta V trained in practice [...].\n\n*In the bandits setting, $V$ is not trained: it is a fixed hyperparameter. The behaviour of the algorithm depends on whether $V$ is larger or smaller than $V^\\mu$.\nIn the LLM setting, as you rightly observe, $V^\\mu = V^{\\mu(\\cdot |x)}$ depends on the prompt $x$. We substract a correction $\\hat V + \\delta V$ to each reward $r(x,y)$. $\\hat V$ is an estimator of $V^{\\mu(\\cdot |x)}$, computed as in GRPO (for example). $\\delta V$, just like $V$ in the bandit setting, is a fixed hyperparameter: it is the degree of freedom of our algorithm. \nThe reward corrected by $\\hat V$, i.e. $r(x,y) - \\hat V$, has expected value $0$ w.r.t. to $\\mu(\\cdot|x)$ : $\\mathrm{E}_{y \\sim \\mu(\\cdot |x)}[r(x,y) - \\hat V] = 0$ (here the notation hides the group sampling). Hence having $\\delta V <0$ is the same as having $V<V^\\mu$ in the bandits setting, and having $\\delta V \\geq  0$ is the same as having $V\\geq V^\\mu$ in the bandits setting (see also our answer to a similar question from Reviewer sBrs).*\n\n* For most of the bounds in the appendix, [...] to be bounded, right?\n\nWe assume that the set of $y$ is finite and that $\\mu$ has full support, so the ratio is necessarily bounded.\nThe ratio $\\pi(y) / \\mu(y)$ does not appear explicitly in our derivations, but the interplay between $\\pi^\\star$ and $\\mu$ can be seen implicitly.\nFor example, if $\\pi^*$ differs significantly from $\\mu$, and the subtracted baseline $V$ is not conservative enough, good arms may be eliminated too early. This can be seen, e.g., in Theorem 4.2: when $V < V^\\mu$, we have\n$$\n    \\operatorname{supp} \\pi^*_{\\mu, V} = \\{y \\mid \\mu(y)(r(y) - V) - \\tau_{\\mu, V} > 0\\}.\n$$"}, {"review_id": "z9yjnFvXcs", "reviewer": "Reviewer_x4kd", "summary": "This paper analyze off policy policy gradient (in the bandit setting) and shows how different baselines can affect the learned policy. They show when the baseline is lower than the behavior value, the learned policy improves; the behavior value is a critical point, using a baseline beyond which can lead to collapsing in policy support. The algorithm is tested in a synthesized bandit problem and training a 8B Llama. The experimental results support the theory's insight.", "questions": "1. In the analyses, are the rewards implicitly assumed to be non-negative?\n\n2. In Theorem 4.3, does the set Y^inf exist always? since the policy can be stochastic.\n\n3. I do not understand the statement of point 3 in Theorem 4.3 from the current writing, and reading the proof doesn't help. Can you expand more on what you wish to convey?\n\n4. Do you have a bound on how large V^inf is? Maybe this is related to the misunderstanding above.\n\n5. How does the convergence rate and V^inf change with V in Theorem 4.3? I", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "S1. To my knowledge, despite many off-policy PG papers, this paper is the first to analyze directly off-policy without importance weight or state occupancy correction. \n\nS2 The theoretical analyses reveal an interesting landscape of off-policy PG's dynamics and is quite complete. It is also good to see the experimental results corroborate with the theoretical analyses. \n\nS3. The paper is overall well-written and well motivated. The analyses here are timely given that PG algorithms are gaining interests again in the LLM era. .\n\nW1. The current analyses assumes a bandit setting, which is different form the contextual setting used in practice. However, I think this limitation is minor given the contribution of the paper. \n\nW2. Some minor writing clarity can be improved.", "was_revised": false, "final_justification": "", "rebuttal": "We are very thankful to the reviewers for their time and for their insightful comments.\n\nWhile the reviewers seem to agree that the papers' contributions are impactful, two shared concerns appear:\n* a lack of diversity in the experimental setup (not enough models, not enough datasets), and\n* a lack of comparison to popular RL algorithms.\n\nWe want to address these shortcomings in this general response (which we include in all individual responses), before answering each reviewer's questions separately.\n\nOur initial intent was for the article to be equally balanced between a theoretical section, a bandits section, and a section on real-world applications, which is why we chose to keep the experimental section concise. However, the reviewers' comments have helped us realize that the paper would be strengthened by the inclusion of additional experiments, and we thank them for that.\n\nWe have run a series of new experiments, which we hope will fully answer their concerns. In particular, we add two additional models (Qwen 2.5 3B and Llama 3.2 3B), a much larger dataset (NuminaMath) and a comparison to GRPO. The resulting tables can be found below (sadly, a recent change in the conference's rules does not allow us to share graphs anymore):\n\n**Table 1: Training Dynamics of the off-VPG Algorithm Across Different Baselines.**\nThis extends our Figure 3 with additional models and datasets (Same hyperparameters. Reported values are the maximum pass@1 accuracy (in %) on the test set observed during training (2000 steps, with evaluation every 200 steps). Some runs crash during training, as in Figure 3; these are marked in bold.)\n\n|Dataset|Model|δV=-0.5|δV=-0.3|δV=-0.2|δV=-0.1|δV=0|δV=0.1|δV=0.2|δV=0.3|\n|-------|-----|--------|--------|--------|--------|-----|-----|-----|-----|\n|MATH|Llama8B|48.7|49.1|49.3|49.3|**48.9**|**47.5**|**46.1**|**43.5**|\n||Llama3B|45.9|46.7|46.4|46.4|**45.4**|**44.5**|**43.9**|**45.0**|\n||Qwen3B|62.2|63.5|65.1|65.6|64.9|**37.8**|**14.7**|**13.0**|\n|NuminaMath|Llama8B|20.5|21.5|22.2|23.6|**23.1**|**17.8**|**18.0**|**17.2**|\n||Llama3B|20.5|21.8|21.8|21.7|22.7|**20.7**|**19.8**|**19.6**|\n||Qwen3B|20.3|37.6|40.6|42.0|40.8|**23.7**|**21.9**|**19.7**|\n\nIn our article, Figure 3 provides empirical evidence of how the choice of baseline δV affects the performance of our off-VPG algorithm. \nWe extend the same experiment to two additional models (Llama3B and Qwen3B) and an additional dataset (a 144k-token subset of NuminaMath)\n\nWe consistently observe the same two phenomena highlighted in Figure 3:\n\n- When the baseline is large (δV ≳ 0), training becomes unstable and often crashes (bold entries in **Table 1**).\n- As the baseline increases, the training becomes faster—reflected in the average left-to-right progression of higher scores within each row (for runs that did not crash).\n\nThe general appearance of the training and test accuracies are also coherent with our earlier experiments. These results reinforce the robustness of our recommendation: to use a slightly negative baseline, such as δV = -0.1.\n\n**Table 2: Comparison of off-VPG and GRPO, off-policy** \n(10000 steps, with evaluation every 200 steps)\n\n|Dataset (Train & Test)|Model|off-VPG (off-policy)|GRPO (off-policy)|\n|------------------|------|--------------------------|----------------|\n|MATH|Llama8B|52.0|50.9|\n||Llama3B|49.0|49.1|\n||Qwen3B|64.8|61.7|\n|NuminaMath|Llama8B|26.1|24.8|\n||Llama3B|25.5|25.1|\n||Qwen3B|44.3|40.8|\n\nWe conduct in **Table 2** a performance comparison between GRPO and off-VPG.\nWe use the GRPO variant from (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Shao et al), with KL regularization coefficient β = 0.001 and symmetric clipping at 0.2.\nFor off-VPG, in line with our previous conclusions, we use a fixed baseline of δV = -0.1.\n\nUnder these conditions, off-VPG appears to perform slightly better on average than GRPO, despite being more simple. All training runs are stable. We also observe that off-VPG achieves faster score improvements than GRPO, especially when training Qwen3B.\n\nNote that we do not yet claim that off-VPG is universally superior to GRPO--a fully rigorous comparison is beyond the scope of this work. Rather, we argue that its performance is competitive and reliable across diverse setups (three models and two datasets), and we plan on pursuing a more systematic investigation of this comparison in future work.\n\nIf the article is accepted, we plan on including two additional graphs in the main text (a variant of Figure 3 with Qwen3B, and a comparison between off-VPG and GRPO with Llama 8B), and to put the other results in the Appendix, as they do not change the article's overall narrative (though they make our conclusions more robust).\n\nNote that we are very willing to run additional experiments in the remaining discussion period, should the reviewers deem it necessary.\n\n--- End of general response ---\n\n* In the analyses, are the rewards implicitly assumed to be non-negative?\n\nNo, the analyses are invariant to any transformation of the rewards of the form $\\alpha \\cdot r + \\beta$ (and the same holds for the baseline $V$).\n\n* In Theorem 4.3, does the set Y^inf exist always? since the policy can be stochastic.\n\nTheorem 4.3 assumes access to the full population of samples (i.e., we know $\\mu$) and the ability to perfectly optimize our surrogate objective (i.e., we can apply ${\\cal T}$ exactly). This removes the stochasticity usually referred to as approximation and optimization error. In this setting, even if $\\pi$ is stochastic, $Y^\\inf$ always exists. We did not address approximation and optimization errors, as they can be handled with classical techniques, which would lengthen the proofs without adding novel insights.\n\n* I do not understand the statement of point 3 in Theorem 4.3 from the current writing, and reading the proof doesn't help. Can you expand more on what you wish to convey?\n\nThe phrasing is indeed a bit ambiguous; thank you for pointing it out, we will rephrase it.\nThe third point of Theorem 4.3 states that the limit reward is going to be optimal if and only if the baseline $V$ is smaller than some threshold $V_{0,\\mu}$ (which depends on the interplay between $\\mu$ and $\\pi^*$).\n\n* Do you have a bound on how large V^inf is? Maybe this is related to the misunderstanding above.\n\nWe have not sought an explicit bound for $V^\\inf$. A natural lower bound is $V^\\inf > \\max_y \\mu(y) r(y)$.\n\n* How does the convergence rate and V^inf change with V in Theorem 4.3?\n\nThis is a very good question that highlights a trade-off: if $V$ is too large, there is a risk of eliminating good arms (indeed, $V^\\inf$ is a decreasing function of $V$); conversely, if $V$ is too small, the constant $c$ becomes small (indeed, $c$ is an increasing function of $V$). In other words, the larger $V$ is, the faster the convergence, but the higher the risk of converging to a suboptimal policy."}, {"review_id": "0j39tbee9D", "reviewer": "Reviewer_ARzT", "summary": "This work discusses the design of off-policy RL for LLMs finetuning. By introducing a baseline V, the paper discusses the impact of policy convergence and explores how different baseline settings affect model performance. The conclusion drawn is that higher optimization weights should be given to positive trajectories, achieved through setting a baseline slightly lower than the expected value of the behavior policy.", "questions": "1. There are relevant discussions on the influence of bias in off-policy contexts within Off-Policy Actor-Critic[1]. It would be better if a reasonable citation were added.\n2.  Off-VPG does not account for the relationship between behavior policy and current policy when utilizing trajectories. Off-policy policy gradient algorithms often correct bias through the importance weights. Should this be considered as a baseline? For instance, naive Off-Policy PG [2].\n3. Should the experiments be conducted on larger datasets to further validate the effectiveness of the Off-VPG? Additionally, the evaluation is only performed on the MATH dataset without assessing the LLM's performance in general textual tasks or other mathematical benchmarks. Lack a more comprehensive experimental validation.\n4. Some specific implementation details for Off-VPG are missing. For example, is the behavior policy completely unupdated or updated with a delay? If it remains unupdated, could it degrade into rejection sampling fine-tuning?\n5. There is an absence of comparisons between on- and off-policy algorithms concerning performance and sample efficiency to substantiate claims such as \"Off-policy methods offer greater implementation simplicity and data efficiency than on-policy techniques.\"\n\n[1]Off-Policy Actor-Critic. Thomas Degris, Martha White, and Richard S. Sutton. ICML 2012.\n\n[2]Off-Policy PG: https://lilianweng.github.io/posts/2018-04-08-policy-gradient/\n\nTrivial:\n1. The organization and formatting of theorem sections make them difficult to read.\n2. The x-axis title in Figure 2 lacks clarity.\n3. Color differentiation in line graphs is too low. It takes effort to discern them.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nUtilizing off-policy RL to optimize LLMs holds research value and may enhance the efficiency and deployment of RL-based fine-tuning. The theoretical analysis provides substantial support for the algorithm.\n\nWeaknesses:\n1. The conclusion that the baselines in off-policy learning not only influence the variance as in the on-policy algorithms but also introduce policy bias is not much novel in the off-policy RL scope. Some similar conclusions have been discussed in previous off-policy RL algorithms. Besides, the final conclusion (optimize more on the positive trajectories) quite aligns with the intuition. These factors somewhat diminish the contribution of the theoretical analysis *slightly*.\n2. The off-VPG is evaluated merely with one model on one dataset. Thus, the experimental support for the algorithm is limited.\n3. There is no comparison or discussion regarding on-policy finetuing. Although off-policy methods may have advantages in efficiency or implementation, is there a trade-off between performance and efficiency?\n4. This off-policy algorithm design does not specifically target LLMs and lacks discussions related to existing off-policy RL algorithms.", "was_revised": true, "final_justification": "This article offers a perspective on how to allocate good and bad data in off-policy data for the strength of LLM updates, which is worth studying. However, the author does not clearly delineate the prior work on off-policy RL in the article, lacks discussion on the applicability compared with on-policy methods in LLMs, and the experiments are insufficient to adequately illustrate the claim. Therefore, I still believe this paper is not ready for NeurIPS 2025 from both the presentation and experimental aspects.", "rebuttal": "We are very thankful to the reviewers for their time and for their insightful comments.\n\nWhile the reviewers seem to agree that the papers' contributions are impactful, two shared concerns appear:\n* a lack of diversity in the experimental setup (not enough models, not enough datasets), and\n* a lack of comparison to popular RL algorithms.\n\nWe want to address these shortcomings in this general response (which we include in all individual responses), before answering each reviewer's questions separately.\n\nOur initial intent was for the article to be equally balanced between a theoretical section, a bandits section, and a section on real-world applications, which is why we chose to keep the experimental section concise. However, the reviewers' comments have helped us realize that the paper would be strengthened by the inclusion of additional experiments, and we thank them for that.\n\nWe have run a series of new experiments, which we hope will fully answer their concerns. In particular, we add two additional models (Qwen 2.5 3B and Llama 3.2 3B), a much larger dataset (NuminaMath) and a comparison to GRPO. The resulting tables can be found below (sadly, a recent change in the conference's rules does not allow us to share graphs anymore):\n\n**Table 1: Training Dynamics of the off-VPG Algorithm Across Different Baselines.**\nThis extends our Figure 3 with additional models and datasets (Same hyperparameters. Reported values are the maximum pass@1 accuracy (in %) on the test set observed during training (2000 steps, with evaluation every 200 steps). Some runs crash during training, as in Figure 3; these are marked in bold.)\n\n|Dataset|Model|δV=-0.5|δV=-0.3|δV=-0.2|δV=-0.1|δV=0|δV=0.1|δV=0.2|δV=0.3|\n|-------|-----|--------|--------|--------|--------|-----|-----|-----|-----|\n|MATH|Llama8B|48.7|49.1|49.3|49.3|**48.9**|**47.5**|**46.1**|**43.5**|\n||Llama3B|45.9|46.7|46.4|46.4|**45.4**|**44.5**|**43.9**|**45.0**|\n||Qwen3B|62.2|63.5|65.1|65.6|64.9|**37.8**|**14.7**|**13.0**|\n|NuminaMath|Llama8B|20.5|21.5|22.2|23.6|**23.1**|**17.8**|**18.0**|**17.2**|\n||Llama3B|20.5|21.8|21.8|21.7|22.7|**20.7**|**19.8**|**19.6**|\n||Qwen3B|20.3|37.6|40.6|42.0|40.8|**23.7**|**21.9**|**19.7**|\n\nIn our article, Figure 3 provides empirical evidence of how the choice of baseline δV affects the performance of our off-VPG algorithm. \nWe extend the same experiment to two additional models (Llama3B and Qwen3B) and an additional dataset (a 144k-token subset of NuminaMath)\n\nWe consistently observe the same two phenomena highlighted in Figure 3:\n\n- When the baseline is large (δV ≳ 0), training becomes unstable and often crashes (bold entries in **Table 1**).\n- As the baseline increases, the training becomes faster—reflected in the average left-to-right progression of higher scores within each row (for runs that did not crash).\n\nThe general appearance of the training and test accuracies are also coherent with our earlier experiments. These results reinforce the robustness of our recommendation: to use a slightly negative baseline, such as δV = -0.1.\n\n**Table 2: Comparison of off-VPG and GRPO, off-policy** \n(10000 steps, with evaluation every 200 steps)\n\n|Dataset (Train & Test)|Model|off-VPG (off-policy)|GRPO (off-policy)|\n|------------------|------|--------------------------|----------------|\n|MATH|Llama8B|52.0|50.9|\n||Llama3B|49.0|49.1|\n||Qwen3B|64.8|61.7|\n|NuminaMath|Llama8B|26.1|24.8|\n||Llama3B|25.5|25.1|\n||Qwen3B|44.3|40.8|\n\nWe conduct in **Table 2** a performance comparison between GRPO and off-VPG.\nWe use the GRPO variant from (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Shao et al), with KL regularization coefficient β = 0.001 and symmetric clipping at 0.2.\nFor off-VPG, in line with our previous conclusions, we use a fixed baseline of δV = -0.1.\n\nUnder these conditions, off-VPG appears to perform slightly better on average than GRPO, despite being more simple. All training runs are stable. We also observe that off-VPG achieves faster score improvements than GRPO, especially when training Qwen3B.\n\nNote that we do not yet claim that off-VPG is universally superior to GRPO--a fully rigorous comparison is beyond the scope of this work. Rather, we argue that its performance is competitive and reliable across diverse setups (three models and two datasets), and we plan on pursuing a more systematic investigation of this comparison in future work.\n\nIf the article is accepted, we plan on including two additional graphs in the main text (a variant of Figure 3 with Qwen3B, and a comparison between off-VPG and GRPO with Llama 8B), and to put the other results in the Appendix, as they do not change the article's overall narrative (though they make our conclusions more robust).\n\nNote that we are very willing to run additional experiments in the remaining discussion period, should the reviewers deem it necessary.\n\n--- End of general response ---\n\n\n* The conclusion that the baselines [...] contribution of the theoretical analysis slightly.\n\n*It is true that the observation that baselines introduce biases is not novel; our main contribution here is a theoretical analysis of the effects of such a bias, as well as empirical confirmations that such biases can have a positive impact.\nRegarding the final conclusion, it is in fact very common to hear that negative trajectories can be just as important, or even more important than positive ones (see e.g. [1]). A common missconception is that penalizing negative samples has the effect of encouraging diversity and exploration, because the probability mass diverted from the penalized action would be somewhat uniformly distributed over other actions. This is in fact not the case, as shown by our results: when penalizing a negative action, most of the probability mass goes to the modes of the distribution $\\pi$, as visible in the proof of Thm 4.2, which leads to the lack of diversity described in Thm 4.2 (\"the rich get richer\").\n[1] The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning, Zhu et al.*\n\n* The off-VPG is evaluated merely with one model on one dataset. Thus, the experimental support for the algorithm is limited.\n* and\n* Should the experiments be [...] Lack a more comprehensive experimental validation.\n\n*We agree; see the additional experiments in our general response.*\n\n* This off-policy algorithm design does not specifically target LLMs and lacks discussions related to existing off-policy RL algorithms.\n\n*We see the simplicity of our algorithm (combined with its good performance), which is not specific to LLMs, as an advantage. Are there any particular works that you think would be good additions to our related works section (besides [1])?*\n\n* There are relevant discussions [...].\n\n*Thank you very much, we will be sure to add this work to our related works section.*\n\n* Off-VPG does not account for the relationship between behavior policy and current policy [...] For instance, naive Off-Policy PG [2].\n\n*It is common in off-policy RL to correct bias with an importance ratio as follows $\\mathrm{E}_{y \\sim \\mu}[\\frac{\\pi(y)}{\\mu(y)}A(y)]$, where $A(y)$ is some advantage term (e.g. the reward itself). In the context of LLMs, the action $y$ is a long sequence of tokens $t_1,\\ldots, t_k$, and the importance ratio $\\frac{\\pi(y)}{\\mu(y)} = \\frac{\\pi(t_1) \\pi(t_2|t_1)\\ldots}{\\mu(t_1) \\mu(t_2|t_1)\\ldots}$ is often almost $0$ or extremely large when $\\pi \\neq \\mu$, which makes it impractical. In practice, the ratio is usually clipped on a per-token basis, e.g. in PPO or GRPO. As GRPO is considered one of the sota losses, we compare it to our algorithm in our additional experiments (see the general response).*\n\n\n* Some specific implementation details for Off-VPG are missing. For example, [...] into rejection sampling fine-tuning?\n\n*In Definition 4.1, we define the algorithm that we call off-VPG, in which the behaviour policy remains fixed. The behaviour of this algorithm is described in Thm 4.2, and illustrated in the bandits setting in Figure 1.\nIn the paragraph \"Policy improvement scheme with off-VPG\", we describe the derived policy improvement scheme in which the behaviour policy is updated with the (current) trained policy $\\pi$ with a delay (i.e. every N gradient steps on $\\pi$, we update $\\mu \\leftarrow \\pi$), and which we also call \"delayed updates setting\". Thm 4.3 applies to this case, and Figures 2, 3 and 4 illustrate it in the bandits and the LLMs setting respectively.*\n\n* There is an absence of comparisons [...] than on-policy techniques.\"\nand\n* There is no comparison [...] between performance and efficiency?\n\n*Our phrasing might have been overly enthusiastic. The goal of this article was not to prove that off-policy RL is better than on-policy RL (e.g. in terms of performance/efficiency trade-off). Our point of view was rather: practitioners often want or have to train off-policy (to benefit from some of the advantages of off-policiness, or because they are constrained by their setup), which makes off-policy RL worth studying. We will rephrase this sentence in a more neutral way.\nThough a rigorous study of the trade-offs between on-policy and off-policy RL is beyond the scope of this article and left to future work, we note that our preliminary experiments suggest that off-policy training (with off-VPG or with GRPO) with an update frequency of 250 gradient steps for the actor policy leads to final scores that are similar to those obtained with on-policy training (e.g. compare Tables 1 and 2 from our general response with Table 3 in our response to reviewer oGYm)*"}, {"review_id": "SyPlTSyAbM", "reviewer": "Reviewer_sBrs", "summary": "The paper analyses a **simple off-policy REINFORCE algorithm (off-VPG)** whose advantage is $A=r-V$, with a **user-tunable baseline V**.\n\n- **Theory** For tabular soft-max policies the authors prove exponential convergence of off-VPG to a limit distribution $\\pi^{*}_{\\mu,V}$ (Theorem 4.2) and show a **phase-transition** at the critical value $V=V^{\\mu}$ (the behavior-policy mean reward) whereby the policy’s support suddenly collapses to a singleton . Repeated application forms a policy-improvement scheme with guaranteed reward increase when $V<V^{\\mu}$ (Theorem 4.3) .\n- **Intuition** Lower $V$ accentuates high-reward (“success”) trajectories, higher $V$ emphasises pushing down low-reward (“failure”) ones; unlike the on-policy case, this baseline **changes the expected gradient direction off-policy** .\n- **Experiments** (i) 100-arm bandit shows the predicted phase transition in reward, support, and entropy when $V$ crosses $V^{\\mu}$ . (ii) Fine-tuning **Llama-3 8B** on the MATH dataset confirms that setting $\\delta V \\approx -0.1$ (i.e. slightly below the prompt-wise mean reward) yields stable learning, whereas $\\delta V\\ge 0$ causes catastrophic collapse .\n\nThe work offers a **computationally cheap knob**—just shift the baseline—to trade off stability versus exploration when using off-policy trajectories for LLM RLHF.", "questions": "1. Lines 596–599: The Lyapunov function is called “$\\Phi_t$ is bounded **above** on the simplex”. Strictly speaking $F$ is concave when $b>0$; here “above” could be re-worded as “has a finite supremum”.\n2. Lines 628–631: The bound $\\|\\nabla^2F\\|\\le2b$ → step-size $<1/b$. Readers may ask why the factor 2 disappears; worth stating that *any* constant $< 2/b$ suffices, but $1/b$ is chosen for margin.\n3. In the third statement of Theorem 4.3, does the author mean that by setting the value of $V$ within a certain range, we can ensure that the policy value converges to the optimal value? If so, the use of the phrase “if and only if” is quite confusing.", "limitations": "1. Proofs rely on a finite-arm bandit with a tabular softmax policy; behavior with function approximation and delayed updates remains open.\n2. The monotonic-improvement claim hinges on picking a baseline $V$ that is less than the unknown expected return $V^{\\mu}$, yet in real-world RLHF that return is buried under reward noise, so telling practitioners to “just choose a slightly smaller $V$” simply dumps the toughest task back onto them.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. **Rigorous characterization** of off-VPG dynamics; explicit closed-form of $\\pi^{*}_{\\mu,V}$ and monotone support shrinkage clarify how baseline controls bias/variance trade-off .\n2. **Actionable guideline**: choose $V$ slightly below the behavior—policy mean to avoid support collapse—validated on both bandits and an 8B LLM .\n3. **Engineering simplicity**: no importance sampling, no critic, minimal code changes, yet competitive stability.\n4. Clear discussion of why on-policy intuition fails off-policy and how this links to practical RLHF sampling delays .\n\n**Weaknesses**\n\n1. **Limited model and task diversity**: only one LLM architecture (Llama-3 8B) and one dataset (MATH).\n2. **Missing baselines**: lacks direct comparison to IS-PPO, GRPO, or regularized PG, so relative performance ceiling is unclear.\n3. **Statistical under-reporting**: LLM results use three seeds without CIs; bandit plots omit error bars.\n4. **Code not yet released**, reducing reproducibility.", "was_revised": true, "final_justification": "The authors have addressed my concerns during the rebuttal phase. I maintain my positive evaluation.", "rebuttal": "We are very thankful to the reviewers for their time and for their insightful comments.\n\nWhile the reviewers seem to agree that the papers' contributions are impactful, two shared concerns appear:\n* a lack of diversity in the experimental setup (not enough models, not enough datasets), and\n* a lack of comparison to popular RL algorithms.\n\nWe want to address these shortcomings in this general response (which we include in all individual responses), before answering each reviewer's questions separately.\n\nOur initial intent was for the article to be equally balanced between a theoretical section, a bandits section, and a section on real-world applications, which is why we chose to keep the experimental section concise. However, the reviewers' comments have helped us realize that the paper would be strengthened by the inclusion of additional experiments, and we thank them for that.\n\nWe have run a series of new experiments, which we hope will fully answer their concerns. In particular, we add two additional models (Qwen 2.5 3B and Llama 3.2 3B), a much larger dataset (NuminaMath) and a comparison to GRPO. The resulting tables can be found below (sadly, a recent change in the conference's rules does not allow us to share graphs anymore):\n\n**Table 1: Training Dynamics of the off-VPG Algorithm Across Different Baselines.**\nThis extends our Figure 3 with additional models and datasets (Same hyperparameters. Reported values are the maximum pass@1 accuracy (in %) on the test set observed during training (2000 steps, with evaluation every 200 steps). Some runs crash during training, as in Figure 3; these are marked in bold.)\n\n|Dataset|Model|δV=-0.5|δV=-0.3|δV=-0.2|δV=-0.1|δV=0|δV=0.1|δV=0.2|δV=0.3|\n|-------|-----|--------|--------|--------|--------|-----|-----|-----|-----|\n|MATH|Llama8B|48.7|49.1|49.3|49.3|**48.9**|**47.5**|**46.1**|**43.5**|\n||Llama3B|45.9|46.7|46.4|46.4|**45.4**|**44.5**|**43.9**|**45.0**|\n||Qwen3B|62.2|63.5|65.1|65.6|64.9|**37.8**|**14.7**|**13.0**|\n|NuminaMath|Llama8B|20.5|21.5|22.2|23.6|**23.1**|**17.8**|**18.0**|**17.2**|\n||Llama3B|20.5|21.8|21.8|21.7|22.7|**20.7**|**19.8**|**19.6**|\n||Qwen3B|20.3|37.6|40.6|42.0|40.8|**23.7**|**21.9**|**19.7**|\n\nIn our article, Figure 3 provides empirical evidence of how the choice of baseline δV affects the performance of our off-VPG algorithm. \nWe extend the same experiment to two additional models (Llama3B and Qwen3B) and an additional dataset (a 144k-token subset of NuminaMath)\n\nWe consistently observe the same two phenomena highlighted in Figure 3:\n\n- When the baseline is large (δV ≳ 0), training becomes unstable and often crashes (bold entries in **Table 1**).\n- As the baseline increases, the training becomes faster—reflected in the average left-to-right progression of higher scores within each row (for runs that did not crash).\n\nThe general appearance of the training and test accuracies are also coherent with our earlier experiments. These results reinforce the robustness of our recommendation: to use a slightly negative baseline, such as δV = -0.1.\n\n**Table 2: Comparison of off-VPG and GRPO, off-policy** \n(10000 steps, with evaluation every 200 steps)\n\n|Dataset (Train & Test)|Model|off-VPG (off-policy)|GRPO (off-policy)|\n|------------------|------|--------------------------|----------------|\n|MATH|Llama8B|52.0|50.9|\n||Llama3B|49.0|49.1|\n||Qwen3B|64.8|61.7|\n|NuminaMath|Llama8B|26.1|24.8|\n||Llama3B|25.5|25.1|\n||Qwen3B|44.3|40.8|\n\nWe conduct in **Table 2** a performance comparison between GRPO and off-VPG.\nWe use the GRPO variant from (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Shao et al), with KL regularization coefficient β = 0.001 and symmetric clipping at 0.2.\nFor off-VPG, in line with our previous conclusions, we use a fixed baseline of δV = -0.1.\n\nUnder these conditions, off-VPG appears to perform slightly better on average than GRPO, despite being more simple. All training runs are stable. We also observe that off-VPG achieves faster score improvements than GRPO, especially when training Qwen3B.\n\nNote that we do not yet claim that off-VPG is universally superior to GRPO--a fully rigorous comparison is beyond the scope of this work. Rather, we argue that its performance is competitive and reliable across diverse setups (three models and two datasets), and we plan on pursuing a more systematic investigation of this comparison in future work.\n\nIf the article is accepted, we plan on including two additional graphs in the main text (a variant of Figure 3 with Qwen3B, and a comparison between off-VPG and GRPO with Llama 8B), and to put the other results in the Appendix, as they do not change the article's overall narrative (though they make our conclusions more robust).\n\nNote that we are very willing to run additional experiments in the remaining discussion period, should the reviewers deem it necessary.\n\n--- End of general response ---\n\n* Limited model and task diversity: only one LLM architecture (Llama-3 8B) and one dataset (MATH).\nand\n* Missing baselines: lacks direct comparison to IS-PPO, GRPO, or regularized PG, so relative performance ceiling is unclear.\n\n*We have ran additional experiments with 2 other models, one additional dataset, and a baseline (GRPO) (see the general response).*\n\n* Statistical under-reporting: LLM results use three seeds without CIs; bandit plots omit error bars.\n\n*Figure 3 was only ran with 3 seeds because training $8\\cdot 3$ 8B models is quite expensive. Do you think that showing intervals representing the variance of these trajectories would improve the figure?\nIn Figure 4, we show all trajectories for 7 different seeds to prove the robustness of the phenomenon studied to randomness.\nIn the bandit setting, we study the expected version of the algorithm: as such, the gradient descent is non-stochastic. The only randomness comes from the drawing of the rewards associated to each arms, i.e. when definining the bandit task. Combining results corresponding to different tasks (e.g. with error bars) yields a graph that is very hard to interpret. We have observed that the general appearance of the curves is very stable w.r.t. the random seed; should we add variants of these curves for other random seeds in the Appendix?*\n\n* Code not yet released, reducing reproducibility.\n\n*Our code is shared with another team within our lab; we should be able to release it a month (perhaps even in time for the camera-ready version).*\n\n* Lines 596–599: The Lyapunov function is called “$\\Phi_t$ is bounded above on the simplex”. Strictly speaking $\\Phi_t$ is concave when $b>0$ ; here “above” could be re-worded as “has a finite supremum”.\n\n\nThank you for this wording suggestion.\n\n * Lines 628–631: The bound $|\\nabla^2 F|\\leq 2b$ $\\rightarrow$ step-size $<1/b$. Readers may ask why the factor 2 disappears; worth stating that any constant $<2b$ suffices, but $1/b$ is chosen for margin.\n\nThank you for pointing out this potential confusion. The factor of 2 disappears because we are using the bound on the step size, $< 2 / \\| \\nabla^2 F \\|$. We are not aware of a tighter result.\n\n\n* In the third statement of Theorem 4.3, does the author mean that by setting the value of $V$ within a certain range, we can ensure that the policy value converges to the optimal value? If so, the use of the phrase “if and only if” is quite confusing.\n\n*As you guessed, the statement should be understood as \"the limit policy is optimal if and only if $V<V_{0,\\mu}$\". We agree that the phrasing is awkward, and will change it.*\n \n \n* Proofs rely on a finite-arm bandit with a tabular softmax policy; behavior with function approximation and delayed updates remains open.\n*The behaviour with respect to delayed updates is described in the case of finite-arm bandits by Thm 4.3. We agree that going from a tabular setting to the case of LLMs is a non-trivial change, which our theory does not fully capture (though our experiments show that the intuition from bandits still applies in the case of LLMs). In our defense, there is no agreed upon mathematical framework that convincingly captures the regularity of transformers with respect to the prompts $x$ (e.g. the fact that similar prompts are treated similarly), and such a framework would be needed to reach theoretical conclusions regarding LLMs.*\n\n* The monotonic-improvement claim hinges on picking a baseline $V$ that is less than the unknown expected return $V^\\mu$, yet in real-world RLHF that return is buried under reward noise, so telling practitioners to “just choose a slightly smaller $V$” simply dumps the toughest task back onto them.\n\n*In the case of LLMs, which is our main motivation, this problem is partially averted thanks to the additive renormalization by the prompt-dependent average reward $V^{\\mu(\\cdot |x)}$, which we estimate with the group-average $\\hat{V} = \\frac{1}{G}\\sum_{i=1}^G r(y_i,x)$:\n$\\mathbb{E}[\\frac{1}{G} \\sum_{i=1}^{G} (r(y_i,x) - (\\hat V + \\delta V)) \\log(\\pi(y_{i}|x))]$, where the expectation is w.r.t. ${x \\sim D, \\{y_i\\}_{i=1}^{G} \\sim \\mu(.|x)}$.\nAs an estimator of $V^{\\mu(\\cdot |x)}$ is substracted to the rewards for prompt $x$, the expected return after the renormalization is $0$, which means that the critical value now becomes $\\delta V = 0$ uniformly across the prompts $x$s.\nOur experiments (including the new ones) show that $\\delta V =-0.1$ seems like a safe bet, though we do not claim that it is optimal.*"}, {"review_id": "hhp2HfMKWx", "reviewer": "Reviewer_oGYm", "summary": "The paper 'Reinforcement Learning From Others' Successes, Not Their Failures' propose to study the off-policy RL surrogate objective and discuss the implication of different choice of baseline value function $V$. The paper develops a clean theoretical analysis of the off-policy vanilla policy gradient (off-VPG) metho. Also provide a simple off-policy algorithm for LLM fine-tuning with almost no engineering overhead. Experimental validation verifies the finding that a small baseline $V$ benefits the training.", "questions": "1. First, I'm curious why the authors chose to study the surrogate loss function (1). As the authors claimed \"As it does not involve any off-policy correction, such as importance sampling, we do not expect this algorithm to converge to an optimal policy in general.\" If the purpose is to study the off-policy algorithm as close to what it is in practice, I'm wondering if the analysis will be altered if a KL regularized or clipped objective is used (as in PPO objective).\n2. It seems not very clear to me what Theorem 4.2 implies. In particular, when $V<V^{\\mu}$, the optimal policy seems to be a reweighting of the sample policy $\\mu$ by reward minus value $r(y)-V$, but for $V\\geq V^{\\mu}$ I feel it very hard to interpret the results. \n3. What if we do adam or other adaptive algorithm in the experiments? Would the effect of different $V$ be mitigated?\n4. The \"Intuition\" paragraph is indeed intuitive, however for LLM finetuning, we often have a continuous reward value than a binary reward. Could the authors comment on the implication of the theory for this kind of rewards?\n5. Does the same collapse phenomenon observed in on-policy algorithms? I'm also curious on how the proposed method performs comparing to on-policy algorithms?", "limitations": "The limitations are reflected in the questions section.\n\nI don't think there are any potential negative societal impact for this work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The paper has clear motivation (off-policy RL) and theoretical analysis. The experiments (both bandit and LLM) are clear and verify the theoretical findings with clear evidence.\n\nWeakness (please respond to the Questions section directly): Unclear why only analyzing the surrogate loss; The theoretical analysis requires more intepretation as it's not intuitive for now; The experiments might not be adequate and requires further comparison with on-policy algorithms.", "was_revised": false, "final_justification": "", "rebuttal": "We are very thankful to the reviewers for their time and for their insightful comments.\n\nWhile the reviewers seem to agree that the papers' contributions are impactful, two shared concerns appear:\n* a lack of diversity in the experimental setup (not enough models, not enough datasets), and\n* a lack of comparison to popular RL algorithms.\n\nWe want to address these shortcomings in this general response (which we include in all individual responses), before answering each reviewer's questions separately.\n\nOur initial intent was for the article to be equally balanced between a theoretical section, a bandits section, and a section on real-world applications, which is why we chose to keep the experimental section concise. However, the reviewers' comments have helped us realize that the paper would be strengthened by the inclusion of additional experiments, and we thank them for that.\n\nWe have run a series of new experiments, which we hope will fully answer their concerns. In particular, we add two additional models (Qwen 2.5 3B and Llama 3.2 3B), a much larger dataset (NuminaMath) and a comparison to GRPO. The resulting tables can be found below (sadly, a recent change in the conference's rules does not allow us to share graphs anymore):\n\n**Table 1: Training Dynamics of the off-VPG Algorithm Across Different Baselines.**\nThis extends our Figure 3 with additional models and datasets (Same hyperparameters. Reported values are the maximum pass@1 accuracy (in %) on the test set observed during training (2000 steps, with evaluation every 200 steps). Some runs crash during training, as in Figure 3; these are marked in bold.)\n\n|Dataset|Model|δV=-0.5|δV=-0.3|δV=-0.2|δV=-0.1|δV=0|δV=0.1|δV=0.2|δV=0.3|\n|-------|-----|--------|--------|--------|--------|-----|-----|-----|-----|\n|MATH|Llama8B|48.7|49.1|49.3|49.3|**48.9**|**47.5**|**46.1**|**43.5**|\n||Llama3B|45.9|46.7|46.4|46.4|**45.4**|**44.5**|**43.9**|**45.0**|\n||Qwen3B|62.2|63.5|65.1|65.6|64.9|**37.8**|**14.7**|**13.0**|\n|NuminaMath|Llama8B|20.5|21.5|22.2|23.6|**23.1**|**17.8**|**18.0**|**17.2**|\n||Llama3B|20.5|21.8|21.8|21.7|22.7|**20.7**|**19.8**|**19.6**|\n||Qwen3B|20.3|37.6|40.6|42.0|40.8|**23.7**|**21.9**|**19.7**|\n\nIn our article, Figure 3 provides empirical evidence of how the choice of baseline δV affects the performance of our off-VPG algorithm. \nWe extend the same experiment to two additional models (Llama3B and Qwen3B) and an additional dataset (a 144k-token subset of NuminaMath)\n\nWe consistently observe the same two phenomena highlighted in Figure 3:\n\n- When the baseline is large (δV ≳ 0), training becomes unstable and often crashes (bold entries in **Table 1**).\n- As the baseline increases, the training becomes faster—reflected in the average left-to-right progression of higher scores within each row (for runs that did not crash).\n\nThe general appearance of the training and test accuracies are also coherent with our earlier experiments. These results reinforce the robustness of our recommendation: to use a slightly negative baseline, such as δV = -0.1.\n\n**Table 2: Comparison of off-VPG and GRPO, off-policy** \n(10000 steps, with evaluation every 200 steps)\n\n|Dataset (Train & Test)|Model|off-VPG (off-policy)|GRPO (off-policy)|\n|------------------|------|--------------------------|----------------|\n|MATH|Llama8B|52.0|50.9|\n||Llama3B|49.0|49.1|\n||Qwen3B|64.8|61.7|\n|NuminaMath|Llama8B|26.1|24.8|\n||Llama3B|25.5|25.1|\n||Qwen3B|44.3|40.8|\n\nWe conduct in **Table 2** a performance comparison between GRPO and off-VPG.\nWe use the GRPO variant from (DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, Shao et al), with KL regularization coefficient β = 0.001 and symmetric clipping at 0.2.\nFor off-VPG, in line with our previous conclusions, we use a fixed baseline of δV = -0.1.\n\nUnder these conditions, off-VPG appears to perform slightly better on average than GRPO, despite being more simple. All training runs are stable. We also observe that off-VPG achieves faster score improvements than GRPO, especially when training Qwen3B.\n\nNote that we do not yet claim that off-VPG is universally superior to GRPO--a fully rigorous comparison is beyond the scope of this work. Rather, we argue that its performance is competitive and reliable across diverse setups (three models and two datasets), and we plan on pursuing a more systematic investigation of this comparison in future work.\n\nIf the article is accepted, we plan on including two additional graphs in the main text (a variant of Figure 3 with Qwen3B, and a comparison between off-VPG and GRPO with Llama 8B), and to put the other results in the Appendix, as they do not change the article's overall narrative (though they make our conclusions more robust).\n\nNote that we are very willing to run additional experiments in the remaining discussion period, should the reviewers deem it necessary.\n\n--- End of general response ---\n\n\n* First, I'm curious why the authors chose to study the surrogate loss function (1). [...] I'm wondering if the analysis will be altered if a KL regularized or clipped objective is used (as in PPO objective).\n\n*It was precisely the simplicity of the proposed loss which raised our interest: we wanted to study the potential impact of the simplest lever of action that we could think of, namely our baseline V. Adding a KL regularization or an importance ratio would certainly dramatically impact the theoretical analysis, though we expect that a KL regularization would mainly have the effect of keeping the trained policy closer to th reference policy, as in other algorithms. We leave this for future work. Note however that we have included preliminary comparisons to GRPO in our general response (Tables 2 and 3), where our loss seems to be competitive.*\n\n* It seems not very clear to me what Theorem 4.2 implies. In particular, when $V<V^\\mu$, the optimal policy seems to be a reweighting of the sample policy $\\mu$ by reward minus value $r(y)-V$, but for $V\\geq V^\\mu$ I feel it very hard to interpret the results.\n\n*In the case $V<V^\\mu$, the limit policy (which is not necessarily optimal) is more precisely the positive part of a reweighting of $\\mu$ minus a constant. The key takeaway is that the probability mass of an action $y$ w.r.t. the limit policy is a somewhat complicated increasing function of $\\mu(y)(r(y)-V)$.\nWhen $V\\geq V^\\mu$, the limit policy is generically going to be a singleton (generically w.r.t. the choice of rewards and of $\\mu$): large $V$s lead to a loss in diversity in the limit policy.*\n\n* What if we do adam or other adaptive algorithm in the experiments? Would the effect of different $V$ be mitigated?\n\n*In fact, our LLMs experiments are already conducted using AdamW, as is common when training large transformers (this is stated in Appendix B). Figure 4 shows that the effect is not mitigated.*\n \n* The \"Intuition\" paragraph is indeed intuitive, [...] for this kind of rewards?\n\n*All of our theorems hold for continuous rewards (we never assume that the rewards are discrete). We only give the example of $\\{0,1\\}$ rewards to help the readers' intuition, but the general idea is the same in the case of continuous rewards: letting $V$ be large puts comparatively more emphasis on the trajectories with the lowest rewards, and vice versa.*\n\n\n* Does the same collapse phenomenon observed in on-policy algorithms? I'm also curious on how the proposed method performs comparing to on-policy algorithms?\n\n*Regarding the impact of the baseline $\\delta V$ on the risk of collapse, we have run some very preliminary experiments in an on-policy setting and observed similar results, but the risk of collapse seems to be more dependent on the choice of model, hyperparameters, and even possibly codebases (as many efficient RL codebases are never 100% on-policy to minimize the downtime of the worker Gpus tasked with generating training trajectories). These are still preliminary experiments from which we do not draw definitive conclusions.\nWe have also run a comparison with GRPO in an on-policy setting, reported in ***Table 3*** below.*\n\n\n**Table 3: Comparison between off-VPG and GRPO, on-policy** \n(10000 steps with evaluation every 200 steps, bold values indicate that the run collapsed during training)\n\n|Model|off-VPG seed 1|seed 2|GRPO seed 1|seed 2|\n|--------------|----------------|--------|--------------|--------|\n|Qwen2.5-3B|65.4|64.2|**61.8**|**61.9**|\n|Llama3.1-8B|52.0|52.4|50.0|**50.1**|\n\nThough it it not the main focus of our work, we also conducted a comparison between the off-VPG algorithm and GRPO in an on-policy settings to assess whether off-VPG remains competitive. The results are presented in **Table 3**. The experiment uses the MATH dataset, with hyperparameters identical to those used in **Table 2**, except for the update frequency of the actor policy. In this experiment, updates occur at every gradient step, unlike previous experiments where updates occurred every 250 steps.\n\nWe run two seeds for each configuration and report the highest score achieved on the test set. All learning curves increase steadily and plateau after approximately 3000 steps. The plateau is consistently higher for off-VPG than for GRPO, across both seeds and models. We also observe a sharp performance collapse during training in 3 out of 4 GRPO runs (highlighted in bold in the table), while no such collapse occurs with off-VPG.\n\nThese results suggest that off-VPG remains a robust and effective choice in on-policy settings, with the added benefit of a simple implementation."}]

Author rebuttal:

Meta-review: The paper analyzes a simple off-policy policy gradient algorithm, showing both theoretically and empirically that emphasizing positive rewards while down-weighting failures yields policy improvement guarantees and more stable performance when fine-tuning LLMs and in bandit settings. The reviewers generally recognized the novelty of the theoretical analysis and appreciated the empirical validation, though some raised concerns about the limited diversity of experiments and missing comparisons to key baselines The additional experiments and comparisons added during rebuttal helped address some of these concerns, but the paper would benefit from a more candid discussion of its limitations. In particular, the reliance on bandit-style analyses and the narrow experimental scope leave open questions about generalization. The authors are encouraged to explicitly acknowledge these limitations and discuss how future work could broaden the empirical validation and strengthen connections to on-policy baselines.
Decision: Accept (poster)