Tapered Off-Policy REINFORCE - Stable and efficient reinforcement learning for large language models
Authors: Nicolas Le Roux
Marc G Bellemare
Jonathan Lebensold
Arnaud Bergeron
Joshua Greaves
Alexandre Fréchette
Carolyne Pelletier
Eric Thibodeau-Laufer
Sándor Tóth
Sam Work
Venue: NeurIPS 2025 poster ()
Keywords: reinforce
large language models
verifier
ppo
off-policy
rlhf
Primary area: deep_learning
TL;DR: A simple general purpose off-policy REINFORCE method which outperforms PPO, DPO and STaR on recent benchmarks.

Abstract: We propose a new algorithm for fine-tuning large language models using reinforcement learning. Tapered Off-Policy REINFORCE (TOPR) uses an asymmetric, tapered variant of importance sampling to speed up learning while maintaining stable learning dynamics, even without the use of KL regularization. TOPR can be applied in a fully offline fashion, allows the handling of positive and negative examples in a unified framework, and benefits from the implementational simplicity that is typical of Monte Carlo algorithms. We demonstrate the effectiveness of our approach with a series of experiments on the GSM8K and MATH reasoning benchmarks, finding performance gains for training both a model for solution generation and as a generative verifier. We show that properly leveraging positive and negative examples alike in the off-policy regime simultaneously increases test-time accuracy and training data efficiency, all the while avoiding the ``wasted inference'' that comes with discarding negative examples. We find that this advantage persists over multiple iterations of training and can be amplified by dataset curation techniques, enabling us to match 70B-parameter model performance with 8B language models. As a corollary to this work, we find that REINFORCE's baseline parameter plays an important and unexpected role in defining dataset composition in the presence of negative examples, and is consequently critical in driving off-policy performance.

TAPERED OFF-POLICY REINFORCE Stable and efficient reinforcement learning for LLMs
Nicolas Le Roux v.V.I Mare G. Bellemare*
Jonathan Lebensold
Arnaud Bergeron Joshua Greaves Carolyne Pelletier Eric Thibodcau-Laufer
Alex Fréchette
Sándor Toth² Sam Work? Mila RRllantt Al
Abstract
We propose new algorithm for fine-tuning large language models using reinforce- ment learning. Tapered Off-Policy REINFORCE (TOPR) uses an asymmetric, tapered variant of importance sampling to speed up learning while maintaining stable learning dynamics, even without the use of KI. regularization. TOPR can be applied in fully offline fashion, allows the handling of positive and negative examples in unified framework, and benefits from the implementational simplicity that is typical of Monte Carlo algorithms. We demonstrate the effectiiveness of our approach with series of experiments on the GSM8K and MATH reasoning bench- marks, finding performance gains for training both model for solution generation as a generative verifier, and on a learning to search task, using the model as a query expander. We show that properly leveraging positive and negative examples alike in the off-policy regime simultaneously increases test-time accuracy and training data efficiency, all the while avoiding the "wasted inference" that comes with discarding negative examples. We find that this advantage persists over multiple iterations of training and can be amplified by dataset curation techniques, enabling match 70B-parane r0B-parameter model performance with 8B language models. As a corollary this work, we find that REINFORCE's baseline parameter plays important and unexpected role in defining dataset composition in the presence of negative examples, and is consequently critical in driving aff-policy performance.
Introduction
Reinforcement learning (RL) and EM-type methods are rapidly becoming the dominant paradigm for fine-tuning LLMs on complex tasks such as chain-of-thouge reasoning. These methods can amplify base model's performance without additional human data and can optimize for synthetic rewards [53] and non-differentiable objectives
[5). While several popular methods rely solely on positive examples to fine-tune an LLM [52] 17, the "trial and error" nature of RL algorithms especially well-positioned to leverage negative examples produced by the model, which are increasingly being recognized as key to efficient learning 39 45 54 [3). In fact, there is mounting evidence that the simplest of all methods, REINFORCE
49 is highly effective approach to fine-tuning LLMs [
However, REINFORCE is essentially on- on-policy algorithm. In the presence of negative rewards, its good behaviour can only be guaranteed when the training data distribution (or reference model) matches, close to, the model's 's own distribution. This limits its ability reuse past data, and
* Equal contribution.
Equal contribution.
Corresponding author: nicolas dicolas@le-roxxnnee
8 Work done at Reliant
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Algorithm 1 TOPP (single iteration)
Pass@1
Input: Language model n, reference /
Input: Prompts -1,-----11 -m
60
....... ... ~ uu( (z.) for i = 1, 1,..,,m
50
Make dataset D {vi, v/
Pass
40
for (2, y) E D do
TOPR:
x ar
if R(x, v) < else
-
REINFORCE
DPO
STOOPGGAAD(A) R(x, v) log ir |x)
PPO
Perform gradient step w.r.t. loss
2000
4000
6.000
8000 10000
end for
Gradient steps
Figure 1: Left: Test set accuracy (Pass@1) on the GSM8K benchmark [7] over the course of off-policy fine-tuning of the Llama 8B model. As training becomes increasingly off- policy, the naive use of the REINFORCE gradient causes substantial performance degradation and PPO stops improving. DPO, which handles negative examples through preference- better but still falls well short of TOPR's performance. Pass @ refers to the usual single-reasoning accuracy. See Section 4 for experimental details. Right: TOPR pseudo-code (Section
formulation, fares
puts pressure on the experimenter to select a just-right set of hyperparameters to avoid problems. Indeed, evidence of the instability of REINFORCE-typ in off-policy training of LLMs can be found everywhere in the literature, from early work [36 to key algorithmic choices made in the training of the Kimi x1.5 [46] and DeepSeek-RR R1 [18] models. While KL regularization the objective can mitigate this instability, it results slower learning and requires additional hyperparameter tuning.
We propose Tapered Off-Policy REINFORCE (TOPR), stable algorithm even when the model differs substantially from the data distribution, while fully leveraging positive and negative examples. TOPR improves language model (or policy) by means of the asymmetric policy gradient
( [M(T)
STOPR(TT)
Σ Vogg(()))ogTT()
Σ P(T) +:R(F)<D
((g)V (I)
r:(())>>0
where is response (or trujectory) sampled from some data-generating policy μ., R(T) is the reward associated with this trajectory, and [=]] = max(mmin(a r b). a) denotes the clipping function. The lack of importance ratio for positive examples means that TOPR will increase their probability, even when that importance ratio is small. The importance ratio is kept for negative examples as it ensures that the model will stop using capacity for them once their probability small enough. Unlike other LLM algorithms in the REINFORCE family, TOPR does not require an explicit KL penalty to guarantee stable behaviour, making both simpler to implement and computationally more efficient. Compared to PPO 440) DPO 33), and the "naive" application of REINFORCE [1, TOPR continues to improve reasoning performance even once differs substantially from l (Figure left).
We characterize the stable off-policy performance of TOPR by using it to train language models to reason within the GSM8K and MATH benchmarks. We use these benchmarks highlight the various factors at play in off-policy reinforcement learning of language models, in particular the importance of positive/negative balance in the training dataset. Surprisingly, we find that REINFORCE'S baseline parameter commonly used as a variance reduction mechanism plays altogether different role of balancing the dataset in this regime, and is essential to good performance. Critically, our results indicate that choosing the best baseline requires taking more than only the mean return into account.
We conclude with series of multi-iteration experiments showing the ability of TOPR to fine-tune language models well beyond their base benchmark performance.
2 Off-policy policy optimization
We consider an autoregressive language model that, given prompt æ, assigns a probability to length-n response y according TTy) II ( vee). Given reward function R(z, y)

---

that measures the quality of the response to z and dataset of prompts ......,, Zm we wish to maximize the expected reward J(²) m Σn= [By=-((1 Imy) R(z,,3) 3) In this paper, we abstract the prompt-response relationship and view this problem through the lens of policy optimization. where т is a trajectory produced by the language model (i.e., the policy). With mild abuse of notation, we thus write (TT)
E R(T).
T~K
The original REINFORCE algorithm 49 maximizes J(T) through the process of on-policy policy optimization. In the simplest form of the algorithm, a single trajectory T is sampled according to т, and the parametrized policy т is updated according to the unbiased gradient estimate (=) = ((T) V log (T), whose expe is J = E R(T) log (T).
In practice, training is rarely truly on-policy, for example because data is generated in a parallel, asynchronous fashion [33] or in a separate "sidecar" process [35 18] It is obviously also desirable to reuse trajectories throughout training, especially when generating these trajectories incurs a substantial computational cost or because they have been generated by different process (c.g. expert trajectories). In the aff-policy policy optimization setting, we assume the existence of reference distribution μu, typically different from π, which produces training trajectories. In the fully online case, which we mainly consider in this work, trajectories are sampled once and never updated. The goal is to find the optimal policy using this fixed set of trajectories. We also consider in our experiments the setting where trajectories are only rarely updated. Our main goal in this paper is to highlight the trajectories in off-policy policy optimization and propose
pitfalls of dealing with negatively-rewan solution TOPR- that avoids these pitfalls to produce performant, stable behaviour when training language models. By way of explaining the algorithmic choices behind TOPR, we review existing solutions and how they fall short of our desiderata.
2.1 The problem with naive REINFORCE
As a warm-up, consider a binary reward function R(T) E {-1,1] and the algorithm that sam- ples trajectory from ja, then updates the policy according the REINFORCE update: (.) = V log (T). This essentially corresponds to the "naive" off-policy application of the REINFORCE update [T. In expectation, this update maximizes the objective
(p) = Σ ((()) ((T) - Σ
(2)
TET-
TET
where T and T are the set of trajectories with positive and negative rewards, respectively. The first term is maximized by making æ as close to j as possible on the positive subset T+, The second term, on the other hand, incentivizes æ being as far from μ as possible. This term unbounded above (in terms of TT) and can be made arbitrarily large by driving the probability of any single trajectory supported by μ to zero. This acts as a destructive force on the the model parameters, driving them to producing infinitely negative logits and without safeguards, eventually causes degenerate behaviour
We show in Section 3.1 that, while the issue can be mitigated by early stopping, the use of baseline parameter, or KL regularization towards μ. these modifications effectively work by fully or partially ignoring negative trajectories and thus limit the amount of learning that can be done off-policy.
2.2 Supervised fine-tuning
A simple solution to avoid the catastrophic failure of the model due to negative trajectories is to remove them from the dataset entirely. This can be interpreted as a form of reward-weighted supervised fine-tuning (SFT). The corresponding objective is
JSFT(T)
Σ ((T) log=())) TET+
where the trajectory has weight R(T) if R(T) is positive, 0 otherwise. If we write (T((T) x R()) then J (T) ) is the cross-entropy loss between PR and
"This issue doesn't 't appear in the on-polic application of REINFORCE because, by definition, a trajectory whose probability (T) is small is unlikely to arise in the dataset.
3

---

Supervised fine- fine-tuning in the usual sense [56] can be viewed as the special case where all positive rewards are equal +1 and j is fixed and independent of the language model. STaR [52], ReST [17]. and ReST-EM 43 use SFT with dataset generated by the L.M itself, i.e. is equal or close to
or by another L.M,, possibly with a filtering step to further enhance dataset quality.
Removing negative examples from the dataset yields an objective that is bounded above, making these methods stable. As they are implemented with cross-entropy loss, they can also quickly learn to mimic the distribution HR. a characteristic that we retain in TOPR. However, omitting negative examples comes at a cost: for challenging problems, there may be few positive examples, and finding them may require additional machinery such as reference- sfference-guieed guided grading [55], and wasted inference cycles. Mathematically, the lack of negative examples means that incentivized stay closer to p. limiting the amount of progress that can be achieved before having to resample from the LLM.
2.3 Truncated importance sampling
Importance sampling perhaps the most common technique to address distribution shift. From =
derive unbiased estimate of the on-policy gradient: R log (r). We call this the ofllccl REINFORCE (OPR) gradient. In theory, that equation provides a convenient algorithm for optimizing the true objective (T)): sample
J(x) B--
R(T)).
P(T)
V (T) = x(r)
u(r)
trajectory T ~ and weight its update by the importance ratio x(r) In practice, it is well-known that
p(7)'
importance sampling is plagued with excessive variance. This is problematic when optimizing over sequences, where the importance ratio is a product of many per-step ratios [38]. Gradient variance matters for positive trajectories whose probability x (TT) increases during training and negative trajectories, where a single excessive ratio can have a destructive effect on the model parameters.
One can mitigate the variance issue by truncating the importance ratios
34
12
14
The correspond-
ing sample gradient is (nns(=)
x(T) [M(T]
R(7)VlogT(?)
Truncated importance sampling (TIS) is an integral part of TOPR. There are, however, situations where following the gradient of J(T) is not desirable, justifying further enhancements. To see
this, note that when the importance ratio m(r) is close to 0, so the norm of the gradient. Should this happen for trajectories with positive rewards, the model will take a long time to increase that trajectory's 's probability. This is not specific to importance sampling and is an issue with the usual on-policy REINFORCE (22). We shall see in our experiments how TIS, while effective, is more
sensitive dataset composition and the choice of reward baseline.
2.4
PPO and other methods
PPO 40
one of the most widely used policy-based methods, optimizes the objective Jreo(T) =
x(()) p(()
R(T), [(r) l+c R(T)) for e E (0, 1). This objective implements an asymmetric
Epon min
treatment of positive and negative rewards and is essentially composed of three parts (Fig. 2 In the near on-policy setting, when only few updates are made before resampling trajectories, this can be
quite effective; GRPO [18), for example, modifies Jrco with a aatch-dependent baseline.
However, the PPO objective applies the importance ratio to the reward rather than to the gradient. As consequence, the gradient of Jero (T) becomes zero outside of the [1 1 + E] range, limiting its usefulness and potentially causing brittleness when more than a handful of updates are made before resampling trajectories. The algorithm is also not incentivized reduce the relative probability of negative trajectories below 1 - c, limiting the potential improvement from fa. Although variants such as sPPO increase this robustness [47]. their performance still drops after many updates.
We focused here on methods that work with batch size i.e that do not need to compare examples. We discuss other methods, including popular ones like DPO, in Appendix A

---

1-E 1
PPO
TOPR
15
Hot 2
Rid
Figure 2: Visualization the TOPR objective (Eq. with a 1. =1 0, = 1) contrasted with PPO, as function of the importance ratio (IS). The two losses are equal on the intervals [1, + c] and [1 1] (positive and negative rewards, respectively). However, PPO stops gradients when the ratio differs substantially from I, which prevents it from being an effective off- off-policy algorithm. TOPR also implements sharper positive-example loss for small importance ratios, accelerating the learning of these examples.
Negative examples
Bounded objective
Low
Fast
b+
b
variance
learning
SFT
0
0
No
Yes
Yes
Yes
Naive REINFORCE Off-policy REINFORCE
1
1
Yes
No
Yes
Yes
0
+00
0
+00
Yes
Yes
No
No
Truncated IS
0
1
Yes
Yes
Yes
No
TOPR
0
Yes
Yes
Yes
Yes
Table TOPR combines the advantages of supervised fine-tuning, REINFORCH, and importance sampling to support stable and efficient off-policy fine-tuning of language models.
TOPR: Tapered off-policy REINFORCE
We now introduce the TOPR algorithm. TOPR uses importance sampling to downweight negative tra- jectories that are not likely under x, while allowing positive trajectories be upweighted irrespective of ग. The framework we consider involves two sets truncation limits, a b+ and a b
T(T) [u(T] R(T) (Tg TET-
[(T) [u(T)
VJ(T) Σ ((T) TET+
Σ P(T)
llgg)))g=(T). (3)
By choosing different truncation limits, we obtain many of the methods introduced in the previous section [T. TOPR itself corresponds to a range of truncation limits that combine the desirable properties each of these methods into one learning rule.
Gracefully unlearning negative trajectories. Setting a = 0 allows the algorithm to progressively reduce the contribution of must eventually lead to model degeneracy with naive REINFORCE (Section 2.1
negative trajectories, as provided by importance sampling. Any a
Quickly learning positive trajectories. Setting + > gives benefits of supervised fine-tuning: we ensure minimum rate of learning for positive trajectories and accelerate their learning when they have a low probability under и. This allows us avoid the "quasi-local minima" issue that plagues REINFORCE in digh-dimensional action spaces.
5

---

Trading off' bias and variance. The upper truncation limits allow us to keep gradient variance under control, as expected from truncated importance sampling. This is important early in training for negative examples, when a few examples may exhibit very large importance ratio, and late in training for positive examples, where we expect the untruncated ratio be much greater than
We follow Occam's principle in defining the canonical form of TOPR as the algorithm where 0 and all other parameters are 1. This yields the expected TOPR gradient:
(T)) [((T) R(r)Vloga(T))
V TTOPA (T)
Σ ((T))(()(VlogT)) (T) E ((T) TET!
SFT update positive examples
TIS update negative examples
which combines the SFT update for positive examples, leading to acceleration, and the TIS update for negative examples, allowing for their handling without brittleness. Algorithm T sketches out an implementation of TOPR in an off off-policy, deep learning setting.
We will demonstrate in Section that this canonical parametrization is highly effective and provides robustness to the choice of data distribution and deep learning hyperparameters. Before doing so, however, we provide theoretical justification for the design choices behind TOPR.
3.1 Analysis
In Section 2.1 we argued that introducing baseline parameter cannot create stable off- off-policy learning behaviour without risking sacrificing performance. We will make this point more precisely in this section and the next. To begin, let us revisit the expected naive REINFORCE update, now introducing a baseline parameter E IR:
V (T)
E c)
TmE
The following establishes the contribution of positive and negative examples as well as the baseline to the expected loss JT))(T).
Proposition 3.1. (T), defined as (T) = (T), is the four-part loss
(T) = C+ R KL( (T) R., ( TT cKL(p (T)
where Pn is the reighee distribution
[(T)
R(F) < <0 otherwise;
(R(()
R,
RT E() TET-
= [r: R(T) < 0}.
and symmetrically utz. and C is a constant independent of n.
Proposition 3.1 shows that the baseline induces KL regularization towards (c < 0) or away (c > 0)
from the sampling distribution [see also 25
47 At Ju = ग, we recover that the baseline has no effect
on the expected on-policy gradient 49 44. In particular, when all rewards are positive (/2 0),
Eq. moves the policy æ towards a reward- wwward-weighted version of the sampling distribution μ [5
Proposition 3.1 also shows that adding a baseline to minimize the impact of negative rewards (c < 0) works by regularizing towards μ. To guarantee stable behaviour, the bascline must in general match the smallest negative reward (c.g., (T(r) = R(T) 2 0). At this point, however, the baseline effectively removes negative trajectories from the objective function - losing the information contained in these trajectories. Using importance sampling instead avoids this issue. To see this, we start by noting that Eq. Bss the gradient of
((()
Σ u(F)p ((()) a .5) R(T). (6)
JTOPR (T) = Σ M(T)p
U(()) (((T). a..)...bb )
R(T)+ +
(p(T)
TET+
TET-
where pl: b): [0, o0) R is the taper function
11 + log = b log )
if < a if b otherwise.
p(x,a,b)

---

The taper function describes the effect of the truncation on the objective optimized by TOPR. It defines a lower bound on the importance ratio, in the sense that for any S
(T()) ((TT)
(T) (((T)
a.
and is equal to this ratio on the interval (a, bj (Fig. 2). For our choice of at = b+ and a positive
reward function R(r) 2 0, TOPR optimizes a lower bound on the true objective (
J ((T) STOPR (T)
E Tmpp
1, R(TT)
The bound follows from the analysis of related algorithms by 8 25 26 and 17
The following
proposition establishes the stable off off-policy behaviour of TOPR for wider range of truncation parameters.
Proposition 3.2. For a = 0, Eq. ois is bounded above: there exists a such that
sup, Jrome(T) S B.
Furthermore, for any a > (roeR( is unbounded above unless R(T) 2 foror all T.
For positive examples, the taper function with at > O maintains substantial gradient even when (r) small, since the weight used for the gradient of the surrogate objective, (u(T), is independent of the current policy æ. This allows the model to recover from low probability x(r) assigned to good trajectory, avoiding a traditional failure of REINFORCE. Empirically, [26] observed lower variance and more efficient learning when optimizing this log-ratio lower bound. With negative rewards, however, replacing the importance ratio with the log-ratio leads to the surrogate objective being an upper bound on (x)) [25] which is different way of expressing the conclusion of Proposition 3.2
4 Results
We study the effectiveness of TOPR at training language models to perform thain-of-thouuht (CoT) reasoning, well preliminary results tne-ttnnng LLM to perform query expansion with the end goal of improving search results. For the most part we focus on the single-iteration, fully offline regime aiming to characterize the relative stability and effectiveness of TOPR for training language models compared to prior alternatives. Our results are naturally complementary to the full gamut of methods that improve language models iteratively.
Our main results are on mathematical reasoning datasets that require step-by-step solutions and are widely used to evaluate the reasoning capabilities of LLMs: GSM8K 77 and MATH (20). As our core model, we use the Llama family of instruction-tune language models [10), using the 8B model unless otherwise specified. More details about the datasets and experimental setup may be found in Appendix
4.1 Fine-tuning chain-of-the of-thuhh reasoning
Our first set of experiments aims to answer the question: Is more careful handling of importance ratios and rewards beneficial in off-policy policy optimization? We begin with a comparison of PPO, DPO, naive REINFORCE, and TOPR. For DPO, pairs of candidate solutions are formed from these so as to obtain up to 16 contrastive pairs. For PPO, we use e = 0.2.
Fig. (left) experimentally demonstrates the limitations of existing methods. Naive REINFORCE performs well at first but, in the absence of KL term, collapses as moves away from μ.. As expected, PPO makes little progress on the objective as most data points quickly fall outside of its [1 - c, + c] range. DPO performs well off-policy, confirming the observations of [35,, especially when measured in terms of Pass @ accuracy. TOPR outperforms all these methods. We explore in more detail in Appendix D. some reasons for TOPR's 's success.
TOPR minimizes reasoning failures. To understand the reasons behind TOPR's success, we measured the proportion of generated solutions that were correct, incorrect, or invalid (the string "The answer is" is not present) during the course of training. Fig. gives strong evidence as to the
7

---

75
Pass@1 score on GSMBK
70
Figure 3: Test set accuracy GSM8K across training when using all examples (TOPR) or pos- itive examples alone. Not shown here, TOPR also yields higher inference efficiency at test time: greater Maj@y @N performance for all N.
65
60
55
TOPR ves only
50
2000
6000
10000
Gradient steps
root cause of REINFORCE's poor performance, whose generations are overwhelmingly degenerate by the end of training. By contrast, TOPR proves effective at teaching the model to avoid incorrect formatting - yielding the desirable property that one can solely rely on RL for solution generation. rather than using additional tools to correctly format them.
Using negative examples improve performance. To understand the impact of negative examples on training, we next formed a "positives only" dataset by removing all negative examples from our base dataset. This procedure mimics some of the design choices of recent work such as STaR that apply SFT as the inner loop of an RL-like procedure. Using this dataset results in stable learning but substantially lower performance than that of TOPR (Fig. 33. This translates into greater self- consistency [88 efficiency: more solutions must be generated at test time to reach the same level of performance (top right). more detailed analysis this experiment can be found in Appendix [D.2
Striking the right balance of positive and negative examples. We now refine the previous analysis by varying the effective proportion of positive examples in the dataset. We start with a dataset of 50,000 examples, 10% of which are positive. We then vary the baseline for each model to reach an effective proportion of positive examples from 1% all the way to 100%. Details on how changing the baseline can affect the effective proportion of positive examples may be found in Appendix B
Fig. shows the test performance on GSM8K using either TOPR or TIS when the effective proportion of positive examples varies. The optimal effective proportion to be around 10-20% for both datasets (GSM8K and MATH) and both algorithms (TIS and TOPR). We posit that good baseline one that achieves such a proportion. We also performed additional experiments in Appendix D.3 showing that this effective proportion is independent of the actual proportion of examples in the training set.
Our result also gives further evidence that the optimal baseline not always the expected return in practical settings, contrary common belief
Acceleration improves robustness to dataset composition. Given the important performance gains from incorporating negative examples to training and the relevance of dataset composition (Section B is natural to ask whether TOPR's ' positive-example acceleration (a* 1) helps when positive examples are outnumbered in the dataset, for example because the problem at hand is very hard. When that proportion is low, the model tends to lower the probability of most trajectories in its training set, leading to the probability of positive trajectories being lowered too. Fig, shows that, thanks to its acceleration (at = 1), TOPR recovers from these cases while TIS cannot. When the effective proportion of positive examples is high, there is virtually difference between TOPR and TIS. We also see that TIS reaches a slightly higher maximum Pass @ accuracy (chosen over all experiments) compared to TOPR. This suggests that TIS may trade robustness for peak performance.
Ratio truncation improves stability. A natural question is whether the truncation of importance ratios done by TOPR is necessary, if standard importance sampling alone suffices, suggested by [.] To study this question, we trained from the base model as before but using standard importance sampling (a+ + =a 0. b+ bb- +0e). Results and analysis may be found in Appendix [.4]
TOPR outperforms across multiple iterations. As a final experiment, we combine insights from our previous experiments to demonstrate that TOPR is effective inner-loop algorithm for iterated
37 and 8 remark similar findings in the on- on-policy setting: see also the empirical study by (6]
8

---

Passe8 score on GSMBK
Pass(1 1 score on MATH
22.35
20.52 20.29
$9.92
70.65
70.5
98
70.19
20
18.98 19.18
18
19.86
65.45
09.05
18
68-
67.7
67.46
Passer
66-
95.32
6.15
@
24.5
64-
18/8
62
10
60
TOPR-10p
TOPR; 10p
TIS 10p
$8.38
6.3
TIS 10p
1
10
100
10 Effective percentage of positive examples
100
Effective percentage of positive examples
Figure 4: Test set accuracy on GSM8K (left) and MATH (right) when the training contains P 10% of positive examples, using either truncated importance sampling (TIS) or TOPR. The x-axis, log scale, represents the effective proportion of positives.
100
Passg1 1 score on GSMBK
Pass@1 score on MATH
Lame3
45
TOPtt Positenss
DeepSeeltth TOPR AK
40
80
Pasxg1
30
Ldfma2 - 700
UamaBb TOPR Uniform
Lammtt TOPR Positiss
Deeps eektth TOM
70
25
20
60
50
5
0
1
2
3
Iteration
Iteration
Figure 5: Pass@1 1 scores on GSM8K (left) and MATH (right) for uniform sampling, positive-only sampling, and Anna Karenina sampling (see main text). By combining TOPR and Anna Karenina sampling, fine-tuned DeepSeek-RI 8B model outperforms Llama 70B.
fine-tuning of language models. Training begins with a base model To. from which a dataset is sampled (14: T;-1, = 1.2...... We then subsample this dataset to create iteration batch of to = 50, 000 data points, and use these to train a new policy मं starting from TT- as a reference. Figure shows how model performance continues to improve over multiple iterations, both for GSM8K and MATH: furthermore, TOPR enables faster learning than positive-only sampling, allowing us to surpass DeepSeek 8B-level Maj @ 16 accuracy within few iterations.
One challenge with fine-tuning models that already perform quite well is that, as training progresses. the model is essentially presented with examples that it already performs quite well on. This can make training quite inefficient. To combat this, we introduce Anna Karenina sampling, based on Tolstoy's famous "All happy families are alike; each unhappy family is unhappy in its own way." For each problem, we sample 64 candidate solutions, of which we only keep the first positive example. The iteration batch is then filled with negative examples chosen at random from those candidates. On GSMSK, this technique enables more efficient learning (79.6% Pass I accuracy) compared to uniform sampling (75.4%) It is less effective early MATH iterations, where the model has low Pass@1 accuracy and every positive example counts. As further evidence of TOPR's effecttiveness, applied to the more recent DeepSeek 8B model [18] it produces a model that rapidly surpasses the 70B version of Llama 3 in Maj 16 accuracy.
cettaset-balaccing technique we call
4.2 Learning to search
We now turn to preliminary results of TOPR's performance on learning to search task. We apply reinforcement learning to replicate responses from black-box API. Many systems that support text-based search rely on synonym and abbreviation expansion, where a user's query automatically improved using large corpus of morphological variants for the same concept [21] While model- based approaches to expanding a user's query have been proposed 27 used RL to learn better covering for user queries.
none of them have

---

Base model
Taned mode
Figure 6: Test performance intersection over union with TOPR (trained model). The trained model offers more set coverage where the base model fails to produce useful query expansions. Values below 0.0 are due the smoothing and should be ignored.
0.0
0.2
2.4
0.6
0.8
1.5
Insursoction over urion
For this purpose we consider the clinicaltrials gov repository clinical trial protocols (CTG). We studied whether a large language model can learn to produce set of synonyms (an expansion) in response given query that minimizes the symmetric set distance between the CTG API and simple internal database containing the same documents but differently indexed.
We used the Llama3-708 model to generate 1,200 distinct in-domain queries related to the field of life sciences, then defined a train-test split of 1,000 and 200 respectively. For each query, we prompted training
the smaller 8B model to propose variations of up to 10 query expansions resulting dataset of 8,000 samples to fine-tune our generator. Expansions were assigned a reward of if they resulted in the same search results, or up 5% relative tolerance for searches with 11 more results.
Figure 6 shows the distribution of search result overlap for the test set, before and after training. We measure this overlap in terms of the intersection over union (IOU) metric, which as the name implies measures the ratio of shared articles to total articles. Thus, score of 1.0 indicates a perfect match, and 0.0, completely different results. We find that, despite the challenging nature of this task, TOPR is able to improve on the base model by reducing the number of complete misses (score: 0.0). The average test IOU was improved from 35.03% (base model) 35.83% (trained model). While preliminary, this result points to the potential value of using reinforcement learning to optimize language models act as agents performing complex, but narrow tasks.
5 Conclusion and future work
Our results show that simple but principled change to REINFORCE is all that is needed to deploy it successfully and stably in the off- policy regime. Our approach is more efficient than existing dataset curation methods: when generating the dataset as all data points are kept; at training time because no KI. regularization required, and negative examples are effectively made use to improve performance; and at test time, because fewer solutions need to be generated. Our theory further provides an alternative, optimization based perspective on truncated importance sampling for RL, which may warrant revisiting other algorithms that make of it (34). Finally, our analysis sheds new light on the role of the baseline parameter and dataset composition in off-policy RL.
From here, there are number of future avenues for research. On one hand, we limited our experi- mental work the setting where μ the model at the beginning of an iteration, and data points are generated in the "self- self-ttuugtt style. It would be beneficial to deploy this in the offline setting [28] with a different μ4, but at first glance this poses numerical challenges. We also limited ourselves to the training of large language models but there is no reason to believe that TOPR would not perform equally in other application areas of RI., from video games to roboties 23
10

---

Author contributions and acknowledgments
NLR and MGB developed the algorithm, provided the theoretical analysis, structured the experimental work, and led the overall project. NLR, MGB, JL, AB, and JG implemented the algorithm, the experimental framework, and performed experiments. NLR, MGB, and JL wrote the paper. JL, JG, AF, CP, ETL, ST, and SW contributed to the technical infrastructure (datasets and software) with which the experimental work was performed.
NLR did this work while working as a consultant for Reliant AI, using their compute infrastructure. This work was partially supported by a Canada CIFAR AI Chair.
The authors would like to thank Matthieu Geist, Sal Candido, Rishabh Agarwal, Michael Bowling, Jesse Farebrother, Nathan Rahn, Harley Wiltzer, Charline Le Lan, and John Schulman for feedback and helpful discussions, as well as Matt Leus, Karl Moritz Hermann, and Nasib Naimi for support on this project.
References
[1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in LLMs, arXiv preprint arXiv:2402.1 14740, 2024.
[2] M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[3] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, June 2013.
[4] D. Calandriello, D. Guo, R. Munos, M. Rowland, Y. Tang, B. A. Pires, P. H. Richemond, C. Lan, M. Valko, T. Liu, al. Human aligmment large language models through online preference optimisation. In Proceedings of the International Conference on Machine Learning. 2024.
[5] Y. Chow, G. Temnenholtz, I. Gur, V. Zhuang, B. Dai, S. Thiagarajan, C. Boutilier, R. Agarwal, A. Kumar, and Faust. Inference-aware fine-tuning for best-of-n sampling large language models, arXiv preprint arXiv:2412.15287 2024.
[6] W. Chung, V. Thomas, M. C. Machado, and N. Le Roux. Beyond variance reduction: Under- standing the true impact of baselines on policy optimization. In International Conference on Machine Learning, pages 1999-2009. PMLR, 2021.
[7] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training Verifiers to Solve Math Word Problems. CoRR, abs/2110.14166 2021.
[8] M. Deisenroth, G. Neumann, J. Peters, et al. A survey policy search for robotics. Foundations and Trends0 in Robotics, 2(1-2):1-142, 2013.
[9] K. D. Dhole, R. Chandradevan, and E. Agichtein. Generative query reformulation using ensem- ble prompting, document fusion, and relevance feedback. arXiv preprint arXiv:2405. 17658, 2024.
[10] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[11] B. Efron and R. Tibshirani. An introduction to the bootstrap. CRC Press, 1994.
[12] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, Dunning, et Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the International Conference on Machine Learning, 2018.
11

---

[13] Y. Flet-Berliac, N. Grinsztajn, F. Strub, B. Wu, E. Choi, C. Cremer, A. Ahmadian, Y. Chandak, M. G. Azar, O. Pietquin, et al. Contrastive policy gradient: Aligning Ilms on sequence-level scores in fupirll fashion. arXiv preprint arXiv:2406. 19185, 2024.
[14) G. Gao, E. Choi, and Artzi. Simulating bandit learning from user feedback for extractive question answering. arXiv preprint arXiv:2203. 10079, 2022.
[15] D. Ghosh, M. C Machado, and N. Le Roux. An operator view of policy gradient methods. Advances in Neural Information Processing Systems, 33, 2020.
[16] N. Grinsztajn, Y. Flet-Berliac, M. G. Azar, F. Strub, B. Wu, E. Choi, C. Cremer, A. Ahmadian, Y. Chandak, O. Pietquin, et al. Averaging log-likelihoods in direct alignment. arXiv preprint arXiv:2406.191 2024.
[17] C. Gulcehre, T. I.. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Abern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling, arXiv preprint arXiv:2308.0000 089988, 2023.
[18] D. Guo, D. Yang. H. Zhang, J. Song. R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseck-rl: Incentivizing reasoning capability in I1ms via reinforcement learning, arXiv preprint arXiv:2501. 12948, 2025.
[19] S. Guo, B. Zhang, T. Liu, T. Liu, M. Khalman, F. Llinares, A. Rame, T. Mesnard, Y. Zhao, B. Piot, J. Ferret, and M. Blondel. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.0 0444424999, 2024.
[20] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NewrlPS Datasets and Benchmarks 2021, 2021.
[21] A. Henriksson, H. Moen, M. Skeppstedt, V. Daudaravičius, and M. Duneld. Synonym extraction and abbreviation expansion with ensembles of semantic spaces. Journal of biomedical semantics, 5:1-25, 2014.
[22]
S. M. Kakade, A natural policy gradient. Advances in neural information processing systems, 14, 2001.
[23] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, 2018.
[24] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[25] N. Le Roux. Efficient iterative policy optimization. arXiv preprint arXiv:1612.08967. 2016.
[26] N. Le Roux. Tighter bounds lead to improved classifiers. In International Conference on Learning Representations, 2017.
[27] Y. Lei, Y. Cao, T. Zhou, T. Shen, and A. Yates. Corpus-steered query expansion with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 393-401, 2024.
[28] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644, 2020.
[29] A. Lewkowyez, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra. Solving quantitative reasoning problems with language models. In Advances Neural Information Processing Systems, 2022.
12

---

[30] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let's verity step by step. In Proceedings of the International Conference on Learning Representations, 2023.
[31] L.. Luo, Y. Liu, R. Liu, Phatale, H. Lara, Y. Li, L. Shu, Zhu, Meng. Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv, 2024.
[32] D. Mahan, D. Van Phung, R. Rafailov, C. Blagden, N. Lile, L. Castricato, J.-P. Fränken, C. Finn, and A. Albalak. Generative reward models. arXiv preprint arXiv:2410.12822 2024.
[33] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the Interaational Conference on Machine Learning, 2016.
[34] R. Munos, T. Steplcton, A. Harutyunyan, and M. G. Bellemare. Safe and efficient off off-policy reinforcement learning. In Advances in Newral Information Processing Systems, 2016.
[35] M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville. Asyn- chronous rlhf: Faster and more efficient off-policy rl for language models. arXiv preprins arXiv:2410. 182522 2024.
[36] R. Y. Pang and H. He. Text generation by learning from demonstrations. In International Conference on Learning Representations, 2021.
[37] J. Peters and S. Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 21(4):682-697, 2008
[38] D. Precup. R. S. Sutton, and S. Dasgupta. Off-policy cemporal-difference learning with function approximation. In Proceedings of the 18th International Conference on Machine Laerning. pages 417-424, 2001.
[39] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2023.
[40] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 1707.00777 2017.
[41] Z. Shao, P. Wang, Q. Zhu, J. S. Runxin Xu, M. Zhang, Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
[42] N. Shazeer and M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the International Conference on Machine Learning, 2018.
[43] A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, X. Garcia, P.J. Liu, J. Harrison, J. Lee, K. Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585 2023.
[44] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for rein- forcement learning with function approximation. Advances in Neural Information Processing Systems, 1999.
[45] Tajwar, A. Singh, A. Sharma, R. Rafailov, J. Schneider, T. Xie, Ermon, C. Finn, and A. Kumar. Preference fine-tuning of lims should leverage suboptimal, on-policy data. arXiv preprint arXiv:2404.14:5 143667 2024.
[46] K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, Li, C. Xiao, C. Du, C. Liao, et al. Kimi k1. Scaling reinforcement learning with 1ms. arXiv preprint arXiv:2501. 12599, 2025.
[47] S. Vaswani, O. Bachem, S. Totaro, R. Müller, S. Garg, M. Geist, M. C. Machado, P. S. Castro, and Le Roux. general class of surrogate functions for stable and efficient reinforcement learning. In AISTATS, pages 8619-8649, 2022.
13

---

[48] X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi. S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. In Proceedings of the International Conference on Learning Representations, 2022.
[49] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinfrrement learning. Machine learning, 8:229-22 256, 1992.
[50] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowiez, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jemnite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing:
System Demonsirations, 2020.
[51] Y. Xia, J. Wu, S. Kim, T. Yu, R. A. Rossi, H. Wang, and J. McAuley. Knowledge-awer query expansion with large language models for textual and relational retrieval. arXiv preprint arXiv:2410.13765 2024.
[52] E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 15:15476-15488, 2022.
[53] L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agarwal. Generative verifiers: Reward modeling next-token prediction. arXiv preprint arXiv:2408. /5240, 2024.
[54] R. Zhang, L. Lin, Y. Bai, and S. Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint urXiv:2404.008868 2024.
[55] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang. Z. Wu, Y. Zhuang. Z. Lin, Z. D. E. Xing, al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 66:46695-46622. 2023.
[56] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909: wwt.i/90990999 08593, 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our contributions are both theoretical and experimental, as reflected in the abstract and introduction. We limit our claims to the learning algorithm.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The paper explicitly mentions that we limit our claims to the case where the starting policy is the one used to draw the samples. We also only make claims about fine-timing of large language models for specific tasks.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer:
[Yes]
Justification: We explicitly stated assumptions for our theorems. Proofs are correct to the best of our knowledge.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In addition to the algorithm, we provided the list hyperparameters used as well the architectural details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
16

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines for Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The training datasets are publicly available, and the parsing code is taken from open source implementations of the benchmarks. The code was written in a proprietary environment and cannot easily be released, however TOPR itself can be obtained with simple modification of any open-source REINFORCE implementations. We are additionally in the process of open-sourcing a version of TOPR.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (h (ht //tn://nipppeepe cc/
public Mudides/CodeSumm ss onPo icyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
(https://nipp)
more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
As explained for the "experimental result reproducibility" question.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
All main claims of the paper are supported by experiments containing error bars. Experiments on learning to verify, which are the appendix, do not have error bars due to the high cost of running them.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: Experiments were run across several months, with many experiments not making it into the final submission. We did not track the total compute used for this work. We also used mix of different providers with different associated costs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every re- spect, with the NeurIPS Code of Ethics EthicsGuidel ines?
Answer: [Yes]
Justification: The research is theoretical question with proposed applications reason- ing tasks, which comply with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
18

---

Answer: [NA]
Justification: We do not argue that there is no societal impact per se but rather that we cannot predict it this stage given the theoretical nature of the work involved. We do not foresee any direct path to a negative application, hence our answer.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Similar to the question above, we do not foresee any particular misuse related to the use our method.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We provided references for the used assets.
Guidelines:
The answer NA means that the paper does not existing assets.
19

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This research did involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This research did not involve human subjects.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Only humans contributed to the important, original and non-standard compo- nents of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
cc cc/Conferences/ 2025 LLM) for what should or should not be described.
Please refer to our LLM policy (https:
21

---

In this appendix, we provide additional theoretical results, experimental results and analyses.
A DPO and and other contrastive losses
We focused in the main text on methods that could work with batch size of 1. This excludes all contrastive losses, which usually use pairs of examples, one positive and one negative. DPO [39], one of the most well-known of these methods, works with pairs of trajectories and maximizes the weighted log probability ratio of these two trajectories. For positive and negative trajectories Tw and Tr, respectively, the DPO objective is
T(Tw) Juro (TT llogo (3 (Blog (T(Tw)
3 log T(1)) (((T))
where a is the sigmoid function. When rewards are either or 1, DPO can be repurposed to handle negative and positive trajectories [19] 4. and is in fact well-suited to off-policy policy optimization
[35]. However, DPO does not directly aim to maximize J(x) and, with finite number of trajectories, it is possible for the objective to increase while the probability of the positive trajectory decreases, as long as the probability of the negative trajectory decreases more. We shall see in our experiments that, while DPO indeed performs well off-policy, it is largely outperformed by TOPR. More recently, CoPG [13 also applied the idea of contrasting negatives and positives and, although their update shares similar form with REINFORCE. the use of carefully crafted baseline makes the method similar to DPO and IPO 2 and we therefore omit it from our analysis.
B Changing the effective positive rate through the baseline
In addition to the choice of the loss, the composition of the training set is critical to the performance of the trained model. In the context of training language models perform chain-of-thought reasoning, for example, dataset curation methods such as STaR, ReST, and ReST-EM differ mainly on which data they include.
As we will see, the relative importance of positive and negative examples in the dataset is equally critical to good performance. Interestingly enough, the baseline parameter can also be interpreted as modulating this relative importance.
Let us again assume binary reward function Ro(r) E (-1,1) and a baseline E [-1,1]. Let p = 1++/((++1+ + [T [T-)) be the proportion of positive examples in the dataset. Substituting
R((T Ro(T) into Eq. 6 we obtain
( U(T))
aaa,bb .6- (-1-c) ( a
JroP((T)) Σ p(T)p
at .bt) (1 c)+ Σ p(T)p
(((T)
+E2+
TET-
T(T) (u(T)
T(T) a a0,b) (U(T)
= (1 (1
Σ P(T)p +
a attbb)
- (1+
Σ u(T)p TET-
With this transformation, the contribution of each positive example to the objective weighted by 1 With some algebra, we find that the effective proportion of positive examples changes from to
p(1 c) -(1-) + (1
p(1 -
With fixed dataset, and 1 - p)) by modifying the baseline according to Eq. 7 This generalizes the result from the previous section that discarding negative examples is equivalent to using baseline of -1.
thus vary the relative importance of positive and negative examples (p
Furthermore, the choice of the baseline e can be viewed as adding a softer version of the KL || x) term to the TOPR objective, again encouraging to stay close to μ when < 0 (see Appendix [F). As a negative baseline c also increases the effective proportion of positive examples in the dataset, we see that a larger proportion of positive examples will decrease the degree of off fff-policyness that is achievable without resampling the training set. Adding negative examples can thus be seen as a way to further improve the policy.
22

---

It is however important to highlight some differences between setting a baseline and weighting positive and negative examples differently. While the expected gradient the same, stochastic estimates can be different, which might impact techniques like gradient clipping or second-order optimization. We expect this difference to be larger for extreme values of the baselines.
We shall show in Section 4 how carefully choosing the effective proportion of positive examples, either through dataset composition or a baseline, can lead boost in accuracy.
C Datasets and experimental setup
C.1 Models and Datasets
We focus on mathematical reasoning datasets that require step-by-step solutions and are widely used to evaluate the reasoning capabilities of L.LMs. As core model, the Llama family of instruction-tunee language models [0]. using the 8B model unless otherwise specified.
GSM8K 77
The GSM8K dataset is composed of short grade-school math problems, requiring basic arithmetic or elementary algebra solve. It contains 1,319 problems for testing and 7,473 for training. Verifying the correctness of model responses is straightforward, as the final answer is typically an integer. When the string is not present, we consider the answer as missing. For each training question we generate
16 candidate solutions using chain-of-thouge (CoT) prompting. using the 8-shot prompt from 488. We parse our model's answer by looking for the magic string "The answer is", matching this few-shot prompt.
The MATH dataset contains problems from high school math competitions, covering wide range of topics such as algebra, geometry, and probability, and is generally harder than GSM8K. We use the split from the original work, which includes 7,500 training problems. For computational reasons, we report performance on the smaller MATH-500 test set [30]. Each problem includes a step-by-step solution, ending in a final answer marked by \boxed in the solution (e.g.. "..so the smallest possible value of is π I").. This marking allows for verification of the correctness of model-generated responses by comparing the final answer the ground truth. We use the script provided by [41] for this purpose.
MATH 20
For each training question we generate = =22 candidate solutions using chain-of-thought (CoT) prompting, using the 4-shot prompt from [2]].
C.2 Experimental setup
[50]. We use data parallelism on a single H100 node with a per-GPU batch size of 1, a constant learning rate of 5e-7 chosen from a small parameter sweep. the Adafactor optimizer [42] to minimize memory usage, and neither weight decay nor KL regularizationf Fl in reinforcement learning terms can be thought of as implementing hyperbolic discounting. We use HuggingFace' s default gradient clipping parameter of 1.0. The reward +1 for a correct answer and -1 for an incorrect answer, and no baseline is used. Candidate generations are produced using vLLM [44] with temperature T = 1, top,- = 1, top, = 500, and a maximum of 512 tokens.
Our training infrastructure is based on HuggingFace's transformers library
We divide the loss by the sequence length [16), which
An iteration of training consists of generating candidates using a model (usually the base model), labelling those candidates with their associated reward, and performing single epoch over this generated dataset. The reference μl corresponds to the model predictions at the beginning of the iteration. Unless otherwise noted, all reported scores and accuracies are with respect to test sets, measured by evaluating the model at the end of the iteration. No early stopping is performed. We use the bootstrap technique [1] to provide confidence intervals: we generate 64 solutions for each question for GSM8K and 16 for MATH due to the cost of evaluating. For each question, we select K answers at random without replacement, then compute the average maj@ or pass@k K performance across the dataset. We repeat this process 100 times and estimated the empirical variance V across
'In addition to allowing us to focus on the relative stability of different learning rules, the removal of the KL term decreases the memory and computational burden of training the language model.
23

---

100
100
80
80
TOPR
60
DPO
60
40
40
20
20
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Gradient steps
Gradient
100
100
80
80
Naive REINFORCE
60
60
PPO
40-
20
%
%
10000
Gradlent steps
Gradlent
Figure 7: Proportion correct (green), incorrect (gray), and invalid (red) generated solutions on the GSM8K test set. Of the four, TOPR the only to significantly reduce invalid generations.
the 100 trials. We compute the standard error correspond to two standard errors.
KV
depicted confidence intervals
D Additional experimental results
D.1 TOPR minimizes reasoning failures
To understand the reasons behind TOPR's success, we measured the proportion of generated solutions that were correct, incorrect, or invalid (the string "The answer is" is not present) during the course of training. Fig. 7 gives strong evidence as the root cause of REINFORCE's poor performance, whose generations effective at teaching the model to avoid incorrect formatting yielding the desirable property that one can solely rely on RL for solution generation, rather than using additional tools to correctly format them.
overwhelmingly degenerate by the end of training. By contrast, TOPR proves
D.2 TOPR vs. Positives only
Fig. 8 allows to further analyze the experiment comparing TOPR, which uses both positives and negatives, to SFT which only uses the positive examples. Breaking down the test results as a function of the number of rationales that conclude in the correct answer ("Correct answer cardinality". bottom left), we find that TOPR's performance gains from using negative examples can be attributed to reducing the number of questions for which no or few solutions majority for self-consistency. We find similar results on MATH, where using TOPR enables us to almost double the @ass@ I accuracy compared to the base model (bottom right). Beyond these results, it is also worthwhile noting that TÓPR is more training inference-effcceen indeed, because all vLLM generations are used to improve the model, training data effectively generated faster rate.
found, guaranteeing strong
D.3 Effectivve vs actual proportion of positive examples
We observed in Sec. 4. that the optimal effective positive rate for GSM8K and MATH was in the 10%-20% range. These results, however, were obtained with a dataset that contained 10% of actual positive examples. One might thus wonder if the optimal performance is obtained when the effective

---

Pa55 score en GSMBK
Maj@21 score on GSMBK
75
70
68-
65
Make1s
60
55
84
TOPR
TOPR
50
-4--
only
Positives only
2000
4000
6000
8000
10000
2000
4000
5000
8000
10000
Gradlent steps
Gradient steps
700
Base model
questions
600
Positives only
17.5
TOPR
500
15.0
12.5
400
10.0
300
Pass
200
5.0
2.5
0.0
14
5-8
9-12
13-16
Base model
Positives only
TOPR
Correct answer cardinality
Figure 8: Top Test accuracy on GSM8K across training when using all examples (TOPR) or positive examples alone. Not shown here, TOPR also yields higher inference efficiency at test time: greater maj @n performance for all n. Bottom left: Distribution of GSM8K problem questions as a function of the number of correct generations. TOPR more effectively reduces the number of questions with none (0) to few (1-4) correct generations. Bottom right: Pass@ accuracy on MATH.
Passi GSMBK
Maja 16 GSMBK
71.40
60-92
70.0 6966
70.19
89
88.91
88.79
89.14
95
88.13
38.0
88.22
87.99
87
57.45
Passi
6.
6.27
86
62:
84
60-
TOPR-10p
69.61
83
TOPR-10p
83
58
58
Effective percentage positive examples
Effective percentage positive examples
Figure 9: Test set accuracy on the GSM8K dataset when the training set contains either = 10% (solid line) or p 50% (dotted line) of positive examples the baseline is varied. The x-axis,
log scale, represents the effective proportion of positives p T+e(1-22)) 2p) Pass@ (left) and Maj @ 16 (right) results are shown.
p(1-c)
positive rate matches that actual positive rate, i.c. with baseline c 0. For that purpose, in addition to the dataset containing 10% actual positive samples (labelled " "10p"), we built one containing 50% of positive examples ("50p"). We then vary the baseline for each model reach an effective proportion of positive examples from 1% all the way to 100%. We see in Figure thatt, once again, TOPR's 's performance is maximal around 10-20% of effective positive examples, regardless of the actual proportion of positive samples in the training. Further, we observe strong correlation between 10% and 50% curves, showing that the effective proportion actual proportion of positive samples. Further, Fig. shows the optimal effective proportion to be around 10-20% for both GSM8K and MATH. We posit that good baseline one that achieves such proportion.
more critical factor than
25

---

75-
70
65
Pass
60
TOPR
55
TOPR (Grad. clip. 100)
IS (Grad. clip. 100)
50
45
1000
2000
3000
4000
5000
Gradient steps
Figure 10: Performance of TOPR and standard importance sampling (IS) in our default experi- mental setting and with higher gradient clipping parameter (100.0). TOPR shows greater robustness to the gradient clipping parameter.
Acceleration improves robustness to dataset composition. Given the important performance gains from incorporating negative examples to training and the relevance of dataset composition (Section[B) B). is natural to ask whether TOPR's positive-cxample acceleration (a = 1) helps when positive examples outnumbered in the dataset, for example because the problem at hand is very hard. Figure&shovee the test performance on GSMSK using either TOPR or TIS when the effective proportion of positive examples varies. When that proportion is low, the model tends to lower the probability most trajectories in its training set. This leads to the probability positive trajectories being lowered as well. Thanks to its acceleration (a* 1). TOPR recovers from these cases while TIS cannot. When the effective proportion of positive examples is high, there is virtually no difference between TOPR and TIS. Interestingly, see that TIS reaches slightly higher maximum Pass@1 accuracy (chosen over all experiments) compared to TOPR. This suggests that TIS may trade robustness for peak performance.
D.4 The importance of truncating the importance ratio
Fig. shows that this results an algorithm that is as performant as TOPR. When we inspected its in-training behaviour, however, found that the average gradient norm became increasingly larger as training became more and more off- off--olicy. The impact of this norm blow-up is mitigated by the use of the default gradient clipping parameter (1.0), well as the relatively low number of negative examples in the training dataset (~33%).. To further demonstrate the stabilizing effect of ratio truncation in TOPR, used the same two algorithms but now with negatively-skewt dataset (60% negatives) and the gradient clipping parameter set to 100.0. The "Grad. Clip. 100" curves depict these results. TOPR is affected these changes but still improves the base model. Standard importance sampling, on the other hand, harms the model's performance producing 31 % of bad reasonings by the end of training against 12 % for the base model.
D.5 Experiments on Learning to verifyy
53 and 32 recently studied the use multiple CoT generations to verify the output of an LLM. In the context of math reasoning benchmarks, this generative verifier acts reward model that is used a best-of-n selection scheme, improving performance over self consistency (48). Our next series of experiments aimed study whether TOPR can improve verifier performance and improve solution quality for harder problems.
For each training sample in the MATH dataset, we used the 70B model to generate 16 solutions per problem, each with 4 verifications. then fine-tuned an 8B model using TOPR to act as generative verifier [53] using total of 480,000 data points. We evaluated this generative verifier
26

---

Verifier
Verifier accuracy
Invalid rate
Weighted SC
None
55.5%
Llama 8B
32.6%
34.2%
56.7%
8B TOPR
70.9%
0.90%
61.5%
Table 2: Performance of verifier trained with TOPR compared to using base model verifier or no verifier. Here "Verifier accuracy" is the Pass@ I accuracy of the verifier (whether it correctly judges that solution is right), "Invalid rate" is the number of verifications that could not be parsed successfully, and "Weighted SC" is weighted self-consistency with 32 generations (see c.g. 31] When verifier is used, Weighted SC indicates the usual maj @ @:32 accuracy.
Pass@1 score
72.35
90
Maj@16
GSMBK
72
68.
70-
60.92
70.65
50.5
65.45
09.05
70.19
67.46
88
B6.58
88.13
D8.42
37.07
M..2
88.91
88.44
88.92
87
62
Pass:01
66.
56.1
86
TOPR-10p
83
TOPR-10p
82.45
T1S-10p
$8.38
T15-10p
10
100
100
Effective percentage positive examples
Effective percentage positive examples
Pass@1 1 score on
M1 scere MATH
20
18 98
20 20.29 29.96
8.59
80
81.89
30.2
86.86
30.66 66
18
17:35
28:31
28.95
Pass@1
25
14
12
98
20
19
10
TOPR 10p
TOPR: 10p
6-
6.3
6.16
TIs 10p
13.52
TIS 10p
10
100
10
100
Effective percentage posilive examples
Effective percenlage positive examples
Figure 11: Test set accuracy GSM8K (top) and MATH (bottom) when the training set contains p = 10% of positive examples, using either truncated importance sampling (TIS) or TOPR. The x-axis, on a log scale, represents the effective proportion of positives ý I+<(1-2p)
p(1-c)
weighted eelf-consistency setting, where four verifications are aggregated into a score for each solution, and the answer with the highest score sum is selected. Table 2 shows that this procedure indeed produces a much more effective verifier for MATH generations, both terms of its verifier accuracy and effect on solution quality, In even more pronounced version of the results from Fig. 7 we find that TOPR fine-tunes the model to output almost no invalid generations - simply because is negatively rewarded for doing
E
Additional Maj @ 16 results
We computed Maj @ 16 results addition to the Pass@ scores from the main text. Similar conclu- sions can be reached, whether it is about the comparison between TOPR and TIS when varying the effective positive rate (Fig. IT or the performance of TOPR when performing multiple iterations (Fig.
F The impact of the baseline on KL regularization in TOPR
In standard REINFORCE, adding baseline to the reward does not change the unbiasedness of the gradient estimate but affects its variance. The same is not true in off off-policy policy optimization.
27

---

100
LiamaBb TOIR Uniform
Lana3: 706
Pass@1 Score on GSMBK
DeepSeekdb TOP
Lamaaoo TOPR
100.0
Liamal F3h
Lonode TOFR Unifom
Maj score on GSMBK BK
Limmalt
DecsScoldb TOPA,
LianaBo TOPR ResRives ony
Landd TOFR
Positives only
95.0
Pessip
80
M51016
92.5-
70
90.0
85.0
50
82.5
Iteration
Ibeeratoon
Pass@] score on MATH
Ma @B score MATH
60
40-
50
35
45
Lumadb ToPR Uniform
Torft Uniform
Uamadb TOPR Postives only
DeepSookBb TOPR
+ Limmdo Pesttives only
DoopSeokeb TOPR
-+-
Ulemat TOPS
RM
35
+ Llemat
20
30
15
25
10
20
1
1
2
3
4
Iteration
Iteration
Figure 12: Pass@1 (left) and Maj@ @aj@16 16 (right) scores on GSM8K (top) and MATH (bottom) for uniform sampling, posltive-only sampling, and Anna Karenina sampling (see main text). By combining TOPR and Anna Karenina sampling, we are able to fine-tune the DeepSeck-RI 8B model to achieve performance slightly superior to Llama 3 70B.
Looking back at Eq. (3. assuming at a =0 b+ b b, see that adding baseline yields
[
VTOPP (T, (T) En(T)
[R(T) - cVV
[u(T)]
[(T)]" [(T))logT((
[((T)
Eu(T) ((T)) (T()
cV (T)
[M(T)
V (T, 0)
c Σ (((T) ((7)
V T()
(MT)
((r(T)]
((F)
c Σ m(7)
V Vlog (T))
mK(T)
T(T) (MT)
[m(T) B [u(T)
ViTOPR c) V (T, 0) Σ ((T)
V log (T)
Assume negative baseline c. Since m(r)
is positive when (T)) > by((T), the additional
m(7)
[
term will decrease TT)) in that case. Hence, a negative baseline will discourage (T)) to be above bp(r) for all trajectories r. In that sense, it acts as a softer version of a KL regularizer that only alters when it deviates too much from μ. Alternatively, a positive baseline will encourage to be large, making the policy more deterministie. Note that this effect is solely due to clipping and goes against the effect due to stochasticity (6).
28

---

G Theoretical proofs
G.1 Proof of Proposition 3.1
As in the main text, we separate positive and negative trajectories as 7 == {T : R(T) 2 0} and
T (T R(T) < 0}. Define the reward-weighted distribution
(R(T) {
[( R₁
if R(T) 0,
R
E m(T) ((()) TET
otherwise;
and symmetrically for 10 and R
We have
T(T) [R(T) c] log
Jp,((z) J(r)
E
tmp
(T(T)
T(T) ((T)
log T(T)
E
R(T) log
E
tmp
p(T))
Σ.
(() (T) + cKL (1 T) + Co.
where (L(2 || T) denotes the the first term of the above equation into its positive and negative components:
divergence from μ to π and Co R. We now break
(og)
log
Σ (()
log
Σ (())
log (T)
TET+
TET-
R
Σ (og)))) R
Σ (T( log
TE
TET-
R KL ( [T)
KL(K || π) Ci.
Putting all together, we see that
( C R KL () (L cKL(M
with C' a constant independent of ा.
G.2 Proof Proposition 3.2
We start by showing that p(z, 0, b) 2 for all x 2 0 and all b 0. If x < b, then 0(0,0,,0 0.
If x > b, then p(x, 0, = b + log ;) 2 0 since 2 1.
Then, starting from
(T) ((T)) at b+ R(T)+
((T) u(T) En(r
JOOPR (TT) = Σ u(T)
(u(T) 0,b R(T),
(8)
TET
TET
we see that the second term is negative as the sum is over trajectories with negative rewards and both (T)) and I(T)) (m(r).00 0, b-) are positive.
Hence,
((() [u(T)" a- ,b, B(())
JTOPE (T) S Σ M(T)p
TET
Σ U(T) T(T)
R(T))
(since (z, at at.bt) s x)
m(T))
T
max R(T)
(since (T)) < 1 for discrete trajectories)
B
This concludes the proof.
'If RJ 0 (resp. Hj 0), argument holds for any # (resp, p7)-
29


Reviews:
[{"review_id": "FrAIwPXZ8B", "reviewer": "Reviewer_PUSX", "summary": "This paper proposes Tapered Off-Policy REINFORCE (TOPR), a policy gradient algorithm designed for the stable and efficient off-policy fine-tuning of large language models. The method uses an asymmetric update rule, applying a SFT objective to positive-reward examples and a truncated importance sampling objective to negative-reward examples. The authors claim this approach ensures stable learning dynamics, even in highly off-policy scenarios, without requiring an explicit KL regularization term.", "questions": "1.  Could you please provide precise details on how the dataset (with absolute rewards) was converted into preference pairs $(yw, yl)$ for the DPO baseline? How were the \"up to 16 contrastive pairs\" per prompt constructed? A fair comparison is essential to validate TOPR's superiority.\n2. Your analysis of the baseline $c$ is insightful. However, how should a practitioner select $c$ to achieve the optimal 10-20% effective positive ratio for a new dataset or model without running an expensive hyperparameter sweep? \n3.  The canonical TOPR algorithm uses an SFT-style update for positive examples ($a+ = 1$). What happens if you use a truncated importance sampling update for positive examples as well (e.g., $a+ = 0, b+ = C$ for some $C>1$)? Does this harm performance, or could it provide a better bias-variance trade-off?\n4.  Have you explored TOPR's performance on tasks with denser or non-binary reward signals, such as summarization (e.g., optimizing for ROUGE scores) or dialogue systems? How does it perform when most examples have a reward between -1 and 1?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The core idea of TOPR—asymmetrically handling positive and negative examples—is elegant, simple to implement, and well-motivated. It cleverly combines the speed of supervised learning with the stability of truncated importance sampling.\n2. The paper offers a compelling re-interpretation of the REINFORCE baseline's role in the off-policy setting. Framing it as a mechanism to control the effective ratio of positive-to-negative examples, rather than purely for variance reduction, is a potentially valuable contribution to the community's understanding.\n\nWeaknesses\n1. The TOPR algorithm is presented as a new contribution. However, a deconstruction of its gradient update reveals that it is a straightforward composition of two well-established techniques. The update for positive examples is identical to reward-weighted supervised fine-tuning (SFT), a method used in prior work like STaR[1]. The update for negative examples is a direct application of truncated importance sampling [2], a known technique for controlling variance in off-policy estimation.\n2. The paper convincingly argues that the REINFORCE baseline $c$ is critical for controlling the \"effective proportion of positive examples\". However, it provides no practical guidance on how a user should set this crucial hyperparameter, short of an expensive experimental sweep, which undermines the algorithm's claimed simplicity.\n\n[1] STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning\n\n[2] Importance Resampling for Off-policy Prediction", "was_revised": true, "final_justification": "Concerns have been addressed.", "rebuttal": "We thank the reviewer for their comments and questions.\n\nWhile we agree that TOPR can be viewed as a mixture of SFT (for the positive examples) and TIS (for the negative examples) on self-generated examples, we are a bit surprised to see this listed as a weakness rather than a strength. Had such a mix been straightforward, one would expect this method to have been developed earlier given its strong results. Most implementations that only rely on positive rewards, such as STaR, ReST, etc. emphasize complex prompting techniques (such as rationalization) and dataset curation, while here we deal with the negative samples directly.\n\nAdditionally, the derivation of TOPR is not a random idea. It is the result of an understanding of the behavior of several well-established methods and their own strengths and weaknesses. We are glad to have made our presentation clear enough that the idea of TOPR seems obvious but we dispute the claim that it is not a new contribution.\n\nOn the other hand, we agree with the statement that we provide no way to easily set the baseline parameter. As we stated in our submission, we found the best effective percentage to be around 10-20% for both GSM8K and MATH. Of note, our result shows that the GRPO baseline is not necessarily optimal. A deeper understanding of the baseline parameter would indeed be most welcome.\n\nMoving on to your questions:\n\n1/ Construction of the dataset for DPO\n\nFor each question, we generate 16 answers, leading to two sets $T_+$ and $T_-$. We then sample with replacement 16 contrastive pairs for each question. When one of the two sets is empty, i.e. all answers are incorrect or all answers are correct, then we cannot generate contrastive pairs and the question is discarded. We do this to compare things fairly across algorithms (same number of generated responses). Although we could attempt a different scheme (e.g., sample 8 distinct pairs for each question), this would be expensive for questions for which the model rarely generates the correct answer.\n\n2/ Setting of the baseline\n\nThere is an explicit formula for the setting of the baseline parameter to achieve a particular effective positive ratio, shown in Eq. 7 in Appendix B. If the proportion of positive examples in the dataset is $p$ and the baseline is $c$, then the effective proportion of positive examples is $\\tilde{p} =\\frac{p(1-c)}{1 + c(1 - 2p)}$. Solving for $c$ readily gives the baseline parameter to use. Not that this is for rewards of +1 and -1 but the formula can be adjusted for any rewards.\n\n3/ TIS vs. TOPR\n\nThe importance of the SFT update for the positive examples, as opposed to simply using TIS for all examples, is of prime importance. This concern was echoed by other reviewers and we thus intend to greatly clarify the difference in the revised submission.\n\nBefore answering your question, we want to clarify what might have been a misunderstanding: the method we call TIS in our submission uses truncated importance sampling for both positive and negative examples, as you suggest.\n\nWe were also surprised by the strong performance of TIS given the predominance of SFT-like methods to finetune LLMs. This said, TIS does struggle when there are few positive examples, because in this case their importance ratio goes to zero faster than the model can positively reinforce those examples. \n\nTIS requires a reference or sampling distribution for the positive examples, allowing it to leverage expert data for instance. As such expert trajectories are often used to do behaviour cloning, TIS is hardly used despite its strong performance when the dataset is entirely generated from the base model. Further, even if we knew the sampling distribution for expert trajectories, some or many importance ratios for these trajectories would be close to 0 if the base model is far from an expert, leading to a small gradient for these trajectories. We would then likely be in a similar situation as that when there are few positive examples, i.e. TOPR would outperform TIS.\n\nIn addition to this property, we did not find a setting where TIS consistently outperformed TOPR. As shown in Figure 4, there are cases where TIS is superior to TOPR, but they require the baseline to be very carefully selected, and thus a potentially expensive hyperparameter sweep.\n\n4/ Non-binary rewards\n\nWe decided to privilege a thorough experimental analysis on a smaller set of tasks rather than provide less conclusive answers on more tasks. We acknowledge the compromises of this decision and we do not have results for non-binary reward signals. Should the reviewer have suggestion on which such dataset, with manageable computational complexity, to include, we will give it strong consideration."}, {"review_id": "dus9xa32m2", "reviewer": "Reviewer_cD7b", "summary": "This paper introduces Tapered Off-Policy REINFORCE (TOPR), an algorithm for RL fine-tuning of LLMs that addresses the issues of other algorithms in the off-policy setting, such as instability and slow convergence. The idea of the method is to update on positive-reward examples without importance sampling (enabling fast learning even when unlikely under the current policy), and use truncated importance sampling (TIS) on negative-reward examples (preventing destructive parameter updates as the policy diverges from the data distribution). The authors demonstrate that TOPR outperforms naive REINFORCE, PPO, and DPO in the off-policy setting on GSM8K and MATH reasoning benchmarks; they also show that TOPR is more robust to dataset imbalance compared to doing TIS on all examples. Other contributions include theoretical analysis of TOPR (boundedness of the objective and stability), a reinterpretation of the REINFORCE baseline parameter through the lens of effective dataset composition, and the effectiveness of a new data sampling strategy which focuses on negative examples.", "questions": "- In a head-to-head comparison with prior art (TIS) on Figure 4, TOPR is shown to be much more robust when effective % of positive examples is small. Would this hold up if we compare TOPR with TIS+downsampling, where we downsample the negative examples to increase % of positives and only then do TIS on it? It is true that some negative examples would be thrown out doing this, but it would be great to research whether this waste would in fact push the performance of TIS significantly below the performance of TOPR in this setting, and by how much.\n- Would Figure 4 look the same with x-axis being the actual proportion of positive examples in the dataset?\n- Table 1 claims that TOPR is faster than TIS - is there experimental evidence to support this?\n- What number of gradient accumulation steps are you using and have you tried different ones? My concern is that it's plausible it could significantly impact the robustness of different methods to the dominance of negative examples in the dataset.\n- The paper says \"All main claims of the paper are supported by experiments containing error bars.\" and describes the bootstrap procedure, but I can't see the error bars in any figures—is it because the confidence intervals are too small to be rendered?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Justifications of TOPR and comparisons with other similar methods in Sections 2 and 3 are clear and insightful, and help to understand and appreciate the new method. They are corroborated by extensive theoretical analysis.\n- TOPR has a number of advantages—it's designed in a very principled way, it's simple to implement, doesn't require regularization, and is shown to be more robust than alternatives off-policy.\n\nWeaknesses:\n- The experimental validation is thorough but I feel like it has not fully convinced me of superiority of TOPR for the following reason. The closest method to TOPR is TIS, as TOPR only differs from it by removing the density ratio for positive examples. It also seems that TIS is also the closest to TOPR in performance, performing better than DPO and PPO. However, the only head-to-head comparison of TIS and TOPR is on Figure 4; it shows performance of these 2 methods for different values of the REINFORCE baseline. It strongly hints at greater robustness of TOPR when negative examples dominate the data. However, I didn't understand why this setting of low effective (not actual) percentage of positives would be a particularly important one in practice. It is also not fully clear to me if this performance of TIS in the setting of 1-2% positives is the best we can do with TIS. Finally, the core value proposition of TOPR seems to be that it avoids \"waste\" of ignoring or throwing out negative examples, but has the pure cost of this waste been computed experimentally, in isolation from differences in the behavior of different methods?", "was_revised": true, "final_justification": "Clearly reasons to accept outweigh reasons to reject, but with a margin that is not very high:\n- It's a new principled and elegant method that seems to work better than others on an important class of problems;\n- The evaluation, however, didn't allow me to fully confirm the superiority of the new method. Many concerns were addressed by the authors, and I find myself directionally agreeing with their arguments, but they still didn't fully allow me to validate the evaluation methodology and results. If it wasn't for this, I would have put 5/6, because the other parts of the work are solid.", "rebuttal": "Thank you for your review and many questions.\n\n\n1/ Comparison between TOPR and TIS for various proportions of positive examples\n\nWhile we did not perform the experiment for TIS, Figure 8 in the appendix gives us insight on the relative impact of downsampling negative examples compared to downweighing them for TOPR. We can see that using 10% positive examples in the dataset performs comparably, albeit slightly better, to using 50% positive examples, regardless of the *effective* percentage of positive examples. Difference is on the order of a couple percentage points, which is less than the difference between TIS and TOPR shown in Figure 4. Should you place this as high priority (because of the incurred cost), we could run the equivalent of Figure 8 for TIS in the final version of the paper.\n\nFigure 8 should also give an idea of what Figure 4 would look like with the actual proportion of positive examples on the x-axis.\n\nAnother big difference between TIS and TOPR comes when the dataset is not composed of trajectories generated from the base model but either from a more powerful model (distillation) or with expert trajectories. In the former case, the importance ratio can be close to 0 if the teacher model is very different from the student model and TIS will assign a very small gradient to these positive examples. In the latter case, one does not even have access to the sampling distribution and cannot use the importance ratio. We believe this is why, even though TIS performs well in our setup, SFT is the predominant method for fine-tuning LLMs.\n\nAs your comment was echoed by all other reviewers, we will strongly emphasize this difference in the final version should our submission be accepted.\n\n2/ Claims of Table 1\n\nBefore discussing evidence, we want to highlight the fact that there would only be a significant difference between TOPR and TIS in the presence of desirable trajectories with low probability under $\\pi$. We identify two cases where this can happen:\n- First, when most of the datasets is comprised of negative examples, the optimization will decrease $\\pi(\\tau)$ for sampled trajectories $\\tau$, which are generally those with larger $\\mu(\\tau)$ with $\\mu$ the sampling policy. Hence, during the stochastic optimization, we might select positive trajectories $\\tau$ for which $\\pi(\\tau)$ is much smaller than $\\mu(\\tau)$. The gradient for these trajectories will be much smaller for TIS than for TOPR, leading to poorer optimization. The evidence for this is in Figure 4, where the performance of TIS drops much more than that of TOPR for low effective percentage of positive examples.\n- Second, when the dataset is composed of examples not drawn from the base model, for instance in the case of behaviour cloning. In that instance, even when there are only positive examples in the dataset, it might happen that $\\pi(\\tau)$ is much smaller than $\\mu(\\tau)$ and the model will not learn them. The empirical evidence for the superiority of TOPR over TIS in that context is that behaviour cloning uses an SFT-like objective, also used by TOPR for positive examples, and not the REINFORCE based one used by TIS.\n\nHowever, in cases where a/ the dataset is generated from the base model and b/ there is a signification proportion of positive examples in the dataset, we can expect $\\pi(\\tau)$ to never be much smaller than $\\mu(\\tau)$ and for TIS and TOPR to perform similarly, as observed on the right-hand side of the two plots in Figure 4.\n\n3/ Gradient accumulation\n\nThat is an interesting question. As experiments ran on 8 GPUs, we accumulated 1 gradient per GPU before applying the update (see section C.2 in the appendix). We did not try other values of gradient accumulation for this submission. However, we are in the process of open-sourcing a version of the code and we tested it with various batch sizes without observing any significant difference so far. Please note however that our experiments with that new codebase are not as thorough as the ones we have done for the submission.\n\nCould you maybe explain why you think different methods might be affected differently by the percentage of negative examples? We have additional theoretical results, which we decided not to include in the submission for conciseness, showing that TOPR is likely to have lower variance than SFT as it minimizes the impact of a gradient step on trajectories not in the batch. We can definitely consider including these results in the Appendix should you think this is important.\n\n4/ Error bars\n\nAll figures except Figure 6 and Figure 9 (in the Appendix) contain error bars corresponding to two standard errors (see section C.2). Error bars are indeed extremely small due to the number of solutions generated. Note that our error bars capture uncertainty in the generation of the test examples, not in the training process nor the creation of the training set as this would significantly increase the total computational cost of the experiments.\n\n5/ Cost of discarding negative examples\n\nThe sole difference between SFT and TOPR is the inclusion of negative examples. Hence, the cost of this waste cannot be isolated from \"differences in the behavior of different methods\" as the different behavior between SFT and TOPR is *precisely* this waste. This is in contrast with, e.g., the difference between TOPR and Naive REINFORCE where all examples are included but with a different update. Thus, Figure 3 highlights this waste."}, {"review_id": "AcLdLhUW9f", "reviewer": "Reviewer_5Gwa", "summary": "The authors introduce Tapered Off-Policy REINFORCE (TOPR), a new algorithm for fine-tuning large language models with reinforcement learning that remains stable even when operating off-policy and does not require KL regularization. TOPR consists of an asymmetric importance-sampling update: positive trajectories receive a supervised fine-tuning–style weight independent of the current policy, while negative trajectories are handled with a truncated importance-sampling correction to prevent destructive gradient updates.Theoretical analysis shows that this “tapered” scheme bounds the learning objective and avoids the need for baselines or KL penalties that can mask valuable negative-feedback information\n\nEmpirically, the authors demonstrate on GSM8K and MATH reasoning benchmarks that TOPR outperforms REINFORCE, PPO, and DPO. Also, in a multi-iteration fine-tuning setup, TOPR combined with “Anna Karenina” sampling (which prioritizes diverse negative examples and keeps just the first correct solution) enables a DeepSeek 8 B-parameter model to exceed the performance of much larger Llama 70 B-parameter model over successive rounds of training.", "questions": "* The paper evaluates TOPR only on GSM8K and MATH reasoning tasks. Could you please include additional tasks to demonstrate generality? Could be code generation, or dialog based tasks or even non-LLM traditional RL tasks.\n* Could you please report the run-time comparison for TOPR vs PPO, DPO, REINFORCE?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths:\n* The paper introduces a “tapered” importance-sampling update that treats positive examples like supervised fine-tuning while truncating negative weights. This is a novel unification of SFT and off-policy RL .\n* The algorithm is presented cleanly and its behavior intuitively illustrated, making both motivation and mechanics easy to follow.\n* The paper offers good theoretical analysis, proving that the TOPR objective is bounded under its tapered importance-sampling scheme, ensuring stable off-policy updates without requiring KL penalties.\n* Across GSM8K and MATH benchmarks, TOPR consistently outperforms strong baselines (REINFORCE, PPO, DPO).\n\n### Weaknesses:\n* Experiments are limited to two mathematical reasoning tasks (GSM8K, MATH), leaving the method’s applicability to broader language tasks untested.", "was_revised": true, "final_justification": "I'm happy with the rebuttal and maintain my positive score.", "rebuttal": "Thank you for the review.\n\n1/ More tasks\n\nWe acknowledge your desire, echoed by other reviewers, to see TOPR tested on more standard benchmarks. Unfortunately, our GPU budget (in dollars) required us to make the choice between shallow evaluation across many datasets, or focusing on a smaller set and going in depth. The current set of experiments was already done at significant expense given their thoroughness, and every additional task requires careful tuning of the baselines to make sure our results are trustworthy.\n\nHowever, while our submission focused on math reasoning datasets, we also did preliminary evaluations of TOPR on query expansion. In these experiments, the website clinicaltrials.gov was used as a ground truth for expanded query terms. For instance, a researcher may search \"IV\" but expect \"intravenous\" to be included in their search results. We fine tuned a Llama 3 8b model to produce a domain-specific set of query expansions after having collected and verified a corpus of positive and negative samples.\n\nTOPR was able to shift the increase the performance of the LLM, measured as the intersection over union of the expanded set of queries generated by the LLM compared to the ground truth. This was done without any domain knowledge. The results remain preliminary but could be included if desired, although they do not correspond to a standard benchmark.\n\n2/ Compute budget\n\nAll methods that we evaluated require the same compute budget for generating samples.\n\nFor training, TOPR, PPO and REINFORCE all require about the same time to perform one gradient step (on a H100 node in our setup, about 1.3s per batch). PPO and TOPR compute the ratio of probabilities from the reference model, which requires an additional forward pass (the reference). We found this to add a small overhead (5-10% of per-step cost), with computing the gradients and the backward pass the dominant cost. DPO is slower because it performs the backward pass on two samples and computes two reference probabilities.\n\nOverall, every figure that uses \"Gradient steps\" as the x-axis can be equivalently thought of as approximately using runtime on that axis."}, {"review_id": "aD1iUpqrah", "reviewer": "Reviewer_3Rpm", "summary": "This work introduces a new RL algo called Tapered Off-Policy REINFORCE (TOPR) to address the instability issues of REINFORCE even when applied in off-policy settings (for fine-tuning large language models). TOPR uses asymmetric importance sampling with different truncation parameters for positive and negative examples, enabling fast and stable learning (without requiring KL regularization).", "questions": "none", "limitations": "ran only on small-ish networks", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Rigorous theory\n  - algo has intuitive-ish justification\n  - analysis through prop 3.1 and 3.2, showing that TOPR optimizes a bounded objective when a = 0 is insightful\n  - asymmetric treatment of positive vs negative examples is well-motivated\n  - simple compared to other RL algos\n- seems to work really well compared to others\n   - stable off-policy learning without requiring KL regularization\n   - effectively uses both positive and negative examples - \n   - simple to implement - just a modification to standard REINFORCE\n\nWeaknesses \n- I would have liked to see more comparisons to more RL algos. Other baselines could be: TRPO, PPO, GRPO.\n- wasn't run on large networks", "was_revised": true, "final_justification": "no revisions needed.", "rebuttal": "Thank you for the kind review. Regarding your two comments:\n\n1/ Comparison to other methods\n\n- We *do* compare to PPO: theoretically in Figure 2 and empirically in Figure 1.\n- While GRPO uses advantages, it does not address the main limitation of PPO which is that the gradient does not flow as soon as the policy deviates significantly from the base policy, making it unsuitable for truly off-policy optimization. We can thus expect to observe the same suboptimal plateau as PPO.\n- Although TRPO can be faster than PPO in terms of updates, making sure the constraint is satisfied can be computationally expensive. \n\n2/ Larger networks\n\nWe appreciate the desire for more thorough experiments, a desire echoed by other reviewers. Sadly, the current set of experiments was already done at significant expense, especially given their thoroughness, for instance to evaluate the influence of the baseline. Larger networks also achieve high accuracy on MATH and GSM8K, leading to most examples being positive. Hence, a meaningful experiment using larger networks should involve harder datasets, leading to even larger compute requirements.\n\nHowever, given the strong demand for the reviewers for such additional tasks, we would love if you collaboratively could have a prioritized list of tasks that should be added. This would allow us to strengthen the submission while keeping computational costs manageable.\n\n3/ Figure 5\n\nIf you have the time, we would love to know how we can improve Figure 5. Are you talking about the right plot, where the legend crosses the lines? In any case, we will update the Figure to improve its readability."}]

Author rebuttal:

Meta-review: This paper presents a relatively simple idea and this is also its biggest weakness.
It adapts the REINFORCE
algorithm to use importance sampling when the reward is negative and
the unaltered algorithm when it is positive.  This gives it some nice
properties including the ability to use positive and negative examples
and the ability to operate stably off-policy.  The paper is well
written and achieves improved results on GSM8K and MATH benchmarks
with a Llama 8B model.  A particular strength of the paper is its
clear and insightful description of the problem and solution method.
Decision: Accept (poster)