Title: Tapered Off-Policy REINFORCE - Stable and efficient reinforcement learning for large language models
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforce

large language models

verifier

ppo

off-policy

rlhf
Abstract: We propose a new algorithm for fine-tuning large language models using reinforcement learning. Tapered Off-Policy REINFORCE (TOPR) uses an asymmetric, tapered variant of importance sampling to speed up learning while maintaining stable learning dynamics, even without the use of KL regularization. TOPR can be applied in a fully offline fashion, allows the handling of positive and negative examples in a unified framework, and benefits from the implementational simplicity that is typical of Monte Carlo algorithms. We demonstrate the effectiveness of our approach with a series of experiments on the GSM8K and MATH reasoning benchmarks, finding performance gains for training both a model for solution generation and as a generative verifier. We show that properly leveraging positive and negative examples alike in the off-policy regime simultaneously increases test-time accuracy and training data efficiency, all the while avoiding the ``wasted inference'' that comes with discarding negative examples. We find that this advantage persists over multiple iterations of training and can be amplified by dataset curation techniques, enabling us to match 70B-parameter model performance with 8B language models. As a corollary to this work, we find that REINFORCE's baseline parameter plays an important and unexpected role in defining dataset composition in the presence of negative examples, and is consequently critical in driving off-policy performance.


Peer reviews:
comments: SUMMARY:
This work introduces a new RL algo called Tapered Off-Policy REINFORCE (TOPR) to address the instability issues of REINFORCE even when applied in off-policy settings (for fine-tuning large language models). TOPR uses asymmetric importance sampling with different truncation parameters for positive and negative examples, enabling fast and stable learning (without requiring KL regularization).

STRENGTHS AND WEAKNESSES:
Strengths:
- Rigorous theory
  - algo has intuitive-ish justification
  - analysis through prop 3.1 and 3.2, showing that TOPR optimizes a bounded objective when a = 0 is insightful
  - asymmetric treatment of positive vs negative examples is well-motivated
  - simple compared to other RL algos
- seems to work really well compared to others
   - stable off-policy learning without requiring KL regularization
   - effectively uses both positive and negative examples - 
   - simple to implement - just a modification to standard REINFORCE

Weaknesses 
- I would have liked to see more comparisons to more RL algos. Other baselines could be: TRPO, PPO, GRPO.
- wasn't run on large networks

QUESTIONS:
none

LIMITATIONS:
ran only on small-ish networks
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Tapered Off-Policy REINFORCE (TOPR), a new algorithm for fine-tuning large language models with reinforcement learning that remains stable even when operating off-policy and does not require KL regularization. TOPR consists of an asymmetric importance-sampling update: positive trajectories receive a supervised fine-tuning–style weight independent of the current policy, while negative trajectories are handled with a truncated importance-sampling correction to prevent destructive gradient updates.Theoretical analysis shows that this “tapered” scheme bounds the learning objective and avoids the need for baselines or KL penalties that can mask valuable negative-feedback information\n\nEmpirically, the authors demonstrate on GSM8K and MATH reasoning benchmarks that TOPR outperforms REINFORCE, PPO, and DPO. Also, in a multi-iteration fine-tuning setup, TOPR combined with “Anna Karenina” sampling (which prioritizes diverse negative examples and keeps just the first correct solution) enables a DeepSeek 8 B-parameter model to exceed the performance of much larger Llama 70 B-parameter model over successive rounds of training.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n* The paper introduces a “tapered” importance-sampling update that treats positive examples like supervised fine-tuning while truncating negative weights. This is a novel unification of SFT and off-policy RL .\n* The algorithm is presented cleanly and its behavior intuitively illustrated, making both motivation and mechanics easy to follow.\n* The paper offers good theoretical analysis, proving that the TOPR objective is bounded under its tapered importance-sampling scheme, ensuring stable off-policy updates without requiring KL penalties.\n* Across GSM8K and MATH benchmarks, TOPR consistently outperforms strong baselines (REINFORCE, PPO, DPO).\n\n### Weaknesses:\n* Experiments are limited to two mathematical reasoning tasks (GSM8K, MATH), leaving the method’s applicability to broader language tasks untested.\n\nQUESTIONS:\n* The paper evaluates TOPR only on GSM8K and MATH reasoning tasks. Could you please include additional tasks to demonstrate generality? Could be code generation, or dialog based tasks or even non-LLM traditional RL tasks.\n* Could you please report the run-time comparison for TOPR vs PPO, DPO, REINFORCE?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Tapered Off-Policy REINFORCE (TOPR), an algorithm for RL fine-tuning of LLMs that addresses the issues of other algorithms in the off-policy setting, such as instability and slow convergence. The idea of the method is to update on positive-reward examples without importance sampling (enabling fast learning even when unlikely under the current policy), and use truncated importance sampling (TIS) on negative-reward examples (preventing destructive parameter updates as the policy diverges from the data distribution). The authors demonstrate that TOPR outperforms naive REINFORCE, PPO, and DPO in the off-policy setting on GSM8K and MATH reasoning benchmarks; they also show that TOPR is more robust to dataset imbalance compared to doing TIS on all examples. Other contributions include theoretical analysis of TOPR (boundedness of the objective and stability), a reinterpretation of the REINFORCE baseline parameter through the lens of effective dataset composition, and the effectiveness of a new data sampling strategy which focuses on negative examples.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Justifications of TOPR and comparisons with other similar methods in Sections 2 and 3 are clear and insightful, and help to understand and appreciate the new method. They are corroborated by extensive theoretical analysis.\n- TOPR has a number of advantages—it's designed in a very principled way, it's simple to implement, doesn't require regularization, and is shown to be more robust than alternatives off-policy.\n\nWeaknesses:\n- The experimental validation is thorough but I feel like it has not fully convinced me of superiority of TOPR for the following reason. The closest method to TOPR is TIS, as TOPR only differs from it by removing the density ratio for positive examples. It also seems that TIS is also the closest to TOPR in performance, performing better than DPO and PPO. However, the only head-to-head comparison of TIS and TOPR is on Figure 4; it shows performance of these 2 methods for different values of the REINFORCE baseline. It strongly hints at greater robustness of TOPR when negative examples dominate the data. However, I didn't understand why this setting of low effective (not actual) percentage of positives would be a particularly important one in practice. It is also not fully clear to me if this performance of TIS in the setting of 1-2% positives is the best we can do with TIS. Finally, the core value proposition of TOPR seems to be that it avoids \"waste\" of ignoring or throwing out negative examples, but has the pure cost of this waste been computed experimentally, in isolation from differences in the behavior of different methods?\n\nQUESTIONS:\n- In a head-to-head comparison with prior art (TIS) on Figure 4, TOPR is shown to be much more robust when effective % of positive examples is small. Would this hold up if we compare TOPR with TIS+downsampling, where we downsample the negative examples to increase % of positives and only then do TIS on it? It is true that some negative examples would be thrown out doing this, but it would be great to research whether this waste would in fact push the performance of TIS significantly below the performance of TOPR in this setting, and by how much.\n- Would Figure 4 look the same with x-axis being the actual proportion of positive examples in the dataset?\n- Table 1 claims that TOPR is faster than TIS - is there experimental evidence to support this?\n- What number of gradient accumulation steps are you using and have you tried different ones? My concern is that it's plausible it could significantly impact the robustness of different methods to the dominance of negative examples in the dataset.\n- The paper says \"All main claims of the paper are supported by experiments containing error bars.\" and describes the bootstrap procedure, but I can't see the error bars in any figures—is it because the confidence intervals are too small to be rendered?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Tapered Off-Policy REINFORCE (TOPR), a policy gradient algorithm designed for the stable and efficient off-policy fine-tuning of large language models. The method uses an asymmetric update rule, applying a SFT objective to positive-reward examples and a truncated importance sampling objective to negative-reward examples. The authors claim this approach ensures stable learning dynamics, even in highly off-policy scenarios, without requiring an explicit KL regularization term.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The core idea of TOPR—asymmetrically handling positive and negative examples—is elegant, simple to implement, and well-motivated. It cleverly combines the speed of supervised learning with the stability of truncated importance sampling.\n2. The paper offers a compelling re-interpretation of the REINFORCE baseline's role in the off-policy setting. Framing it as a mechanism to control the effective ratio of positive-to-negative examples, rather than purely for variance reduction, is a potentially valuable contribution to the community's understanding.\n\nWeaknesses\n1. The TOPR algorithm is presented as a new contribution. However, a deconstruction of its gradient update reveals that it is a straightforward composition of two well-established techniques. The update for positive examples is identical to reward-weighted supervised fine-tuning (SFT), a method used in prior work like STaR[1]. The update for negative examples is a direct application of truncated importance sampling [2], a known technique for controlling variance in off-policy estimation.\n2. The paper convincingly argues that the REINFORCE baseline $c$ is critical for controlling the \"effective proportion of positive examples\". However, it provides no practical guidance on how a user should set this crucial hyperparameter, short of an expensive experimental sweep, which undermines the algorithm's claimed simplicity.\n\n[1] STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning\n\n[2] Importance Resampling for Off-policy Prediction\n\nQUESTIONS:\n1.  Could you please provide precise details on how the dataset (with absolute rewards) was converted into preference pairs $(yw, yl)$ for the DPO baseline? How were the \"up to 16 contrastive pairs\" per prompt constructed? A fair comparison is essential to validate TOPR's superiority.\n2. Your analysis of the baseline $c$ is insightful. However, how should a practitioner select $c$ to achieve the optimal 10-20% effective positive ratio for a new dataset or model without running an expensive hyperparameter sweep? \n3.  The canonical TOPR algorithm uses an SFT-style update for positive examples ($a+ = 1$). What happens if you use a truncated importance sampling update for positive examples as well (e.g., $a+ = 0, b+ = C$ for some $C>1$)? Does this harm performance, or could it provide a better bias-variance trade-off?\n4.  Have you explored TOPR's performance on tasks with denser or non-binary reward signals, such as summarization (e.g., optimizing for ROUGE scores) or dialogue systems? How does it perform when most examples have a reward between -1 and 1?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}