Title: Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement learning

off-policy RL

LLM finetuning

bandits
Abstract: Reinforcement learning (RL) is increasingly used to align large language models (LLMs). Off-policy methods offer greater implementation simplicity and data efficiency than on-policy techniques, but often result in suboptimal performance. In this work, we study the intermediate range of algorithms between off-policy RL and supervised fine-tuning by analyzing a simple off-policy REINFORCE algorithm, where the advantage is defined as $A=r-V$, with $r$ a reward and $V$ some tunable baseline. Intuitively, lowering $V$ emphasizes high-reward samples, while raising it penalizes low-reward ones more heavily.
We first provide a theoretical analysis of this off-policy REINFORCE algorithm, showing that when the baseline  $V$ lower-bounds the expected reward, the algorithm enjoys a policy improvement guarantee. Our analysis reveals that while on-policy updates can safely leverage both positive and negative signals, off-policy updates benefit from focusing more on positive rewards than on negative ones.
We validate our findings experimentally in a controlled stochastic bandit setting and through fine-tuning state-of-the-art LLMs on reasoning tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper 'Reinforcement Learning From Others' Successes, Not Their Failures' propose to study the off-policy RL surrogate objective and discuss the implication of different choice of baseline value function $V$. The paper develops a clean theoretical analysis of the off-policy vanilla policy gradient (off-VPG) metho. Also provide a simple off-policy algorithm for LLM fine-tuning with almost no engineering overhead. Experimental validation verifies the finding that a small baseline $V$ benefits the training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper has clear motivation (off-policy RL) and theoretical analysis. The experiments (both bandit and LLM) are clear and verify the theoretical findings with clear evidence.\n\nWeakness (please respond to the Questions section directly): Unclear why only analyzing the surrogate loss; The theoretical analysis requires more intepretation as it's not intuitive for now; The experiments might not be adequate and requires further comparison with on-policy algorithms.\n\nQUESTIONS:\n1. First, I'm curious why the authors chose to study the surrogate loss function (1). As the authors claimed \"As it does not involve any off-policy correction, such as importance sampling, we do not expect this algorithm to converge to an optimal policy in general.\" If the purpose is to study the off-policy algorithm as close to what it is in practice, I'm wondering if the analysis will be altered if a KL regularized or clipped objective is used (as in PPO objective).\n2. It seems not very clear to me what Theorem 4.2 implies. In particular, when $V<V^{\\mu}$, the optimal policy seems to be a reweighting of the sample policy $\\mu$ by reward minus value $r(y)-V$, but for $V\\geq V^{\\mu}$ I feel it very hard to interpret the results. \n3. What if we do adam or other adaptive algorithm in the experiments? Would the effect of different $V$ be mitigated?\n4. The \"Intuition\" paragraph is indeed intuitive, however for LLM finetuning, we often have a continuous reward value than a binary reward. Could the authors comment on the implication of the theory for this kind of rewards?\n5. Does the same collapse phenomenon observed in on-policy algorithms? I'm also curious on how the proposed method performs comparing to on-policy algorithms?\n\nLIMITATIONS:\nThe limitations are reflected in the questions section.\n\nI don't think there are any potential negative societal impact for this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyses a **simple off-policy REINFORCE algorithm (off-VPG)** whose advantage is $A=r-V$, with a **user-tunable baseline V**.\n\n- **Theory** For tabular soft-max policies the authors prove exponential convergence of off-VPG to a limit distribution $\\pi^{*}_{\\mu,V}$ (Theorem 4.2) and show a **phase-transition** at the critical value $V=V^{\\mu}$ (the behavior-policy mean reward) whereby the policy’s support suddenly collapses to a singleton . Repeated application forms a policy-improvement scheme with guaranteed reward increase when $V<V^{\\mu}$ (Theorem 4.3) .\n- **Intuition** Lower $V$ accentuates high-reward (“success”) trajectories, higher $V$ emphasises pushing down low-reward (“failure”) ones; unlike the on-policy case, this baseline **changes the expected gradient direction off-policy** .\n- **Experiments** (i) 100-arm bandit shows the predicted phase transition in reward, support, and entropy when $V$ crosses $V^{\\mu}$ . (ii) Fine-tuning **Llama-3 8B** on the MATH dataset confirms that setting $\\delta V \\approx -0.1$ (i.e. slightly below the prompt-wise mean reward) yields stable learning, whereas $\\delta V\\ge 0$ causes catastrophic collapse .\n\nThe work offers a **computationally cheap knob**—just shift the baseline—to trade off stability versus exploration when using off-policy trajectories for LLM RLHF.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. **Rigorous characterization** of off-VPG dynamics; explicit closed-form of $\\pi^{*}_{\\mu,V}$ and monotone support shrinkage clarify how baseline controls bias/variance trade-off .\n2. **Actionable guideline**: choose $V$ slightly below the behavior—policy mean to avoid support collapse—validated on both bandits and an 8B LLM .\n3. **Engineering simplicity**: no importance sampling, no critic, minimal code changes, yet competitive stability.\n4. Clear discussion of why on-policy intuition fails off-policy and how this links to practical RLHF sampling delays .\n\n**Weaknesses**\n\n1. **Limited model and task diversity**: only one LLM architecture (Llama-3 8B) and one dataset (MATH).\n2. **Missing baselines**: lacks direct comparison to IS-PPO, GRPO, or regularized PG, so relative performance ceiling is unclear.\n3. **Statistical under-reporting**: LLM results use three seeds without CIs; bandit plots omit error bars.\n4. **Code not yet released**, reducing reproducibility.\n\nQUESTIONS:\n1. Lines 596–599: The Lyapunov function is called “$\\Phi_t$ is bounded **above** on the simplex”. Strictly speaking $F$ is concave when $b>0$; here “above” could be re-worded as “has a finite supremum”.\n2. Lines 628–631: The bound $\\|\\nabla^2F\\|\\le2b$ → step-size $<1/b$. Readers may ask why the factor 2 disappears; worth stating that *any* constant $< 2/b$ suffices, but $1/b$ is chosen for margin.\n3. In the third statement of Theorem 4.3, does the author mean that by setting the value of $V$ within a certain range, we can ensure that the policy value converges to the optimal value? If so, the use of the phrase “if and only if” is quite confusing.\n\nLIMITATIONS:\n1. Proofs rely on a finite-arm bandit with a tabular softmax policy; behavior with function approximation and delayed updates remains open.\n2. The monotonic-improvement claim hinges on picking a baseline $V$ that is less than the unknown expected return $V^{\\mu}$, yet in real-world RLHF that return is buried under reward noise, so telling practitioners to “just choose a slightly smaller $V$” simply dumps the toughest task back onto them.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work discusses the design of off-policy RL for LLMs finetuning. By introducing a baseline V, the paper discusses the impact of policy convergence and explores how different baseline settings affect model performance. The conclusion drawn is that higher optimization weights should be given to positive trajectories, achieved through setting a baseline slightly lower than the expected value of the behavior policy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nUtilizing off-policy RL to optimize LLMs holds research value and may enhance the efficiency and deployment of RL-based fine-tuning. The theoretical analysis provides substantial support for the algorithm.\n\nWeaknesses:\n1. The conclusion that the baselines in off-policy learning not only influence the variance as in the on-policy algorithms but also introduce policy bias is not much novel in the off-policy RL scope. Some similar conclusions have been discussed in previous off-policy RL algorithms. Besides, the final conclusion (optimize more on the positive trajectories) quite aligns with the intuition. These factors somewhat diminish the contribution of the theoretical analysis *slightly*.\n2. The off-VPG is evaluated merely with one model on one dataset. Thus, the experimental support for the algorithm is limited.\n3. There is no comparison or discussion regarding on-policy finetuing. Although off-policy methods may have advantages in efficiency or implementation, is there a trade-off between performance and efficiency?\n4. This off-policy algorithm design does not specifically target LLMs and lacks discussions related to existing off-policy RL algorithms.\n\nQUESTIONS:\n1. There are relevant discussions on the influence of bias in off-policy contexts within Off-Policy Actor-Critic[1]. It would be better if a reasonable citation were added.\n2.  Off-VPG does not account for the relationship between behavior policy and current policy when utilizing trajectories. Off-policy policy gradient algorithms often correct bias through the importance weights. Should this be considered as a baseline? For instance, naive Off-Policy PG [2].\n3. Should the experiments be conducted on larger datasets to further validate the effectiveness of the Off-VPG? Additionally, the evaluation is only performed on the MATH dataset without assessing the LLM's performance in general textual tasks or other mathematical benchmarks. Lack a more comprehensive experimental validation.\n4. Some specific implementation details for Off-VPG are missing. For example, is the behavior policy completely unupdated or updated with a delay? If it remains unupdated, could it degrade into rejection sampling fine-tuning?\n5. There is an absence of comparisons between on- and off-policy algorithms concerning performance and sample efficiency to substantiate claims such as \"Off-policy methods offer greater implementation simplicity and data efficiency than on-policy techniques.\"\n\n[1]Off-Policy Actor-Critic. Thomas Degris, Martha White, and Richard S. Sutton. ICML 2012.\n\n[2]Off-Policy PG: https://lilianweng.github.io/posts/2018-04-08-policy-gradient/\n\nTrivial:\n1. The organization and formatting of theorem sections make them difficult to read.\n2. The x-axis title in Figure 2 lacks clarity.\n3. Color differentiation in line graphs is too low. It takes effort to discern them.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyze off policy policy gradient (in the bandit setting) and shows how different baselines can affect the learned policy. They show when the baseline is lower than the behavior value, the learned policy improves; the behavior value is a critical point, using a baseline beyond which can lead to collapsing in policy support. The algorithm is tested in a synthesized bandit problem and training a 8B Llama. The experimental results support the theory's insight.\n\nSTRENGTHS AND WEAKNESSES:\nS1. To my knowledge, despite many off-policy PG papers, this paper is the first to analyze directly off-policy without importance weight or state occupancy correction. \n\nS2 The theoretical analyses reveal an interesting landscape of off-policy PG's dynamics and is quite complete. It is also good to see the experimental results corroborate with the theoretical analyses. \n\nS3. The paper is overall well-written and well motivated. The analyses here are timely given that PG algorithms are gaining interests again in the LLM era. .\n\nW1. The current analyses assumes a bandit setting, which is different form the contextual setting used in practice. However, I think this limitation is minor given the contribution of the paper. \n\nW2. Some minor writing clarity can be improved.\n\nQUESTIONS:\n1. In the analyses, are the rewards implicitly assumed to be non-negative?\n\n2. In Theorem 4.3, does the set Y^inf exist always? since the policy can be stochastic.\n\n3. I do not understand the statement of point 3 in Theorem 4.3 from the current writing, and reading the proof doesn't help. Can you expand more on what you wish to convey?\n\n4. Do you have a bound on how large V^inf is? Maybe this is related to the misunderstanding above.\n\n5. How does the convergence rate and V^inf change with V in Theorem 4.3? I\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRL has been very successful in aligning LLMs for various downstream tasks. However, the most popular RL methods are on-policy and have several drawbacks. This paper proposes a new off-policy method to address the limitations of on-policy RL methods when aligning LLMs. The paper proves two theorems: one regarding the limits of the algorithm in relation to various baselines, and the other concerning policy improvement. Additionally, the paper conducts several experiments using LLaMA on mathematical tasks. The results of the experiment show that the various baselines, as shown in the proof, empirically affect the models' downstream performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe ability to train off-policy for LLMs alignment is a very important problem. The paper's proposal is scalable and has the potential to impact training models in an asynchronous manner. The paper comes with guarantees that hold in practice, which makes it a promising direction.\n\nWeaknesses:\n\nThe paper has several shortcomings. First, it omits some key baseline algorithms necessary for a thorough comparison. The evaluation is also limited to a single model and task, which raises concerns about its robustness and applicability to other tasks. Moreover, without including baseline algorithms, it’s unclear how the proposed method compares to existing approaches in the literature, such as [1]. Finally, some of the theorems in the appendix seem underexplained. \n\n[1] Fine-Tuning Language Models with Advantage-Induced Policy Alignment by Banghua Zhu et al. 2023\n\nQUESTIONS:\n- How does this paper compare to [1], PPO [2] in terms of the importance ratio enabling off-policy updates, and GRPO?\n- The original motivation of the paper was efficiency, but there are no comparisons to on-policy algorithms to demonstrate efficiency in any way.\n- For equation (1), how are you avoiding the importance sampling issue?\n- How do you choose the N to use for delayed updates? How sensitive is the proposed algorithm with respect to N?\n- Are the bounds in the bandit RL setting or the multi-step RL setting?\n- Could you provide more motivation for why you have the additional delta V term? The fact that the value of V varies for each prompt is expected, so I'm not sure why adding the correction term is necessary.\n- How is delta V trained in practice because it seems to be different than V^\\mu, which is computed using multiple rollouts.\n- For most of the bounds in the appendix, there are no assumptions made regarding the ratio of \\pi(y)/\\mu(y), which could be bad if the ratio is not assumed to be bounded, right?\n\n[1] Fine-Tuning Language Models with Advantage-Induced Policy Alignment by Banghua Zhu et al. 2023\n\n[2] Batch size-invariance for policy optimization by Jacob Hilton et al 2022\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}