Title: The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Reinforcement Learning

LLM Reasoning
Abstract: Reinforcement learning with verifiable rewards (RLVR) is a promising approach for training language models (LMs) on reasoning tasks that elicit emergent long chains of thought (CoTs). Unlike supervised learning, it updates the model using both correct and incorrect samples via policy gradients. To better understand its mechanism, we decompose the learning signal into reinforcing correct responses and penalizing incorrect ones, referred to as **P**ositive and **N**egative **S**ample **R**einforcement (**PSR** and **NSR**), respectively. We train `Qwen2.5-Math-7B`, `Qwen3-4B` and `Llama-3.1-8B-Instruct` on a mathematical reasoning dataset and uncover a surprising result: training with only negative samples — without reinforcing correct responses — can be highly effective: it consistently improves performance over the base model across the entire Pass@$k$ spectrum $k$ up to 256), often matching or surpassing PPO and GRPO. In contrast, reinforcing only correct responses improves Pass@1 but degrades performance at higher $k$, due to reduced diversity. These inference-scaling trends highlight that solely penalizing incorrect responses may contribute more to performance than previously recognized. Through gradient analysis, we show that NSR works by suppressing incorrect generations and redistributing probability mass toward other plausible candidates, guided by the model's prior beliefs. It refines the model's existing knowledge rather than introducing entirely new behaviors. Building on this insight, we propose a simple variant of the RL objective that upweights NSR, and show that it consistently improves overall Pass@$k$ performance on MATH, AIME 2025, and AMC23. Our code is available at [`https://github.com/TianHongZXY/RLVR-Decomposed`](https://github.com/TianHongZXY/RLVR-Decomposed).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into the effectiveness of negative reinforcement in enhancing the reasoning capabilities of LLMs. It dissects RLVR into PSR and NSR, exploring their distinct impacts on model behavior and generalization. The authors demonstrate that NSR, which solely penalizes incorrect samples, can surprisingly elevate model performance across the Pass@k spectrum, often matching or surpassing traditional RL algorithms like PPO and GRPO. Based on these insights, the paper proposes W-REINFORCE, a modified RL objective that emphasizes NSR, achieving notable performance improvements across multiple reasoning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper presents a novel perspective by decomposing RLVR into PSR and NSR, offering valuable insights into their individual roles in shaping model behavior.\n- The authors conduct extensive experiments to validate their claims, providing a comprehensive analysis of how PSR and NSR affect model performance across different metrics.\n- The findings have important implications for the design of RL algorithms for LLMs, particularly in preserving diversity and promoting generalization.\n\n**Weaknesses:**\n- While the decomposition of RLVR into PSR and NSR is insightful, the overall concept of this method is not entirely new and has been touched upon in previous studies [1].\n- The proposed W-REINFORCE algorithm, despite showing promise, requires careful tuning of the weighting factor $\\lambda$ . The paper does not provide sufficient guidance on how to optimally set this parameter in practice.\n- The claim that NSR (using only negative samples) outperforms PSR is a bit counterintuitive and confusing. The authors should provide a clearer explanation for this result (Table 1).\n\n[1] Xiong, Wei, et al. \"A minimalist approach to llm reasoning: from rejection sampling to reinforce.\" arXiv preprint arXiv:2504.11343 (2025).\n\nQUESTIONS:\n1. Can the authors elaborate on the relationship between SFT and PSR? \n2. How does the proposed W-REINFORCE algorithm compare to other RL variants in the literature that also emphasize negative reinforcement? What specific advantages does it offer over these existing methods?\n3. The choice of the weighting factor $\\lambda$  in W-REINFORCE seems crucial for its performance. Could the authors offer more detailed guidance or a systematic approach for selecting an appropriate $\\lambda$ value, especially for different models and tasks?\n4. The authors mention that NSR refines the model's existing knowledge rather than introducing entirely new behaviors. Could they provide more in-depth analysis or visualization to help readers better understand this refinement process?\n5. Why does W-REINFORCE underperform PPO in pass@1? An explanation from the authors would be appreciated.\n6. How does the proposed method generalize to OOD datasets? *(This will not influence my score)*\n7. What is the specific value of $n$ in the pass@k evaluation metric? (Is it 256, as suggested in line 111?)\nIf $k=256$, based on the definition in Eq. (5), the pass@256 values in Table 1 should be binary (either 0 or 1). This is because if we assume $n=k=256$, the term $\\binom{n}{k}$ becomes $\\binom{256}{256} = 1$. Consequently, the term $\\binom{n-c}{k}$ becomes $\\binom{256-c}{256}$, which evaluates to 0 for any $c > 0$ and to 1 only when $c=0$. This seems to contradict the results shown in Table 1.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nStudying RL with verifiable (binary output-based) rewards, the authors analyze the impact of positive and negative reward signals on model performance and compare to standard methods such as PPO and GRPO. The authors focus the evaluation on math reasoning using the MATH, AIME2025, and AMC23 datasets.\n\nThe authors empirically show using Qwen2.5-Math-7B with a full spectrum of Pass@k with k in {1, 2, 4, 8…, 256} that only using positive reward samples during training leads to better performance for pass@1 while reducing diversity of responses and overfitting for larger values of k. The authors find that using negative reward samples performs similarly to PPO and GRPO methods in terms of larger k values and that it outperforms other methods in preserving model entropy (response diversity) by reducing the likelihood of wrong responses and reallocating the probability mass to others. The authors also use token-level gradient dynamics to verify these claims.\n\nThe authors also introduce a weighted variant of REINFORCE that prioritizes negative sample that contributes to the loss function with a linear hyperparameter, which performs comparably to more elaborate algorithms like GRPO and PPO on the tested benchmarks.\n\nBased on these results, the authors claim that negative reward samples naturally avoid overfitting (unlike positive reward samples), allow the model to effectively explore and search according to base model prior beliefs, and that the accuracy and diversity trade-off in RLVR is a balance issue between positive and negative rewards (which can be solved by re-weighting).\n\nSTRENGTHS AND WEAKNESSES:\nReasons to accept:\n\n- This work potentially provides crucial insight into outcome-based rewards in RL for LLMs that can have a significant impact for similar methods that are limited to not using dense/progress rewards. Understanding the importance of negative samples can also have a significant impact on alignment and safety methodologies, where constructing negative samples based on a rule or reason is easier than constructing positively rewarded ones.\n\n- Good methodology using entropy and token-level gradients (quality). \n\n- The paper is well written and, to the best of my knowledge, original work.  \n\nReason to reject:\n\n- My main concerns are related to the lack of statistical significance tests or statistical uncertainty estimates (quality, uncertain significance). First, the authors claim to have fulfilled the NeurIPS requirement for statistical significance/uncertainty estimates with the reason ”Our results are based on 256 random samplings and calculated with an unbiased estimator.”. *Just using an unbiased estimator does not eliminate the requirement for uncertainty estimates/statistical significance tests!* \n\nDoing a few estimates, (assuming k=10, n=256, for a few c values), I find that the statistical uncertainty could be in the order of magnitude of 3-8 percent, i.e., in the order of magnitude the gained improvements in Table 1 are claimed or for which the differences in Figures 2 and 3 are used as evidence. I could be wrong, though, so I would require the authors to demonstrate that their results are statistically significant using the method they deem most appropriate.\n\nWithout demonstrating that the empirical results used in Figures 2, 3, and Table 1 are statistically significant and not just statistical trends, I have to question the validity of the results of the paper. The statistical trends are definitely there, but it has not been shown that these trends are actually statistically significant (at least 2-sigma or 95% confidence level deviations). For example, all claims in section 3.2 should either be highlighted as just observed trends (not results) or require the derivation of an uncertainty estimator + reported uncertainties. Table 1 should also include the statistical uncertainties. This is my main concern, and fully addressing this point will make me increase my score to a 4 or 5 (together with the quality and significance score).\n\n- The authors only tested one model (quality, uncertain significance). I realize that expecting multiple RL training runs for models above 7B in academic papers is unreasonable. However, I think calling for testing at least a second 7B or some of the smaller 3B models from different model families to support universal empirical RLxLM training dynamics claims is not.\n\nQUESTIONS:\n- What are the implications of splitting rewards into “only positive” and “only negative”? Besides the direct impact of positive and negative samples on training curves, how does this simplification neglect the indirect impact of having positive and negative samples during training? My question is motivated by the assumption that LMs in RLVR encounter more wrong than correct answers and, e.g., does encountering a positive sample every 10 negative samples impact the overall trajectory differently when comparing trajectories with only positive or negative samples? \n\n- To what extent are the results in Figure 2 explained by PPO and GRPO training encountering more negative than positive reward samples during training?\n\n- To what extent is the trend of RL models losing their advantage at larger decoding budgets solely explained by negative reward samples (as argued in lines 136-141)? Are there no other possible explanations?\n\n- It is unclear to me if the authors agree with this statement “[line 265] RLVR primarily adjusts the model’s output distribution toward high-reward responses, rather than eliciting new reasoning abilities beyond the base model.” Based on the discussed results, I think this is ambiguous, as it matters not just that the probability mass is redistributed, but how.  Please clarify your position and argumentation.\n\n- What can we learn or hypothesize from your study about RL algorithms using dense/progress-based rewards?\n\nMinor comment:\n- The x-axis labels are missing in Figure 3.\n\nLIMITATIONS:\nThe authors provide a very brief limitation section in Appendix C. *[For the AC]* I am not sure whether the limitation section is supposed to be in the main body of the paper or not.\n\nThe authors acknowledge the limitation of only using one model to build their results and that they neglect of dense/intermediate rewards.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper decomposed the training signals of reinforcement learning into two independent parts: \"rewarding correct answers\" (positive reinforcement, PSR) and \"punishing incorrect answers\" (negative reinforcement, NSR), and made an unexpected key discovery. The study shows that merely training the model by punishing incorrect answers (NSR) is sufficient to match or even surpass more complex algorithms like PPO, because it can enhance the reasoning ability while effectively protecting the output diversity of the model, which is crucial for complex problems that require multiple attempts to solve. On the contrary, merely rewarding correct answers (PSR) can improve the single-shot accuracy rate, but it will damage diversity due to overconfidence. Based on this, the paper reveals that negative reinforcement works by gently correcting the model's prior knowledge, and proposes a simple and efficient W-REINFORCE algorithm, which achieves excellent results in multiple mathematical benchmark tests by effectively balancing positive and negative feedback.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper decomposes the training signals of reinforcement learning into two independent parts: \"rewarding correct answers\" (positive reinforcement, PSR) and \"punishing incorrect answers\" (negative reinforcement, NSR). Its most significant finding is that \"only punishing errors\" can achieve excellent results, which challenges the common perception that people have previously placed more emphasis on rewarding correct answers, and reveals the huge value of negative feedback that has been underestimated.\n\nWeaknesses：\n1.The conclusion of the paper heavily relies on the Qwen-Math type of \"expert models\" that are already extremely powerful. The reason why negative reinforcement works is that the model already has the ability to distinguish \"suboptimal solutions\". If it were replaced with an ordinary, non-professional basic model, the effectiveness of this method would likely be greatly reduced, and thus its universality is questionable.\n2.This method is highly suitable for tasks in mathematics that have clear, binary (right or wrong) feedback signals. However, it is difficult to apply to tasks such as creative writing, dialogue systems, etc., where the reward signals are ambiguous, subjective or continuous. This is because it is impossible to clearly distinguish between \"positive\" and \"negative\" samples.\n3.Long-term use of negative reinforcement alone leads to performance degradation, indicating that it is not a stable training strategy in the long run. Additionally, the new algorithm W-REINFORCE proposed by them lacks sufficient ablation analysis for the selection of key hyperparameters.\n\nQUESTIONS:\n1.The new algorithm W-REINFORCE proposed in the paper sets the weight of positive samples to λ = 0.1. The experimental results show that this setting is very effective. However, the paper does not seem to provide an ablation study on the selection of this hyperparameter. How was λ selected? Is it sensitive to different models and tasks? What would the results be if λ is set to 0.05 or 0.2? A robust method should be less sensitive to the hyperparameters, or at least should clearly demonstrate its sensitivity.\n\n2.In the experiment, only the Qwen2.5-Math-7B and Qwen3-4B models were used for the experiment here. It is unknown whether this method is applicable in different model sizes as well.\n\n3.The paper, in the appendix, acknowledges that long-term training of NSR leads to performance degradation. However, this should not merely be regarded as a \"limitation\", but rather a core experimental result that deserves in-depth analysis in the main text. Readers cannot discern the \"decline\" trend of NSR from the charts in the main text because the experiment may only have reached its peak performance. The stability of an algorithm is a crucial aspect in evaluating its value.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper study the effect of the gradient of positive and negative samples on the policy. It conduct an experiment to show the influence of tdifferent gradient on different Pass@k accuracy. It then design an algorithm to improve Pass@k accuracy by giving different weight to different part of the loss and gradient.

STRENGTHS AND WEAKNESSES:
Strengths:
The paper is clearly written, and the experiments effectively demonstrate the main observations.

Weaknesses:

1. Personally, I find that the phenomenon observed is somewhat aligned with common intuition in the field. Additionally, the technique of reweighting the negative samples lacks novelty in my view. However, I acknowledge that other reviewers may differ in their judgment on whether this insight constitutes a meaningful contribution beyond what is already understood without formal validation.

2. The paper primarily focuses on the on-policy setting. However, off-policy settings—such as those used in popular algorithms like DPO—may exhibit different behaviors. The lack of consideration for off-policy scenarios limits the generality and significance of the findings.

QUESTIONS:
Why doesn't consider the setting of off-policy?

LIMITATIONS:
1. The paper primarily focuses on the on-policy setting. However, off-policy settings—such as those used in popular algorithms like DPO—may exhibit different behaviors. The lack of consideration for off-policy scenarios limits the generality and significance of the findings.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-14