The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
Authors: Xinyu Zhu
Mengzhou Xia
Zhepei Wei
Wei-Lin Chen
Danqi Chen
Yu Meng
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Reinforcement Learning
LLM Reasoning
Primary area: deep_learning

Abstract: Reinforcement learning with verifiable rewards (RLVR) is a promising approach for training language models (LMs) on reasoning tasks that elicit emergent long chains of thought (CoTs). Unlike supervised learning, it updates the model using both correct and incorrect samples via policy gradients. To better understand its mechanism, we decompose the learning signal into reinforcing correct responses and penalizing incorrect ones, referred to as **P**ositive and **N**egative **S**ample **R**einforcement (**PSR** and **NSR**), respectively. We train `Qwen2.5-Math-7B`, `Qwen3-4B` and `Llama-3.1-8B-Instruct` on a mathematical reasoning dataset and uncover a surprising result: training with only negative samples — without reinforcing correct responses — can be highly effective: it consistently improves performance over the base model across the entire Pass@$k$ spectrum $k$ up to 256), often matching or surpassing PPO and GRPO. In contrast, reinforcing only correct responses improves Pass@1 but degrades performance at higher $k$, due to reduced diversity. These inference-scaling trends highlight that solely penalizing incorrect responses may contribute more to performance than previously recognized. Through gradient analysis, we show that NSR works by suppressing incorrect generations and redistributing probability mass toward other plausible candidates, guided by the model's prior beliefs. It refines the model's existing knowledge rather than introducing entirely new behaviors. Building on this insight, we propose a simple variant of the RL objective that upweights NSR, and show that it consistently improves overall Pass@$k$ performance on MATH, AIME 2025, and AMC23. Our code is available at [`https://github.com/TianHongZXY/RLVR-Decomposed`](https://github.com/TianHongZXY/RLVR-Decomposed).

The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
Xinyu Zhu Mengzhou Xia²
Zhepei Wei! Danqi Chen? Yu Meng'
Wei-Lin Chen'
Computer Science Department, University of Virginia Princeton Language and Intelligence (PLD, Princeton University {xinyuzhu, zhepei wei, wwwcen, yumeng5) (evirngg/vviiiiiiii edu (mengzhou, danqic]@cs www.e..p..netttee princeton. edu
Abstract
Reinforcement learning with verifiable rewards (RLVR) promising approach for training language models (LMs) on reasoning tasks that clicit emergent long chains of thought (CoTs). Unlike supervised learning, it updates the model us- ing both correct and incorrect samples via policy gradients. To better under- stand its mechanism, we decompose the learning signal into reinforcing correct responses and penalizing incorrect ones, referred to Positive and Negative Sample Reinforcement (PSR and NSR), respectively. We train Qwen2. 5-Math-7B. Qwen3-4B and Llama-3.. Ins and uncover surprising result: training with only negative samples-withou reinforcing correct responses- can highly effective: consistently improves performance over the base model across the entire Pass @ k spectrum (k up to 256), often matching surpassing PPO and GRPO. In contrast, reinforcing only correct responses improves Pass@1 but degrades performance at higher kt due reduced diversity. These inference-scaling trends highlight that solely penalizing incor- responses may contribute more performance than previously recognized. Through gradient analysis, we show that NSR works by suppressing incorrect generations and redistributing probability mass toward other plausible candidates, guided by the model's 's prior beliefs. It refines the model's existing knowledge rather than introducing entirely new behaviors. Building on this insight, we propose a sim- ple variant of the RI. objective that upweights NSR, and show that it consistently improves overall Pass @k k performance on MATH AIME 2025, and A.C223. Our code is available at https: // /////ttiat ggggtuu........ com htt.:/aiinnonn///LWW/W /RLVR Wew.-----eeooodd.....o.
IenB-Instruce truct on a mathematical reasoning dataset
Introduction
Language models (LMs) have recently demonstrated remarkable capabilities in various complex reasoning tasks, including mathematics [7, 17], coding [21, 66), and scientific reasoning [37, 40). A key technique in achieving such success is reinforcement learning with verifiable rewards (RLVR) [15, 20, 23, 47]. which is particularly effective in domains where the correctness of outcome can be automatically verified via tools functions. RLVR typically employs binary reward (+1 1) based on the objective correctness of model responses. This simple yet effective mechanism not only mitigates reward hacking [33, 45] but also eliminates the need for extensive human annotations and complex reward model training [27, 77].
RLVR's appeal multifaceted: it offers a conceptually simple formulation [23], exhibits notable sample efficiency [12, 28, 52], and enables inference-time scaling behaviors [13, 35, 60, 68. 74]. However, the precise mechanisms driving its effectiveness remain underexplored, particularly how utilizes correct and incorrect samples. To address this, we decompose RLVR into two learning paradigms: Positive Sample Reinforcement (PSR) and Negative Sample Reinforcement (NSR), as
39th Conference Neural Information Processing Systems (NeurIPS 2025).

---

E: : Alice has 3 apples. She buys 2 more and gives 1 to her friend. How many apples does she have now?
yi Ta (3/ x))
Positive Sample Reinforcement (PSR)
y/: 3 + 4. So Alice has 4 apples.
max (3)
Pass@
22: Start 3, buy 2 (now 5), give (now 4).
8
Pass
LLM Te (3/ ())
y3: 2 So the answer apples.
Negative Sample Reinforcement (NSR)
max (y()) 0
Pass@1
y4 We can't tell how many she has
Pass @32
Figure 1: Decomposing learning signals in RLVR into positive and negative reward components. Positive Sample Reinforcement (PSR) increases the likelihood of correct responses and improves Pass@1 1, but reduces output diversity and hurts Pass @ass@k k for large k. Negative Sample Reinforcement (NSR) discourages wrong responses and redistributes probability mass according to the model's 's prior knowledge, improving the full Pass@ l: spectrum.
illustrated in Figure This decomposition prompts a natural question: What roles do PSR and NSR play in shaping model behavior and generalization?
To empirically isolate the effects of PSR and NSR, we train two LMs, Qwen2. 5-Math-7B and Qwen3-4B, either PSR or NSR exclusively, and evaluate their inference-time performance across range Pass@k & metrics. We find that PSR-only training improves Pass@1 but hurts Pass @ at larger k values, indicating loss of output diversity and exploration capacity. More strikingly, NSR-only training consistently improves performance over the base LM across the entire Pass @ spectrum and, in many cases, matches or even surpasses the performance of PPO [41] and GRPO [15, 43].
To understand why NSR alone is so effective, we conduct a token-level gradient analysis and demonstrate that NSR works by suppressing incorrect reasoning steps and redistributing probability mass towards other plausible candidates already favored by the model's prior. This effectively refines its existing knowledge without aggressively teaching new behaviors. PSR, in contrast, sharpens the output distribution around sampled correct paths [1. 16, 61]. often at the cost of suppressing alternative valid solutions. This suggests that NSR plays a pivotal role in preserving diversity and promoting generalization, especially when the base model already encodes strong reasoning priors.
Motivated by this insight, we propose Weighted- REINFORCE, simple yet effective variant of the REINFÖRCE [58] objective that upweights its NSR contribution. We demonstrate that this adjustment consistently improves the Pass @ l performance on MATH, AIME 2025, and AMC23, yielding an overall better result than strong baselines including PPO [41] and GRPO [15, 43].
Weighted-REINFORE
Our contributions in this work are follows:
We decompose RLVR into two components, PSR and NSR, and investigate their distinct impacts on model behavior and generalization measured by range of Pass@ k: metrics.
We empirically demonstrate the surprising effectiveness of NSR-only training and use gradient analysis to show that NSR refines the model's prior by suppressing incorrect reasoning steps and preserving plausible alternatives.
We propose Weighted-REINNFORCE a simple modification to the RI. objective that upweights NSR. yielding consistent gains across complex reasoning benchmarks including MATH, AIME 2025, and AMC23.
RLVR Objective and Decomposition
2.1 Reinforcement Learning with Verifiable Rewards
Reinforvement learning with verifiable rewards (RLVR) has recently emerged as a powerful paradigm for enabling LMs to self-improve tasks with objectively verifiable outcomes. The reward is given by deterministic verification function which assesses whether the model's response y to the prompt æ is correct or not. All tokens {y1, /2,...,, ..r. in a response receive the same reward (e.g., +1 for correct responses and 1 for incorrect ones).

---

Formally, given an LM with parameters 8, set of prompts D from which a is sampled, and verifiable reward function r, RLVR learns a policy Te to minimize the following objective:'
CRLVR (0) - D. (a, v)]. r(x, y) E {-1,+1}.
(I)
In common RL algorithms (e.g., PPO [41], GRPO [15, 43]), rewards are further normalized to stabilize training, enforcing zero mean per batch B (i.e., y (r), y)] = 0).
2.2 Decomposing RLVR into Positive and Negative Sample Reinforcement
While RLVR has demonstrated promising empirical results, its underlying learning dynamics remain underexplored. In particular, it is unclear how the model updates its behavior under this binary outcome reward setting. What the model learns when receiving both positive and negative rewards can be entangled and hard to interpret. To better understand these dynamics, we begin by decomposing the RLVR objective into two distinet learning paradigms: learning from correct responses and learning from incorrect responses. This decomposition allows us to investigate how positive and negative reward signals shape model behavior, which we will show in the next section.
The RLVR objective optimizes the expected eeward-weighted likelihood:
[RIVR(0) = -EAND
Σ r(x, v) To To(())))
r(x, v) E (-1,++1}
(2)
-EEw
x))
EEND
Σ
To (y(
(P)
Cusa (0)
where we define two sub-objectives representing each learning paradigm:
CPSR(() = -EEND
Σ
To (3/ [z)
(3)
(=r(2,,))=1
LNSR(0) = -BE-D
Σ
No (2(
(4)
We refer to these two learning paradigms as positive sample reinforcement (PSR) and negative sample reinforcement (NSR). The positive reward case resembles supervised fine-tuning (SFT), where the model is updated to increase the likelihood of correct responses. In contrast, the negative reward case mirrors likelihood minimization that reduces the probability assigned to incorrect responses. Notably, PSR and NSR on-policy-the responses are sampled from the model itself during training.
Thus, the full RLVR objective decomposes into two sub-objectives LRVVR (0) LrsR (0) + LNSR(() This decomposition reveals that RLVR jointly performs PSR on positively rewarded samples and NSR on negatively rewarded ones. To better understand the individual effects of these two learning paradigms on model behavior, we conduct experiments train LLMs with each sub-objective independently, as well as the full RLVR objective for comparison.
3 Positive and Negative Sample Reinforcement for LLM Reasoning
3.1 Experimental Setup
Models. To understand how different training objectives affect the model's behavior, we train the model with PSR and NSR and evaluate their inference scaling performance on reasoning tasks. Specifically, we use Quen2. 5-Math-7B [65], Qwen3-4B 4B [64] and Llama-3. 1-w.--Innnttuueee [11]
'hile regularization mechanisms such as KL penalties clipping are commonly used practice to stabilize training. the fundamental learning signal still stems from the verifiable reward. Therefore, to clearly understand how the model learns from success and failure, we base our analysis primarily on the reward learning loss. We discuss in Section 4.3 how analysis applies to PPO and GRPO.

---

MATH
AIME2025
AMC23
160
60
100-
00
30-
80-
Pooli
Pee
80
70-
Makk
Maki
7PO
70-
cecuples)
sampuue)
of recuples)
Figure 2: Pass@A & curves of Q 5-Math-7B trained with PPO, GRPO, PSR, and NSR. NSR is comparable other methods across different k values and outperforms them at k 256.
as the base models. Qwen3-4B has two modes (thinking and non-thinking), and we use the non- thinking mode for training and inference.
Compared algorithms. We compare the performance of PSR and NSR with commonly used RL algorithms, including PPO [41] and GRPO [15, 43]. PSR and NSR are implemented by selectively updating the policy model using only correct or incorrect responses, respectively. As a result, PSR and NSR are trained on fewer samples per batch than standard RL algorithms (e.g., PPO and GRPO) that use both correct and incorrect responses. The training objectives of these algorithms can be found in Appendix D.1. We also report the performance of the base models for reference.
Training setup. For the training set, we use MATH [17]. which contains 7,500 problems. We train the models using the verl framework [44]. The prompt batch size is 1,024, with 8 rollouts generated per prompt. The sampling temperature during training is set to 1.0, and the maximum context length set to 4,096, 4,096 and 32,768 tokens for 5eean..--Mah---- Qwen3-4B respectively. We update the model with a mini-batch size of 256 and a learning rate of 1e-6. More hyperparameter settings can be found in Appendix D.I.
ama-333. -3 1-81-I8Inmn --B-Insttuct and
Evaluation setup. We evaluate three widely used math reasoning benchmarks, including the test sets of MATH, AIME 2025 and AMC23. During evaluation, we sample 256 responses per prompt for 1eeen2.5-Math-22 and Llama-3. 1-....8-BBimm 1-8B-Instruct with a temperature of 0.6 and a top- p of 0.95, and 64 responses for Qwen3-4B with temperature of 0.7, top-p of 0.8 and top-k of 20, prompt templates can be found in Appendix D.2. For evaluation metric, recent work [18] highlights that accuracy based on greedy decoding can be unreliable. For this reason, we adopt full spectrum of Pass @k k as main evaluation metric, using & E 21 4. 8, 16, 32, 64, 128, 256} for Qwen2. 5 G-Math-7B and & E {1, 2, 4, 8, 16, 32, 64) for Qwen3-4B. Pass @ is defined as the fraction of problems for which at least one correct response is produced in k independent trials. However, directly computing Pass k using only k samples per example often suffers from high variance. We follow the unbiased estimator proposed by [5], which generates n samples per problem (n 2 k), counts the number of correct responses and computes an unbiased estimate of Pass @ k as:
Pass@k E
(5)
Notably, varying k provides insights into different aspects of model behaviors. Pass@1 approximates greedy decoding accuracy, reflecting how confidently the model can produce a correct response in single attempt, essentially reflecting exploitation. In contrast, Pass@k with large A evaluates the model's ability to generate diverse correct responses across multiple attempts, capturing its exploration ability and reasoning boundary.
3.2 Inference Scaling Trends Under Different Training Objectives
NSR alone is surprisingly effective. As shown in Figures 2 and 3, NSR exhibits unexpectedly strong performance across the full range of l values. Despite being trained solely negative samples, consistently improves Pass@ compared the base model. While PPO, GRPO, and PSR explicitly reinforce correct responses and naturally outperform the base model at Pass 1, it is surprising that NSR achieves comparable Pass @ without training any correct responses. This suggests that

---

MATH
AIME2025
AMC23
160
80
Pheeill 60
Pooli
00
80
7TO
85-
70-
of comsples)
sxmpues)
of comalees)
Figure 3: Pass @ curves of Qwen3-4B (non-thinking mode) trained with PPO, GRPO, PSR, and NSR. NSR consistently performs competitively across varying & values, while PSR does not improve the base model.
MATH
AIME2025
AMC23
90-
GEENO
GEEO
GELOO
30
158
[ssmmbee stmpppe))
[sumbbee sampuus)
comuless
Figure 4: Pass @asss&& & curves of Llama-3 3. Innttrccttt trained with PPO, GRPO, PSR, and NSR. All the methods underperforms the base model, while NSR retains the most performances.
NSR is able to reinforce correct responses indirectly by suppressing incorrect ones and redistributing probability mass toward plausible alternatives.
NSR outperforms or stays comparable to the base model at a large & value. At larger decoding budgets (e.g., Pass @ 256), recent work [72] shows that RI--taieee models often lose their advantage, and in some cases underperform the base model. This trend is generally observed in our experiments with PPO, GRPO, and PSR, especially in Figure 2. NSR, on the other hand, maintains a comparable or even better Pass @ 256 performance than the base model, generally outperforming the other algorithms. These results suggest that NSR promotes exploration and preserves the output diversity.
PSR improves accuracy at the cost of diversity. In contrast, PSR, which only reinforces correct samples, displays a more polarized behavior. It improves Pass@ I, particularly on AIME 2025 and AMC23 in Figure 2. However, this precision comes at a cost: as k increases, Pass @ improves more slowly than other methods and eventually falls below the base model for k 8. As shown in Figure 5a, PSR improves greedy decoding accuracy rapidly during early training but plateaus quickly and is overtaken by other methods. This behavior indicates that PSR overly concentrates probability mass on early correct responses, leading to overconfidence and a collapsed output distribution and ultimately limiting the model's ''s ability to generate diverse correct responses when allowing for more test-time compute.
PSR fails to unlock the model's latent reasoning capabilities. Figure 3 demonstrates the per- formance of training Qwen3-4B under non-thinking mode to simulate scenario where the model's underlying knowledge known to be strong (i.e., its learned thinking mode triggered by the '<think>' tag). While one may intuitively expect all algorithms to casily transfer the model's reasoning ability across different prompt formats (i.c., from thinking to non- non-thinking mode), our results demonstrate significant performance differences across different algorithms. PSR fails activate these latent capabilitics and does not improve the performance, even degrading Pass & on MATH and AMC23 significantly. This highlights a fundamental limitation of PSR: it reinforces the currently dominant be- havior the output distribution, suppressing potentially stronger underlying capabilities. In contrast, NSR and GRPO significantly improve the performance of the base model across all Pass@ k metrics, achieving thinking mode capabilities. Specifically. Qwen3-4B in thinking mode achieves a Pass @1 of 94.5 and Pass @ 64 of 97.8 on MATH test set. NSR and GRPO closely match this performance. with NSR reaching 94.0 (Pass @ 1) and 98.0 (Pass @ 64), and GRPO achieving 93.9 (Pass@) @ 1) and 98.2 (Pass @64).
5

---

19
76
0.10
76-
Accurocy
Entropp
R.88
124
0.91
GRPO
GRRP
0U92
NSR
XSR
20
80
100
20
90
10C
121
(a)
(b)
Correct Semple Ratio
0.8
Fully Solved Ratio
0.7
0.4-
0.6
PPO GRP)
F2O
GRPO
PSR
PSR
0.6-
20
100
120
010
20
60 80 300
120
(c)
(d)
Figure 5: Training dynamics Quen2. 5--n2.5-M on MATH under PPO, GRPO, PSR and NSR across training steps, including (a) greedy decoding accuracy on the test set, (b) the model's entropy on the test set, (c) the ratio of correct responses per batch on the training set, and (d) the proportion of fully-solved prompts per batch (i.e., all rollouts correct) the training set. NSR achieves competitive performance in greedy decoding accuracy while maintaining substantially higher entropy throughout training, suggesting greater exploration. NSR improves both the correct sample ratio and the fulle-solved sample ratio, though less aggressively compared to other algorithms.
RL training hurts Llama models' inference scaling performance. As shown in Figure 4, ap- plying different RL algorithms to Llama-3. 1-8B- Instruct leads to degraded inference-scalle performance compared to the base model. Although @ass@1 improves on MATH and AMC23, Pass @ 256 drops substantially after RL training. This contrast between Qwen and Llama models highlights that backbone model itself often determines whether RL can bring benefits, consistent with recent works [72, 53, 68]. Among methods, however, NSR causes the least degradation to the base model's scaling behavior.
Understanding the Effectiveness of Negative Sample Reinforcement
To better understand the learning mechanisms of PSR and NSR, and explain why NSR consistently demonstrates strong inference scaling performance, we analyze their training dynamics through both empirical observations and gradient analysis.
4.1 Entropy as Lens on Inference Scaling Performance
Since diversity is crucial for strong @ass@k ceeermancee-seccialllly at large k-we quantify model diversity by tracking its entropy on a held-out test set throughout training, aiming to understand how entropy evolves under different training algorithmss In addition, monitor two complementary metrics on the training set: the correct sample ratio, which measures the proportion of correct. responses per batch, and the fully solved ratio, which measures the fraction of problems per batch where all rollouts correct.
Figure 5b shows that NSR maintains high level of entropy on the held test set throughout training, closely matching that the base model, while PSR leads rapid and substantial drop in entropy. PPO and GRPO offer middle ground: they gradually reduce entropy during training, with levels that fall between those of PSR and NSR. Nevertheless, their entropy still declines considerably, which likely limits output diversity and helps explain why their Pass @ 256 performance remains below that of the base model. Interestingly, PPO exhibits slight rebound in entropy during the later

---

Positive Sample Reinforcement (PSR)
Roward:
Negative Sample Reinforcement (NSR)
Reward:
Apples cost $2 each oranges each,
cost 57.
Apples cost $2 each and oranges $1 each, apples and 3 oranges cost $6.
Logit
Exploitation: Sharpen cornect token'' probsbiiity supprees others
Logit
Exploration: Redistribute logits propartinn to alternative tokenss current probsbillt
pC
A
&C
A
A
Logii
ÖLogit
Vocabulary
Vocabulary
"go"
"on"
"go"
suppress correct atternatives
promote correct atternatives
80 apples and 3 oranges cost
=1
2 apples and oranges cost x
31
Figure 6: Gradient dynamics of PSR and NSR under math word problem example. Bars indicate token logits before the update, and arrows indicate gradient direction and magnitude. Left (PSR): The model generates the correct response ("7") and receives +1 reward. Gradients increase the logit of the sampled token while suppressing others, including potentially correct alternatives like "2", resulting in sharpened, overconfident distribution. Right (NSR): The model generates an reward. Gradients demote the sampled incorrect token and incorrect response ("66) and receives
proportionally reallocate logits to other tokens (e.g.. "7", "2") based on their current probabilities, thereby promoting exploration on alternative correct paths and preserving diversity.
stages of training- trend not seen in GRPO. This divergence may stem from PPO's use of a critic model, which can provide potentially more fine-grained and exploratory advantage estimates.
As shown in Figures 5c and 5d, NSR consistently improves the correct sample ratio but maintains a lower proportion of correct samples and fully solved problems throughout training. This indicates that the model avoids becoming overconfident in the observed correct responses. By preserving uncertainty. NSR enables stronger scaling performance in Pass @ k. as demonstrated in Figure In contrast, PSR rapidly increases the number of correct samples in each batch and achieves higher fully solved ratio compared to other methods, suggesting tendency to overfit to the observed correct responses. While this leads to better @ass@1 performance, significantly reduces output diversity and results in weaker Pass k performance as k increases.
These trends underscore the importance of preserving output entropy during RL and highlight key insight: reinforcing negative samples alone can improve accuracy without compromising the base model's generation diversity.
4.2 Token-Lever Gradient Dynamics of PSR and NSR
To gain deeper understanding of the training dynamics of PSR and NSR, we conduct token-level gradient analysis. The loss for both PSR and NSR training instance (ær, y) takes the form:
exp(2ve)
[(0) -
T Σ Tolye yet) RR
Evey exp(zxr))
where R r(æ, y) +-1 denotes the reward assigned to the sampled trajectory, and 2. denotes the logit corresponding to token v in the vocabulary V.
To analyze the effect of these objectives on the model's token distribution, we compute the gradient of the loss with respect token-level logits each step. Let π. To(v) y<r)) denote the probability of token time step t. the gradient descent directions of PSR and NSR are shown Equations (7)
and (8) (the full derivation is provided in Appendix A), which are illustrated in Figure 6.
ÖLPSR Izn
[ To (1- TT)
if e = Br (sampled token)
-TTy. Te
i Fv ye (unsampled token)
This formulation clearly shows how PSR increases the logits of tokens appearing in correct responses while decreasing the logits of all other tokens. As training progresses, it repeatedly amplifies the probability of observed correct alternative generations. This continual sharpening of the output distribution can lead to reduced entropy and overfitting shown in Figure 5b, especially in on-policy settings where the same
pushing their likelihoods toward 1, while suppressing
7

---

examples may be encountered frequently. Over time, the model's behavior may collapse onto narrow set of responses, limiting its ability explore generalize beyond what has been reinforced.
① ÖLNSR
Te
(1 TT)
if e (sampled token)
Zzy
ATg
v + yt (unsampled token)
In contrast, NSR works by penalizing the logits of tokens incorrect responses and softly redistribut- ing probability mass other candidate tokens. Importantly, NSR increases other tokens' logits in proportion to their current likelihoods. This objective has several desirable properties:
Key Insights into NSR via Gradient Analysis
When the model assigns high probability to certain tokens (i.e., Ty, 1) that appear in incorrect outputs (c.g., common grammatical or linguistic constructions), the negative gradient from NSR is scaled by (1 - Ty. resulting small updates. This allows NSR to penalize mistakes without erasing fundamental knowledge learned during pretraining.
Preserving high-confidence priors:
2. Prior-guided probability redistribution: NSR performs a soft reranking of the output distribution by boosting unsampled tokens' logits 2, in proportion to their current proba- bilities k.. This allows the model effectively explore and search for better candidates according to their prior beliefs.
3. Implicit regularization against overfitting: NSR updates only when the model generates incorrect responses. Once the model consistently avoids these mistakes, NSR naturally halts further updates those samples. This stopping criterion prevents the model from overfitting or collapsing diversity in examples it has already mastered.
These analyses highlight an important strength of NSR: preserves the model's prior knowledge over plausible tokens and stops updating once errors are corrected, effectively locking in successful experiences. This is unique advantage of NSR, as we demonstrate in Appendix B via gradient analysis that simply applying an entropy bonus in the policy loss, despite promoting diversity, cannot preserve model's 's prior. We also include a comparison with the widely used unlikelihood training objective, previous approach designed to suppress undesirable or incorrect generations [57, 26).
4.3 Extending Gradient Analysis to PPO and GRPO
Our earlier analysis was based on simple REINFORCE-like objective [58]. We now analyze how extends to PPO and GRPO. The losses for GRPO and PPO are as follows:
To(Ue 12; V&c) Tou U v.c
To(yel Vce) ToalUle m. yer) E, 1 Ar).
(0) T
Σ
min
As, clip
6
To( Vi.ce) ToaaVi pr vt)
To(yiie Miicce) Rota(Vii x ve.ca)
(ouro(!)
Σ
min
e, +e Au.
T
1m1
where denotes the advantage. Comparing the PPO and GRPO objectives Equation (6), there are three key differences: (1) policy loss clipping, (2) KL regularization (reward penalty PPO, loss in GRPO), and (3) advantages instead of raw rewards. We analyze their impact on the gradient dynamics below:
1. Clipping constrains update magnitude when the new policy diverges significantly from the old one, but preserves the gradient update direction, thus leaving our analysis qualitatively unchanged.
2. KL regularization discourages deviation from the reference policy. While it may dampen positive and negative updates, its practical impact is often negligible: for reasoning tasks, the KL coeffi- cient either very small or completely removed, leading to better performance, as demonstrated in recent works [6, 62, 69].
3. The advantage of GRPO A; mean(r) acts as rescaling of the gradient and retains the sid(r)
reward's s sign. In PPO, the advantage is computed relative to value function: tokens that

---

Table 1: Pass & results MATH, AIME 2025 and AMC23 with 5wen2.5-Math-- Bold and underlined numbers denote the best and second- best results for each k.
Method
Pass@k
k
8
16
32
64
128
256
MATH
Base Model
63.2
76.0
83.7
88.4
91.6
93.7
95.2
96.2
96.9
PPO
76.6
82.6
86.7
89.6
91.7
93.4
94.7
95.6
96.3
GRPO
76.3
81.7
85.6
88.4
90.6
92.3
93.6
94.7
95.5
REINFORCE
74.8
79.1
82.4
84.9
86.9
88.5
89.9
91.1
92.0
PSR
74.1
78.3
81.6
84.1
86.2
87.9
89.3
90.4
91.2
NSR
75.7
82.4
86.9
90.1
92.4
94.1
95.3
96.2
96.9
W-REINFORCE
76.6
82.8
87.1
90.2
92.4
94.1
95.3
96.1
96.7
AIME 2025
Base Model
6.1
9.7
13.8
17.9
22.2
26.5
30.8
36.6
46.7
8.5
18.0
22.5
26.6
30.3
GRPO
10.3
14.7
19.4
24.0
28.4
32.8
37.3
42.5
50.0
REINFORCE
9.2
12.5
16.3
21.0
26.4
32.3
38.6
44.7
50.0
PSR
11.6
14.1
16.2
18.6
21.7
25.7
30.9
36.9
43.3
NSR
10.0
24.1
29.3
34.6
46.0
W-REINFORCE
10.6
15.3
20.0
24.7
29.7
34.6
40.5
47.8
56.7
AMC23
Base Model
41.0
56.2
69.2
78.9
85.1
89.1
92.9
97.2
100.0
PPO
62.0
70.0
76.1
80.9
85.3
89.5
93.1
96.0
97.5
GRPO
61.7
68.7
74.6
80.0
85.1
89.7
93.4
95.9
97.5
REINFORCE
61.8
68.5
73.4
77.0
80.8
85.1
89.3
91.9
92.5
62.6
69.9
74.5
77.5
80.3
83.5
87.2
90.6
92.5
NSR
60.9
70.0
77.4
83.2
87.6
91.1
94.5
97.9
100.0
W-REINFORCE
62.0
70.0
77.0
83.1
87.8
91.8
95.2
97.1
97.5
outperform the baseline are reinforced, while other tokens are penalized. This yields finer-grained credit assignment but does not alter the overall gradient syynamics-postine reinforcement still reduces diversity, and negative reinforcement promotes alternatives under the model prior.
Therefore, while PPO and GRPO introduce modifications that stabilize learning, the core gradient behaviors identified in our previous analysis still hold. We further discuss the differences between RLVR and RL with reward models in Appendix
5
Balancing Positive and Negative Reinforcement
Our previous analyses reveal trade-off in reinforcement learning objectives: PSR improves Pass quickly but sacrifices performance on Pass @k & for larger k, whereas NSR preserves @ but may underperform when & is small. To strike better balance between accuracy and diversity, we propose simple weighted combination of PSR and NSR: we scale down the reward magnitude for PSR by a factor of and combine it with NSR, allowing the model to learn from both correct and incorrect samples. When À = 1, it exactly the same REINFORCE, We refer to this method as Weighted-REINFORCE (W-REINFORCE),
(w-REINFORCE(() EEED
X TT(3) (rr(2,))=1
Σ
-EAD
Σ
(3(()
(9)
A-
Even (0)
We apply À 0.1 in our experiments. As shown in Table 1, W-REINFORCE consistently delivers favorable trade-off across a range of values on MATH. AIME 2025 and AMC23, On MATH. W W-REINFORCE matches the best Pass @ score 76.6 with PPO and has the highest performance for all & < 64, while still preserves competitive performance at & 256. On AIME 2025, W- the best result across all & values except = 1. These results confirm that W-REINFORCE, simple weighted extension of the classic REINFORCE algorithm, strikes an effective balance between the strengths of PSR and NSR. By
REINFORCE performs even more atrongly-achievin

---

merely scaling down the weight of positive rewards, it achieves strong performance while preserving diversity. Despite its simplicity, W-REINFORCE consistently outperforms strong RL algorithms such as PPO and GRPO across most Pass k. while the vanilla REINFORCE underperforms. These findings suggest that this simple variant of REINFORCE can serve competitive alternative to more complex RL algorithms when the model prior is strong (e.g.. Qwen models). We provide an ablation study on the choice of À in Appendix E.
6 Related Work
Reinforcement learning with verifiable rewards. Reinforcement learning has shown great promise in improving large language models, as demonstrated by the success of reinforcement learning from human feedback (RLHF) and from AI feedback (RLAIF), which align model responses with human preferences [25, 32, 36, 39j. More recently, reinforcement learning with verifiable rewards [8. 14. 16, 23, 31, 43, 48, 52, 63, 69, 70, 71, 76, 54] has attracted growing attention for its ef- fectiveness in incentivizing reasoning in L.LMs with rule-based rewards [ [12, 15, 22, 47, 74, 10, 24, 34). Notably, DeepSeek-KR R1 [15] and Kimi K1.5 [47] demonstrate that RLVR can elicit emergent rea- soning behaviors such as long chain of thought and self-reflectionn and achieve strong performance across diverse reasoning tasks such as math and coding problems. Yeo et al. [68] further explore the emergence of long CoT across different RLVR setups.
Despite these advances, many prior works mainly focus on evaluating the model's Pass@1 1 or greedy decoding performance, which might overlook the underlying change model behavior (e.g., inference scaling performance). More importantly, the mechanisms behind RLVR for driving reasoning and generalization remain underexplored. In this work, we take a step further by decomposing the RIVR objective into two distinct learning paradigms- incorrect responses. We analyze the learning signal at the token level through gradient analysis and evaluate the models extensively using Pass @k k with wide spectrum of k. Our findings highlight the critical yet previously underappreciated role of negative rewards in RLVR.
from correct responses and learning from
Inference-time scaling behaviors. Inference-ting scaling has emerged promising direction for enhancing model performance [2, 3, 4, 29, 35, 38, 42, 46, 49, 50, 51, 56, 59, 80, 67, 73, 78, 55] particularly in reasoning tasks where generating multiple candidate solutions [7. 80] or longer reasoning traces [35, 68, 75] can help hit the correct answer. Through the lens of inference-time scaling, recent studies have raised questions about whether RL-trained models are better than the base models [72. 18]. A recent work by [72] suggests that RLVR primarily adjusts the model's 's output distribution toward high-reward rather than eliciting new reasoning abilities beyond the base model. By adopting Pass@k l as metric. they find that RL-trained models have inferior inference- -ime scaling performance than the base models. These findings align with our findings that reinforcing correct samples hurts Pass @ l larger k. Another concurrent work by [9] studies supervised fine-tuning (SFT), and finds that diversity collapse during SFT adversely affects inference-tine scaling performance. They show that SFT reduces generation diversity, leading to degraded Pass @ k performance. and that interpolating weights from early, potentially less overconfident checkpoints and later ones can effectively restore diversity and improve both Pass 1 and Pass
In this work, we highlight the underestimated role of negative reinforcement in preserving and improving inference-time tireeneee-teeeeeeee scaling performance, and that the accuracy and diversity trade-off in RLVR can be effectively balanced by adjusting positive and negative reward weights.
7 Conclusion
In this work, we investigate the mechanism underlying RLVR for LM reasoning. By decomposing RLVR into positive and negative sample reinforcement, we reveal a surprising finding: solely penalizing incorrect samples can effectively enhance LM reasoning capabilities while preserving generation diversity. Experimental results show that NSR consistently improves performance across a wide Pass@k k spectrum and in many cases matches or outperforms strong RL. algorithms such as PPO and GRPO. Our gradient analysis demonstrates that NSR works by suppressing incorrect responses and redistributing probability mass toward plausible alternatives based on the model prior. Building that
on these findings, we proposed a simple variant of REINFORCE, Weighted-RFINFORR upweights the negative sample reinforcement. Empirical results show that it achieves a good balance between PSR and NSR, and yields consistent Pass @ improvements across multiple reasoning benchmarks. We discuss limitations and future work directions in Appendix F.
10

---

Acknowledgments
We would like thank the members of the Princeton NLP Group for their valuable feedback and discussions. We also thank the anonymous reviewers and the area chair for their time, effort, and constructive suggestions, which have greatly improved the quality of this paper. This research is partially funded by the National Science Foundation (IIS-2211779), the NVIDIA Academic Grant, the Lambda Research Grant, and the OpenAl Superalignment Fast Grant, MX is supported by the Apple Scholars in AI/ML PhD Fellowship.
References
[1] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 5366-5376, 2017.
[2] Vidhisha Balachandran, Jingya Chen, Lingjiao Chen, Shivam Garg, Neel Joshi, Yash Lara, John Langford, Besmira Nushi, Vibhav Vineet, Yue Wu, et al. Inference-time time scaling for complex tasks: Where we stand and what lies ahead. arXiv preprint 1.XXi:22504002294, 2025.
[3] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large Language Monkeys: Scaling inference compute with repeated sampling. arXiv preprint 2rXiv:2407.217887 2024.
[4] Lingjaa Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei A Zaharia, and James Y. Zou. Are more LLM calls all you need? Towards the scaling properties of compound Al systems, Advances in Neural Information Processing Systems, 37:45767-45799 457900, 2024.
[5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating Large Language Models trained on code. arXiv preprint arXiv:2107.03374, abs/2107.03374 2021.
[6] Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. GPG: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint wrXiv:2504.02544 2025.
[7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[8] Muzhi Dai, Chenxu Yang, and Qingyi Si. S-GRPO: Early exit via reinforcement learning in reasoning models. arXiv preprint M.Xiv:2505.07666. 2025.
[9] Xingyu Dang. Christina Back, Kaiyue Wen, Zico Kolter, and Aditi Raghunathan. Weight ensembling improves reasoning in language models. arXiv preprint arXiv:2504. 10478, 2025.
[10] Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF Workflow: From reward modeling to online RLHF, arXiv preprint 2rXiv:2405.07863. 2024.
[11] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle. Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson. Ava Spataru. Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne
11

---

Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits. David Esiobu. Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino. Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, llyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang. Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li. Kenneth Heafield, Kevin Stone, and et al. The Llama 3 Herd of Models. CoRR, 0/bs/2407.2178. 2024.
[12] Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.051185 2025.
[13] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint wxXi:2553300300 2025.
[14] Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective RI. reward at training time for LLM reasoning. arXiv preprint arXiv:2410.1511: 2024.
[15] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. ReepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint 220::229....948.. 2025.
[16] Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.0400 04642, 2024.
[17] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurlPS Datasets and Benchmarks, 2021.
[18) Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. arXiv preprint arXiv:2504.0700
2025.
[19] Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Shum. Open-Reasoner-Zerro An open source approach scaling reinforcement learning on the base model. https://p/ thub. com/ Opaan Rewe--Raassneeeeeeee Opeeerrr/pp/eeee Setaaannnr--eer//ppee--aaaaaeeee Reasoner-Zero 2025.
[20] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed BI- Bl-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. OpenAI ol system card. arXiv preprint arXiv:2412.1672 2024.
[21] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
[22] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Unlocking RL potential for LLM reasoning through refined credit assignment. arXiv preprint arXiv:2410.0167, 2024.
[23] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. TÜLU 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.133
2024.
[24] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. RewardBench: Evaluating reward models for language modeling, arXiv preprint 2403 2024.
12

---

[25] Harrison Lee, Samrat Phatale, Hassan Mansoor. Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, al. RLAIF vs. RLHFF Scaling reinforcement learning from human feedback with Al feedback. In International Conference on Machine Learning, pages 26874-26901. PMLR, 2024.
[26] Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
pages
[27] Wendi Li and Yixuan Li. Process reward model with Q-value rankings. In The Thirleenth International Conference on Learning Representations, 2025.
[28] Xuefeng Li, Haoyang Zou, and Pengfei Liu. LIMR: Less is more for rl scaling, arXiv preprint arXiv:2502. / 1886, 2025.
[29] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
[30] Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in large language models. CoRR, abs/2505. 24864,, 2025.
[31] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding Rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
[32] Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122235, 2024.
[33] Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. InfoRM Mitigating reward hacking in RLHF via information-theoreti reward modeling. In The Thirty- eighth Annual Conference Neural Information Processing Systems, 2024.
[34] Youssef Mroueh. Reinforcement learning with verifiable rewards: GRPO's effective loss, dynamics, and success amplification. arXiv preprint arXiv:2503.06633) 2025.
[35] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501. 19393, 2025.
[36] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo A Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[37] Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's 's last exam. arXiv preprint arXiv:2501.14249, 2025.
[38] Yuxiao Qu, Matthew YR Yang. Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine-tuning. arXiv preprint Xxiv:2503.00572. 2025.
[39] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53377 -33411,, 2023.
[40] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level google proof q&a benchmark. In First Conference on Language Modeling, 2024.
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017."
Policy Optimization algorithms. arXiv preprint
13

---

[42] Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification RL is suboptimal. arXiv preprint urXiv:2502.121/6 2025.
[43] Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.0338 2024.
[44] Guangming Sheng, Chi Zhang. Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. arXiv preprint arXiv:2409. 19256, 2024.
[45] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and charac- terizing reward gaming. In Advances in Neural Information Processing Systems, volume 35, pages 9460-9471, 2022.
[46] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute op- timally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.
[47] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjuu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi K1.5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501. /2599, 2025.
[48] Prime Intellect Team, Sami Jaghouar, Justus Mattern, Jack Min Ong, Jannik Straube, Manveer Basra, Aaron Pazdera, Kushal Thaman, Matthew Di Ferrante, Felix Gabriel, et al. INTELLECT- 2: reasoning model trained through globally decentralized reinforcement learning. arXiv preprint arXiv:2505.0779) 2025.
[49] Jun Wang, Meng Fang. Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. OpenR: An open souree framework for advanced reasoning with large language models. arXiv preprint arXiv:2410.00 19.7991,, 2024.
[50] Peiyi Wang. Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 9426-9439 (0669993,,400 2024.
[51] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self- Colf-Consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
[52] Yiping Wang, Qing Yang, Zhiyuan Zeng. Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571, 2025.
[53] Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu, Thinkrin Mid-training Incentivizes Reinforcement Learning Scaling. CoRR, abs/2506.2055 20512, 2025.
[54] Zhepei Wei, Xiao Yang, Kai Sun, Jiaqi Wang. Rulin Shao, Sean Chen, Mohammad Kachuee, Teja Gollapudi, Tony Liao, Nicolas Scheffer, Rakesh Wanga, Anuj Kumar, Yu Meng, Wen-tau Yih, and Xin Luna Dong. TruthRL: Incentivizing truthful L.LMS via reinforcement learning. arXiv preprint 25X9::2509227660 2025.
[55] Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, and Lihong Li. WebAgent-Rl: Training web agents via end-to-end multi-turn reinforcement learning. In The 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), 2025.
[56] Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ila Kulikov, and Zaid Harchaoui. From decoding to meta-generation: Inference-time algorithms for large language models. arXiv preprint arXiv:2406. 16838, 2024.
[57] Sean Welleck, Ili Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference on Learning Representations, 2020.
14

---

[58] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8:229-256, 1992.
[59] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for LLM problem-solving. In The Thirteenth International Conference on Learning Representations, 2025.
[60] Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in LLMs. arXiv preprint arXiv:2502.07266, 2025.
[61] Wei Xiong, Jiarui Yao. Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, al. A minimalist approach to LLM reasoning: from rejection sampling to reinforce. arXiv preprint urXiv:2504. 11343, 2025.
[62] Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not all rollouts are useful: Down- sampling rollouts in LLM reinforcement learning, arXiv preprint arXiv:2504.1388 2025.
[63] Yuhui Xu, Hanze Dong, Lei Wang, Doyen Sahoo, Junnan Li, and Caiming Xiong. Scalable chain of thoughts via elastic reasoning. arXiv preprint SEtv:2505.00315., 2025.
[64] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.nm 093888 2025.
[65] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-Mait technical report: Toward Mathematical Expert Model via Self-Improvement. arXiv preprint arXiv:2409. 12122, 2024.
[66] John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Olir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528-500 500552, 2024.
[67] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36:1 1809-1 2023.
[68] Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue, Demystifying long cfain-of-thought reasoning in LLMs. arXiv preprint ar 23Xiv:2502.03373, 2025.
[69] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang. Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjin Liu, Wei- Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao. Yonghui Wu, and Mingxuan Wang. DAPO: An open-source Ilm reinforcement learning system at scale. arXiv preprint ar arXiv:253.. 14476. 2025.
[70] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. VAPO: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint a5xiv:2504.05//6 2025.
[71] Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind PPO's collapse in long-CoT? value optimization holds the secret. arXiv preprint arXiv:2503.0149)) 2025.
[72] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
[73] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STaR: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488 2022.
[74] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma. and Junxian He. SimpleRL-Zooo Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar arXiv:2503. 18892, 2025.
15

---

[75] Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Yunhua Zhou, and Xipeng Qiu. Revisiting the test-time scaling of ol llke models: Do they truly possess test-time scaling capabilities? arXiv preprint 250.2250... 2025.
[76] Xiaojiang Zhang, Jinghui Wang. Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, et al. SRPO: A cross-domain imple- mentation of large-scale reinforcement learning LLM. arXiv preprint arXiv:2504. 142866, 2025.
[77] Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu. Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. ArXiv, abs/2501 .07301, 2025.
[78] Eric Zhao, Pranjal Awasthi, and Sreenivas Gollapudi. inference-time search by scaling verification. arXiv preprint arXiv:2502.01839, 2025.
Sample, scrutinize and scale: Effective
[79] Chujie Zheng. Shixuan Liu, Mingze Li, Xionghui Chen, Bowen Yu, Chang Gao, Kai Dang. Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group Sequence Policy Optimization. ArXiv, https/257.1807777 2025.
[80] Xinyu Zhu, Junjie Wang. Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. Solving Math Word Problems via Cooperative Reasoning induced Language Models. In ACL (1), pages 4471 4485. Association for Computational Linguistics, 2023.
16

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and socictal impact. Do not remove the citcklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims have been validated with extensive empirical results and theoretical analyses.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix F.
Guidelines:
17

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We provide theoretical results in Section 4.2 and Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes],
Justification:
We provide detailed experimental settings Section 3.1 and Appendix D.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
18

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All the datasets used in our paper are open-access, and we also provide our codes in the supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/ public/ poolic/guides.. Caliees/CoSSuuubise
CamPooliyy icyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cc/ccc/pplco cobolcc/guides http://ooSSuubimeeeeeee wtt.Summmiiioooooicoyy.o for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
19

---

Justification:
We provide training and test settings in Section 3.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropriate
information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our results based 256 random samplings and calculated with an unbiased estimator.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computer resources used in this work in Appendix D.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / htt///nuuri.../. h/tppp/ccc//chhcccccseeeeeellel
Answer: [Yes]
20

---

Justification:
Our research was conducted in accordance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is methodological and theoretical paper on understanding the reinforcement learning for improving large language model reasoning and does not have direct societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work does not involve releasing data models, there is not high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
21

---

12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credited the creators of the assets used in the paper, and provide URLs to all assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release our code, used datasets and provided detailed documentation the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/modell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
22

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowd-sourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our work only uses LLMs for writing, editing, and formatting purposes that not impact the core methodology, scientific rigor, and originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy pettp::/neerrtie for what should or should not be described.
23

---

A Gradient Derivation
Gradient of Equation (6). For simplicity notation, let Tu = no(ua Bee), and we omit the constant 1 in the equation which effectively scales the gradient magnitude.
T
0C 0z.
1(v ye) exp(2 &g. Evee exp(ze) - exp(2n exp(2+)
R
VEv (x)((2)))
For the sampled token u = y/. the gradient simplifies to
Oz, = R. exp(z,) E Evevens exp(2er) exp(2x)2
dC
Evev exp exp(2x')) ))²
=-R. exp(zx) E EEE exp( zor) exp(=x))
EvEV exp(2x))
exp(zx)
EveEy Ener exp(2v) exp(ze)
R
Eveye (2v)
EVED exp(zv)
R To - mp))
For the unsampled token v & yt. the gradient simplifies to
ac R exp(2y, exp(2x)
Oz.
(Evev expl 2n.)) )) exp(an))
exp(an)
R Evey exp(zz)) Every exp(zv.)
= R Ty. Th
In summary, the gradient of Equation (6) can be expressed as
0L
- -R. n.)
if y/
(sampled token)
02v
R Ty.
if + yt
(unsampled token)
B NSR differs from Entropy and Unlikelihood Training in Gradients
Gradient of entropy regularization. Entropy regularization is also one way to induce diverse output probability distribution. We compute and analyze its gradient dynamics reveal its difference from the NSR objective as follows. The entropy loss is
H(mT)
Σ
Ter log Twl
24

---

and the gradient ascent direction to maximize entropy is
0H
Ohe'
Σ
(1 + log Tet)
dz.
dz.
v'EV
Σ
TH (I(v = v0') - To) (1 + log mp))
N'EV
To (1 + log - To Σ
Tor (1 + log Te,))
J'EV
TT
log T.
Σ N'EV
Tel log Te
can be seen that the gradient sign and magnitude depend on how the bog-probability of token v. log Te. compares the expected log probability over the vocabulary: when token is significantly more probable than expected, it receives a larger negative gradient, pushing its logit down more strongly. In contrast, when token is less probable than average, it receives positive gradient that boosts its logit.
As result, entropy maximization systematically flattens the output distribution by suppressing high- confidence tokens and boosting low-confidence ones. This behavior can conflict with the model's semantically appropriate
prior knowledge: confidently correct tokens eakens-such&a syntactically completions -are are penalized more for being too probable, while completely implausible tokens may be promoted simply because they are unlikely under the current policy.
Gradient of unlikelihood training objective. While NSR shares the goal of penalizing undesirable outputs with unlikelihood training [57, 26], their formulation and resulting dynamics are fundamen- tally different: NSR penalizes incorrect outputs using their probabilities, rather than minimizing -log(1 Te (y/ (=) as in unlikelihood training. This distinction results in different gradient behavior and training dynamics as shown below:
T
Lunlikelihood (0)
Σ
log (Ve x, B2))
(10)
T
Taking the derivative w.r.t token logits 2y yields:
To
if v = yt
(sampled token)
Lunlikelhhodd
Oz
Ty
Te
if + yt (unsampled token)
(11)
Ty.
Comparing Equation (11) against Equation (8), it can be seen that Equation (1I) induces a markedly different update: it pushes down the sampled token ye 's logit and redistributes probability mass to other tokens in proportion to their current probabilities π.. Consequently, confident predictions are penalized more aggressively, which may erode prior knowledge. In contrast, the gradient of NSR (Equation (8)) is dampened by
π.) factor. This has the opposite effect: as confidence in sampled token grows, the gradient update shrinks when that token appears in an incorrect answer. By redueing the penalty on high-confidence tokens, NSR avoids destructive updates and effectively preserves pretrained knowledge.
C RLVR vs. RL with Reward Models
Different from RLVR, many RL setups require training reward models to assign scalar values that indicate the quality of generation (c.g.. in RLHF [36]). During policy optimization, these rewards are often normalized within a batch by subtracting the mean, so that sample's final reward depends on its relative ranking compared to others in the same batch. As a result. a sample's 's reward can change sign (from positive to negative or vice versa) depending on the overall quality of the batch, making it difficult to interpret rewards as absolute indicators of positive or negative samples. In contrast, RLVR
25

---

applies a binary reward (+1/-1). The signs of the rewards are inherently tied to the correctness of individual sequences, each token in a sequence receives the same reward and the batch average reward always remains within [-1,1] thus reward normalization will preserve the original sign. This inherent sign preservation in RLVR is critical for our analysis, as it enables a clean and unambiguous separation into positive and negative learning paradigms based the reward signs.
D
Implementation Details
D.1 Objectives and Training Hyperparameters
The objectives of PPO, GRPO, PSR and NSR are as follows:
(0) T
1
min L
Tol x; Ve.) TTan(y ae, Meee)
A., clip
(eo(vel V Meee)
+ 21)
Tou (Br (e, me
G
To To(U Vlreed
To To 20, Us,ce) (old(Vi. (e, Mi.co) 1 1, + Ant
(0) =
min
Ac,,
T
Told (y1.c Ml.sse)
[KL(Tall
1
T
To (z, Voce)
To(yl x Ucc)) (Vte (r, Vec))
c, c.tt)). r(x,
(o)
min
clip
T
(Jt (ar, Ur.y V2t
t=
(N)
D min
(oe (x, vec) Touu [x. 22)
clip
To(yel ve)) Told a, ye
-11+ -
r(a, y) 11
where is the clip ratio, and 8 the KL penalty coefficient. For PPO, the KL penalty applied to the advantage. For GRPO, the KL penalty added to the final loss. Both PPO and GRPO use a KL penalty coefficient of le-3. For PSR and NSR, we do not apply KI penalty, which we find to result better performance. The learning rate of the critic model in PPO is le-5, The clip ratio is to 0.2. We also apply entropy bonus to all the above objectives, with a coefficient of le-4. Our experiments conducted over single node with 8 NVIDIA H200 GPUs.
D.2 Prompt Templates
We adopt the prompt templates for Qwen models following [74), as shown in Tables 2 and 3.
Table 2: Prompt template for Qwen2. (rren2.5MMie 5-Math--77
<lim. <lim_startll startbsssste
You are helpful assistant.<lim _nddl> <lim startbuuse
[input]
Please reason step by step, and put your final answer within \boxed() (aeeed))iimm__... (illil_mmend
<lim_ hin_startbassistant
D.3 Advantage Normalization
The implementation computing advantage in verl includes an advantage normalization. However. for PSR and NSR, since we exclude the KL penalty, the advantage is equal to the raw reward (i.c.. +1 for PSR and -1 for NSR). In this case, applying normalization would cause the advantage to be zero, thus eliminating the learning signal. As result, we disable this normalization in PSR and NSR's s implementation. A recent work [61] also suggests that the reward normalization may not be necessary for certain settings.
26

---

Table 3: Prompt template for Qwen3-4B non-thinking mode.
<lim_ <lim_start>>sy startb>sstemm
You are helpful assistant.<lim Lnd__ndd>>>>>eee
starttrrb (input]
Please reason step by step, and put your final answer within \boxed <<lim_endl>>>>>>
endl>
<im startD>assistant
<think>
</think>
Table 4: @ results on MATH, AIME 2025 and AMC23 with different À using
Qwen2.5-Math 5-M Bold and underlined numbers denote the best and second second-best results for each k.
Pass@
2
4
16
32
64
128
256
A
MATH
0
75.7
82.4
86.9
90.1
92.4
94.1
95.3
96.2
96.9
0.05
75.6
82.2
86.7
89.9
92.2
94.0
95.3
96.3
97.1
0.1
76.6
82.8
92.4
94.1
95.3
96.1
96.7
0.2
75.8
81.2
85.2
88.3
90.7
92.6
94.0
95.1
95.9
I
74,8
79.1
82.4
84.9
86.9
88.5
89.9
91.1
92.0
A
AIME 2025
O
10.0
14.6
19.2
24.1
29.3
34.6
40.2
46.0
53.3
0.05
9.0
13.6
18.3
22.9
27.8
33.3
39.4
46.2
53.3
10.6
15.3
20.0
29.7
34.6
40.5
56.7
0.2
9.4
13.2
17.3
21.8
26.5
31.1
35.7
39.6
43.3
9.2
12.5
16.3
21.0
26.4
32.3
38.6
44.7
50.0
À
AMC23
0
60.9
70.0
77.4
83.2
87.6
91.1
94.5
97.9
100.0
0.05
61.6
70.9
77.8
82.9
87.1
91.2
95.1
98.1
100.0
0.1
62.0
70.0
77.0
87.8
91.8
95.2
97.1
0.2
60.4
68.0
73.9
79.3
84.5
89.5
93.6
96.4
97.5
1 61.8
68.5
73.4
77.0
80.8
85.1
89.3
91.9
92.5
E Impact of Reward Weighting Coefficient X
To examine how sensitive W-REINFORCE to the choice of we conduct an ablation study with À E 0, 0.05, 0.1, 0.2, on the MATH dataset, with À = O corresponding to NSR and À = 1 corresponding to vanilla REINFORCE. As shown in Table 4, performance is relatively stable when & 0.2, while larger values gradually degrade @ass@k across all k. This confirms that the key to W-REINFORCE 's effectiveness lies in moderately down-weighting the positive reward, which prevents the model from collapsing its distribution toward the tighest-probability answers. When
= 1, Pass @ 256 drops substantially, underscoring that excessive emphasis on positive rewards harms exploration and diversity. Overall, these results suggest that W-REINFORCE is robust to the choice of and small À can offer better trade-off between Pass and Pass@ @56..
F Limitations
We discuss the limitations of our work and future work directions as follows.
1. Performance degradation with extended NSR training. We observe that extensive training with NSR (c.g., over hundreds of gradient steps) leads noticeable decline performance, whereas W-REINFORCE does not exhibit this issue. However, such instability is not unique USS-sttandar RL algorithms like GRPO also experience similar degradation training
27

---

collapse under extended updates. as studied in recent works [30, 19, 79J. This suggests that NSR's implicit mechanism for preserving prior knowledge may be insufficient ensure stable training over the long term, and incorporating some degree of PSR may be necessary. Future work could explore more sophisticated methods for integrating NSR and PSR to design more robust RL algorithms, and investigate NSR potential warm-up phase before standard RL training given that improves the full Pass @ spectrum.
2. Beyond sparse and binary rewards. In this work, we focus on the RLVR setup with sparse and binary outcome rewards. However, many real-world tasks require dense reward signals that offer more fine-grained feedback (e.g., evaluating intermediate reasoning steps subjective tasks). It remains an open question how PSR, NSR, W-REINFORCE would perform in such settings, where the learning signals are continuous rather than discrete and potentially more difficult to interpret.
28


Reviews:
[{"review_id": "Rb11Fxk2LY", "reviewer": "Reviewer_Mh3W", "summary": "This paper study the effect of the gradient of positive and negative samples on the policy. It conduct an experiment to show the influence of tdifferent gradient on different Pass@k accuracy. It then design an algorithm to improve Pass@k accuracy by giving different weight to different part of the loss and gradient.", "questions": "Why doesn't consider the setting of off-policy?", "limitations": "1. The paper primarily focuses on the on-policy setting. However, off-policy settings—such as those used in popular algorithms like DPO—may exhibit different behaviors. The lack of consideration for off-policy scenarios limits the generality and significance of the findings.", "rating": 2, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper is clearly written, and the experiments effectively demonstrate the main observations.\n\nWeaknesses:\n\n1. Personally, I find that the phenomenon observed is somewhat aligned with common intuition in the field. Additionally, the technique of reweighting the negative samples lacks novelty in my view. However, I acknowledge that other reviewers may differ in their judgment on whether this insight constitutes a meaningful contribution beyond what is already understood without formal validation.\n\n2. The paper primarily focuses on the on-policy setting. However, off-policy settings—such as those used in popular algorithms like DPO—may exhibit different behaviors. The lack of consideration for off-policy scenarios limits the generality and significance of the findings.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful feedback. We address your raised issues as follows.\n> **Personally, I find that the phenomenon observed is somewhat aligned with common intuition in the field. Additionally, the technique of reweighting the negative samples lacks novelty in my view.**\n\nTo the best of our knowledge, our work is the first to find that solely penalizing mistakes through negative reinforcement learning, without relying on positive reward signals, can effectively improve LLM reasoning. If we missed any relevant work, we are happy to discuss and cite them accordingly.\n\n\n> **The paper primarily focuses on the on-policy setting. However, off-policy settings—such as those used in popular algorithms like DPO—may exhibit different behaviors. The lack of consideration for off-policy scenarios limits the generality and significance of the findings.**\n\nOn-policy RL training has been increasingly used in the latest state-of-the-art models. Notably, models such as DeepSeek-R1 [1], Kimi [2], and Qwen3 [3] all have adopted **fully on-policy training** without any off-policy RL components. Moreover, recent RLVR research [4,5] predominantly adopts on-policy RL algorithms like PPO and GRPO. Therefore, we believe our findings under the on-policy setting are both significant and relevant to current practice.\n\n[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025  \n[2] Kimi k1.5: Scaling Reinforcement Learning with LLMs, 2025  \n[3] Qwen3 Technical Report, 2025  \n[4] Demystifying Long Chain-of-Thought Reasoning in LLMs, 2025  \n[5] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild, 2025"}, {"review_id": "mKwlYdqcJK", "reviewer": "Reviewer_CHWn", "summary": "The paper decomposed the training signals of reinforcement learning into two independent parts: \"rewarding correct answers\" (positive reinforcement, PSR) and \"punishing incorrect answers\" (negative reinforcement, NSR), and made an unexpected key discovery. The study shows that merely training the model by punishing incorrect answers (NSR) is sufficient to match or even surpass more complex algorithms like PPO, because it can enhance the reasoning ability while effectively protecting the output diversity of the model, which is crucial for complex problems that require multiple attempts to solve. On the contrary, merely rewarding correct answers (PSR) can improve the single-shot accuracy rate, but it will damage diversity due to overconfidence. Based on this, the paper reveals that negative reinforcement works by gently correcting the model's prior knowledge, and proposes a simple and efficient W-REINFORCE algorithm, which achieves excellent results in multiple mathematical benchmark tests by effectively balancing positive and negative feedback.", "questions": "1.The new algorithm W-REINFORCE proposed in the paper sets the weight of positive samples to λ = 0.1. The experimental results show that this setting is very effective. However, the paper does not seem to provide an ablation study on the selection of this hyperparameter. How was λ selected? Is it sensitive to different models and tasks? What would the results be if λ is set to 0.05 or 0.2? A robust method should be less sensitive to the hyperparameters, or at least should clearly demonstrate its sensitivity.\n\n2.In the experiment, only the Qwen2.5-Math-7B and Qwen3-4B models were used for the experiment here. It is unknown whether this method is applicable in different model sizes as well.\n\n3.The paper, in the appendix, acknowledges that long-term training of NSR leads to performance degradation. However, this should not merely be regarded as a \"limitation\", but rather a core experimental result that deserves in-depth analysis in the main text. Readers cannot discern the \"decline\" trend of NSR from the charts in the main text because the experiment may only have reached its peak performance. The stability of an algorithm is a crucial aspect in evaluating its value.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper decomposes the training signals of reinforcement learning into two independent parts: \"rewarding correct answers\" (positive reinforcement, PSR) and \"punishing incorrect answers\" (negative reinforcement, NSR). Its most significant finding is that \"only punishing errors\" can achieve excellent results, which challenges the common perception that people have previously placed more emphasis on rewarding correct answers, and reveals the huge value of negative feedback that has been underestimated.\n\nWeaknesses：\n1.The conclusion of the paper heavily relies on the Qwen-Math type of \"expert models\" that are already extremely powerful. The reason why negative reinforcement works is that the model already has the ability to distinguish \"suboptimal solutions\". If it were replaced with an ordinary, non-professional basic model, the effectiveness of this method would likely be greatly reduced, and thus its universality is questionable.\n2.This method is highly suitable for tasks in mathematics that have clear, binary (right or wrong) feedback signals. However, it is difficult to apply to tasks such as creative writing, dialogue systems, etc., where the reward signals are ambiguous, subjective or continuous. This is because it is impossible to clearly distinguish between \"positive\" and \"negative\" samples.\n3.Long-term use of negative reinforcement alone leads to performance degradation, indicating that it is not a stable training strategy in the long run. Additionally, the new algorithm W-REINFORCE proposed by them lacks sufficient ablation analysis for the selection of key hyperparameters.", "was_revised": true, "final_justification": "The author's detailed reply addressed most of my concerns. So I raise my rating. The additionally conducted experiments demonstrating the effectiveness of NSR on different model families and the robustness of W-REINFORCE.", "rebuttal": "Thanks for your thoughtful feedback. We address your raised issues as follows.\n\n> **The conclusion of the paper heavily relies on the Qwen-Math type of \"expert models\" that are already extremely powerful… and thus its universality is questionable.**\n\nTo address your concern, we conduct experiments under the same setting with a different model family. Following are Pass@k results with Llama-3.1-8B-Instruct as the backbone model on MATH\n\n| Method | 1        | 2        | 4        | 8        | 16       | 32       | 64       | 128      | 256      |\n| ------ | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| PPO    |  52.0     |   59.9     |   66.3     | 71.7     | 76.1     | 79.6     | 82.6     | 85.2     | 87.4     |\n| GRPO | **54.1** | **61.7** | **67.7** | 72.6     | 76.7     | 80.1     | 83.0     | 85.5     | 87.5     |\n| PSR    |   50.2     |   56.3   |     61.7    | 66.3     | 70.4     | 73.9     | 76.9     | 79.6     | 81.8     |\n| NSR    |   52.8     |   60.8   |   67.4    | **72.9** | **77.4** | **81.2** | **84.3** | **86.7** | **88.7** |\n\nAs shown in the table, NSR is also effective and comparable or outperforms other methods with Llama model.\n\n> **This method is highly suitable for tasks in mathematics that have clear, binary (right or wrong) feedback signals. However, it is difficult to apply to tasks such as creative writing, dialogue systems, etc., where the reward signals are ambiguous, subjective or continuous. This is because it is impossible to clearly distinguish between \"positive\" and \"negative\" samples.**\n\nAs stated in the paper, our focus is on reinforcement learning with **verifiable rewards** (RLVR), where the reward is inherently unambiguous and verifiable. This setup aligns with prior RLVR literature that uses binary rewards [1,2,3,4]. Open-ended tasks like creative writing and dialogue are typically addressed using RLHF frameworks [5] which represent a different training setting (e.g., Qwen 3 [4] has separate training stages for verifiable and non-verifiable tasks). Therefore, extending our analysis and method to non-verifiable tasks is an interesting direction for future work, but is beyond the scope of RLVR.\n\n> **Long-term use of negative reinforcement alone leads to performance degradation, indicating that it is not a stable training strategy in the long run. Additionally, the new algorithm W-REINFORCE proposed by them lacks sufficient ablation analysis for the selection of key hyperparameters.**\n\nWe explicitly acknowledged this training stability observation in our paper to promote transparency and to encourage further research in RLVR. That said, we also note that training instability or performance degradation over extended RL training is a well-known challenge in the broader RL-for-LLM-reasoning research, not unique to our method. For example, standard RL methods (e.g., PPO) also suffer from training instability or collapse over long horizons for LLM training, as shown in Figure 2 of [2]. \n\nAs for hyperparameter of W-REINFORCE, we provide ablations with different λ as follows:\n\nMATH (Pass@k)\n\n| λ    | 1        | 2        | 4        | 8        | 16       | 32       | 64       | 128      | 256      |\n| ---- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| 0.05 | 75.6     | 82.2     | 86.7     | 89.9     | 92.2     | 94.0     | 95.0     | **96.3** | **97.1** |\n| 0.1  | **76.6** | **82.8** | **87.1** | **90.2** | **92.4** | **94.1** | **95.3** | 96.1     | 96.7     |\n| 0.2  | 75.8     | 81.2     | 85.2     | 88.3     | 90.7     | 92.6     | 94.0     | 95.1     | 95.9     |\n\nAs shown in the above table, we test λ=0.05, 0.1, 0.2, large λ makes the method closer to PSR and small λ makes it closer to NSR. Results show that W-REINFORCE is robust with different λ. \n\n[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025  \n[2] Demystifying Long Chain-of-Thought Reasoning in LLMs, 2025  \n[3] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild, 2025  \n[4] Qwen3 Technical Report, 2025  \n[5] Training language models to follow instructions with human feedback, 2022"}, {"review_id": "0lhUzfmL08", "reviewer": "Reviewer_hPq3", "summary": "Studying RL with verifiable (binary output-based) rewards, the authors analyze the impact of positive and negative reward signals on model performance and compare to standard methods such as PPO and GRPO. The authors focus the evaluation on math reasoning using the MATH, AIME2025, and AMC23 datasets.\n\nThe authors empirically show using Qwen2.5-Math-7B with a full spectrum of Pass@k with k in {1, 2, 4, 8…, 256} that only using positive reward samples during training leads to better performance for pass@1 while reducing diversity of responses and overfitting for larger values of k. The authors find that using negative reward samples performs similarly to PPO and GRPO methods in terms of larger k values and that it outperforms other methods in preserving model entropy (response diversity) by reducing the likelihood of wrong responses and reallocating the probability mass to others. The authors also use token-level gradient dynamics to verify these claims.\n\nThe authors also introduce a weighted variant of REINFORCE that prioritizes negative sample that contributes to the loss function with a linear hyperparameter, which performs comparably to more elaborate algorithms like GRPO and PPO on the tested benchmarks.\n\nBased on these results, the authors claim that negative reward samples naturally avoid overfitting (unlike positive reward samples), allow the model to effectively explore and search according to base model prior beliefs, and that the accuracy and diversity trade-off in RLVR is a balance issue between positive and negative rewards (which can be solved by re-weighting).", "questions": "- What are the implications of splitting rewards into “only positive” and “only negative”? Besides the direct impact of positive and negative samples on training curves, how does this simplification neglect the indirect impact of having positive and negative samples during training? My question is motivated by the assumption that LMs in RLVR encounter more wrong than correct answers and, e.g., does encountering a positive sample every 10 negative samples impact the overall trajectory differently when comparing trajectories with only positive or negative samples? \n\n- To what extent are the results in Figure 2 explained by PPO and GRPO training encountering more negative than positive reward samples during training?\n\n- To what extent is the trend of RL models losing their advantage at larger decoding budgets solely explained by negative reward samples (as argued in lines 136-141)? Are there no other possible explanations?\n\n- It is unclear to me if the authors agree with this statement “[line 265] RLVR primarily adjusts the model’s output distribution toward high-reward responses, rather than eliciting new reasoning abilities beyond the base model.” Based on the discussed results, I think this is ambiguous, as it matters not just that the probability mass is redistributed, but how.  Please clarify your position and argumentation.\n\n- What can we learn or hypothesize from your study about RL algorithms using dense/progress-based rewards?\n\nMinor comment:\n- The x-axis labels are missing in Figure 3.", "limitations": "The authors provide a very brief limitation section in Appendix C. *[For the AC]* I am not sure whether the limitation section is supposed to be in the main body of the paper or not.\n\nThe authors acknowledge the limitation of only using one model to build their results and that they neglect of dense/intermediate rewards.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Reasons to accept:\n\n- This work potentially provides crucial insight into outcome-based rewards in RL for LLMs that can have a significant impact for similar methods that are limited to not using dense/progress rewards. Understanding the importance of negative samples can also have a significant impact on alignment and safety methodologies, where constructing negative samples based on a rule or reason is easier than constructing positively rewarded ones.\n\n- Good methodology using entropy and token-level gradients (quality). \n\n- The paper is well written and, to the best of my knowledge, original work.  \n\nReason to reject:\n\n- My main concerns are related to the lack of statistical significance tests or statistical uncertainty estimates (quality, uncertain significance). First, the authors claim to have fulfilled the NeurIPS requirement for statistical significance/uncertainty estimates with the reason ”Our results are based on 256 random samplings and calculated with an unbiased estimator.”. *Just using an unbiased estimator does not eliminate the requirement for uncertainty estimates/statistical significance tests!* \n\nDoing a few estimates, (assuming k=10, n=256, for a few c values), I find that the statistical uncertainty could be in the order of magnitude of 3-8 percent, i.e., in the order of magnitude the gained improvements in Table 1 are claimed or for which the differences in Figures 2 and 3 are used as evidence. I could be wrong, though, so I would require the authors to demonstrate that their results are statistically significant using the method they deem most appropriate.\n\nWithout demonstrating that the empirical results used in Figures 2, 3, and Table 1 are statistically significant and not just statistical trends, I have to question the validity of the results of the paper. The statistical trends are definitely there, but it has not been shown that these trends are actually statistically significant (at least 2-sigma or 95% confidence level deviations). For example, all claims in section 3.2 should either be highlighted as just observed trends (not results) or require the derivation of an uncertainty estimator + reported uncertainties. Table 1 should also include the statistical uncertainties. This is my main concern, and fully addressing this point will make me increase my score to a 4 or 5 (together with the quality and significance score).\n\n- The authors only tested one model (quality, uncertain significance). I realize that expecting multiple RL training runs for models above 7B in academic papers is unreasonable. However, I think calling for testing at least a second 7B or some of the smaller 3B models from different model families to support universal empirical RLxLM training dynamics claims is not.", "was_revised": true, "final_justification": "The authors successfully resolved my concerns/reasons to reject, in particular, by providing statistical significance tests, running new experiments, and resolving uncertainties I've had. My original reasons to accept remain unchanged and are strengthened by the provided author revisions and rebuttal.", "rebuttal": "Thanks for your thoughtful feedback. We address your raised issues as follows.\n\n> **My main concerns are related to the lack of statistical significance tests or statistical uncertainty estimates (quality, uncertain significance)...** \n\nWe report the 95% confidence intervals on MATH with Qwen2.5-Math-7B in the table below:\n\n| Method | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 | 256 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| GRPO | 76.3 ± 1.0 | 81.7 ± 0.9 | 85.6 ± 0.9 | 88.4 ± 0.8 | 90.6 ± 0.7 | 92.3 ± 0.7 | 93.6 ± 0.6 | 94.6 ± 0.6 | 95.5 ± 0.6 |\n| PPO | **76.6 ± 1.0** | 82.6 ± 0.9 | 86.7 ± 0.8 | 89.6 ± 0.7 | 91.7 ± 0.7 | 93.4 ± 0.6 | 94.7 ± 0.6 | 95.6 ± 0.5 | 96.3 ± 0.5 |\n| PSR | 74.1 ± 1.1 | 78.3 ± 1.0 | 81.6 ± 1.0 | 84.1 ± 0.9 | 86.2 ± 0.9 | 87.9 ± 0.8 | 89.3 ± 0.8 | 90.3 ± 0.8 | 91.2 ± 0.8 |\n| NSR | 75.7 ± 0.9 | 82.4 ± 0.9 | 87.0 ± 0.8 | 90.1 ± 0.7 | **92.4 ± 0.6** | **94.1 ± 0.6** | **95.3 ± 0.5** | **96.2 ± 0.5** | **96.9 ± 0.5** |\n| W-REINFORCE | **76.6 ± 0.9** | **82.8 ± 0.9** | **87.1 ± 0.8** | **90.2 ± 0.7** | **92.4 ± 0.6** | **94.1 ± 0.6** | **95.3 ± 0.5** | 96.1 ± 0.5 | 96.7 ± 0.5 |\n\nAs shown in the table, NSR/W-REINFORCE consistently matches or outperforms the state-of-the-art RL algorithms PPO and GRPO across a wide range of pass@k values. Notably, for $k > 4$, both W-REINFORCE and NSR achieve statistically significantly higher pass@k scores than GRPO. We'd like to highlight that W-REINFORCE achieves this competitive performance with a much simpler design: no value model (unlike PPO) or sophisticated group-wise advantage normalization (unlike GRPO). Given these advantages, we believe our method offers a good trade-off between performance and simplicity, and can serve as a strong baseline for future research.\n\n> **The authors only tested one model (quality, uncertain significance)...**\n\nTo address your concern, we conduct experiments under the same setting with a different model family. Following are results with Llama-3.1-8B-Instruct as the backbone model on MATH:\n\n| Method | 1        | 2        | 4        | 8        | 16       | 32       | 64       | 128      | 256      |\n| ------ | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| PPO    | 52.0     | 59.9     | 66.3     | 71.7     | 76.1     | 79.6     | 82.6     | 85.2     | 87.4     |\n| GRPO   | **54.1** | 61.7     | **67.7**     | 72.6     | 76.7     | 80.1     | 83.0     | 85.5     | 87.5     |\n| PSR    | 50.2     | 56.3     | 61.7     | 66.3     | 70.4     | 73.9     | 76.9     | 79.6     | 81.8     |\n| NSR    | 52.8     | **60.8** | 67.4 | **72.9** | **77.4** | **81.2** | **84.3** | **86.7** | **88.7** |\n\nAs shown in the table, NSR is also effective and comparable or outperforms other methods with Llama model.\n\n> **What are the implications of splitting rewards into “only positive” and “only negative”... does encountering a positive sample every 10 negative samples impact the overall trajectory differently when comparing trajectories with only positive or negative samples?**\n\nThe positive and negative rewards in RLVR are typically entangled and difficult to interpret directly, which motivates our decomposition into “only positive” and “only negative” rewards. This controlled setup enables us to isolate and understand the independent effects of each reward signal. As shown in Figure 3, we present the training trajectories for three scenarios: both positive and negative rewards (PPO/GRPO), only positive rewards (PSR), and only negative rewards (NSR). \n\nWe believe PPO and GRPO’s trajectories reflect the practical training dynamics of mixed rewards because they are designed to handle varying ratios of correct and incorrect rollouts using advantage normalization to adapt to the natural distribution of rewards. We do not study the scenario with a manually fixed positive-to-negative ratio (e.g., 1:10) as it introduces a sampling bias that deviates from the actual training dynamic and is not a realistic representation of on-policy learning.\n\n> **To what extent are the results in Figure 2 explained by PPO and GRPO training encountering more negative than positive reward samples during training?**\n\nWe clarify that PPO and GRPO do not encounter more negative than positive reward samples. Instead, as shown in Figure 3(c), PPO and GRPO encounter more positive than negative reward samples during training. This leads to increasing probability of generating observed correct samples and causes a reduction in model entropy, as shown in Figure 3(b). The narrowing of the output distribution limits the model’s exploration, which explains the inferior  pass@k performance for larger k shown in Figure 2.\n\n> **To what extent is the trend of RL models losing their advantage at larger decoding budgets solely explained by negative reward samples (as argued in lines 136-141)? Are there no other possible explanations?**\n\nThe major difference between NSR and GRPO/PPO is that NSR relies solely on negative reward samples while excluding positive ones, while PSR represents the opposite extreme. This tightly controlled setup allows us to directly attribute the observed trends to the presence or absence of positive and negative rewards.\n\nWhile we cannot claim with absolute certainty that no other factors contribute in any way, our controlled experimental design is a standard scientific method for isolating a primary cause (i.e., positive reward only training yields pass@k degradation). The fact that we can (a) turn the degradation effect on and off by manipulating a single variable (the positive reward signal) and (b) provide a clear, low-level mechanistic explanation for it (the gradient dynamics) makes our explanation robust and direct, minimizing the likelihood of other major confounding factors.\n\n> **It is unclear to me if the authors agree with this statement “[line 265] RLVR primarily adjusts the model’s output distribution toward high-reward responses, rather than eliciting new reasoning abilities beyond the base model.”**\n\nWhether RLVR elicits new reasoning abilities or primarily adjusts the output distribution depends on the specific setup and task, and generally there isn’t a clear-cut answer. Our paper does not aim to resolve this question. Instead, our focus is to highlight a previously under-appreciated role of the negative sample reinforcement in shaping model behavior in RLVR. An in-depth investigation into the emergence of reasoning ability is orthogonal to our contributions and beyond the scope of this work.\n\n> **What can we learn or hypothesize from your study about RL algorithms using dense/progress-based rewards?**\n\nIn dense reward cases, where the reward may be in the range of -1~1, the findings of our paper can still transfer to such a scenario: positive reward will reinforce observed rollouts, negative reward will promote other possible candidates."}, {"review_id": "HgSrvAQLf9", "reviewer": "Reviewer_nxZP", "summary": "This paper delves into the effectiveness of negative reinforcement in enhancing the reasoning capabilities of LLMs. It dissects RLVR into PSR and NSR, exploring their distinct impacts on model behavior and generalization. The authors demonstrate that NSR, which solely penalizes incorrect samples, can surprisingly elevate model performance across the Pass@k spectrum, often matching or surpassing traditional RL algorithms like PPO and GRPO. Based on these insights, the paper proposes W-REINFORCE, a modified RL objective that emphasizes NSR, achieving notable performance improvements across multiple reasoning benchmarks.", "questions": "1. Can the authors elaborate on the relationship between SFT and PSR? \n2. How does the proposed W-REINFORCE algorithm compare to other RL variants in the literature that also emphasize negative reinforcement? What specific advantages does it offer over these existing methods?\n3. The choice of the weighting factor $\\lambda$  in W-REINFORCE seems crucial for its performance. Could the authors offer more detailed guidance or a systematic approach for selecting an appropriate $\\lambda$ value, especially for different models and tasks?\n4. The authors mention that NSR refines the model's existing knowledge rather than introducing entirely new behaviors. Could they provide more in-depth analysis or visualization to help readers better understand this refinement process?\n5. Why does W-REINFORCE underperform PPO in pass@1? An explanation from the authors would be appreciated.\n6. How does the proposed method generalize to OOD datasets? *(This will not influence my score)*\n7. What is the specific value of $n$ in the pass@k evaluation metric? (Is it 256, as suggested in line 111?)\nIf $k=256$, based on the definition in Eq. (5), the pass@256 values in Table 1 should be binary (either 0 or 1). This is because if we assume $n=k=256$, the term $\\binom{n}{k}$ becomes $\\binom{256}{256} = 1$. Consequently, the term $\\binom{n-c}{k}$ becomes $\\binom{256-c}{256}$, which evaluates to 0 for any $c > 0$ and to 1 only when $c=0$. This seems to contradict the results shown in Table 1.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- The paper presents a novel perspective by decomposing RLVR into PSR and NSR, offering valuable insights into their individual roles in shaping model behavior.\n- The authors conduct extensive experiments to validate their claims, providing a comprehensive analysis of how PSR and NSR affect model performance across different metrics.\n- The findings have important implications for the design of RL algorithms for LLMs, particularly in preserving diversity and promoting generalization.\n\n**Weaknesses:**\n- While the decomposition of RLVR into PSR and NSR is insightful, the overall concept of this method is not entirely new and has been touched upon in previous studies [1].\n- The proposed W-REINFORCE algorithm, despite showing promise, requires careful tuning of the weighting factor $\\lambda$ . The paper does not provide sufficient guidance on how to optimally set this parameter in practice.\n- The claim that NSR (using only negative samples) outperforms PSR is a bit counterintuitive and confusing. The authors should provide a clearer explanation for this result (Table 1).\n\n[1] Xiong, Wei, et al. \"A minimalist approach to llm reasoning: from rejection sampling to reinforce.\" arXiv preprint arXiv:2504.11343 (2025).", "was_revised": true, "final_justification": "The authors' responses addressed most of my concerns.", "rebuttal": "Thanks for your thoughtful feedback. We address your raised issues as follows.\n\n> **While the decomposition of RLVR into PSR and NSR is insightful, the overall concept of this method is not entirely new and has been touched upon in previous studies [1].**\n\nAccording to NeurIPS policies, works posted after March 1st, 2025 are considered concurrent. Reference [1] was posted on arXiv on April 15 and should therefore be treated as concurrent, not prior work.\n\nMore importantly, the methodology in [1] is fundamentally different from ours. Their approach is based on **prompt filtering**: removing prompts whose responses are entirely correct or incorrect, this means that the training data still contains a **mix of both positive and negative reward examples**. In contrast, our work introduces a **complete objective decomposition** of RLVR into PSR and NSR, fully isolating positive and negative reward examples. This clean setup allows us to clearly examine how each reward type individually shapes model behavior.\n\n> **The proposed W-REINFORCE algorithm, despite showing promise, requires careful tuning of the weighting factor**\n\n> **The choice of the weighting factor in W-REINFORCE seems crucial for its performance. Could the authors offer more detailed guidance or a systematic approach for selecting an appropriate value, especially for different models and tasks?**\n\nWe provide hyperparameter study of W-REINFORCE with different λ as follows:\n\nMATH (Pass@k)\n\n| λ   | 1        | 2        | 4        | 8        | 16       | 32       | 64       | 128      | 256      |\n| ---- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| 0.05 | 75.6     | 82.2     | 86.7     | 89.9     | 92.2     | 94.0     | 95.0     | **96.3** | **97.1** |\n| 0.1  | **76.6** | **82.8** | **87.1** | **90.2** | **92.4** | **94.1** | **95.3** | 96.1     | 96.7     |\n| 0.2  | 75.8     | 81.2     | 85.2     | 88.3     | 90.7     | 92.6     | 94.0     | 95.1     | 95.9     |\n\nRL algorithms generally require hyperparameter tuning, but our experiments show that W-REINFORCE is not highly sensitive to λ.\n\n> **The claim that NSR (using only negative samples) outperforms PSR is a bit counterintuitive and confusing. The authors should provide a clearer explanation for this result (Table 1).**\n\nThe explanation why NSR can outperform PSR is described in Line 142 and further supported by empirical evidence in Figure 3 and analysis in Section 4.2. As shown in Figure 3, PSR quickly overfits to the observed correct rollouts, which reduces output diversity and harms exploration, resulting in inferior pass@k performance when k is large. In contrast, NSR penalizes incorrect outputs and maintains a high entropy, preserving a broader search space. Section 4.2 further explains this with a principled gradient analysis. It reveals that PSR suppresses all possible alternatives except the highest probability token, while NSR encourages exploration by promoting all plausible candidates based on model’s prior. These results and analysis explain why NSR can outperform PSR.\n\n> **Can the authors elaborate on the relationship between SFT and PSR?**\n\nSFT typically refers to off-policy training, whereas PSR is fully on-policy. Moreover, PSR follows a standard RL setting that incorporates importance sampling and clipping in its loss function.\n\n> **How does the proposed W-REINFORCE algorithm compare to other RL variants in the literature that also emphasize negative reinforcement? What specific advantages does it offer over these existing methods?**\n\nTo the best of our knowledge, our work is the first to find that solely penalizing mistakes through negative reinforcement learning, without relying on positive reward signals, can effectively improve LLM reasoning. If we missed any relevant work, we are happy to discuss and cite them accordingly.\n\nThe advantages are that it can preserve models’ generation diversity and strike a good balance between pass@1 and pass@k performance when k is large (i.e., more decoding budgets).\n\n> **The authors mention that NSR refines the model's existing knowledge rather than introducing entirely new behaviors. Could they provide more in-depth analysis or visualization to help readers better understand this refinement process?**\n\nSection 4.2 provides a principled gradient analysis on why NSR is refining existing knowledge. It reveals that the gradients of NSR promote all plausible alternatives based on models’ prior. Figure 4 offers a visualization of this process.\n\n> **Why does W-REINFORCE underperform PPO in pass@1? An explanation from the authors would be appreciated.**\n\nAs shown in table 1, W-REINFORCE is either comparable to PPO (on MATH and AMC23) or outperforms PPO (on AIME 2025) measured by pass@1.\n\n> **How does the proposed method generalize to OOD datasets?**\n\nWe train models on the MATH dataset and evaluate them on AIME 2025 and AMC 23. MATH consists of PhD-level problems, while AIME and AMC feature Olympiad-level math questions with different formats and difficulty distributions. These test sets can be considered OOD relative to the training data. The results on various benchmarks suggest our method can generalize to OOD datasets.\n\n> **What is the specific value of n in the pass@k evaluation metric?**\n\nYes, n is 256. Pass@256 is computed by averaging over the entire evaluation set. For example, the MATH test set contains 5000 problems. For each problem, pass@256 is either 0 or 1 depending on whether at least one of the 256 generations is correct. The final pass@256 metric reported in Table 1 is the mean across all problems, which yields a number between 0 and 1.\n\n[1] Xiong, Wei, et al. \"A minimalist approach to llm reasoning: from rejection sampling to reinforce.\" 2025."}]

Author rebuttal:

Meta-review: The paper studies reinforcement learning with verifiable rewards by decomposing signals into positive (PSR) and negative (NSR) reinforcement. It finds that NSR alone can substantially improve LLM reasoning, often matching or surpassing PPO/GRPO, while preserving diversity. The authors propose W-REINFORCE, a simple variant emphasizing NSR, which performs competitively across reasoning benchmarks (MATH, AIME 2025, AMC23).

**Strengths:**

- Clear and well-structured, with strong empirical evidence.

- Provides novel insight into the value of negative reinforcement in LLM training.

- Demonstrates improved diversity and robustness compared to PSR-only methods.

- W-REINFORCE is simple yet effective, rivaling more complex baselines.

- Rebuttal added significance tests, additional model experiments, and ablations.

**Weaknesses:**

- Generality limited to strong base models and binary/verifiable tasks.

- NSR alone degrades with long-term training.

- Hyperparameter sensitivity and novelty relative to concurrent work remain mild concerns.

--

Reviewers initially raised concerns on statistical rigor, model diversity, stability, novelty, and $\lambda$ sensitivity. The authors provided confidence intervals, tested on a second model family (Llama), added $\lambda$ ablations, and clarified scope and distinctions from concurrent work. These responses resolved most concerns, leading to higher scores and a clear positive consensus.

The key contribution (that penalizing mistakes alone can improve reasoning) is interesting. Rebuttal experiments convincingly addressed statistical, robustness, and novelty concerns. Three reviewers moved to acceptance; only one remained negative with limited engagement. On balance, this is a solid contribution meriting acceptance.
Decision: Accept (poster)