On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
Authors: Wenlong Deng
Yi Ren
Muchen Li
Danica J. Sutherland
Xiaoxiao Li
Christos Thrampoulidis
Venue: NeurIPS 2025 poster ()
Keywords: Large language model
Reasoning
Reinforcement Learning
Primary area: deep_learning

Abstract: Reinforcement learning (RL) has become popular in enhancing the reasoning capabilities of large language models (LLMs), with Group Relative Policy Optimization (GRPO) emerging as a widely used algorithm in recent systems. Despite GRPO's widespread adoption, we identify a previously unrecognized phenomenon we term Lazy Likelihood Displacement (LLD), wherein the likelihood of correct responses marginally increases or even decreases during training. This behavior mirrors a recently discovered misalignment issue in Direct Preference Optimization (DPO), attributed to the influence of negative gradients. We provide a theoretical analysis of GRPO’s learning dynamic, identifying the source of LLD as the naive penalization of all tokens in incorrect responses with the same strength. To address this, we develop a method called NTHR, which downweights penalties on tokens contributing to the LLD. Unlike prior DPO-based approaches, NTHR takes advantage of GRPO’s group-based structure, using correct responses as anchors to identify influential tokens. Experiments on math reasoning benchmarks demonstrate that NTHR effectively mitigates LLD, yielding consistent performance gains across models ranging from 0.5B to 3B parameters.

On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
Wenlong Dengl: 1.2, Yi Ren', Muchen Li!
Danica J. Sutherland 13 Xiaoxiao Li Christos Thrampoulidis"
University of British Columbia, Vector Institute, Amii
*Corresponding author
Abstract
Reinforcement learning (RL) has become popular in enhancing the reasoning capa- bilities of large language models (LLMs), with Group Relative Policy Optimization (GRPO) emerging widely used algorithm in recent systems. Despite GRPO's widespread adoption, we identify a previously unrecognized phenomenon we term Lazy Likelihood Displacement (LLD), wherein the likelihood of correct responses marginally increases or even decreases during training. This behavior mirrors a recently discovered misalignment issue in Direct Preference Optimization (DPO), attributed to the influence of negative gradients. We provide theoretical analysis of GRPO's 's learning dynamic, identifying the source of LLD as the naive penaliza- tion of all tokens in incorrect responses with the same strength. To address this, we develop a method called NTHR, which downweights penalties on tokens con- tributing to the LLD.. Unlike prior DPO-based approaches, NTHR takes advantage GRPO's group-based structure, using correct responses as anchors identify math reasoning benehmarks demonstrate that
influential tokens. Experiments NTHR effectively mitigates LLD, yielding consistent performance gains across models ranging from 0.5B to 3B parameters.
Introduction
Reinforcement learning (RL) has become increasingly popular for improving reasoning capabilities of large language models (LLMs) 4 9 23 In particular, Group Relative Policy Optimization (GRPO) [22] has emerged as a widely adopted algorithm RL training pipelines for reasoning tasks. Models such DeepSeck-R1 R1 44, DeepSeek-Maun
[22] Med-R1 [11], and Search-R1 [10] have successfully employed GRPO to achieve notable performance gains in code generation, mathematical problem solving, medical reasoning, and retrieval-augmented generation. These successes highlight GRPO's growing importance as a tool for aligning models to task-specific behaviors through rule- based or heuristic rewards.
Despite its empirical success, the optimization behavior GRPO remains insufficiently studied, in part due to its recency. In this work, we focus on the impact of negative gradients in GRPO, which arise when the advantage function average action at given state (e.g., incorrect suboptimal response). In such cases, the learning algorithm decreases the probability selecting this action. We draw a conceptual connection to Direct Preference Optimization (DPO) [88] which introduces negative gradients when penalizing
that the selected action is worse than the
dis-preferred responses. Recent studies 19 30 20 have revealed that negative gradients can lead arguably counterintuitive behavior: simultaneous decrease in the probabilities of preferred responses t) Motivated by this insight, we examine whether similar gradient dynamics in GRPO might cause analogous issues and investigate their impact on policy learning. This effect,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

termed misalignment [20 or likelihood displacement [19]. is undesirable as it often degrades model performance by diverting probability mass away from optimal responses. In this work, we empirically observe (see Fig. D that GRPO can suffer from what we call Lazy Likelihood Displacement (LI.D): failure to sufficiently increase, or even a decrease in, the likelihood of correct answers during training. To explain this phenomenon, we provide an analysis of GRPO's ss update dynamics and identify that LLD stems from penalizing certain tokens in dispreferred responses. These updates inadvertently reduce the likelihood of correct answers due to shared structural or semantic features between positive and negative responses.
While several approaches have been proposed to mitigate the reduced probabilities of preferred responses (y ) DPO [0],, they either compromise data efficiency or rely on training procedures that are not directly compatible with GRPO's online learning paradigm. For instance, Razin et al. [19]
recommend discarding training examples where the likelihood of positive responses y decreases, thereby avoiding harmful updates- but this strategy reduces data utilization and does not address cases where the increase in the likelihood of y is insufficient. Similarly, Ren and Sutherland [20] propose modifying DPO into a more online-compatible setup by training both y* and y which effectively mitigates the "squeezing effect" imposed by off-policy negative gradient. However, GRPO already employs online response sampling, and our empirical analysis reveals that it still suffers from the LLD effect in a subset of training examples (see Fig.
To effectively address LLD in GRPO, we introduce negative token hidden reward (NTHR) for selective token penalization, which reduces the penalty on tokens in incorrect responses that most strongly contribute to lowering the likelihood of correct ones. Interestingly, we find that these influential tokens often align closely with key reasoning steps in the correct responses (see Fig. (3). Our approach NTHR provides fine-grained and targeted mitigation strategy that mitigates LLD during training. We evaluate GRPO+NTHR through per-sample training and compare it against baseline methods such as random token dropping and standard GRPO across multiple model sizes. We demonstrate that GRPO+NTHR successfully mitigates LLD and leads to improved generalization on held-out mathematical reasoning problems. In summary, our contributions are as follows:
GRPO exhibits LLD for correct responses. We empirically show that GRPO suffers from Lazy Likelihood Displacement (LLD), an extended form of the misalignment phenomenon previously observed only in DPO.
Identifying the source of LLD, We provide a theoretical explanation for LLD in GRPO, identifying its cause penalization of shared reasoning or correct tokens in dispreferred responses.
NTHR selective token penalization. We introduce Negative Token Hidden Reward (NTHR) for selective token penalization that effectively mitigates LLD without sacrificing data efficiency.
Empirical validation on math reasoning tasks. We demonstrate that GRPO+NTHR consistently outperforms GRPO on math benchmarks across model sizes ranging from 0.5B to 3B.
2 Preliminaries and Related Work
2.1 GRPO
GRPO loss, introduced in Math 44 and DeepSeek-RI [22]. enhances fine-tuning by refin- ing how reward and loss are calculated. Concretely, unlike traditional Proximal Policy Optimization (PPO) 21), GRPO eliminates the need for value function estimation, employing group-relative rewards for more nuanced optimization process.
For query-answer pair (ær, a), the policy my samples G responses {y:) Each y: consists of sequence of lysl tokens, and we denote yi,ck the subsequence of the first & tokens. Let r; denote the reward for response y:- The advantage of the i-th response is computed by normalizing the group- level rewards {rx) and is the same for each token k l,.... [/s]- Concretely, A == ===,,
with μ = [((r)) and a Var((()) being the empirical average and standard deviation of

---

Effect segutive gradent (Owen 0.30 Nyth)
regative gradlent
Effect of negattie gradier SCwen Morth 50, Marto
Ony
Lag Uhclhood
Loop Lestreddd
0.44
Peggera
0:0
2.00
0.5
Sample Orders (Samed Likelhood Change!
Sample Orders (Serted Liaeiheed
Sample (Serted Livel Change)
(a) Qwen-0.5B-Instruue
(b) Deepseek-1.SB
(c) Qwen-Math-1.5B
We show that negative gradients can lead to small or reduced likelihood change of positive samples GRPO. The oogllikeldoodd gains achieved by Pos Only training (orange) are significantly higher than those from GRPO (blue) for Qwen-0.5B-In (a) and Deepseek- 1.5keekkk1.BBB (b). In Qwen-Math- 1.5B (c), samples with small or reduced (ax) (left) are primarily influenced by negative gradients, as evidenced by their larger ((x) in the Pos Only setup. However, some samples the right show smaller (x) than in GRPO, indicating that negative gradients are not always harmful.
Figure I:
the rewards. The GRPO objective (O) is then defined as:
G
l
E (a
min (0)A,)) clip -,, (I)
G (3: Tot (az)
where is clipping hyperparameter, clip(-) is the clipping operation, and (0)
x0 is the likelihood ratio between the current policy me and the old policy Te..
new (M M..c))
2.2 Positive and Negative Gradient
The GRPO loss includes several regularization mechanisms designed to stabilize training, such the KL term and the clipping operation. However, it is common to treat the log-likelihood component of the generated responses as responsible for "gathering knowledge," while viewing the regularization components responsible for "stabilizing the training procedure." Without loss of generality, we focus on the online training setting for GRPO, in which "i and min(-, clip(-)) min(-,clip()) can be safely neglected. This simplification is justified by two observations. First, GRPO inherently operates in near-online fashion, as generates new samples on-the-fly and typically uses mini-batches that are comparable in scale to the full batch. Second, prior work [ has shown that omitting the clipping operation does not degrade performance. Additionally, related studies [ g have demonstrated that the KL term can be omitted when other hyperparameters are carefully tuned. With this simplification, we can better understand the interactions between positive and negative responses one roll-out. The subtle differences between GRPO and its variants, e.g., DAPO 29), Dr.GRPO [3] GPG (.
etc., could also be well interpreted.
Specifically, the GRPO's 's objective gradient (oJaRRo( with respect to 0 can be approximated as AL which using VOTT mo Vo log my further simplifies to
Val
E
(L Vo log (o xx V..kk)
1= yii
constant
This expression allows us to interpret GRPO using a similar framework as proposed by Ren and Sutherland (20) i.e., imposing a positive negative pressure on Vi,k- Since %, is ratio of two probabilities (and hence must be positive), the sign of the equivalent learning rate on yi,k is determined by Á. Then, following common practice that the reward is for correct responses and O for incorrect ones, is safe to conclude that all the tokens in the correct responses impose positive gradients while tokens in the wrong responses impose negative ones.
3

---

3 The Effect of Negative Gradients
We begin by empirically analyzing the impact of negative gradients in GRPO 4 on the likelihood of generating correct responses. Let each question æ: be associated with G generated samples, of which N := N (a=) give the correct answer so we call them positive and denote by y, E [N and the rest N'. L G N are incorrect we call them negative and denote by y,+ : E [N Positive/neeative samples are given rewards r; and r respectively.
We conduct experiments using math-reasoning tasks to assess how negative gradient in GRPO training influences the log-likelihood of correct responses. Specifically, we examine range of model-datasee combinations that differ in terms of dataset difficulty and model scale: Qwen-2.5-o
[25]. Qwen-
2.5Math-1.5D [26] with the MATH dataset 77 and Deepseek- 1.5B 4 with the AIME dataset. For each question, we generate 8 response rollouts, filtering out samples where all responses are either entirely correct or incorrect. We retain only those examples containing a mix of both.
Our analysis focuses on the GRPO baseline (illustrated by the blue bars in Fig. 4). To precisely characterize the learning dynamics of the algorithm, we reinitialize the model parameters A for each individual sample, perform a single GRPO update to obtain the updated parameters 0', and evaluate the average change in the loglliellhoood of the correct responses:
1 N+ 1=1 [n Te (y. x) In x)] Σ
N+
A
for an input question æ with N correct responses y E [N+]. As illustrated in Fig. | some correct responses experience either a decline or only a marginal improvement in log-likelln lig-liielhhood after training. For example, in Deepseck- likelihoods (negative values in the plot). Similarly, in Qwen-Math-Ie M.5h- and Qwen-O.5B. many samples show only minor changes in likelihood. A closer analysis shows the following.
(Fig. [bb, significant portion of samples exhibit reduced
Negative gradients lead to small or reduced likelihood change. In order to isolate the effect of negative gradients, we introduce a variant which we call Pos Only, where negative advantages are masked (i.e., values of Å. < 0 are set to 0), effectively removing the influence of negative gradients during training. Fig. shows the Pos Only variant in orange bars. The likeliihd gains under Pos Only training are substantially higher than those from GRPO p00t-paalllllarlyy the left side of the plots, where many samples show several-fold improvements.
Negative gradients are not always harmful. It is important to note that negative gradients are not inherently harmful. In terms of the likelihood change A (x), it is possible, as shown in Fig. Ic] that under Pos Only training, some samples exhibit even smaller A(x) than GRPO, which uses both positive and negative gradients. This decrease may result either from negative gradients "squeezing" the likelihood mass toward confident y (20), or from certain negative gradients increasing A(x), as suggested by our Theorem 4.4 below. Moreover, in terms of performance, we demonstrate that Pos Only actually yields a 1.3% drop in average performance compared to GRPO. The result is shown in Tab. 2 for finetuning Qwen-Math-1.55 1.5B on the math dataset using GRPO and Pos Only respectively.
What do negative gradients of questions with small likelihood change penalize? We find that the minimal likelihood change occurs in correct responses of those questions the dataset that have partially correct (but eventually incorrect) responses or have correct responses. In Fig. 2 we present two representative examples from Qwen-O.5B, where the incorrect responses are partially correct and the corresponding correct responses show the smallest likelihood change. The left example, highlighted in red, shows an answer that is nearly correct and corresponds to the red-boxt bed-boxed sample in Fig. Ia The right example provides correct answer but does not follow the expected output format (highlighted in red); it corresponds to the blue-boxed sample in Fig. Ta Additional examples are provided in Appendix. This suggests that penalizing entire partially correct responses is suboptimal.
Controlling negative gradients at the token level can improve performance. While negative gradients do not necessarily hurt performance and penalizing entire incorrect responses is suboptimal, we show Sec. 5 that carefully reducing negative gradients at a token level can mitigate small likelihood change and lead to improvement boosts. Our approach specifically leverages the previous observation that partial responses might be partly correct by filtering out and selectively penalizing their consistuent tokens.

---

Aluoott Correct
Foruatt Issue
Question: greatest integer valte
Math Question: What imtege that
Bu do 7
which the expreation has dotrain real sumbers. Ground-trui fruth Answer: 5
Sample: (Mdath datevs)) Model (Qeas e 0.iB-ied!
Greund truch Answer:
Sample jodash detouet Madel (Qwwe-t.---iee
To salve this problem, we seed to determtäne the valuei for which the given
solve the inequality
need to the values
expression has domain of all real numbers. Let's anayye the expression step by step:
satidyy the inequality, Since the inequality is inequality, we should first find the
roots of the correspondling equation x -in - 20- Lets solve the equation
< its 34
the incquality the discrimiiant less For the quadiatic equation
Factor the quadratic equation. af Thes 21(s R2-..
noots, we of
each factor equal to zero and solve
Solve insquality for Salving for i, we gee. Taking the square
Solve for ot a- or in ana. Therefore, the roots of the equatioo
root of both sides, we get: c Since is coetficiont in polynomini, it must be con-negative. Therefore, we can multily the emiine inquulity by -] (which reverses the incculiity: a -
x its 24 are w- or a-a.
..
For (--m m) let's tost n- 11000 24200
find the approxamate value VI2; 4SS 1,06 So, ao howw Ideniify greatest integer
For ni (3.6), lef' test n- 5f 1109) 54 25 66 -i.c... 3. For wr (Ros)) let's test
greatest integer greater
at- 1110) 24 M1 W o Therefore, the incqualtty xf Lix 5 holds true for asass. Hence, the greatest integer answer is
Therefore, greatest integer value SbS for expression sof 4af 1ix+? al Re
has domtin afl real numbers 6. Final arswer.
that sutisfies the inoquality
boxed foruus, the
Figure 2: Inspecting negative (incorrect) samples of questions with small average likelihood change Axx) (Eq. (2) reveals that they are either nearly correct (Left) or get the correct response in a wrong answer format (Right). Thus, penalizing entire negative sample responses might be suboptimal. Red dashed lines denote omitted reasoning steps.
Lazy Likelihood Displacement
We have shown that negative gradient in GRPO can cause a marginal or even reduced likelihood of correct responses y². Here, we formalize this observation by introducing the concept of Lazy Likelihood Displacement (LLD)-4 phenomenon in which the probability of correct answers decreases or increases only marginally after training. Our experiments reveal that LLD frequently emerges in group- based policy optimization settings, indicating that GRPO and its variants also subject to the same limitations observed in preference optimization frameworks such as DPO 20 19
Definition 4.1
Let Tew and Thy, denote the initial and final language models, before and after optimizing a preference learning objective I (e.g., Eq. (1)) over dataset D. such that I [6pm. < [(mm). We say that LLD occurs for tuple (x, y
for small nonnegative constant 2 O.
In Tep (g) (x) < In To (3 xx) +c.
4.1 Understanding LLD
We begin by showing that GRPO effectively performs a weighted group preference optimization between two groups of responses: correct and incorrect ones (detailed proof in the appendix).
When reward is binary, GRPO performs preference optimization between two distinct groups: the group of correct responses (r; = 1) and the group of incorrect responses (v; = 0). Specifically, the optimization objective reduces the following:
Lemma 4.2
N
To (x)
N
Tel TT (x)
4+
min
+
Σ J=1
max
T8 (y: (x)
Te Uj (x)
where p - p(x) A
1 Eiela 1[ra(z)= 1] denotes the correctness rate for given input, N+ = pG
and N = (1 G) are the sizes of the correct and incorrect response groups respectively, and the group-specific weights are defined as p+ 1-p_ and p
vp(1-p)
Vp(1-p)
Next, we introduce an assumption unconstrained features, which allows us to focus our analysis on the flnal-layer hidden embedding by E IRd and the token unembedding matrix W E RR///dd where V is the vocabulary of tokens.
5

---

Assumption 4.3 (Unconstrained Features) Expressive (enough) neural networks can produce wn- constrained embeddings bx R independent of the architecture 's specific complexities [27 T6
34
These embeddings are subsequently transformed into logits by token unembedding matrix The resulting logits are passed through a softmax function to yield a probability distribution over possible next tokens. To assign probabilities to sequences y E V the language
19V.
W E
model Ta operates in an autoregressive manner, i.e., Toly x) = -
y) Softmax Whx.y
Here, he yes is the embedding of sequence (x, y=*) and Softmax(-)) (- is the yk-th entry of the V- -imeenionnl softmax map. We focus on the online training setting for GRPO, state in Sec. S.2 the effect of the clipping operation is effectively removed. We assumee that the objective is optimized via gradient flow and theoretically analyze the dynamies of the log-likelihood of positive response, d In (o(s) (y x). yielding the following characterization (see the appendix for a proof). di
Theorem 4.4 For any question æ, at any time i 2 0 of training, and any correct response y i d In ToCy (x)
[]+], in addition to the dependence on token unembeddings, the likelihood change at exhibits increased laginess (that is, has smaller magnitude) the following quantity increases:
My
lu
, O (Dx.vlex x.y, sKc
OK (bx.vlee h bx (5)
impact of negatiie gradient
Here, K2,33' and og. are token-level prediction error similarity weights, which quantify the similarity of token-level prediction error across responses (see Appendix for formal definitions. ). We refer to the quantity in Eq. (5) as Group Weighted Hidden Embedding Score (GWHES).
The first term in Eq. (5) captures the influence of negative gradients the likelihood of the correct re- sponse. Specifically, this shows that a negative token with a large value of o x"i h ".
more likely to cause the LLD of the correct responses. This motivates the following corollary.
Corollary 4.5 For any question x, the negative gradient associated with the hidden embedding of token ke in any incorrect response Yj j E [,-), will exert greater adverse effect on the likelihood d In (f(())y) (x) as the following quantity getslarger: change ds
O (h
(hx.vl.e Px.
(6)
This can occur when the negative and positive tokens have high embedding similarity, theoretically supporting the observation Fig. that LLD samples are often nearly correct.
4.2 Identifying LLD Samples
Theorem 4.4 suggests using AGWHES as a met- ric identify samples that cause LLD. Here, we verify the validity of this metric experimen- tally. For Qwen-1.5B ..ee-deeseeck, we generate responses for the first 100 questions from the AIME dataset (1983-2023) and compute the GWHES scores for those questions with non- extreme predicted probabilities (0 < < 1), resulting in 57 valid questions. For Qwen- 2.5Math- 1.5B, we use questions from the Math dataset and retain 47 valid questions. We then calculate the Top-K K overlap accuracy between two rankings of questions: one based on the ascending
Top-K
Qwen- -e5B-deeseeck
Qwen- Qwenth.5B-math
GWHES
Random
GWHES
Random
10
50%
17.5%
60%
21.3%
15
75%
26.3%
75%
31.9%
Table 1: Ranking questions by GWHES results in significantly higher Top- overlap with the likelihood-change ranking compare to ranking ran- domly. This demonstrates the effectiveness of GWHES in identifying L.DD samples.
'Theorem 4.4 can be seen as an extension of 19 Thm.3] for DPO to the more challenging GRPO setting with braeed structure. Unlike DPO, we will exploit this grouppstructur to selectively penalize tokens rather than responses.

---

order of likelihood changes Axx) as defined in Eq. 2 (denoted O2 and the other based on the descending order of GWHES scores as defined in Eq. ( (denoted 2 The overlap accuracy is computed as 1 The results in Tab. indicate that the overlap achieved using GWHES is significantly greater than that of the random ranking baseline, highlighting strong alignment
K
between high GWHES scores and LLD samples.
5 Selective Token Penalization in Incorrect Responses
Theorem 4.4 identifies samples prone to causing LLD and Corollary 4.5 quantifies the influence of individual tokens' negative gradients on the likelihood of correct responses. The goal of this section to explore whether appropriately controlling negative gradients can mitigate LLD and further lead to performance improvements over vanilla GRPO. One approach could be using AGWHES to identify negative samples y; that cause LLD and entirely discard them during training. However, this compromises data efficiency and fails to address cases where samples only marginally improve yy Instead, we propose a more targeted approach: identify tokens that negatively impact A A(x) and reduce the penalty applied to them.
Math Question: How many elements are the interscction of the set of all the prime numbers less than 30 and the set of all the odd numbers greater than zcro?
Ground-truth Answer:
Sample from [Math dataset]
Correct Response
Incorrect Response Contnin Correct Steps
To determine Step 1: Identify the set of all prime
To determine than 3 0 00:2 2, 2.5,777 3,577, 17,11,11 11,13. 17. 19, 2 3.29.9922 9 So, the set of all prime numbers less than i (2.3.5.7. 113, 1 17,1 1,,1,, ,,23,29 2
First, let 's list the prime numbers less
numbers less than 30. The prime numbers less than 30 ure: 2, 11, 13, 17, 19, 23, 29. So, the of all prime numbers than (2, 3, 5, 11, 13, 17, 19, 23.
Next, let list zero The set of odd numbers greater than zero is { 3,5,7.9, 11, 11,,,,,1,,,,,,,,,, Now, we find the intersection of these two sets The intersection consists of numbers that are both in the set of prime numbers less than 3 0 and in the set of all odd numbers greater than
all odd numbers greater than
Step Identify the set all odd numbers greater than zero. The set of all odd numbers greater than zero is: {1.
19, 21 2,,,,33, ,,,
3, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, ...]. So,
the set of all odd numbers greater than zero is {1, 3, 7.
91, 13, 15, 17, 19, 23, 25, 27, 29, ...}.
Step 3: Find the intersection of the two sets ... Prom the
From the lists above we can see that the only numbers that both prime and odd are the odd prime numbers.
two sets, we see that the common elements 3, 5,
23, and So, the interscction (3, 5. 7, 11, 13, 17, 19, 23, 29).
11, 13,
Therefore, the intersection is (3.5,7, 13,5,7,11,13,17,
Step 4: Count the number of elements in the intersection. The number of elements in the intersection is 9. Therefore, the number of elements in the infersection of
3 } To find the number of elements in the intersection we count the elements in the list {3, 5 5.7.11, 1,11,11 3, 1 7, 9, } There are 8 elements in the intersection Thus, the number of elements in the intersection is !boxed 8
the set of all the prime numbers less than 30 and the sel of all the odd numbers greater than zero is \boxcd (9).
Figure 3: Key insight: Tokens of negative samples (incorrect responses) can be logically or step- correct. Tokens with high NTHR tend to strongly correlate with these types of tokens (highlighted in red). yThe bold dots represent omitted reasoning.
5.1 Negative Token Hidden Reward (NTHR)
Motivated by Corollary 4.5 we define the impact that an individual token k of incorrect response vイ E [N-] has on the likelihood of a group of correct responses as follows:
N+
ly
(Bevt h by
which defines relative influence from imposing a negative gradient on yj to all tokens in all positive responses. Specifically,, a token from an incorrect response can negatively affect the likelihood of group of group of correct responses when J 2 with the magnitude of 57.ck indicating the severity of this effect. We refer to this value as the negative token hidden reward (NTHR):
7

---

it quantifies token's 's detrimental impact on correct responses' likelihood. In Fig. 3 we visualize tokens with high $ JR values- highlighted in red. We find that many of these influential tokens are logically or stepwise correct terms (such as "odd," "prime," and "intersection" in the specific example). These terms exhibit strong semantic alignment with the correct responses.
5.2
NTHR selective token penalization.
As shown in Corollary 4.5) tokens with high positive NTHR values tend to cause signifi- cant reductions in the likelihood of generated correct responses. To mitigate this adverse effect, we introduce a selective penalization strategy (detailed in Algorithm[]] and complex- ity discussion in Appendix) that attenuates the penalty on tokens from negative responses whose NTHR scores exceed a threshold т. Concretely, we define the resulting set of se- lected tokens as:
Algorithm NTHR selective token penalization
I: Input:
Responses' hidden embeddings
{h NN
(h] 11=1 coefficient a, scale factor
17 and 3.
[[
2: for 1' to N
do
87 Eq.
end for (i-mingeNt]] 8/-
6: Select negative tokens: for each negative response j do
Vj (yTA $j.ck' > >
(8)
8: V + + Jj.c
end for
To determine the threshold т, an effective prat- ical strategy is computing the minimum av- erage token-level influence exerted by each correct response all other correct responses.
10: Apply selective penalization:
11: for each retained token y j, E V do
12:
AJ, < 17 AJ
13: end for
Formally, we set 7 = B minve[N $
where 38 is scale factor and b measures the average impact of the i'-th correct response's to- kens on the likelihoods of other correct responses:
N+ lytt
(t) (t (0), Der (1)
k=1
A larger क indicates a stronger mutual influence among positive responses, which could be interpreted as an estimation of the "local clasticity" of a deep neural network (6. Finally, to apply selective penalization, define the advantage of each retained token using scale factor 1, resulting in A J. 17 A The scale factor will reduce the penalty selected negative tokens.
5.3 NTHR Selective Token Penalization Mitigates LLD
Impact af UTA Uletthood Changes Ohean C.50 Mado
Inppctt of WTH on Ltodl and Changes Jowen 30 Nato
2000
Oargehh (ay
Oharge Byy Log Lidfrod
25 1. R. 2.25 2.2
201000000000000 250000 Zample Orders Lladihood Change)
Sample (Camtdd by Weeeeed Change)
(a) Qwen-O.5BB 0.5B-IInsrrue
(b) Qwen-3B
(c) Qwen- Dween-1.B--ae 1.5B-Math
Figure 4: GRPO+ NTHR consistently improves likelihood change of correct responses, indicated by the green bars exceeding the blue hars. While GRPO Random offers only modest improvements, GRPO+NTHR consistently outperforms it, highlighting the effectiveness NTHR in identifying LLD tokens.
To evaluate the effectiveness of our NTHR selective token penalization in mitigating LLD, we conduct experiments using the same setup described in Sec. 4 Specifically, we identify the selected

---

token set y in incorrect responses using scale parameter 3 = 1, and apply no penalization to these tokens by setting 1 = We refer to this variant GRPO+NTHR (green bars). We then compare -iRPO-Random Random (orange bars), where we randomly select V tokens and set their advantage (thus avoid penalizing them in optimization).
the following two baselines: (1) Standard GRPO (blue bars), and (2) GRPO-
The results Fig. show that GRPO+NTHR consistently improves likelihood change across all samples, as evidenced by the green bars surpassing the blue bars. Notably, while removing negative gradients entirely- in the Pos Only in Fig. Ic can sometimes reduce the likelihood change compared to GRPO in Owen-Math- Math-1.5B 1.5B, our GRPO+NTHR approach (Fig. (e) maintains consistent improvements. Furthermore, unlike GRPO Random that provides only modest gains across all models, GRPO+NTHR delivers consistent and substantial improvements, highlighting the effectiveness of our proposed NTHR identifying L.I. tokens.
5.4 Performance of NTHR Selective Token Penalization
Having demonstrated the impact of negative gradients on likelihood change and the effectiveness of method in mitigating the LLD issue in Sec. 5.3 now study the effect of negative gradient model performance. For this, we finetune models with different sizes using MATH dataset (levels 3-5) 7 and use greedy decoding to evaluate finetuned models on five math benchmarks: AIME24, AMC, MATH500, Minerva, and Olympiad (detailed implementation details in Appendix).
We find that NTHR consistently boosts model performance
various sizes, with detailed results presented in Tab. 2 Notably, even for Qwen2.5-Matt which shows a substantial improvement of 20% after RL fine-tuning outpacing gains in other models-NTHR still provides and Qwen2.5-1.5BB
NTHR consistently outperforms GRPO by 1.1% and 1.5% average respectively. Similar improvements are observed with a larger model Qwen2.5-3B, where NTHR reaches 36.30 % compared to GRPO's 33.88 and the base model's 31.36 %. We further fine-tune Qwen2.5-Man Mathh on the more challenging DeepScaler [14] dataset and observe that NTHR outperforms GRPO by 1.8%.
average improvement of 0.8%. For Qwen2.5-0.5BB
Base model + Method
AIME24
AMC
MATH500
Minerva
Olympiad
Avg.
Owen2.5-Math-1.51
Base
3.3
20.0
39.6
7.7
24.9
19.10
GRPO
13.3
57.5
71.8
29.0
34.1
41.14
Pos Only
10.0
57.5
70.6
30.1
31.0
39.84
NTHR
16.7
57.5
70.8
30.5
34.2
41.94
Qwen2.5-0,5B-Ins
Base
0.0
2.5
33.4
4.4
7.0
9.46
GRPO
0.0
7.5
33.8
9.2
8.1
11.72
NTHR
0.0
10.0
36.6
8.1
8.6
12.66
Qwen2.5-1.5B-Ins
Base
0.0
22.5
53.0
19.1
20.7
23.06
GRPO
3.3
32.5
57.2
18.8
23,0
26.96
NTHR
35.0
58.8
21.0
20.9
28.48
Qwen2.5-Math-1.51 (deepscaler)
Base
3.3
20.0
39.6
7.7
24.9
19.10
GRPO
10.0
42.5
72.4
32,4
31.9
37.80
NTHR
16.7
47.5
73.2
29.4
31.4
39.60
Qwen2.5-3B
Base
10.0
37.5
58.6
26.1
24.6
31.36
GRPO
6.7
35.0
66.6
31.2
29.9
33.88
NTHR
10.0
47.5
65.6
31.6
26.8
36.30
Table 2: Results across selected math benchmarks for different Qwen2.5 models and methods. NTHR consistently provides average performance gains various models.
Conclusion
In this work, we study how negative gradients affect the likelihood correct responses in GRPO.
We identify phenomenon term Lazy Likelihood Displacement (LLD), where penalization of

---

incorrect responses inadvertently reduces or lead to small likelihood change of correct ones. To address this, we propose the negative token hidden reward (NTHR) along with selective token-level penalization strategy. NTHR reduces the penalty on tokens in incorrect responses that contribute most to lowering the likelihood correct responses, thus successfully mitigating the LLD issue. Through experiments on math reasoning tasks across models ranging from 0.5B to 3B, we demonstrate that NTHR improves GRPO, validating the effectiveness of our approach in addressing LLD and enhancing performance.
This work was partially funded by the NSERC Discovery Grant RGPIN-202 1- 03677, Alliance Grant ALLRP 58 1998-222, 2, the Natural Science and Engineering Research Council of Canada (NSERC), the Canada CIFAR AI Chairs program, the Canada Research Chair program, an I1TP grant funded by MSIT, the BC DRI Group, and the Digital Research Alliance of Canada.
Acknowledgments:
References
[1] Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprini a2X55 2025.
[2] Wenlong Deng, Yize Zhao, Vala Vakilian, Minghui Chen, Xiaoxiao Li, and Christos Thram- poulidis. Dare the extreme: Revisiting delta-parameter pruning for fine-tuned models. arXiv preprint arXiv:2410.09344, 2024.
[3] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self self- improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.00 1tt::2553001077, 2025.
[4] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945, 2025.
[5] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-leve bilingual multimodal scientific problems. arXiv preprint MxXi:22402.14008 2024.
[6] Hangfeng He and Weijie Su. The local elasticity of neural networks. In International Conference on Learning Representations, 2020.
[7] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
[8] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner- zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprini arXiv:2503.24220. 2025.
[9] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, et al. Openai ol system card, arXiv preprint arXiv:2412. 16720 2024.
[10] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning, arXiv preprint arXiv:2503.0951 2025.
[11] Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-rl: Reinforce- ment learning for generalizable medical reasoning in vision-language models. arXiv preprint arXiv:2503. 13399, 2025.
[12] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
10

---

[13] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv: arXiv:2503.20783, 2025.
[14] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang. Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol ol-preview with a 1.5b model by scaling rl, 2025. Notion Blog.
[15] Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, and Dacheng Tao. The energy loss phenomenon in rlhf: A new perspective mitigating reward hacking. arXiv preprint arXiv:2501.19335. 2025.
[16] Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory. Signal Processing, and Data Analysis, 20(2):11, 2022.
[17] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402. 13228, 2024.
[18] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
[19] Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, and Boris Hanin. Unintentional unaligmment: Likelihood displacement in direct preference optimization. arXiv preprint arXiv:2410.08884, 2024.
[20] Yi Ren and Danica Sutherland. Learning dynamics of Ilm finetuning. arXiv preprint arXiv:2407.10490 10490, 2024.
[21] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[22] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 40Xiv:2402.03300, 2024.
[23] Gemini Team, Rohan Anil, Sebastian Borgcaud, Jean- Baan-Baptiste Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
[24] Hemish Vecraboina. Aime problem set 1983-2024, 2023, URL httph::tpt///tt www. kaggle com/ dataastss/henisseveerbbbbbn -ttp:/aaeeeemeeeemmmaeeaemmmmeee -983--8-
[25] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1511 15/15 2024.
[26] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint wxXiv22499.1212 2024.
[27] Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax bottleneck: A high-rank mn language model. arXiv preprint arXiv:1711.03953, 2017.
[28] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. /2284, 2023.
[29] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint ar 2533 2025.

---

[30] Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing Ilm reasoning generalists with preference trees. arXiv preprini 00X0022404.02078, 2024.
[31] Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang. TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.051I1 2025.
[32] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
[33] Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcom- ing reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation. arXiv preprint arXIv:2403.05177.2222
[34] Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis. Implicit geometry of next-token prediction: From language sparsity patterns to model representations. arXiv preprint arXiv:2408. 15417, 2024.
12

---

7 Appendix
Notation. For any time é 2 0, use (t),, w. (t), and hg (t) denote the token unembedding matrix, unembedding of a token x E V. and hidden embedding of 2 E V", respectively. We let and #< be the first l 1 tokens in 2. For a question æ, the old policy
24 be the k-th token Tie generates group of G samples and resulting (z, fv," N- { hx-), where N + +N G.
Lastly, we denote by e; E IR Ml the standard basis vector corresponding
V.
7.1 Proof of Lemma 4.2 GRPO Group Preference Optimization
In this section, we demonstrate that training with GRPO constitutes preference optimization. We adopt binary reward system, assigning 1 to correct responses and 0 to incorrect ones, consistent with recent works 13 [29 is notable that this also applies to its variants, e.g., DAPO (2)]
Dr.GRPO [3] GPG M etc.
For single question æ, we simplify the expected loss ( in Eq. by omitting the token normalization term
yielding:
E 1y 18 No (-1az)
Σ min (7r, AL, clip 1
(10)
The success probability for question x is Pi(ri,r = 1) ~ P(r 1) where we use a population
approximation for large enough G. In this case, we take = and = /p(1 - p). Consequently,
the advantage A becomes:
1-p Vpll-p)
if = 1,
A
(11)
if O.
Vp(1-p)
Since
min (7rx(0)A,, A clip (0), 1 1 E, 1+
equivalent tos
Ai, min (1), 1 +)
if A > 0 + r 1=11
A.. max (7H, (0); e),
if A <0 r: 0,
the expected loss then becomes:
1-p
To To (xx) TBe (y (x)
E E( (1))
E Etg 1G No
min
1, 1
Vp(1 p)
p
( (x)
E
max
1 -
1 r,
Vp(1- - p)
Te (y/ (x)"
(12)
When generating G sampled responses, the loss becomes:
N
(3, (x) 1 + p max
Toly, (x) Th (yj (x)
(13)
Teaa (y:" (x:
where N PG, N (1-p)G, pt Vp(1-p)
and p
This represents group
VM(-p)"
preference optimization, i.c., increasing the likelihood of correct responses while penalizing incorrect ones.
7.2 Proof of Theorem 4.4
Assume that all responses sequences whose first tokens are distinct from each other. Then analyze the likelihood change for correct response yi by measuring a In (Tcc x). According
13

---

to the chain rule, using the loss in Eq. 11 and denoting Tck = =o(y; xx y.,c) Th (y., Xx, yt.ck) for brevity, we can obtain:
and Told,
d
dQ(l) dt
(V, [x) = In (o (x),
(14)
dt
(V la mocey (v, 100,
lu
p=
Teck
(1+c-
Tick
Vin Vin 2, vrrck)
<old
Told,
Tec 8 NeWK
v (I 2, (j.ck))
Told,<k'
Told,<kk
(15)
if x 2 0 if x < 0
where ((x)
is the gradient of the clip function. We assume Te(e) = since GRPO
0
is online or nearly online due to sampling new responses at each iteration. Thus we obtain:
N+
d
In Toce) (2, (20)
v In (2 (x))
p In To(o) To( (y (x)) P In To
dt
As per the unconstrained features Assumption
the model's trainable parameters are
0= 0=(w W whr, ( (EeN 1uti (ha. BEENN-,.E........ 13
Here, we also used the (mild) assumption that all responses differ in their first token. Unfolding the gradients with respect to these parameters yields:
N+
N
In To(() (w) = Vw ln Toce x),p*
Σ
Vw Toce [x) p
Σ
Vw In (x)
dl
N+
N
Vn, In Ta(e) (2111
Vn, In (TC p
Vha In (000))((1..11
j=1
V matee
In (yt x x1
(16)
For softmax model output the gradients can be easily computed follows:
|=|
Vw In (o(e)
Σ (- z<1)) hT (t)
In addition, the gradient with respect to the hidden representation each position is:
Vhe In (2(e) Wze (1)
Σ
(e(o =<x) wz((), A E (1,...,,12))
ZEV
Putting this back in (16) together with a few algebra steps, yields
In To(o (v."" (x) = (I) (II) (III) + (IV)
(17)
dt
14

---

where:
(1) = p
k. h (neviie (t), he Mear (t)
(18)
l,,1
(II)
p
(t) (ha.viee hz)
M. sicer (8))
(19)
(III)
w Via
(1)
Σ
(2(0))
wz)(), Σ x 1
Σ j=1
p
w
(20)
4;11
SEV
1
(IV) =
Σ
W Mi,k
(t)
Σ
To(e) yl -w (t(())
(21)
2=
zEV
where af (t) = (e -Tcc) ci vv - To(e) (-12, y ve,e)))) and (t,)(()
=
(ext.
VJj, TT( (- ))
Specifically, (I) and (II) capture how token embeddings influence changes in likelihood, while (III) and (IV) reflect how the geometry of token unembeddings governs such changes. Our focus on (I) and (II) because:
Token embeddings encapsulate the contribution of all network parameters excluding the token unembedding layer.
Token embeddings are influenced by the words in the sample, which span a broader space than token unembeddings.
Our results in Tab. furthrr validate the effectiveness of these terms. Thus we arrive at Theorem (II) (I) increases, the likelihood change decreases.
7.3 Implementation Details
Dataset. For training, we use the MATH dataset (levels 3-5) train the model. Additionally, we include a subset of the DeepScaler dataset [14], which contains more challenging problems. For evaluation, we assess the reasoning capabilities of the fine-tuned models on five standard math
benchmarks: AIME 2024 (24), AMC, MATH500, Minerva Math 12] and OlympiadBench (5)
Models. We we select models range from 0.5B-3B (25). We use 0.5B-in and 1.5B-ins models as 32 found small base model may not follow the format prompt well. We also use Qwen2.5- 1.5B (26) show the influence of math knowledge. For 3B model, we use Qwen-2.5-3B (25 base model. For all models, we conduct reinforcement fine-tuning using GRPO and NTHR using the same hyperparameters.
Hyperparamters. For the 0.5B model, we use two A6000 GPUs with a batch size 32, maximum rollout length of 2500 tokens, learning rate -7 and mini-batch size of 16, resulting in two iteration updates per training step. We use math dataset to train the model for 105 steps. For the larger models, mini-batch size of 64, leading to four iteration updates per step, for the math dataset, we train for 40 steps, which approximates one epoch. For the DeepScaler dataset, which contains more data, we train for 45 steps.
utilize four A00 GPUs with batch size of 256, learning rate le 6 and
Across all models, we generate rollouts per prompt. We use default sampling temperature of 1.0, clipping ratio of 0.2, and set the KL loss coefficient to 1 x 10-4 The Qwen-math model (26) uses its full context length of 3072 tokens for rollouts, while all other models use a maximum rollout length of 4000 tokens. To accelerate training, we employ dynamic sampling [29]. which filters out samples with zero advantage.
I Lastly, we use 3 = 1 and define the weighting factor as r 2 x |0.5 pl. where p is the success rate. This formulation penalizes questions with success rates near 0.5 less, avoiding excessive reward for easy samples while not over-relying difficult questions, whose scarce correct responses may
15

---

not yield effective token selection. Finally, we set the temperature to 0 to perform greedy decoding, following standard practice in math evaluation tasks 28 2 [3]. This enables the assessment of the model's most confident output, which is essential for math tasks requiring exact correctness and providing users with consistent answers to repeated queries.
7.4 Complexity Discussion
In this section, we introduce several techniques to enhance the computational efficiency of NTHR, which is essential for practical implementation.
Output last-layer embeddings with old policy probabilities. Since GRPO relies on the old policy Told to compute the old probabilities used in the clip(-) operation. only need to extract the last- layer embeddings during the forward pass of the old policy. This approach introduces no additional forward passes and thus keeps the computational overhead minimal. Moreover, last-layer embeddings have been widely adopted in prior work 33 15 for constructing robust reward signals.
Calculate summations first. We more conveniently rewrite 7 matrix inner product.
N |
- (-2,, b" b' (Grji - (-E,, V) (Z a a J.
h"
Importantly, our reformulation involves calculating the summations over i, k first before taking the inner product. This reduction reduces the overall complexity for compute by 3.
Focus on vocabulary of the responses. As the formulation involves computing the outer product between the prediction error vector (e.g., ev. Toce) v(,e))) and the hidden embedding, which
incurs a computational complexity of O(D) d). Since the probability mass is primarily concentrated on the output words, for each question æ, we restrict the computation the vocabulary V associated with its generated responses. Since V < [V/, this significantly lowers the overall cost to O(D (d).
Running time of each module. We also track the average time cost of each module during training, as reported in Tab. 3 Notably, the data generation (Data Gen) module that using dynamic sampling accounts for the majority of the total training time. In contrast, the overhead introduced by NTHR minimal, contributing only small fraction to the overall cost. Notably, although deepseek- 1.5BB has a longer average output length of approximately 3,400 tokens, our NTHR still maintains low time
overhead.
Madeltd
Data Gen
Model Upd
THR
Ref
Old Prob
Total (Sec)
Owen2.55-Mth-- (Math)
250)
140
22
55
55
522
Table 3: Average running time (per step, in seconds) of each module for different models and tasks.
Okk = (ev!.. - TT(. x, y v) Byy - TT(- x, y<k))
(22)
As result, our method does not introduce additional inference overhead, and the overall complexity remains small.
7.5 Performance across training iterations
In this section, we present the performance of various models across training iterations. As illustrated in Fig. 5, although performance fluctuates throughout training, NTHR consistently outperforms GRPO. Notably, for the Qwen-2.5-1.5BB 1.55--1.BBim Ins model, report results update step 100 (corresponding to 25 training steps), as performance begins to decline beyond this point.
7.6 Ablation study
In this section, we conduct ablation studies to shed more light on the role played by different modules.
Abltion on 3. We conduct ablation study on 3, i.e.. the scaling factor for the threshold T in Eq. 88. which is then used to control how bad the negative tokens should be masked out. We set B with oc, 0, 0.1, 1, and train with Owen2.5-Math- 1.5B using Math dataset. Choosing 8 -00 corresponds apply 7) to all incorrect tokens. As shown in Tab. 4 this removal results in worse
16

---

013
2000
Gh10
000
Mile:
801
20
00
Qwast
44:Quca5551 5-1 5-1-5Baa
Qwca2
Figure 5: Performance across training iterations for various models, NTHR consistently outperforms GRPO for most of the training process.
performance compared to GRPO. As further shown in Tab. 4 incorporating NTHR consistently improves performance over GRPO. While different 3 values result in varying degrees of improvement, all settings outperform GRPO, indicating that reducing the influence of identified negative gradients beneficial.
Base model + Method
AIME24
AMC
MATH500
Minerva
Olympiad
Avg.
GRPO
13.3
57.5
71.8
29.0
34.1
41.14
NTHR (3 -00
13.3
50.0
71.8
30.5
34.4
40.00
NTHR (3 0)
20.0
55.0
70.0
29.8
32.9
41.54
NTHR 0.1)
13.3
57.5
71.4
30.9
34.4
41.50
NTHR (3 1.0)
16.7
57.5
70.8
30.5
34.2
41.94
Table Ablation study on B. Although changes in 3 (except 0o) lead to variations performance,
they consistently improve GRPO, indicating that reduce influence of identified negative gradients is beneficial.
Ablation on n7. We conduct an ablation study on 1) using three variants: 17 1 p, 7) p. and 2 10.5 pl. As shown in Tab. while the optimal choice of varies across different models, the variant 2- |0.5 pl consistently yields strong performance across all models. Therefore, we
adopt 2 |0.5 pl in our training.
Base model + Method
AIME24
AMC
MATH500
Minerva
Olympiad
Avg.
Qwen2.5Math-1.55
GRPO
13.3
57.5
71.8
29.0
34.1
41.14
NTHR (7) = p))
13.3
55.0
72.4
29.0
33.3
40.60
NTHR (J) 1 11-7
13.3
62.5
72.8
30.5
34.2
42.66
NTHR (1) 2-
10.5 |0.5 p)
16.7
57.5
70.8
30.5
34.2
41.94
Wwen2.5-3B
GRPO
6.7
35.0
66.6
31.2
29,9
33.88
NTHR (1) p)
10.0
47.5
64.6
33,8
26.8
36.54
NTHR (1) 1
6.7
42.5
63.8
32.4
28.0
34.68
NTHR (1j 2 10.5 p)
10.0
47.5
65.6
31.6
26.8
36.30
Table 5: Ablation study on f).
7.7 Results on Llama
filtered, filtered_ wwtprrrespposseeeeeeeeeeeeeeeeee lengonseeeength/mmean
As shown in Fig. 6 the response length of Llama3.2- IB Instruct declines rapidly after a few epochs, with the av- erage length dropping from about 1.2K tokens to roughly 500. This reduction may stem from the model's 's limited cognitive behaviors 3. but can match Qwen's 's trajectory of seff-improvement (3). De- spite this, We still test NTHR using Llama3.2-18 1 and observed consistent improvements when applying our NTHR method shown in Tab. 6
outa Mccoefo.0001, entcoefo..00 www..rooseeoooooooo meta llam
1200
equipped with them, they
1000
800
600
7.8 NLL Loss Complementary
Figure Response length dynamics of Llama3.2- I---tuuttt across different stages of GRPO training.
Recently, Yue et al. 31 proposed directly applying an NLL loss to positive samples, which aligns closely with the
17

---

Method
math500
minerva math
olympiad
aime24
amc23
avg
GRPO
37.0
7.4
9.3
3.3
20.0
15.4
NLL Loss
36.0
9.6
11.0
3.3
25.0
16.98
NTHR
37.4
8.1
11.0
3.3
20.0
15.96
Table 6: Performance comparison across math reasoning benchmarks on Llama. Bold numbers
indicate the best performance per column.
LLD issue identify, as it increases the likelihood of correct responses. Specifically, incorporating a supervised NLL term reinforces the model's confidence in generating correct outputs. Motivated by this, augmented GRPO with an additional NLL loss and trained using Llama-3.22 1----1.-21---ttii- I-Inttuuct. As shown in Tab. 66 this modification leads to improved performance across multiple tasks. The NLL objective tackles LLD from complementary serrective-strrengttennnng correct responses through positive gradients -hiie NTHR mitigates harmful negative updates. Both methods consistently yield performance gains.
7.9 Results Deepseck-1.50
We utilize four A100 GPUs with a batch size of 256 and a mini-batch size of 64, resulting in four update iterations per training step. Each prompt generates 8 rollouts. We adopt default sampling temperature of 0.6, set the clipping ratio to 0.2, and use a KL loss coefficient of 1 x 10-4 The maximum rollout length is capped at 4000 tokens. We set / 0 and define the weighting factor as = 2. 00.5 pl. We limit training to 35 steps, as we observed a consistent decline in response length with continued training.
Since the context window length limited, the results are very unstable, as a results, we calculate the last two checkpoints" average performance and report the results in the table below. Although this setting may not be optimal or robust, we include the results and observe that our NTHR consistently outperforms GRPO on average.
Base model + Method
AIME24
AMC
MATH500
Minerva
Olympiad
Avg.
Deepseek-1 1.5BB
Base
20.0
50.0
68.8
20.6
29.6
37.8
GRPO
25.0
65.0
81.10
34.6
40.0
49.1
NTHR
25.0
70.0
80.60
33.7
40.7
50.0
Table 7: Results across selected math benchmarks for deepseek- 1.5B.
7.10
Examples of LLD Incorrect Responses
We present the top-ranked incorrect response based on LLD for 2.thh-.5Math-1.5B in Fig. Ic As shown in Tab, g the answer is nearly correct, with the only issue occurring in the final step.
Similarly, Fig. lb shows the top-ranked incorrect response by LLD for 1repscek-1.5B. As detailed in Tab, [ the response includes the correct answer multiple times; however, due to continued self- reflection, it becomes truncated by the context window limit, resulting in final answer that is wrong.
7.11 Words with high NTHR
We present additional examples of words with high NTHR scores. As shown in Tab. TO using the same response from Tab. g|
we observe that the correct parts are associated with high NTHR values, while the incorrect portion in step 5 receives low NTHR values (in black). We also show response with incorrect output format in Tab. [I]] using the same response from Fig. 2 we observe that core steps and correct answer 8 are with high NTHR.
18

---

Model & Data
Problem & Answer
Incorrect Response
Swen2.5mathe
Question: A portion of the graph of quadratic function [(x) is shown.
To solve the problem, we need to analyze the given functions
1,5B
f(x), g(x), and h(x) and determine the points of intersection between the graphs of these functions.
Math (order I)
Graph of fz)):
Let g(z) -fx). h(r)
The graph of f(z) is quadratic function. From the given graph, we can observe the following:
f(-x). a the number
points where the graphs y f(x) and y = ((z) intersect.
The roots f(x) are at and a
b number of points where graphs y f(x)
The vertex of the parabola at =
Graph of g(x) ((fr)--ffee -(x);
and y h(z) intersect, what
The graph of g(a)
reflection the graph of [(x) across
10a + b?
x-axis. Therefore:
Answer: 21
The roots g(z) are also at z ar 3.
The vertex the parabola for g(x) at 2.
Graph h(z) = f(-r):
The graph of lu(r) the reflection the graph (2r) across
y-axis. Therefore:
roots h(()) are at and
vertex parabola for h(()) is x
Next, we determine the points of intersection:
Intersection f(x) and g(z);
Since g(r) (x)) the graphs of f(x) and g(x) intersect
the points where f(x)) f(z). This simplifies
f(x) 0
The roots of f(x) are at x = and x = 3. Therefore, the graphs of f(x) and g(x) intersect at two points:
a=2
Intersection f(x) and h(x)):
Since h(z) f(-z). the graphs of f(x) and h(z) intersect at the points where f(x) = =(-x). This implies that f(x) is function, and its graph is symmetric about the y-axis. and x = 3, but these not symmetric about the y-axis. Therefore, the graphs of f(x) and (((r) do not intersect. Thus: b = 0
The roots of (xx
at
Finally, we calculate 10a b. 10a + b 10 10-2+0 20 Thus, the final answer is:
20
Table 8: Incorrect response example from qwen2. 5math 1. 1.5B, The incorrect response is almost correct.
19

---

Model & Data
Problem & Answer
Incorrect Response
Deepseek- 1.3B
Question: One commercially available ten-button lock may be opened by pressing - any or- der the correct five buttons. The sample shown below has 2, 9} as its combination. Suppose that these locks designed so that sets of as many nine buttons or few button could serve com- binations. How many additional combinations would this allow? Answer: 770
To solve the problem, explain your reasoning step by step, and put your final answer within boxed, Alright, so have this problem about ten-button locks and figuring out how many additional combinations redesign would allow. Let me try to break it down step by step. First, the original lock. It seems like ten-button lock where you need to press five buttons in any order open it. The combination given is 1.2.3.6,9. which is a set of five buttons. So, the question is asking about how many additional combinations would be possible if we redesign the locks to allow sets of many nine buttons or as few as one button. I think I need to figure out the total number of possible combinations before and after the redesign and then find the difference. That should give me the number of additional combinations. Starting with the original lock
AIME (order I)
Therefore, the number of additional combinations is 1022 252 = 770. thats the same answer as before. So, perhaps 770 the correct answer. But wait, let me make sure. Is the original number of combinations one, or 252?
Then, the redesigned lock allows for any number of buttons from nine, 2 2 1022 possible combinations, but the original combination was just one. So, the numher of additional combinations would be 1022 I 1021. But that seems like lot more. Wait, which is it? Is the original combination one specific combination, or is it 252 unique combinations? In the prohlem, it says, "one commercially available ten-button lock may be opened by pressing - in any order - the correct five buttons." "
After redesign, the numher of possible comhinations is 1022. So, the number additional combinations 1022 1021. But wait, the redesign allows any combination from one to nine buttons. So, the number possible combinations is 1022, which includes the original 252 comhinations. Therefore, the
number of additional combinations is 1022- 252 770.
...
The redesigned allows for more combinations, but if you consider that each combination is set of buttons, then the number of additional combinations is 1022 -252 = 770. But the problem says, "the sample shown below has 1,2,3,6,9 as its combination." So, the sample is a specific combination, so thats one combination. So, originally, the number of possible combinations was 252, including the correct one. So, if you redesign it, the number of possible combinations is 1022, which includes all the original 252 combinations plus 1022 252 770 additional combinations. Therefore, the number of additional combinations is 770. But to make sure, let me think another approach. the original lock had only one combination, the number of possible combinations 252, each being unique of five buttons. Then, if you redesign it so that you can have any number of buttons from one nine, the number of possible combinations is Stop due to limited context window length.
Table 9: Incorrect response example from deepseek- 1. 5B on AIME: Although the response in- cludes the correct answers, excessive self-refiection causes the output exceed the context window, preventing it from being formatted correctly. Green dashes means omitted reasoning.
20

---

Model & Data
Problem & Answer
Incorrect Response
Qwen2.5math-
Question: portion the
solve the problem, need analyze the given functions
1.5B
graph of quadratic function [(x) is shown.
f(z), g(z). and h(x) and determine the points of intersection
Math
between the graphs of these functions.
1. Graph of [xx): The graph of (ir) is quadratic function. ohserve the following: The From the given graph, and 3. The vertex of the
Let g(x) -f()), Mar
f(-x). If a is the number of points where the graphs of y f(x: and = g(x) intersect, and b is the number of points
roots of f(z) are
parabola is at 2. Graph of g(a)) - [a:)::The graph of g(r)) is the reflection of the graph of f(x across the x-axis. Therefore: The roots of g(x) ure also at a = 1 and x = 3. - The vertex of the
where the graphs of f(x) and y b(x)) intersect, what is 10a +
parabola for g(x) is at = 2.
3. Graaphh ((x)) = f( f(-x):= The graph k(x) is the reflec- tion of graph of f(x)) across the y-axis. Therefore: The roots of h(x) are parabola h(()) at
Answer: 21
The vertex the
Next, we determine the points of intersection:
4. Intersection of f(x) and g(z]: Since g(z) -f(x).
the graphs of [(x and g(x) intersect the points where f(x) [xx). This simplifies
The roots of f(x) are at a - and x 3. Therefore, the graphs of (ar) and g(r) intersect two points:
a ==2
5. Intersection of f(x) and h(z): Since h(x) f(-x). the graphs of (ar and h(())) intersect the points where f(x) f(-x). This implies that f(x) is even function, and its graph is symmetric about the y-axis. The roots of f(x) at = 1 and x 3. but these not symmetric about the y-axis. Therefore, the graphs of (ic and ls(2) do not intersect. Thus:
b b=
Finally, we calculate 10a b:
10a b 10 +
Thus, the final answer
Table 10: High NTHR (highlight in red) example from qwen2. 5math-1 11.58 5B. Core steps in correct parts are with high NTHR. The erroneous words of step are with low NTHR values.
21

---

Model & Data
Problem & Answer
Incorrect Response
Qwen2.5-0.5B.
Question: What is the greatest integer such that n² 11n + 24 S 0?
To solve the incquality n' 11m + 24 S O, we need to find the values of that satisfy the incquality. Since the inequality is an inequality, we should first find the roots of the corresponding equation n² 11n + 24 = 0. 's solve the 1. Factor the quadratic equation: n² 111n + (n 3)(n 8) or a the equation
Ins Math
Answer:
equation n² 1111
each factor equal Solve
to zero solve for
n:
Therefore, the and 8. Now, need to find
n²- 11-112+221 the values of that satisfy the inequality n² 11n 24 0. Since the inequality is inequality, should check the intervals between the roots. The roots = 3 and divide the number line into three intervals: (-xc, 3). [3, 8). and [8, oo). Let 's test point in each interval to determine where the inequality holds true. 1. For n in (-oo.3), let test a = Q: 0 114 24 > 0. For n (3,8), let test == 52 == +4 -5 +4 = -4 -4 0. 3. For n in (8, co). let 's test n 9: 92 -
8
81 09
1In 8.Hence, the greatest integer n that satisfies the incquality is 8. In boxed format, the answer S.
0. Therefore, the inequality
S holds
true
Table II: High NTHR (highlight in red) example from qwen2. 5-0. 5B Ins. This example have correct answer but in wrong format. Core steps and correct answer 8 are with high NTHR.
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
We give clear motivation and summarized the contribution of our work in introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
In conclusion, discussed the limitaions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

We provide the main theory in main paper and proof and assumptions in appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide algorithms and details in appendix to ensure reproducibility.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer:
[Yes]
Justification:
The data are all opensource data, we plan to open-source all the code after the
paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp cc/public/guu getggiddddeeeeeebbbnninnnn SttSSSabmiiiioooiiiioiooooii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We report the details in appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We run on different models, some model we report the average performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, the paper provides sufficient information on the computer resources needed to reproduce the experiments appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research conducted the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:[Yes]
Justification:
this can help achieve better LLM reasoning.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All foundation models and datacts used are properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any new assets such as code, data, or models. Therefore, there no documentation provided alongside assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasettcoode/modll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
The paper does not involve crowdsourcing experiments or research with human subjects. Therefore, it does not include instructions given to participants, screenshots, or details about compensation.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve any study participants, crowdsourcing experiments, research with human subjects. Therefore, it does not describe potential risks incurred by study participants, disclose such risks to subjects, or obtain Institutional Review Board (IRB) approvals.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: we don't 't rely on LLM in developing our method.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neurre http://ariiaaccomooeeeeeeeee//// cc//oonferencce//2225/LLL for what should or should be described.
29


Reviews:
[{"review_id": "kw3uwvFqme", "reviewer": "Reviewer_JEcX", "summary": "This paper investigates the effect of negative gradients in Group Relative Policy Optimization (GRPO), a popular reinforcement learning method for fine-tuning large language models. The authors identify a phenomenon called Lazy Likelihood Displacement (LLD), where the likelihood of correct responses fails to increase—or even decreases—during training. To address this, they propose Negative Token Hidden Reward (NTHR), a token-level selective penalization strategy that reduces the impact of harmful negative gradients. Theoretical analysis and extensive experiments on math reasoning benchmarks demonstrate that NTHR effectively mitigates LLD and improves model performance across various scales.", "questions": "No", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis paper identifies and analyzes a previously overlooked issue—Lazy Likelihood Displacement (LLD)—in GRPO-based reinforcement learning for LLMs, and proposes a novel solution, NTHR, that selectively reduces penalties on certain tokens to mitigate this effect. The work is well-motivated, theoretically grounded, and supported by comprehensive experiments across multiple model sizes, demonstrating consistent performance gains and strong practical relevance.\n\nWeaknesses:\n\n1. The authors should extend and adapt the existing methods proposed to mitigate the reduced probabilities of preferred responses in DPO to the GRPO setting, and include comparisons with the proposed NTHR strategy. This would better support the related claims made in the introduction and highlight the advantages of the proposed approach.\n\n2. Intuitively, directly applying an NLL loss to the positive samples can increase their probabilities and thereby alleviate the identified Lazy Likelihood Displacement phenomenon, as has been done in [1]. How does this approach compare to the proposed NTHR strategy in terms of advantages and disadvantages?\n\n3. I am curious whether the NTHR strategy might lead to under-penalization of incorrect responses, potentially resulting in the retention of erroneous patterns and degradation of model behavior. \n\n4. The proposed strategy relies on token embedding similarity to identify tokens for reduced penalization, which may be sensitive to the quality of the embedding space. Therefore, it would be beneficial for the authors to evaluate the generalization and stability of the method on a wider range of models, such as the LLaMA-3 series.\n\n5. The authors could cite [2–3] to better support the use of final-layer embeddings.\n\nIf the authors address my concerns through additional experiments and discussion, I would be happy to raise my score to 4 or 5.\n\n[1] VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv 2025\n\n[2] Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation. NeurIPS 2024\n\n[3] The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking. ICML 2025", "was_revised": true, "final_justification": "My concerns—such as the comparison with DPO-style methods, generalization of the embedding-based strategy, clarity of certain figures, and the need for stronger empirical analysis—have all been addressed well.", "rebuttal": ""}, {"review_id": "BwKqRpjTU9", "reviewer": "Reviewer_Nhas", "summary": "This paper investigates the impact of negative gradients on the likelihood of correct responses in GRPO. The authors identify a phenomenon they call Lazy Likelihood Displacement (LLD), where the penalization of incorrect responses can inadvertently reduce or lead to small likelihood changes of correct ones. To address this issue, they propose a method called Negative Token Hidden Reward, which selectively penalizes the advantage of tokens in incorrect responses that contribute most to lowering the likelihood of correct responses. The authors demonstrate the effectiveness of their approach while improving performance on math reasoning tasks across models of varying sizes (0.5B to 3B parameters). The key contributions of this paper are: (1) the identification of LLD in GRPO, (2) the development of their method to address LLD, and (3) the empirical validation of their method's effectiveness in improving performance on math reasoning tasks.", "questions": "Questions:\n- 1) Can you guarantee convergence to an optimal policy with this new loss?\n- 2) Can you relate this new advantage estimation to classical GAE?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strenghts:\n- The paper is well-structured and provides a clear, step-by-step explanation of the problem.\n- The paper is sound, the theorem and the lemma are correct.\nWeaknesses:\n- The paper lacks some theoretical analysis of the impact of using the new loss (convergence to an optimal policy).\n- The experiments are limited to Qwen2.5 on math reasonning. It has been shown that this model can exhibit unexpected behavior, especially in this setting, compared to others [1]. It would be better to confirm that the method also works with other models.\n- The significance and novelty are a bit limited. The proposed method introduces a token-level advantage estimation coming from the LLM itself instead of an independent critic. Therefore, one would expect a comparison with a method using classical token-level advantage estimation like PPO.\n- The method introduces 3 new hyperparameters to GRPO. I acknowledge that an ablation study of those is provided in the appendix. However, it is often easy to beat a method by introducing hyperparameters to it that are tuned to beat the base version on a specific problem.\n[1] Shao, Rulin et al. “Spurious Rewards: Rethinking Training Signals in RLVR.” (2025).", "was_revised": true, "final_justification": "My main concerns were addressed, so I've raised my score.", "rebuttal": ""}, {"review_id": "zQOgjyXwy4", "reviewer": "Reviewer_RWsJ", "summary": "This paper investigates an problem in Group Relative Policy Optimization (GRPO) termed Lazy Likelihood Displacement (LLD), where the likelihood of correct responses sees marginal or even negative updates during training. To address this problem, this paper proposes a novel method called Negative Token Hidden Reward (NTHR), which selectively reduces the penalty on tokens within incorrect responses that are identified as contributing to LLD.", "questions": "Both theoretical analysis and algorithm design involve unconstrained embedding $h$. However, it's unclear to me what the author defines as unconstrained embedding $h$ in the context of the code. In other words, how does a neural network model translate into unconstrained embedding $h$ in your code?", "limitations": "See weakness and questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe theoretical analysis in this paper is robust, although Assumption 4.3 might be somewhat oversimplified.\n\nWeaknesses:\n\n- The experimental analysis lacks comprehensiveness, as the paper only tests on mathematical datasets. \n\n- Although LLD appears intuitively correct that it might lead to decreased performance. Nevertheless, the paper neither provides theoretical analysis nor experimental examples to support this claim.", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "bNTwpcKPBZ", "reviewer": "Reviewer_rZbD", "summary": "The paper identifies the Lazy Likelihood Displacement problem, where the probability of generating correct answers increases slowly or even decreases during RL fine-tuning for LLMs, and analyzed theoretically the origin of this problem.\nThe authors then proposed Negative Token Hidden Reward (NTHR) to account for how positive tokens and negative tokens are related, and perform more fine-grained control over the estimated advantage.\nEmpirical results indicate that the proposed method can improve LLM math reasoning.", "questions": "1. It seems to me that a symmetric argument can also be made about \"positive gradients\", can the authors explain why only negative gradients were considered?\n2. Since the proposed fix is essentially biasing the policy gradient, can the authors elaborate on what are the implications (e.g., convergence, bias-variance tradeoff) of it in the context of RL?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is clearly written and easy to follow.\n2. The qualitative results are compelling. In particular, the finding that NTHR tokens can be used to identify correct tokens in incorrect responses is quite interesting, and may open up new ways to design RLVR algorithms.\n\n\nWeakness:\n\n1. The theoretical motivation appears to rest on the assumption that \"negative gradients decreasing the likelihood of correct response is inherently problematic\"; however, from an RL perspective, that this is not necessarily the case. \nSince we ultimately care about maximizing the expected return, and the policy gradient only guarantees that, locally, following the gradient can increase the expected return of the policy.\nIt is, therefore, acceptable that the gradient decreases the likelihood of positive-reward trajectories, provided this is outweighed by the reductions in the likelihood of negative-reward trajectories.\n2.  Note that every time the policy changes, it can also discover new correct responses.\nConsequently, it is not clear to me whether Eq (2) is a good metric, as it is only computed over previously collected responses.\n3. The proof of Theorem 4.4 is problematic. \nAssumption (line 750) should be put into theorem statements rather than the proof.\nAdditionally, it is not clear to me why (III, Eq 20) and (IV, Eq 21) can be omitted.\n4. The empirical results show marginal improvements over GRPO. While small improvements are not necessarily a concern, the lack of error bars makes it difficult to assess the reliability of the reported gains.", "was_revised": true, "final_justification": "My major concerns regarding the lack of theoretical justification and marginal gain of the empirical performance are not addressed. Consequently, I'm maintaining my original score.", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper identifies Lazy Likelihood Displacement (LLD) phenomena in Group Relative Policy Optimization (GRPO), where the likelihood of correct answers stagnates. The authors trace this to uniform penalties on incorrect responses and propose Negative Token Hidden Reward (NTHR) to selectively penalize only the tokens causing LLD. Experiments on 0.5B, 1.5B, and 3B models show that NTHR improves performance.

The main point of contention was the paper's lack of theoretical grounding. Reviewers noted the analysis lacked the explanatory clarity of traditional RL theory. The authors defended their work as a behavioral analysis, which is a fair perspective. However, a key finding complicates their narrative, a simpler variant (THR) performs even better, suggesting the paper's focus on negative token penalization is an incomplete explanation.

Given that a complete theoretical treatment of modern LLMs is often intractable, the valuable empirical insights this paper provides into a real training problem are sufficient. It sheds light on an interesting phenomenon even if it doesn't provide all the answers. Therefore, I recommend acceptance.
Decision: Accept (poster)