Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs
Authors: Zhihe Yang
Xufang Luo
Zilong Wang
Dongqi Han
Zhiyuan He
Dongsheng Li
Yunjian Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Large Language Models
Generative Models
Post Training
Chain of Thought
Primary area: foundation or frontier models, including LLMs
TL;DR: We identify the issue of over-dominance of low-probability tokens in RL training for LLMs, and propose two effective methods accordingly which evidently enhance the performance of RL-trained LLMs across various models and datasets.

Abstract: Reinforcement learning (RL) has become a cornerstone for enhancing the reasoning capabilities of large language models (LLMs), with recent innovations such as Group Relative Policy Optimization (GRPO) demonstrating exceptional effectiveness. In this study, we identify a critical yet underexplored issue in RL training: low-probability tokens disproportionately influence model updates due to their large gradient magnitudes. This dominance hinders the effective learning of high-probability tokens, whose gradients are essential for LLMs' performance but are substantially suppressed. To mitigate this interference, we propose two novel methods: Advantage Reweighting and Low-Probability Token Isolation (Lopti), both of which effectively attenuate gradients from low-probability tokens while emphasizing parameter updates driven by high-probability tokens. Our approaches promote balanced updates across tokens with varying probabilities, thereby enhancing the efficiency of RL training. Experimental results demonstrate that they substantially improve the performance of GRPO-trained LLMs, achieving up to a 46.2% improvement in K&K Logic Puzzle reasoning tasks. Our implementation is available at https://github.com/zhyang2226/AR-Lopti.

Published as a conference paper at ICLR 2026
Do NOT LET LOW-PROBABILITY TOKENS OVER-DOMINATE IN RL FOR LLMS
Zhihe Yang' Xufang Luo Zilong Wang? Dongqi Han?
Zhiyuan He?
Dongsheng Li² Yunjian Xu Xu
'The Chinese University of Hong Kong, Hong Kong SAR, China. Microsoft Research Asia, Shanghai, China.
3 The Chinese University of Hong Kong, Shenzhen Research Institute (SZRI), Guangdong, China. @mae. cuhk. edu.hk
lhhynng, @mxu
xuuflu,, wangzilong, dongqihan, dongsli [ongslijmmicoosftt.com emicrosoft.ccoommmooooooooo.o com
ABSTRACT
Reinforcement learning (RL) has become cornerstone for enhancing the rea- soning capabilities of large language models (LLMs), with recent innovations such as Group Relative Policy Optimization (GRPO) demonstrating exceptional effectiveness. In this study, we identify critical yet underexplored issue in RL training: wwwprobbbbility tokens disproportionately influence model updates due to their large gradient magnitudes. This dominance hinders the effective learning of high-probability tokens, whose gradients are essential for LLMs' performance but are substantially suppressed. To mitigate this interference, we propose two novel methods: Advantage Reweighting and Low-Probability Token Isolation (Lopti), both of which effectively attenuate gradients from low-probability tokens while emphasizing parameter updates driven by high-probability tokens. Our approaches promote balanced updates across tokens with varying probabilities. thereby en- hancing the efficiency of RI. training. Experimental results demonstrate that they substantially improve the performance of CERPOOtrrinee LLMs, achieving up a 46.2% improvement in K&K Logic Puzzle reasoning tasks. Our implementation is htt.tsuuu..oo/..iieaaeeeee hetpo///hyagg2222a20 6/A-//2/////////////// i
available
INTRODUCTION
The reasoning capabilities large language models (LLMs) have recently achieved a milestone breakthrough with the integration of reinforcement learning (RL) during post-training phase (Jaech) ett al. 2024; Guo et al.) 2025 Team et al., 2025). Intuitively, the vast vocabulary size and the auto-regressive generation mechanism of L.LMs pose significant challenges for effective exploration due to the exponentially large state space. DeepSeck-R1 (Guo et al.] (025) climinates this bias, demonstrating that "simple RL with rule-based reward' can significantly enhance the reasoning abilities of LLMs without relying on scaffolding techniques such as Monte Carlo Tree Search (MCTS) (Xie al., 2024b) Chen et al. 2024) Progress Reward Modeling (PRM) (Lightman) et al. [022]Waag et al. 2024). Moreover, they introduce a novel algorithm, Group Relative Policy Optimization (GRPO) (Shao et al., 2024), which has proven highly effective in the domains of mathematics and code, inspiring numerous follow-up studies.
Yu et al. (2025) and
and[Liu [ii al. (2025) consistently report that GRPO training leads to progressively longer response lengths, while the increase does not correspond to proportional improvement in the model's performance. They attribute this trend to the bias in update weights related to response length inherent in GRPO's objective. Xiong al. (2025) conduct comparison between GRPO and Proximal Policy Optimization (PPO). They find that the instability of PPO. compared to GRPO, arises from its unnecessary bias toward entirely incorrect responses on overly difficult prompts. In contrast, GRPO mitigates this issue by discarding such prompts through a within-prompt normalization operation. These findings highlight the substantial impact of update bias training outcomes.
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Sampling & Rating
Training
updated and froeen grob-interve)
idcally tokens in
should NOThen probubiiity change
(a) :07
(c)
Updating
Prssabityy Change
Low-Prob
Charge
0.1
Proqeatcy
10--
Nete: Prob-change
in iti almost the same as updoding (c).
0.25
1.36
-00
(b)
(d)
Updatiing High- Prob Tobana Only
Redeenio updatinn
Advnntage
1.
Note: Prob-changa
1.
prob tokans
even smaller than
where they are
0.5
Proseteiity
Probudailty Intarnal
Probetaitty
Figure 1: Experimental analysis on the K&K Logic Puzzle dataset during GRPO training of 5- ruct- Tokens are divided into four groups based on probability quartiles. (a) Token probability distribution and (b) corresponding advantages. (c) Token probability changes after updates (using SGD with Ir=le-3) and (d) gradient norms for each probability group. Effects of selective updates: (c) Probability changes when only tokens in the lowest quartile (probability < 0.25) are updated, and (f) when only tokens in the highest quartile (probability > 0.755) are updated. To ensure clarity, the top I % of outlier samples in the violin plots for token probability changes are excluded. Results are averaged over 10 randomly sampled batches.
In this study, we identify another important source of update bias in RL training, which is orthogonal to aforementioned ones and has rarely been noted in prior research. This bias arises from the gradient perspective and is strongly correlated with the token probabilities. As shown in Figure I during GRPO training, tokens divided into four groups based probability quartiles. The policy gradient is conducted with the advantage presented in Figure bb). Figure ['d) shows that low-probability tokens generate disproportionately larger gradients compared high-prrobbbility ones. Since each RL update involves hundreds of thousands of tokens with interacting gradients, low-probability tokens are expected to have a greater influence. To verify this, we independently update tokens from the lowest and highest quartiles, as shown in Figures I e) and (f). The pattern in (e) closely matches (c), while (f) looks significantly different. Interestingly, ly, in (e), even though high-probabillty tokens were not updated, their probabilities changed more significantly than when they were updated (as shown in (I). Thus, we conclude that wwwpobbbbbllty tokens dominate model updates during RL training and that this dominance may impede the precise adjustment of the probability distribution across all tokens. Notably, we observe that high-proban probability tokens are much less likely to be updated in the correct direction compared to low- www-robabilltyy tokens (cf. Figure
By deriving the gradients induced by individual tokens, we reveal a key property of RL training that explains the phenomenon illustrated in Figure T Specifically, for an L.M comprising a benign neural network, the gradient norm of any intermediate activation corresponding to a single token is bounded between two values proportional to (1 (1-x). x), where π is the token's probability. This property underscores that tokens with lower probabilities result in larger gradient magnitudes, whereas tokens with probabilities approaching yield gradients that are nearly negligible.
To mitigate the over-dominance of low-probability tokens and promote more efficient updates, we propose two simple yet effective methods: Advantage Reweighting, which reduces the weight as- signed to low-probability tokens, and Low-Probability Token Isolation (Lopti), which separates oow-probability tokens and updates them prior to high-probability tokens. Both methods attenuate gra- dients from low-probability tokens while emphasizing parameter updates driven by high-probabille tokens. Notably, the first one incurs almost no additional computational cost. These methods can be applied independently, each providing benefits, or together, with the potential for further performance improvements. Experimental results demonstrate the effectiveness of the proposed methods across various datasets. In particular, on K&K Logic Puzzle dataset, they enhance the performance of naive GRPO (trained from Qwen2. 5- 3B- Inst ruct) by 35.9% and 38.5%, respectively, and by 46.2% when used together.
2

---

Published as a conference paper at ICLR 2026
In summary, our contributions threefold: (1) We identify critical issue in RL training for LLMs that has received limited attention: lowppbbbayylity tokens disproportionatel dominate the updates due to their large gradient contributions. (2) We provide concise theoretical explanation for this phenomenon. (3) Based on the identified issue, we propose two simple yet effective methods, which significantly improve the downstream performance of GRPO-trained L.MMs across various datasets.
RELATED WORK
As fundamental technique driving recent advancements in LLMs, reinforcement learning is at- tracting increasing attention from researchers. In this section, we provide a concise overview on the development of RL in the context of LLMs.
RL was pioneered by OpenAl as the final step of post-training to further align fine-tuned large models with human preferences (Christiano et al., 2017 Ziegler et al.) 2019 Stiennon et al.) 2020: Ouyang iet al. (022). By leveraging vast amounts of human preference data and stable RL algorithms such as PPO (Schulman al. 2017), numerous enterprise-level language models have benefited from this approach and have been widely adopted. Notable examples include ChatGPT (Brown et al.
2020
AAchiam et al., 2023), , LLaMA (Touvron et al., 2023a by Dubey et al. 2024). Qwen (Bai et al.
2023 Chu et al.] 20233 Yang et al., 2024), Gemini (Team et al. [023] 2024). and Claude (Anthropic
2024). Nevertheless, the challenges of collecting high-quality data that accurately reflect human
preferences, the limited performance of open- source LLMs, and the computationally intensive training requirements of PPO-like online RL algorithms pose significant barriers for further exploring RL's potentiality in the domain of L.LMs. Most studies have focused simplifying RI. algorithms and directly leveraging preference data to optimize models. Representative works include Direct
Preference Optimization (DPO) (Ratailov et al.. 2023 2024), related analyses (Xu et al.) (024b)] Zhong et al.] 2024 Ren & Sutherland 22025), and improved variants such as ORPO (Hong et al. 2024). CPO Xu al. 2024a) and SimPO (Meng et al. 2024).
Recently, the emergence of www.h-oof-tthggggeee (CoT) (Wei et al. 2022) reasoning and its integra-
tion into both pre-training and post-training processes have significantly advanced the foundational capabilities of LLMs. OpenAl-ol (Jaech et al. 2024) was the first to demonstrate the remarkable potential of combining RL. with CoT, enabling LLMs to surpass human cognitive abilities and tackle complex mathematical and coding tasks for the first time. Shortly thereafter, Deepseek-R1 (Guo et al., 2025) fully harnessed the potential RL+CoT through simple yet highly effective reinforcement learning algorithm GRPO (Shao et al. 2024). Their findings revealed that LLMs exhibit human-like "aha moments' during RL training. This achievement quickly garnered significant attention, inspiring
extensive replication efforts (Luo et al. 2025 Xie al. 2025 Hu et al., 2025 Zeng et al. 2025)
stimulating further research on enhancing GRPO (Yu et al. 2025 Liu et al.) (025) and PPO Yuan
al. 2025 [Sii et al. 2025). ) as well as comparative analyses between the (Xiong et al. 2025).
Nevertheless, most existing improvement solutions focus on enhancing sample quality, balancing
response length, and preventing entropy collapse. To the best our knowledge, this work the first to improve RL training from the gradient- inndiiet-disprpooriioalii perspective.
3 PRELIMINARY
Large Language Models. Most existing LLMs are based on a transformer decoder-only architec- ture /Vaswani al., 2017), typically denoted Ty. where 0 E Rd represents the model parameters. The fundamental unit of LLMs is the token, a discrete textual element that may correspond to a word, subword, or character, and is drawn from a finite vocabulary V {v' vo.....,o 'N), where N denotes the vocabulary size. During text generation, the model outputs probability distribution over the vocabulary, conditioned on the given prompt q and the sequence of previously generated tokens O.c1. The next token o; is then sampled from this distribution, expressed mathematically as Or ~ T q, O.z). The generation process is autoregressive, proceeding iteratively until either an end-of- (EOS) token is produced or predefined maximum sequence length tmaz is reached. The resulting sequence of tokens denoted as o.
Practical LLMs often required to align with human preferences exhibit strong reasoning capabilities, which cannot be easily achieved through naive pre-training and supervised fine-tuning. If a reward function r(g, o) is available quantitatively capture these objectives, the optimization of
3

---

Published as a conference paper at ICLR 2026
an LLM can be formulated as reinforcement learning task. In this framework, the generation of each token is treated as an action, while the prompt and the previously generated tokens are treated as the state. Accordingly, the optimization objective of the LLM is expressed as max, Do [r(g. o)], where D is pre-collected dataset.
Group Relative Policy Optimization. As widely used algorithm in early-stage research, PPO (Schulman et al.) 2017) requires value model with as many or even more parameters as the model being trained. The value model must be trained in conjunction with LLMs, and its initialization adds complexity and uncertainties to the RI. training process. To address these chal- lenges, DeepSeek introduces GRPO (Liu al., 2025), which eliminates the need for a value model entirely by estimating value through group-relative comparison. Specifically, for each question q- GRPO samples a group of outputs {01, 02....,oc'' and estimate the expected return under the question through V mean(r(q, 01). o2)....). During the training process, the estimated advantage is set to be consistence within each responses (Ai,t Aa), and is calculated through A; 3.
Compared to PPO, GRPO reduces GPU memory overhead by 50% and decreases single-step RI. training time by over 60% (Xie et al. 2025). In this work, we adopt variant of GRPO to optimize the policy model my. The optimization objective is expressed follows:
r(q.o,)-V(g
JGRPO = E E E Molld
lo.1
min
clip(r,.(0);;11 (1 (ee))AA en) 1
lRL [Tol] (Tolltrer)]
E oii
To(or, (a, oi,ct)
Tres(on,la, on.e))
-log Tref g 01, no(oi, q, oe)
with r.,(0)) Tota(or,e q, oi,et)"
and DEL [no [=re]]=
-1,
Toloi,ea oi.et)
(1) where Told denotes the policy used to sample the responses, Tref represents the initial policy prior to RL training, and C, B are manually defined hyperparameters. Note that the original implementation of GRPO normalizes the token update weights based on the response length, which introduces significant bias toward shorter responses during updates. In line with verl (Sheng et al. 2025) and most follow-up work (Zeng et al.) 2025 Liu et al., (2025), we remove this operation and conduct normalization among all tokens within the same query-batch.
METHODOLOGY
4.1
EXPLANATION ON COW-PROBAYILIIYY TOKENS' DOMINANCE
In this section, we provide a theoretical explanation for why tokens with lower probabilities tend to dominate updates during RL training. The learning objective in equation can be interpreted as weighted cross-entropy loss. For simplicity, we use the notation ((os,)) to denote T(o,,l o;,<t). By evaluating the gradient, we obtain the following expression (cf. Appendix A. for derivation):
lod
VaJGRPO(0) = 1o-
ΣΣ
To((i.t) Acs Itraal ( (o(0)))) Tota(O1,)) Tota(On.))
And) + Fres( B Vo molo.d) mo(or.)
(2)
W4,1
A. 0 and mp(oi.e) Totglore)
+ Ea
where Leruat (o(o).) Tota(
A..))
if A., < 0 and awtalor,e) otherwise
(o(o
We represent LLM as a composite function == Jz V1-10.... SI, where each Le (with E e {1,... ..... L)) corresponds to a distinct layer of the network. Let a denote the input and a denotes the output of 9th layer. We further define the Jacobian matrix of the éth layer with respect to its input as Jr @f((a=1)
dat-1
Assumption 4.1. For every layer, the Jacobian Jr is well-defined and the fe is locally differentiable. Furthermore, assume that for each layer, there exist two constants Ct O and de > such that
4

---

Published as a conference paper at ICLR 2026
(Jt) Ct and o max I de, where m-in(-) and mmax(:) denote the minimum and maximum singular values of the given matrix, respectively.
Assumption 4.1 not restrictive, as aligns with the standard design and training principles of neural-networks, ensuring stable gradients flow through well-defined and non-degenerate Jacobians.
Proposition 4.2. Under Assumption 4.1 let de(or,) == ae
denote the gradient of the
GRPO objective with respect to activation ag at any layer for a single token Oi,t. Let || denote the spectral norm, and define the vocabulary size as N. Then, for each layer E, the following inequalities always hold:
L II oy S..lwa.l.V 1-
L
N N - .1 (0T((o))) & [(
& II dy-luceal .11..V22211 =o((o,))). (3)
1m2++
Refer to Appendix A.2 for the detailed proof. Proposi- tion 4.2 demonstrate that, for a single token, the gradient norm with respect to activation ag any layer is bounded. Specifically, is confined within the truncated conical re- gion illustrated in Figure 2 In equation 3 apart from the term (1 To ( ), all other components in these bounds be regarded as constant. (Although wi, depends on To((),)) , it is approximately equal to A., in most cases.) This result highlights that tokens with lower probabilities lead to larger gradient magnitudes, whereas tokens with probabilities approaching / produce gradients that are
[o(()w)
S Fesstate range of o816510..1
-hats de au
G Moss of boud
G--
(o(o
1.0
Figure 2: Diagram of Proposition nearly zero. The experimental evidence presented in Figure D corroborates this relationship. demon- strating a roughly proportional correspondence between the gradient norm of all LLM parameters and - (o,,)))
Notably, during the RL training process, the gradients are averaged over hundreds of thousands of tokens for each update. Typically, the gradients are not sparsely distributed, leading to mutual influence among them. In such cases, low- www-prbbability tokens tend to dominate the gradient updates. Nevertheless, the gradients of boob-prrobbbiity tokens are equally important and should not be neglected (see Section for for details). To the best of our knowledge, no prior study has explicitly investigated the gradient interference between low-probability and high high-probability tokens.
4.2 MITIGATING THE OVER-DOMINANE OF LOW-PROBABILITY TOKENS
Adverse Effect of the Dominance. A natural question arises: what are the consequences if the gradient of low-probability tokens over-dominates the update process? Experimental re- sults Xiong et al.) al. (2025) suggest that positive samples (i.e., responses/tokens with an advantage greater than 0) play more significant role than those negative ones. Theoretically, the probability of tokens with positive advantage should increase after each update. Thus, we record the proportion of positive tokens with increased probabilities during a single RL train- ing step, as shown in Figure 3 In line with expectations, as the probability of token grows, the proportion of updates in the correct direction decreases. In particular, the propor- tion of correct update directions for tokens with probability greater than 0.75 is even slightly less than 50%. To mitigate the over-dominance of low- cow-roobyyllity tokens and promote more efficient updates for high-pro -robbabbibbby tokens, we introduce the following two methods.
GRPO (taseline)
lopti
GRPO reweight
0.56
Conect Rato
0.54
0.50
0.48
0.25
0.58
Comod hace
0.54
0.52
0.50
0.48
0.5
0.75
Probwbility Interval
Figure The proportion of posi- tive tokens updated in the correct di- rection for different updating meth- ods, under the same experimental settings in Figure
Advantage Reweighting. A straightforward approach to ad- dress this issue is to reweight the advantage of tokens based on their probabilities. Specifically, we re-calculate the advantage
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 GRPO with Advantage Reweighting and Low-Probability Token Isolation
Require: Initial LLM Re Kref, datasets D {q). reward function r(q. o), reweighting hyperparamter o,
isolation threshold fj
1: for each dataset epoch do
2: for each RI. step, sample {q) D do
3:
Auto-regress sampling G responses {or for each question within {q)
4:
Record the old probability for each token Tota(OLe) To (o(.).))
5:
Calculate the reward for each response with reward function r(g. oi)
Calculate the advantage for each token (response) through A = A: r(a.0,) monnrri
6:
ad(r(g.))))
7:
Reweight Advantage through equation
8:
for each RL epoch, sample mini batch >v {q, ([ 1/ do Update the policy with mini batch through equation end-for
9:
10:
Record old Advantage Ág! -Ai,
Mask high-proet coobhrbrbbbiltty tokens through AL, Agid [(Toid(Os;.) n)
each RI. epoch, sample mini batch ~ {q. {{Ai,s Totalo 1t-1h=l M do
14:
Update the policy To with mini batch through equation D end for
15:
16:
Mask low pr--abbbbbbttt tokens Ar, Ag @ (1 [(TTaa(OLa)) En)
for each RL epoch, sample mini batch {q. {f (A Rota ( 1 1 J
do
Update the policy my with mini batch through equation end for end for
19:
20:
21:
end for
return Final policy me
of each token as follows:
ALe = (0000)))) + -))] Ane.
(4)
where a E [0, 1] is manually-defined hyperparameter. This formulation assigns linearly smaller update weights tokens with lower probabilities. As shown in the upper panel of Figure 3 it can significantly reduce the errors in update directions for positive high-probability tokens.
Now-Probability Tokens Isolation (Lopti). In addition Advantage Reweighting, we also plored an alternative method, referred to as Lopti. Specifically, for a sampled mini-batch RI, we predefine a probability threshold 77 E (0, 1) divide tokens into two groups: low-probability tokens and high-probability tokens. We first update the low-probability tokens, followed by the high-prohde probability tokens. For detailed implementation, please refer to lines 11-19 of Algorithm [ With universal hyperparameter setting of = 0.5, this method achieves comparable effect to Advantage Reweighting, as shown in the lower panel Figure
The intuition behind Lopti is follows: during the first stage, updates on low-probabilityy tokens indirectly influence the distribution of the remaining nigh-probability tokens that have not yet been updated (as in ee). If positive high-probabilin token is affected in the correct direction (i.e., its probability increases), its gradient becomes smaller in the subsequent stage when high-probability tokens are updated. Conversely, if its probability decreases, its gradient will dominate within the high-pne gigb-rrobabblity token group, thereby receiving greater attention during the update process. Note that the order of updates cannot be reversed. The corresponding ablation is presented in Section 5.3
is worth noting that Advantage Reweighting and Lopti can operate concurrently and may even lead to further improved downstream performance. In Algorithm we detail how to integrate these two techniques with GRPO. Note that the original GRPO update step (the gray section with strikethrough in lines 8-10) should be skipped if Lopti activated. The computational cost requirements are detailed in Appendix C.2 Since Lopti splits the tokens and performs updates twice, it results in higher computational costs, which a limitation of our method (ef. Appendix F).

---

Published as a conference paper at ICLR 2026
GAPO
Difficulty by Number of People
GRPO + Reweight
Model
3
4
5
6
7
Avg.
GRPO -oopli
GRPO Reweight + Lopti
GPT-4o
0.57
0.49
0.32
0.23
0.21
0.36
Qwen- -ntn--B--nstruct
01-2024-12-17
0.51
0.38
0.38
0.35
0.30
0.38
2.0
Deepseek-R1
0.73
0.77
0.78
0.75
0.88
0.78
1.5
Qwen2.5-3B-Instee
0.09
0.10
0.03
0.05
0.02
0,06
1.0
+ GRPO
0.60
0.45
0.33
0.34
0.23
0.39
0.5
+ GRPO + Reweight
0.67
0.62
0.53
0.44
0.37
0.53 (195.9%)
+ GRPPO + Lopti
0.74
0.67
0,56
0.42
0.30
0,54 (198.5%)
+ -RPO Reweight Lopti
0.72
0.66
0,55
0,52
0.40
0.57 (148.2%)
3.0
2.5
0.22
0.15
0.08
0.10
0.02
0.11
+ GRPO
0.91
0.91
0.77
0.65
0.61
0.77
2.0
+ GRPO + Reweight
0.97
0.98
0.89
0.83
0.80
0.89 (115.6%)
1.5
+ GRPO + Lopti
0.95
0.94
0.84
0.80
0.76
0.86
(79.19)
144
280
+ GRPO Reweight Lopti
0.95
0.94
0.91
0.87
0.87
0.91 (118.2%)
RL stegs
Figure 4: Experimental results on the K&K Logic Puzzles benchmark. For Advantage Reweight, a = 0.3, and for Lopti, 7) = 0.5. The reward curve during training (left) is truncated to exclude the first epoch and smoothed with an exponential moving average (coefficient: 0.95). The evaluation accuracy on the test set (right) are averaged over the last three checkpoints to mitigate randomness.
EXPERIMENTAL RESULTS
To validate the effectiveness of our proposed method, we first conduct experiments on the Knights and Knaves (K&K) Logic Puzzles dataset (Xie et al. 2025 2024a) using GRPO, as described in Section [.] We then extend the experiments to the math-related dataset (Luo et al. 2025 Shi et al. 2025). as detailed in Section 5.2 Finally, we present a series of critical ablation studies, as outlined in Section [.3 Note that our methods are not restricted to GRPO and hold great potential actoss all
Policy-Gradient based RL algorithms. For experiments utilizing REINFORCE++ (Hu 2025) please refer to Appendix D
5.1 EXPERIMENTS ON K&K LOGIC PUZZLES
The K&K logic puzzles, first aggregated into a benchmark for LLMs by Xie et al. (0024a). are class of reasoning problems rooted in classical logic game (Smullyan 1986 Johnson Laird & Byme 1990). These puzzles involve a fictional scenario where inhabitants of an island are either Knights, who always tell the truth, or Knaves, who always lie. The objective is to determine the identity of each inhabitant (Knight or Knave) based on set of statements they make about themselves and others. Please refer to Appendix I.I.I for detailed introduction. The K&K logic puzzles are highly challenging, with only the most advanced LLMs demonstrating strong performance (Xie et al. 2024a). Additionally, it is not exposed in the model's 's pre-training phase, allowing the model to demonstrate continual learning behavior during training. As training progresses, both the training reward and test accuracy gradually improve, rather than converging rapidly. These characteristics make this benchmark an ideal choice for verifying RL performance.
Following Logic-RL. (Xie et al. 2025), we construct the training set by combining logic puzzles with 3 to players and adopt its rule-basee reward function, which consists two components: (1) Format score, assigned I if the model provides CoT reasoning within <think> < </think>>>>// tags and the final answer within <answer></ //nswerw////w>>>>>>>//////>/>>>> answer>> tags, and otherwise; (2) Answer reward, assigned 2 for perfect match with the ground truth -1.5 for partial correctness. and -2 for an completely incorrect answer. We use Owen2. 5- 3--3BBIee rnstrttt and Owen2. - 1M as starting points. Without employing curriculum learning, we directly expose the model to the mixed training set and train for a total of 5 epochs. The experimental results are reported in Figure g Detailed hyperparameter settings are provided in Appendix
B and comprehensive experimental records can
be found in Appendix C.I.I
During the early stages of GRPO training, the reward increases rapidly, but the growth slows signifi- cantly after the first epoch. Subsequently, the improvements introduced by Advantage Reweighting and Lopri become progressively more evident, particularly after 4 epochs. Interestingly, for simpler tasks (involving fewer players), the performance gap between the baseline GRPO and the GRPO enhanced with Advantage Reweighting and/or Lopti is minimal. However, for more complex tasks
7

---

Published as a conference paper at ICLR 2026
(a)
Ittatement
Analysis
Statement
Causal Indicators
Conclusion Indicators
Assumption
Assertion
20
50
Requercy3000 toienns
Speemrat
100-
Opearran pl 4.333
Opeamran
30
Scarman
20
5
20
10
40-
10
10
Raoward
Reward
Reward
Raward
Reward
Namard
(b)
resreight
Lopti
reweight Lopti
megancyyonn toeess
12.5
30-
10.0
10.0
75-
10
5.-
5.0
5.0
2,5
0.0
0.0
Figure 5: (a) The relationship between the frequency of six categories of inference-related words and the corresponding sample rewards for Bwven-2.5-7B-Instuuue 1M trained with naive GRPO. The Pearson correlation coefficient (r) and Spearman rank correlation coefficient (p) are annotated. (b) A comparison of the frequency of the six categories words across the starting point (Qwen-2.5-7B- Instruct-IM) LM), naive GRPO, and GRPO enhanced with Advantage Reweighting and/or Lopti.
with more players, the performance gap becomes significant. In challenging tasks, positive samples are typically fewer and thus more valuable. As analyzed in Section 4.2, high-probabilityy tokens in these rare positive samples are not effectively amplified under standard GRPO training. Our method addresses this limitation, thereby resulting in substantial performance improvements.
In addition, we perform linguistic analysis to investigate the correlation between the model's reasoning behavior and its final performance. Specifically, we use the model trained with naive GRPO to generate responses for the 500 prompts in the test set, sampling 8 responses per prompt, resulting in total of 4,000 samples. For these samples, we analyze the frequency of six categories of inference-related words (see Appendix C.I.I for details) and their corresponding rule-based rewards, as illustrated in Figure a). The analysis reveals positive correlation between the frequency of words in the categories Analysis, Statement, and Causal Indicators and the samples' rewards. Conversely, the frequency of words in the categories Conclusion Indicator, Assumption, and Assertion exhibits negative correlation with the rewards.
It is worth noting that the statistical patterns observed in these six categories of words indirectly highlight the enhancement effects of our proposed Advantage Reweighting and/or Lopti mechanisms on GRPO training, as shown in Figure b)). Notably, the frequency of words positively correlated with reward in the samples generated by our method is significantly higher than that of the baseline, while the frequency of words negatively correlated with reward is substantially lower.
5.2 EXPERIMENTS ON MATH- MATHERELATED DATASETS
To assess the generalization capability of our proposed methods, we conduct additional experiments on math-related datasets. Consistent with the majority of prior studies, we utilize Qwen2. 5-77 as the base model and employ straightforward rule-based reward. Specifically, score of 1 is assigned for completely correct answers, while score of 0 is given for all other cases. We experiment with two different datasets. The first one a subset containing 10k problems introduced by AdaRFT (Shi et al. 2025). which is sampled from DeepScaleR (Luo et al. 2025). This dataset, referred to as DSR- Uniffrm, evenly covers problems across all difficulty levels and is specifically designed for Owen2.5-72 We train this dataset for 5 epochs. The second one is dataset containing 57k problems introduced by Open- Reasoner-Zero (ORZ) (Liu et al.) 2025). For this dataset (ORZ), train for 1 epoch. Apart from the number of training epochs, all other hyperparameters (cf. Appendix B are kept consistent across both datasets.
We evaluate the LLMs after training five benchmarks: Olympiad Bench (He al. 2024). Minerva (Lewkowyez et al.) (022). MATH-500 (Hendrycks et al. (02T). AMC 2022-2023 and AIME 2024. For the first three benchmarks, we use greedy sampling for evaluation. For the last
8

---

Published as a conference paper at ICLR 2026
Table 1: Experimental results math-related datasets (DSR for DeepScaleR and ORZ for Open- Reasoner-Zero) Zero). For Advantage Reweight, is set to 0.1, and for Lopti, 1) is set to 0.5. The evaluation accuracy(%) are averaged over the last three checkpoints mitigate randomness.
Olympiad Bench
MATH
AMC
AIME24
AIME24 Avg.
Dataset Algorithms
Minerva
500
avg @ 16
pass@ 16
avg@ 16
all
Qwen2.5-7B
27.64
18.38
63.00
22.21
30.00
5.00
27.71
+ +GRPO
36,50
29.66
74.67
47.72
28.89
16.46
38.9
DSR
+ GRPO Reweight
37.00
29.66
75.47
48.32
35.56
14.03
40.01
Uniform
GRPO Lopti
36.60
30.27
76.53
47.69
32.22
14.24
39.59
GRPO
38.23
27.69
78.33
49.57
32.22
12.92
39,83
ORZ
GRPO Reweight
40.81
29.04
77.80
49.07
33.33
16.46
41.09
+ GRPO Lopti
38.63
29.78
78.53
47.29
34.44
15.28
40.66
two benchmarks, following prior works, we sample 16 responses for each question and report the average accuracy (avg@ 16). Notably, AIME 2024 is extremely challenging; therefore, we also report pass@1 16, which considers question correctly answered least one of the 16 responses correct.
The experimental results are summarized in Table In contrast to the continual learning behavior ob- served the K&K Logic Puzzle dataset, the test accuracy curve on the math-related dataset converges to specific value within 100 steps and subsequently exhibits only minor fluctuations. Despite this, the improvements introduced by our Advantage Reweighting and Lopti remain observable. is worth noting that the combined application of these two techniques does not result in further performance gains; therefore, we recommend using them individually for optimal results. The underlying reasons for this phenomenon discussed in Appendix to Appendix C.1.2
C.3
For detailed experimental records, please refer
5.3 ABLATION STUDIES
To better convey our motivation and demonstrate the effectiveness of the proposed methods, we perform ablation studies on the K&K Logic Puzzles dataset. The key conclusions derived from these studies are summarized in the following three points.
hiihh-probability tokens matter in RL training. Although the results in Figure and Figure suggest that the gradients of high-iigooee groobbibbbbylityy tokens are almost suppressed by low-probability tokens during updates, the high-probability tokens remain crucial and cannot be disregarded. As shown in Figure a), masking sighbhbbbbbiltty tokens leads to a significant degradation in the performance of the baseline GRPO. Therefore, reducing the influence of low- low-probability tokens high-probability ones holds great potential for advancing RI. training, as anticipated.
The update order is the key for Lopti. The intuition behind Lopti, introduced in Section 4.2 stems from the low-probability dominant effect of incorrectly reduced positive high-probabiilty tokens. To confirm this intuition and rule the possibility of random gains, we reverse the update order by processing high-probabiln probability tokens first, followed by low low-probability tokens, as shown in Figure 6 This modification leads significantly worse performance compared the GRPO baseline, with training even collapsing after the 4th epoch.
Proper hyperparameter tuning essential for Advantage Reweighting and Lopti. As intro- duced in Section 4.2) Advantage Reweighting involves the hyperparameter while Lopti depends on the hyperparameter 7). For the K&K Logic Puzzles dataset, the recommended ranges are a E [0.2, 0.3] and 7 E 0.3, 0.5). values outside these ranges may result in inferior performance compared to the GRPO baseline. is worth noting that the hyperparameter setting for Advantage Reweighting task-sensitive, whereas Lopti demonstrates greater robustness in this regard. For math-related datasets, the optimal hyperparameter for Advantage Reweighting is a 0.1, while Lopti maintains its robustness with y = 0.5.

---

Published as a conference paper at ICLR 2026
(a)
updating
(b)
Lopti high
Updating prob-00.7
Updating tolions prob-x
Lopti proo
Training Reward
Test Accuracy
Trining Reward
Testl Accuracy
2.0
0.75
2.0
0.75
1.0
1.0
0.25
0.25
Tran porkpe hoary
0.0
0.0
72
144
216
Traihing
Training
(c)
GRPO
(d)
GIPO
Reweight 5-6.1
Reweight
ght
+Lopti n-0.3
0-0.5
Training Reward
Test Accuracy
Training Reward
Test Accuracy
1.0
2.0
0.7%
6.75
1.0
1.0
0.25
0.25
0.0
0.0
141
216
200
360
72
1.4
216
200
140
Training
Training
Steps
Figure 6: Ablation studies on the K&K Logic Puzzles dataset. (a) Effect of restricting updates to nigh-probabiliyy tokens. (b) Effect of the token update order in Lopti. (c) Effect of the hyperparameter in Advantage Reweighting. (d) Effect of the hyperparameter in Lopti.
CONCLUSION
In this paper, identify crucial issue in RL training for LLMs: the over-dominance of low- probability tokens in model updates due to their disproportionately large gradient magnitudes. We substantiate this issue through both empirical observations and rigorous theoretical analysis. To address this imbalance, we propose two novel approaches: Advantage Reweighting and Lopti. These methods effectively mitigate gradient disparities by diminishing the undue influence of low-probabilliy tokens, thereby facilitating more balanced and efficient updates for high-probability tokens. Extensive experiments demonstrate the effectiveness of these approaches, showing consistent improvements in GRPO-trained LLMs across diverse base models and datasets.
ACKNOWLEDGMENTS
This work was supported in part by the General Research Fund (GRF) project 14200720 of the Hong Kong University Grants Committee. This work was also partially supported by Microsoft Research.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint arXiv:2303.08771 2023.
Anthropic. The Claude 3 model family: Opus, sonnet, haiku. Technical Report, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 34, 2020.
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: Process supervision without process. In Advances in Neural Information Processing Systems, volume 38, 2024.
10

---

Published as a conference paper at ICLR 2026
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 31, 2017.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint XrXiv:231 11022311.07999 1.09119, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The LLaMA 3 herd of models. arXiv preprint arXiv:2407.217 21783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-RI: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501. 19448, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadhench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
Jiwoo Hong. Noah Lee, and James Thorne ORPO: Monolithic preference optimization without refer- model. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, 2024.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint Stti:::5011002622 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zzero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24220
2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, et al. OpenAl ol system card. arXiv preprint arXiv:2412.16720, 2024.
Philip N Johnson-Laird and Ruth MJ Byrne. Meta-logical problems: Knights, knaves, and rips. Cognition, 36( 36(1):69-84, 1990.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quanti- tative reasoning problems with language models. In Advances in Neural Information Processing Systems, volume 36, 2022.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint 2rX1v:2503.20783, 2025.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol -preview with 1. 1.5b model by scaling rl. Notion Blog, 2025.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference- free reward. In Advances in Neural Information Processing Systems, volume 38, 2024.

---

Published as a conference paper at ICLR 2026
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. In Advances in Neural Information Processing Systems, volume 36, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 37, 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to Q": Your language model is secretly a Q-function. In First Conference on Language Modeling, 2024.
Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning. In The Thirteenth International Conference on Learning Representations, 2025.
John Schulman. Approximating KI. divergence. Technical Blog, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: Stt.::177006377. 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. Proceedings of the Twentieth European Conference Computer Systems, 2025.
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuning via adaptive curriculum learning, arXiv preprint urXiv:2504.05520 2025.
Raymond Smullyan. What is the name of this book? Touchstone Books Guildford, UK, 1986.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 34, 2020.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Bapiin Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 2. 11805, 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl.5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501.12599, 2025.
Hugo Touvron, Thibaut Lavril Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint Sttii::20770022888 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Information Processing Systems, volume 30, 2017.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms by-p-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties, 2024.
12

---

Published as a conference paper at ICLR 2026
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong-Hun Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. In The Thirty-ninth Anmual Conference on Neural Information Processing Systems, 2025.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny prompting elicits reasoning in large language models. In Advances
Zhou, et al. in Neural Information Processing Systems, volume 36, 2022.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229- 2269 1992.
Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123, 2024a.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong. Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502. 14768, 2025.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. The First Workshop on System-2 Reasoning at Scale, NeurlPS 24, 2024b.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint ar Xiv::2500 1 1343, 2025.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of 11m performance in machine translation. In International Conference Machine Learning, volume 41, 2024a.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? comprehensive study. In International Conference on Machine Learning, volume 41, 2024b.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, al. Qwen2 technical report. arXiv preprint 1077,,, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue. Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Yufeng Yuan, Qiying Yu, Xiaochen Zuo. Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. VAPÓ: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118. 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 200: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint wxXiv:2503.18892 2025.
Han Zhong, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He. Jiang Bian, and Liwei Wang. DPO meets PPO: Reinforced token optimization for RLHF, In ICML 2024 Workshop Models of Human Feedback for AI Alignment, 2024.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint Xrv: 1 009900859...9 2019.
13

---

Published as a conference paper at ICLR 2026
A
THEORETICAL INTERPRETATIONS
A.1 GRADIENT DERIVATION FOR THE GRPO OBJECTIVE
For clarity, re-state the objective function of GRPO below:
JGRPO (0) =E
gDD, (o))) Fold
od
min
DeL. ( [[
Ei oi
(5)
Jx
Spoticy
(o())))) and Der. [TT]]] [eel[Tee]] Tre (or,)) log (res( (T(().e)
with (0) (ot.)
(o(e)))
We begin by analyzing the policy loss term Jpoticy (0), which originates from the PPO clipping mechanism (Schulman al. (0077). Note that for samples with positive advantage estimates (i.e., Á, 0), the clipping activated only when (0) 1 + Eb. Conversely, for samples with negative advantage estimates (i.c., A, < 0), the clipping becomes active only when rs,e(0) < 1+ et- Consequently, when clipping is active, the gradient VoJpolicy (0) is zero; otherwise, it simplifies to (0) Å.. In summary, we can express the gradient of Jpoticy (0) as
V VeJpolicy = VOTo(Ows) AA. Iurust To(o),))
A.)
Tota(Oi,))
Told(Oi,t)
To(or.e) A lerust To(oi.)
A.) Vo (o),e)
Tota(Oi.)
(oie()
(6)
if Å and
ma(og.))
> 11110 Eb
To(Oi.)) Tola(os.))
Tazalor.)
where Irsust
(
A.)
if A, < and ( otherwise
Fota(O..))
-
Next, we consider the KL constraint term JKL. (0), commonly referred ka estimation (Schulman 2020). It provides an unbiased estimate of the KL divergence between the current policy and the reference policy. The gradient of JKL(O) given by:
voJKI(()) = 3V, (re)( To (oi,))
+ V log (oLL))
Tre/ (or,e)
VoTT((o,e) + AV log
Te(vi.)) )2
Tre (or,e)
3
V log (or.)).
Te(o),)
By combining equation 6 and equation] 7 we finally obtain the gradient of GRPO objective in the following form.
VoJGRrO(0) = E BaND woald 5 10:
To(O).) ds Iermat To(0).)) Tota(Oue)) (otd(or..)
A.) + (rer(
3 Va Logommolos.) (o(u).)
(o())..)
(8)
if Å. > 0 and
xolos,)) kota(or,))
> 1
where Hrrsst To(O As) (ota((i,))
if Å. < 0 and otherwise
*ptor,)
< 1
<ota(oi,1)
14

---

Published as a conference paper at ICLR 2026
A.2 PROOF FOR PROPOSITION 4
Proof. As introduced in Section 4.1 we denote LLM as a composite function - fi, where each fe (with & E [I,...> b}} corresponds to distinct layer of the network. dg-1 denotes the input and ag denotes the output of éth layer, and the Jacobian matrix of the fth layer with respect to its input is expressed as Je (f)( For any token Or, denote the gradient of GRPO objective with respect to the activations ag at fth layer as )e(0r,)) = VCRPOOO())).) According to the rule of backpropagation, we have:
da;
8e(0r,)) = J2410e41
II JJ oz(o))). j=
Note that the gradients of all intermediate layers are back- bacck-prpaggted from the last layer of LLM, thereby we discuss the gradients of the last layer (oi,)) first. The last-layer output of an LLM is the logits a (az, a ... , ). which corresponds finite vocabulary V {u², v....., oN The output probability the corresponding token calculated through softmax operation:
ea
To (v=)
for Vn E (1,
(10)
_m=1 e
Given token ot,s let l denote the index of the logits head corresponding to this token (i.e., vt 0z,1).
To obtain the gradient of last layer of LLM, we have:
ÖJenro(ou)) Day,
& Wie
a log no (o
day
a log (o(as)
@mo(v")
# Wit
(11)
(Te(Um))
day
a log no (oi.e)
Omo (v²)
618
Wi,t (v*) Oaz
Oaz
Here, equality (i) follows from equation 8 equality (ii) is obtained by applying the chain rule during backpropagation; and equality (ii) holds because Ô log (o))),)) ( = 0 for all m # k. Next, consider the following two cases for the gradient on the logits head aj (n (1,2, ...,,....)) NJ..
Case 1: the logits head corresponds to the sampled token (n k))
@Jenro(or)) Wi,t
Dati
Te (vk)
dat
Ea N
mm 8% 2a%
was To (vk)
(2 e
N
(12)
- To (T((")) TT (1*)
= W.. (1 (1- (
Case 2: the logits head corresponds to the un-sampled token (n + k)
@JGRPolOLs)
1
(
W4.s (
Daz
Oaz
-eat 2a%
1
-ii Te(u²)
(EA eaF
(13)
1
Witt To(U²)
(T(
(-To(1²))
Wi.t (-TO (T)))
15

---

Published as a conference paper at ICLR 2026
For simplicity, denote the vector distribution output across the vocabulary p, and denote [(,,e) as the one-hot vector with its only non-zero component at kth position (i.e., the position correspondence to token o,,)). We have the following expressions
p(oce) = (TT("") To To(o") >. To To("")) E E
I (Os,e) = (0, 0,
0) E R..
(14)
kth
Combining equation 12 and equation
13 and utilizing the notation defined in equation 14 we
obtain:
(o,,) V a, JORPO(O),)) = wAa (T(oi,) p(o,,))
(15)
Considering the lower bound for the gradient norm, have:
[|,2(os,))1 |wael [p((r.)) I(os,))l
|wrrl
(1 =T(x²))2
+ Σn Th TT((2
n
2 |waael
(1 Ta(
+ N N-1 Σ nisk
To mo(on))
(16)
|wa,el
(1 To(b)))2
+
N (1 To(()))))
N
=laa,l
N (1 TT(o),)))
where the inequality follows from the Cauchy-Setie -Schwaz inequality. The equality holds holding if and only if (v() is uniformly distributed for all k.
By substituting equation 16 into equation
9 we obtain:
| = - Jf 62(0).)
-III
j=e+1 min (JF) (10r.(0.)))
(17)
II I J= 9 (1,2(0s)))
N
(1 (==))
where inequality (i) follows from the variational characterization of singular values, inequality (ii) is consequence of Assumption 4.1 and inequality (iii) results from equation 16
Next, considering an alternative direction, we derive an upper bound for the gradient norm:
Σ N
(ox,))l We,el / -x(()))
nglk (um)2
v no {)²))) + Σm nyt To(un)2 2/2+ + +2 + N
no(0") ) (
(18)
= -|waall
V (1 =o( + E nssk To 2
| v (v()
))(x2))²
= -|w,al v2 (1 (1-Tr(oo..)) (o()).))
where the inequality holds because (o(")) 2 0 for all 2, .... The equality is achieved if
and only there exists an index m such that Ty (b = - To (T() and To (o("") = 0 for all n * m and n. + kt
16

---

It should be noted that the hyperparameter settings for our proposed methods are related to the task Published as a conference paper at ICLR 2026
Similarly, substituting equation 18 into equation
9 have
||
=+t Jf 62(0))))
jmt+l mmax (JJ) |
(19)
II j=(+1 dj 100
j=t dj -l,,, v2 (2 --)
where the inequalities hold for the same reasons in equation equation 19 establiss the result of Proposition 4.2
Together, equation 17 and
B HYPERPARAMETER SETTINGS
As described in Section 4.2 our proposed Advantage Reweighting and Lopti require only minor modifications to the existing GRPO training framework. Our implementation is built upon the verl library (Sheng et al. 2025). The key hyperparameter configurations for GRPO training
detailed
in Table Note that we adopt the "clip higher' technique from DAPO Yu et al. 2025) to stabilize entropy and mitigate entropy collapse. All other hyperparameters adhere to the default settings provided by verl.
The hyperparameter configurations specific to Advantage Reweighting and Lopti are summarized in Table 3 As reported in Section while the joint application of the two techniques generally yields improved results for the K&K Logic Puzzle dataset, this not the case for the Math dataset. Please refer to Appendix
C.3 for detailed explanation. Consequently, using either technique individually is recommended for the math-related dataset.
specification, but not to the base model utilized. As we concluded in Appendix CC. for reasoning tasks where the model uses natural language for inference, low- cow-probability tokens occur more frequently in the sampled batch, and therefore their dominant effect is more pronounced. Consequently, integrating Advantage Reweighting and Lopti can yield better results. On the other hand, for tasks such as mathematics, where the model uses specialized mathematical notation for inference, low-probabiliyy tokens occur less frequently. The dominant effect not as pronounced under such cireumstances. Regarding the hyperparameter a for Advantage Reweighting, for tasks where low-probability tokens exhibit a greater dominant effect (such as reasoning tasks), should be to 0.2-0.3 to achieve the best performance. For tasks where the dominant effect is weaker (such as math tasks), it should be set to 0.1 to achieve the best performance. As for the hyperparameter r) for Lopti, it robust across tasks, and setting the value to 0.5 is sufficient for all tasks.
For consistency, the same seed is used across all experiments. We save checkpoint every 20 RL steps, and all evaluation accuracies reported on the test set in this paper are averaged over the last three checkpoints. The detailed implementation can found our code
C EXPERIMENTAL DETAILS
C.1 TASK DESCRIPTION
C.I.I K&K LOGIC PUZZLE
As introduced in Section 5.I the K&K logic puzzles involve fictional scenario where inhabitants of an island are either Knights, who always tell the truth, or Knaves, who always lie. The objective of the LLMSs is to determine the identity of each inhabitant (Knight Knave) based on set of statements they make about themselves and others. Following Logic-RL (Xic et al. 2025), we utilize the LLMs
http://hhhuu.coomollccggggg
We provide the source code in supplementary materials.
17

---

Published as a conference paper at ICLR 2026
Table 2: Key hyperparameters for GRPO training, with the corresponding variable names in the verl configuration indicated in brackets.
Value K&K
Hyperparameter
Math
Rollout-related
Sampling temperature (temperature)
0.7
1.0
Question num per batch (ppo_m)
Answer num per question lato_mmni_bbath_sse____
64
128
Max tokens num per response (max_response_Length)
4096
Training-related
Update batch size airoo_iioe www.coooatcchsssii.
256
512
Optimizer (optim. (eptimtppp))))
adamw
Learning rate (optim. 1r)
1e-6
KL divergence coefficient (k1_ ikl_losss_ceff
0.001
Lower clipping threshold (clip_ratio_le)
0.2
Upper clipping threshold (clip caclppratii___e high)
0.24
Table 3: Hyperparameter settings for Advantage Reweighting and Lopti.
Value
Hyperparameter
K&K
Math
Advantage Reweighting (a)
0.3
0.1
Lopti (1)
0.5
0.5
Joint operation for better results
True
False
after instruction fine-tuning (Qwen2. 5- 3---BBIInstts and Qwen2. 557B-Innstrue -MM as starting point. The prompt specifically designed for the LLMs is follows.
Prompt
systemin You are helpful assistant. The assistant first thinks about reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and<answer> </answer> tags, respectively, ie.. <think> reasoning process here </think><<answe>> answer here </answer>, Now the user asks you solve logical reasoning problem. After thinking. when you finally reach a conclusion, clearly state the identity of each character within <answer> </answer> tags. i.c., <answer> (1) Zoey a knightin (2) ... </answer>.) in userin (problem Ann in assistanthn <think>
To encourage LLMs to exhibit chain-of-thouugh (CoT) reasoning, Logic-RL Xie et al. 2025) designs a reward function consisting two components, as outlined in Table 4 The output format deemed completely correct if LLMs include CoT reasoning enclosed within <think thhikk<</thinii tags and the final answer enclosed within <answer>< /twper////////////////// tags.
Table 4: Reward design for K&K Logic Puzzle proposed in Logic-RL. (iie al. 2025 Format Reward
Answer Reward
Completely Correct
2
Patially Correct
-1
-1.5
Completely Wrong
-1
-2
For the K&K Logic Puzzle dataset, the number of players (ranging from to 7) can be adjusted to control the difficulty level, with greater number of players resulting in higher difficulty. To provide an intuitive illustration, we present an easy example with players and challenging example with 7 players below. Without utilizing curriculum learning, directly train the LLMs the mixed training set for a total 5 epochs.
18

---

Published as a conference paper at ICLR 2026
An Example of K&K Puzzle with 3 people
Problem:
very special island is inhabited only by knights and knaves. Knights always tell truth, knaves always lic. You meet 3 inhabitants: Alexander, Lily, and Samuel. Alexander remarked, "Lily is a knave or Lily is a knight". : In a statement by Lily: "Samuel a knight if and only if Lily knight". Samuel was heard saying, "Lily is knight". So who is knight and who is a knave?
Example Reasoning Process:
Assume Alexander is knight. No contradiction is found in their claim that Lily is a knave or Lily is knight.
Assume Lily is knight. No contradiction is found their claim that Samuel a knight if and only if Lily is knight.
Assume Samuel is a knight. No contradiction is found in their claim that Lily is a knight.
Standard Solution:
(1) Alexander is knight, (2) Lily is knight, (3) Samuel is knight
An Example K&K Puzzle with people
Problem:
A very special island is inhabited only by knights and knaves. Knights always tell the truth, and knaves always lic. You meet inhabitants: Harper, Emma, Mia. Luke, Alexander, David, and Ethan. As Harper put it, "David not knight". In Emma's 's words: "David is knight". Mia said that If Emma is a knight then Emma is a knave. Luke said, "If Alexander is a knave then Emma a knight." Alexander was heard saying, "If David is knight then Harper is knave". "A "Alexander is not knight" David. "Harper is knight." Ethan mentioned. So who knight and who is knave?
Example Reasoning Process:
Assume Harper is knight. No contradiction is found their claim that David is not a knight. David cannot be a knight, because this would contradict the claim of Harper that David is not a knight.
Assume David is a knave. No contradiction is found in their false claim that Alexander is not knight.
Assume Alexander a knight. No contradiction is found their claim that David knight then Harper a knave.
Emma cannot be knight, because this would contradict the claim of their own that David is knight.
Assume Emma knave. No contradiction is found in their false claim that David knight.
Assume Mia is knight. No contradiction is found their claim that Emma a knight then
Emma a knave.
Assume Luke is a knight. No contradiction is found in their claim that If Alexander is a knave then
Emma knight.
Assume Ethan is knight. No contradiction is found in their claim that Harper is knight.
Standard Solution:
(1) Harper knight (2) Emma knave (3) Mia knight (4) Luke knight (5) Alexander
knight (6) David knave (7) Ethan is knight
The detailed training records for Qwen2. 5- 5-33-Instren and Owen2. 5-7B-Instrrcct-11
are presented in Figure and Figure 8 respectively. In addition the points discussed in Section [5.] is worth noting that our Advantage Reweighting and Lopti approaches slightly increase the response length while significantly reducing the gradient norm compared to the naive GRPO. Both observations empirically suggest that the RL training process is further stabilized.
For the six categories of inference-related words used in the linguistic analysis, the detailed word lists are provided in Table S It is important to note that for the nouns and verbs listed the table, account for the plural forms
their conjugated forms also included in the analysis. Specifically, of nouns as well as the past tense and past participle forms of verbs. Additionally, uppercase and lowercase letters treated equivalently.
19

---

Published as a conference paper at ICLR 2026
GRPO
GRPO Reweight
GAPO Lopli
GRPO Reweight Lopti
Training Reward
Response Length
Gradient Norm
Test Accuracy
0.6
750
1
1.0
0.4
500
250
0.2
0
0
72
244
216
288
360
0
72
144
216
268
360
72
144
216
288
360
72
144
216
286
360
Test 3ppl
Test 4ppl
Test Sppl
Test 6ppl
Test 7ppl
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.5
0.5
0.4
0.4
0.2
0.2
0.2
0.2
0
72
144 216
288
360
0
72 144
216 288 360
72 144 216 288 360
72
144
216
288
360
144 216 288 360
Figure 7: Experimental records of Qwen2. 5- IBstrrrrctt trained with GRPO the K&K Logic Puzzle dataset. The training curve is smoothed through exponential moving average with coefficient of 0.95.
GRAD
GAPO
Reweight
GRPO
Lopti
GRPD
Reweight
Lopti
Training
Response Length
Norm
Accuracy
2000
1.0
0.75
1500
1-
1000
0.25
500
Test 3ppl
Test 4ppl
Test 5pp!
Test 6ppl
Test 7ppl
0.8
0.8
0.8
0.8
0.8
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0-
0-
144
216
280
360
72
144
216
280
360
72 144
216
200
360
72 144 216
200
360
144 216 200 360
Figure 8: Experimental records of 2 -nst ruct -1 M1M trained with GRPO on the K&K Logic Puzzle dataset.
C.1.2 MATH DATASET
As discussed in Section 5.2 we perform additional experiments two math-related datasets, DSR- Uniform and ORZ. Consistent with the majority of prior studies, we use Owen2. 5-7B as the starting point. It is important to note that Qwen2. 5 undergoes post-training. This setup is therefore referred to as "cold-start" and denoted as RL-Zero (Guo et al. (025). No instruction -owtetiiiooooooiinnnggo templates employed; instead, use the following straightforward prompt.
20

---

Published as a conference paper at ICLR 2026
Table 5: Six categories of reference-relatte words associated with L.MMs' performance the K&K Logic Puzzles dataset.
Category
Words (Nouns and verbs include their conjugated forms)
Analysis
'analyze', 'consider', 'look at', 'check', 'examine"
Statement
XXX's "statement"
Causal Indicator
'since', 'because", 'due to', "given that" 'so", 'thus', 'hence', "as a result', 'consequently, 'therefore" aassume', 'he... 'must be', 'definite"
Conclusion Indicator
Assumption
Assertion
Prompt
(problem) Let's think step by step and output the final answer within Wboxed
LLMs that have not undergone post-training typically exhibit poor performance in adhering to specific output formats. result, format-related points were not included during training. Additionally, math problems are generally not partially correct, making a binary reward sufficient for evaluating the LLMs' output. Specifically, a reward of is assigned when LLMs produce the correct answer, and 0 otherwise.
The detailed experimental results for the DSR-Uniform and ORZ datasets are presented Figure and Figure I0, respectively. Notably, the training curve for learning trend, with the reward progressively increasing over time. In contrast, this is not observed for ORZ, where the reward converges rapidly within 100 steps. However, the test accuracy curves for both datasets converge to a stable value within 100 steps, after which they exhibit only minor fluctuations. Despite these patterns, the improvements achieved by proposed methods, Advantage Reweighting and Lopti, remain clearly observable.
demonstrates continual
GRPO
GRPD Reweight
GRPO
Training
Response Length
Gradient Norm
0.8
1500
0.6
1000
0.4
500
0.1
304
380
76
Test Olympied bench
Test Minerva
Test Math-300
0.4
0.35
0.35
0.3
0.75
0.3
0.25
0.25
0.2
0.65
304
76
Test AMC2022-2023 avg@16
Test AIME2024 pass@16
Test AIME2024 avg@16
0.5
0.45
0.15
0.4
0.3
0.35
0.25
0.05
304
380
76
Figure 9: Experimental records of 1 trained with GRPO on DSR-uniform dataset. The training curve is smoothed through exponential moving average with coefficient of 0.95, and the testing curve is smoothed with window size of 3.
21

---

Published as a conference paper at ICLR 2026
GRPO
GRPO
Reweight
GRPO Lapti
Training Reward
Response Length
Gradient
0.8
1500
0.3
0.4
500
0.1
0.2
110
220
330
440
110
220
330
440
0
110
220
330
440
Olymplad
0.45
0.35
0.8
0.4
0.75
0.25
0.7
0.3
110
440
110
330
440
330
AMC2022-2023 avg@16
pass@16
avgg16
0.55
0.45
0.2
0.5
0.15
0.1
110
330
Figure 10: Experimental records of Qwen2. 5B trained with GRPO on ORZ dataset.
C.2 COMPUTATIONAL COSTS
Our experiments are conducted on single machine equipped with an AMD EPYC 7V13 64-Core CPU and four NVIDIA A100 80GB PCIe GPUs. The experiments on the K&K Logic Puzzle dataset require approximately 16-22 hours to complete (excluding testing during the training process), while those on the math-related dataset take around 37-48 hours.
The Advantage Reweighting involves only recalculating the advantage of tokens, with time overhead in the range of milliscconds. However, this efficiency does not apply to Lopti, as it splits the tokens in batch into two groups and performs updates twice. Consequently, the updating process requires twice the amount of time, as detailed in T
Table 6: Computational cost comparison of Lopti operation over the first 50 training steps K&K Logic Puzzle Dataset.
Time (s)/step Bree2.5-3B-Inntree
Procedure
Srepseek-Distill--e
Qween:5577B-Instruct-Il
w/o Lopti
w/ Lopti
w/o Lopti
w/ Lopti
w/o Lopti
w/ Loptp
Sampling 140.9
141.7
25.4
27.8
68.5
69.3
Training 100.8
179.2
17.6
35.3
61.4
116.8
Others 10.1
9.4
2.4
2.8
10.3
10.2
Total 251.8
330.3
45.4
65.9
140.2
196.3
C.3 INCOMPATIBILITY OF AR AND LOPTI FOR SIMULTANEOUS APPLICATION TO MATH-RELATER DATASETS
As reported in Section 5 although the joint application of the two techniques (Advantage Reweighting and Lopti) generally yields improved results the K&K Logic Puzzle dataset, this is not observed for the Math dataset. To investigate the underlying cause of this discrepancy, we perform an analysis the DeepScaleR- Unniforr- dataset analogous to that in Figure and present comparison between the K&K Logic Puzzle and DeepScaleR -niffrrm datasets in Figure
When the task involves "logic puzzle", LLMs encounter greater number of "high-entropy" positions during the auto-regressive generation process. This remains true even when we lower the temperature 0.7 (as opposed to the typical setting of 1.0 used in mathematical tasks). As result, the proportion
22

---

Published as a conference paper at ICLR 2026
K&K Logic Puzzle
DeepScaleR-Unifom
12
Pressency (mpscaee)
10
5.74%
0.00
0.25
0.50
0.75
1.00
0.8
0.25
Gradkert hore
0.8-
0.15
0.10
T
0.0
0.75
1.
0.00
0.25
0.75
Probatelly Intarva
Probabiity Intenval
Figure 11: Comparison of token probability distributions and gradient norms during GRPO training of Owen2. 7B- wnstruuttttnt between the K&K Logic Puzzle dataset and the DeepScaleR- Uniform dataset. The proportion of sampled low-probability tokens during training is substantially smaller for the DeepScaleR-Unm Uniform dataset than for the K&K Logic Puzzle dataset.
of flow-probabiliyy tokens" in the sampled batch increases. In such scenarios, the dominance effect of low-probability tokens becomes more pronounced, allowing both "Advantage Reweighting' and "Lopti" to perform effectively. When used independently. there is significant difference in performance; when combined, they can even yield superior results.
In contrast, for math-related tasks, high-entropy positions encountered less frequently during generation, leading to a lower proportion of low-probability tokens in the sampled batch compared logic puzzles. This may be attributed to the greater textual inertia and determinism inherent in mathematical language. Consequently, in this setting, combining the two methods can overly suppress the contribution of oooprbbbyyyyity tokens to the gradient updates, ultimately resulting in suboptimal performance.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 ADDITIONAL POLICY-GRADIENTTBASSI RL ALGORITIIMS
In addition to GRPO, our proposed methods, Advantage Reweighting and Lopti, are also well- adapted to other Policy Gradient-basee RL algorithms. In this section, extend our methods to REINFORCE+ (Hu 2025), and DAPO (Yu et al. 2025).
D.1.1 REINFORCE++
REINFORCE+ Hu 2025 is a widely recognized algorithm that builds upon the conventional REINFORCE (illhamm 1992 while incorporating various stabilization techniques introduced by
PPO (Schulman et al. 2017).
Similar to GRPO, REINFORCE+ also eliminates the need for value model, thereby reducing computational costs compared to PPO. The key differences between GRPO and REINFORCE++ lie in how they estimate the advantage and constrain the distance between the RL-trained model and the initial (or reference) model. GRPO estimates the advantage based on the difference between the reward and the group-relative expected return, incorporating the KL constraint directly into the objective function (cf. Section 3 for details). In contrast, REINFORCE++ does not emphasize the concept of 'goupp' under the same prompt. Instead, it estimates the advantage directly from the reward and treats the KL constraint as penalty term added to the reward. Specifically, REINFORCE++ estimates the advantage as follows:
23

---

Published as a conference paper at ICLR 2026
A. - MM σ
with A rr(q,
Σ DkL
(20)
where μ.d. and A represent the mean and standard deviation of the advantages all tokens within the RL-sampled batch, respectively. The KL divergence term is computed using the k1 estima- tion (Schulman, 2020 DKL Fre((s))) = no (re)))= The optimization objective
of REINFORCE++ is:
JR++ (0) Egg ".:D:(o)) Moold
lo.1
min (r)(0)A clip(r...));) - -v, e. (..1
Σ
0
(21)
To( q. 0,,c))
with (0)
Told (i. Oi,ct)
Similar to the experiments conducted with GRPO, we validate two base models as starting points: Qwen2. 5- 3B rnstruct and Qwen2. 7-----0-- In-IIntruun InBBIInttutt-tt FORCE++ are kept consistent with those used for GRPO, as described Appendix B The only difference is that, the K&K Logic Puzzle dataset, the optimal hyperparameter setting for Advantage Reweighting is a = 0.1 for REINFORCE++, and a = 0.3 for GRPO.
All hyperparameters of REIN-
The evaluation results the test set are reported in Table Notably, the performance of naive REINFORCE+ is slightly worse than that of naive GRPO (cf. Figure This observation aligns with the findings of Xiong al. (2025), the advantage normalization method in REINFORCE++ may introduce unnecessary bias toward entirely incorrect responses on overly difficult prompts. Nevertheless, the improvements achieved by proposed methods, Advantage Reweighting and Lopti, remain significant. For more details on the training process, please refer to the records presented Figure 12 and Figure 13
Table 7: Experimental results of REINFORCE++ the K&K Logic Puzzles dataset. For Advantage Reweight, a = 0.1, and for Lopti. 1J = 0.5. The evaluation accuracy on the test set are averaged over the last three checkpoints to mitigate randomness.
Difficulty by Number of People
Model
3
4
5
6
Avg.
Qwen2.5-3B-Inntruue
0.09
0.10
0.03
0.05
0.02
0.06
REINFORCE++
0.37
0.31
0.20
0.21
0.06
0.23
REINFORCE+ with Reweight
0.53
0.44
0.31
0.26
0.14
0.34 (146.1%)
REINFORCE+ with Lopti
0.47
0.36
0.26
0,26
0.12
0.29 (127.8%)
REINFORCE+ with Reweight & Lopti
0.61
0.49
0.38
0.34
0.21
0.41 (176.5%)
Qwenn-5-77--nssruutt-MM
0.22
0.15
0.08
0.10
0.02
0.11
REINFORCE+
0,68
0.72
0.54
0.42
0.43
0.56
REINFORCE+ with Reweight
0.81
0.77
0.66
0.62
0.48
0.67 (119.7%)
REINFORCE++ with Lopti
0.89
0.85
0.71
0.66
0.51
0.72 (129.7%)
REINFORCE+ with Reweight Lopti
0.87
0.88
0.81
0.71
0.69
0.79 (141.9%)
24

---

Published as a conference paper at ICLR 2026
GRPO
GRPO Reweight
GAPO Lopli
GRPO Reweight Lopti
Training Reward
Response Length
Gradient Norm
Test Accuracy
0.6
750
1
2.0
0.4
500
250
0.2
0
72
244
216
288
360
0
72
144
216
268
360
72
144
216
288
360
72
144
216
286
360
Test 3ppl
Test 4ppl
Test Sppl
Test 6ppl
Test 7ppl
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.5
0.5
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0
72
144 216
288
360
0
72 144 216 288 360
72 144 216 288 360
72 144
216
288
360
144 216 288 360
Figure 12: Experimental records of Qwen2. Instrtcctttt trained with REINFORCE++ the K&K Logic Puzzle dataset. The training curve smoothed through exponential moving average with coefficient of 0.95.
Reweight
Lopti
Reweight Lopti
Training Reward
Length
Gradlent Norm
Test
3
1250
1.5
1.0
0.5
750
0.5
0.25
500
144
216
288
360
144
216
288
360
72
144
216
288
360
216
288
360
4ppl
5ppl
6pp!
7ppl
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.2
0.2
0.2
D.2
0.2
360
72 144
216 200
360
Figure 13: Experimental records of Qwen2.5-7B -ww----Inntttuaa - 1M trained with REINFORCE++
on the K&K Logic Puzzle dataset.
1.1.2 DAPO
DAPO Yu et al. 2025) is recently proposed algorithm that builds upon GRPO (Liu et al. 2025). Beyond the foundation of GRPO, further introduces four key techniques to accelerate and stabilize training:
Clip Higher, which establishes a higher clip threshold for PPO-style importance sampling clipping positive samples. This proves effective in promoting system diversity and preventing entropy collapse;
Dynamic Sampling, which filters out prompts that yield homogeneous responses with identical rewards, thereby improving training efficiency and stability;
25

---

Published as a conference paper at ICLR 2026
Token-Level Policy Gradient Loss, which averages the loss all tokens within a batch rather than over sequences, a modification that is critical in long-chain-of thouugh RL scenarios;
Overlong Reward Shaping, which mitigates reward noise caused by hard truncation when responses exceed length limits, thereby reducing noise and stabilizing training.
Table 8: Experimental results of DAPO on the K&K Logic Puzzles dataset.
Difficulty by Number of People
Model
6
Avg.
Qwen2.5-3B-Inste
0.09
0.10
0.03
0.05
0.02
0.06
+ DAPO
0.68
0.61
0.45
0.42
0.31
0.49
+ DAPO + Reweight
0.72
0.68
0.52
0.51
0.47
0.58
(117.4%)
+ DAPO Lopti
0.77
0.73
0.60
0.58
0.50
0.64
(128.7%)
+ DAPO Reweight Lopti
0.85
0.88
0.71
0.70
0.58
0.74 (150.6%)
1renn.5--B-Instructt-l
0.22
0.15
0.08
0.10
0.02
0.11
+ DAPO
0.96
0.93
0.81
0.76
0.67
0.82
+ DAPO + Reweight
0.97
0.97
0.92
0.87
0.86
0.92 (111.1%)
DAPO Lopti
0.99
0.99
0.97
0.92
0.90
0.95 (116.6%)
DAPO Reweight Lopti
1.00
0.99
0.98
0.96
0.91
0.97 (117.2%)
DAPO
DAPO
Reweight
DAPO Lepti
DAPO + Reweight + Lopti
Training Reward
Response Length
Gradient
Test
1000
3.0
0.6
750
1
0.4
500
0-
250
1.0
0.2
360
216
288
286
360
3ppl
4ppl
5ppl
6ppl
7ppl
1.0
1.0
1.0
0.8
0.8
0.9
0.8
0.6
0.6
0.5
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0:
0-
144
216
288
360
288 360
72 144 216
144 216 288 360
216 288 360
Figure 14: Experimental records Qwen2. owen2.5-3BB 5-..5--3--10 -II--IInnnuuttt trained with DAPO on the K&K Logic Puzzle dataset.
DAPO utilizes the same training objective defined in Eq. In our experiments, we validate DAPO on the K&K Logic Puzzle dataset using two base models: Qwen2 5-3B- In--nnttttt ruct and Owen2.
1M. All hyperparameters of DAPO are kept consistent with those used for GRPO, as described in Appendix B It is important to note that during the sampling process, DAPO filters out prompts that yield homogeneous responses. This filtering operation results in variability in the total number of training steps across different experimental settings, as the filtered prompts differ each training run. ensure fair comparison, we do not fix the number training epochs at 5; instead, we fix the total number of training steps. For the 3B model, we set the training steps to 360, consistent with the experiments conducted for GRPO and REINFORCE++. However, for the 7B model, DAPO converges after 300 steps, and the prevalence of homogeneous responses during the sampling process triggers early stopping. Consequently, we set the training steps to 300 for the 7B model. The experimental results presented in Table and Figures 15
26

---

Published as a conference paper at ICLR 2026
DAPO
DAPO
Reweight
DAPO Lopti
DAPO Reweight
Training Reward
Response Length
Gradient Norm
Test Accuracy
2000
1.0
2
1500
1.0
0.75
1000
0.5
500
0-
0
0
60
120
160
240
300
0
60
120
180
240
300
60
120
180
240
300
80
120
160
240
300
Test 3ppl
Test 4ppl
Test Sppl
Test 6ppl
Test 7ppl
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.5
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0-
120
180
240
300
60
120
180
240
300
60
120
180
240
300
60
120
160
240
300
120 180
240
300
Figure 15: Experimental records of I-sttr-ttttt 1M trained with DAPO the K&K Logic Puzzle dataset.
D.2 ADDITIONAL LARGE LANGUAGE MODELS
To demonstrate the generalization capability of our proposed method beyond Qwen-series models, further evaluate its performance LLaMA- series (Dubey et al., 2024) and SeepSeek-series (Guo
et al. 2025) models in Appendices D.2.1 and D.2.2 respectively.
D.2.1 LLAMA
We adopt two models from the LLaMA-series (Dubey et al. 2024): LLaMA-3 1--------nnt-- and LLaMA-3. 1- ruct for experiments. We evaluate methods with GRPO the K&K Logic Puzzle dataset. All other hyperparameters are kept consistent with those in Appendix B
The experimental results presented in Table Figure and Figure 17 These results closely resemble those obtained with Qwen-series models, as shown in Table
Fi Figure 7 and Figure
Table 9: Experimental records of 1. -23 & LLaMA-3. 3 1-------nnsttnt trained with GRPO on the K&K Logic Puzzles dataset. For Advantage Reweight, a 0.3, and for Lopti, = 0.5.
Difficulty by Number of People
Model
3
4
5
6
7
Avg.
LLaMA-32-3B-Insite GRPO
0.00
0.00
0.01
0.00
0.01
0.00
0.47
0.33
0.29
0.27
0.13
0.30
GRPO Reweight
0.58
0.51
0.33
0.36
0.23
0.40
(133.3%)
GRPO Lopti
0.73
0.70
0.59
0.49
0.43
0.59 (196.7%)
GRPO Reweight Lopti
0.73
0.76
0.58
0.55
0.45
0.62 (+106.7%)
LLaMM-3..-SB-Imstrruc
0.05
0.01
0.03
0.00
0.00
0.02
GRPO
0.86
0.88
0.77
0.72
0.68
0.78
GRPO Reweight
0.89
0.92
0.86
0.81
0.78
0.85 (19.0%)
GRPO Lopti
0.90
0.95
0.89
0.87
0.84
0.88 (112.8%)
GRPO Reweight + Lopti
0.94
0.97
0.89
0.86
0.82
0.90 (115.4%)
27

---

Published as a conference paper at ICLR 2026
GRPO
GRPO Reweight
GRPO Lopti
GRPO Reweight Lopti
Training Reward
Response Length
Gradient Norm
Test Accuracy
1000
0.6
750
1
2.0
0.4
500
250
0.2
0
0
72
144
216
286
360
0
72
144
216
288
360
72
144
216
208
360
72
144
216
286
360
Test 3ppl
Test 4ppl
Test Sppl
Test 6ppl
Test 7ppl
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.5
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0
144
216
288
360
72
144
216
288 360
72 144 216 288 360
72 144
216
286
360
144 216 288 360
Figure 16: Experimental records of -...MM-3.2-33. -----BIIInruttt trained with GRPO the K&K Logic Puzzle dataset.
Reweight
Lopti
Reweight Logti
Training Reward
Response Length
Gradient Norm
Test Accuracy
2000
1.5
1.0
1000
0.5
0.5
500
0.25
0-
144
216
208
360
72
144
216
208
360
72
144
216
288
360
144
216
286
360
Test 4ppl
1.0
1.0
1.0
1.0
1.0
0.6
0.6
0.6
0.5
0.6
0.2
0.2
0.2
0.2
0.2
72 14M
Figure Experimental records of L.L.A-3.1 3..--..111--11 ruct trained with GRPO on the K&K
Logic Puzzle dataset.
D.2.2 SHEPSEEK-RI-DiSTII
As discussed in Section 5.2 and illustrated in Figure 10 the test accuracy curve on the math- related dataset converges to specific value within 100 steps and subsequently exhibits only minor fluctuations. Consequently, the improvements introduced by Advantage Reweighting and Lopti not significant. This phenomenon is common for base models with limited reasoning capability, as they encounter similar data during the pre-training phase and their capability is constrained by the number model parameters.
To observe the continuous learning behavior the math-related dataset and reveal the capability of our proposed methods, we conduct experiments on the ORZ dataset with DepsSek-RR--istill11 1B... All hyperparameters are set the same those in Appendix B
28

---

Published as conference paper at ICLR 2026
The experimental results recorded in Table and Figure 18 It is evident that the gap between the baseline GRPO and GRPO enhanced with Advantage Reweighting or Lopti becomes increasingly larger for both the reward curve and the test curve. Compared with the experimental results shown in Figure TO the improvements brought by our proposed methods are more pronounced.
Table 10: Experimental results on Srepseek-RR-Distiine
trained with GRPO on
ORZ dataset.
Olympiad Bench
MATH
AMC
AIME24
AIME24
Avg.
Dataset Algorithms
Minerva
500
avg@ 16
pass@ 16
avg@ 16
all
KeepSeek-RI-Distill-e
24.81
9.93
55.40
34.64
23.33
9.58
26.28
GRPO
35.84
19.51
73.90
49.85
33.26
14.78
37.86
ORZ
+ GRPO Reweigght
40.02
19.73
77.20
53.97
32,22
17.50
40.11
GRPO Lopti
37.25
20.34
75.67
51.66
34.44
14.72
39.01
GRPO
GRPO + Reweight
GRPO + Lopti
Training Reward
Response Length
Gradient
0.7
3500
0.1
0.6
3000
0.05
2500
2000
110
220
330
440
110
220
330
110
220
330
440
Test Olymplad bench
Test Minerva
Test Math-500
0.45
0.2
0.75
0.35
0.15
0.7
0.25
0.1
0.65
Test AMC2022-2023 avg@16
Test AIME2024 pass@16
Test AIME2024 avg@16
0.55
0.45
0.5
0.4
0.1
0.3
0.35
0.25
0.05
110
220
330
440
110
220
330
440
0
110
220
330
440
Figure 18: Experimental records DeepSee--R1-Ditti11-- trained with GRPO on the
ORZ-math dataset.
E ON THE CONTRADICTORY CONCLUSIONS OF THE 80/20 RULE
is noteworthy that the conclusions from our paper are somewhat contradictory those the 80/20 Rule of RL introduced by Wang et al. 22025). Specifically, in this work, we emphasize that high- probability tokens are equally important low-probability ones, and that the over-dominant effect of low-probability tokens may impede the learning of high-probabilitg tokens. In contrast, Wang al. (2025) argue that only high-entropy tokens (which are somewhat equivalent low- probbabbbblity tokens) matter, and that all low-entropy tokens (high-probability ones) can be completely dropped during the training process.
We would like to emphasize that the fundamental reason lies in the hyperparameter settings used in the experiments. We follow the settings from Logic-RL (Xie et al. 2025). while Wang et al. (2025) follow those from DAPO Yu et al. 2025). In our setting, we set the update batch size to half of the total samples per RL step, i.e., the sampled trajectories are separated into n = batches for updating. This configuration large models, switching between inference and training engines can extremely slow due to heavy I/O costs. For this reason, DAPO samples large batch and updates it separately across 16
common before RL scaled up to models of extremely large size. For extremely
29

---

Published as a conference paper at ICLR 2026
gradient steps. This means that only the first batch is on-policy, while the rest are off-policy, causing the clipping mechanism to be activated.
observed that in such off policy cases, low-probability tokens clipped to a large extent, leading to overemphasis less likely to be clipped the later batches of updates. This phenomenon can be mitigated by the 80/20 rule. However, for relatively on-policy training (when is small), the 80/20 rule does not work. To verify this analysis, we conduct experiments as shown in Figure [9] It is noteworthy that the 80/20 rule degrades the performance of the baseline DAPO for S 8 and only improves performance when 16.
high-pree high-probability tokens, which are
DAPO PeM---aselineeee n=2
80/20 nm2
80/20 nm4
80/20 nm8
80/20 n-16
Clip Fraction
Test 3pp!
Test 4pp!
0.04
0.8
0.8
0.02
0.4
0.4
0-
0
60
120
180
240
300
60
120
180
240
300
0
60
120
180
240
300
Test Sppl
Test 6pp!
Test 7ppl
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0
0
60
120
180
240
300
60
120
180
240
300
0
60
120
180
240
300
Figure 19: The effect of the 80/20 rule (Wang et al. 2025) with different numbers of sampled batch shards on K&K Logic Puzzle dataset.
F LIMITATIONS
One limitation of our study lies in the additional computational overhead introduced by Lopti. As detailed in Appendix C.2, the updating process requires twice the amount of time as it splits the tokens in batch into two groups and performs updates twice. However, also propose alternative method, Advantage Reweighting, which incurs negligible computational cost while achieving even greater improvements on the math-related dataset compared Lopti.
30


Reviews:
[{"review_id": "YNBlo7i6bW", "reviewer": "Reviewer_A6Dz", "summary": "This paper claims that low-probability tokens have higher gradient norms which over-dominates the gradient update when doing RLVR to an extent that high-probability tokens might update even in the opposite direction. After showing this empirically, the show via mitigating this via isolating or re-weighting the low-probability tokens, we have a better RLVR algorithm.", "questions": "1-Is there evidence that K&K is a hard reasoning benchmark? \n2-Have you tried this method on native reasoning models, not base models, and observe improvement?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think this paper's main claim is new to the field: the gradient of the low probability tokens are dictating the changes in the probability of high probability tokens (not the advantages of those high-probability tokens) because their gradient magnitude is large. The paper shows mitigating this improves results on K&K puzzle. I am not familiar with this puzzle, but it seems it is a challenging task. I think the K&K results are the strongest in supporting the evidence.", "weaknesses": "The paper points out a very interesting observation. However, the math results are not consistent with the theory: their methods and GRPO are almost achieving the same score. I think it is because they are testing this on a R1-Zero style scenario where they start from a base model. Papers that do RL on base models show this quick recovery of some performance the curves are flat afterwards. I think the paper should have been done on a native reasoning model on R1-Distill-1.5B as they show usually continued progress even in math problems. Therefore, I would say the experiments are not providing the  significant signal that this algorithm scales. In terms of the reasoning to support the conclusion that mitigating the dominance of low-probability tokens is key in RLVR, while the K&K results support the claim, we don't know if this holds in more serious scenarios like math problems. I don't know if K&K is considered a task that its performance forecasts the performance on downstream hard reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YW38nXaixX", "reviewer": "Reviewer_5xRJ", "summary": "The paper identifies and analyzes an overlooked source of update bias in RL training of LLMs: low-probability tokens produce larger gradient magnitudes and can therefore dominate parameter updates. The authors (1) derive bounds showing token-wise gradient norms scale with $1−\\pi$, (2) propose two mitigation methods: Advantage Reweighting and Low-Probability Token Isolation (Lopti), and (3) demonstrate consistent empirical gains on several benchmarks.", "questions": "1. The paper evaluates only naive GRPO and REINFORCE++ baselines. It’s better to consider whether widely used GRPO variants (for example, DAPO) might interact with or mitigate the same gradient-magnitude imbalance.\n2. To demonstrate broader generalization, It’s better to add experiments across different model families and sizes (for example, Qwen-3 and Llama-3.1).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The theoretical derivation and supporting experiments are well aligned; the motivation is lucid and convincing.\n2. The proposed methods are low-cost, effective, and easy to implement in practice.\n3.  The authors provide careful experiments and analysis, including evaluations on multiple algorithms, multiple domains, and several ablations.", "weaknesses": "See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ucY5SWPrIK", "reviewer": "Reviewer_jZRG", "summary": "This research identifies a critical but previously overlooked inefficiency in the reinforcement learning (RL) process for Large Language Models (LLMs): the disproportionate dominance of low-probability tokens in policy gradient updates. The paper argues that this phenomenon, termed the \"tyranny of the unlikely token,\" actively hinders the model's ability to refine its core knowledge. To address this, the paper proposes the advantage reweighting and Low-probability Token Isolation (Lopti). Experiments on complex reasoning and math tasks using GRPO demonstrate that both methods significantly outperform the baseline.", "questions": "The solutions are presented without comparison to existing techniques for managing noisy gradients.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The research is built on a solid theoretical foundation, with Proposition 4.2 mathematically demonstrating that a token's gradient norm is inversely related to its probability. This elevates empirical observation to a predictable phenomenon.\n2. The paper excels at communicating a complex technical subject. The narrative is logical, and the framing of the issue as the \"tyranny of the unlikely token\" is both memorable and effective.", "weaknesses": "1. The author claim that low-probability tokens dominate model updates during RL training and that this dominance may impede the precise adjustment of the probability distribution across all tokens.   In fact, the high-entropy minority tokens drive effective reinforcement learning for llm reasoning, we do not need to adjust the probability distribution across all tokens based on RL.\n\n2. RL Algorithm Specificity: Experiments were conducted exclusively with GRPO. All experiments used models from the Qwen2.5 family.  There is no other strong comparable baselines on the benchmark.\n\n3. Acknowledging that Lopti doubles update time is treated as a minor caveat. A 100% increase in backward pass cost is prohibitive for most large-scale training operations, rendering Lopti impractical for all but the most specialized applications.\n\n4.The paper offers no guidance on how to co-tune the two new hyperparameters (α and η).\n\n\n\n\n[1] Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning, arXiv 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K2o7aER7Rv", "reviewer": "Reviewer_xEgW", "summary": "This work first identifies the pattern that tokens with low probability could induce significantly higher gradients than those with high probability tokens. Basing on this observation, this work then theoretically prove that the gradient norms are proportional to (1-$\\pi$), where $\\pi$ is the token probability.  Then, this work proposes two algorithms, advantage reweighing and low-probability token isolation to alleviate the issue.\n\nThe proposed method are evaluated on two kinds of tasks: K&K logic puzzles and mathematical reasoning. On both tasks, the proposed methods can outperform the original GRPO. Furthermore, on K&K, the results show that further improvement can be obtained by combining advantage reweighing and low-probability token isolation.", "questions": "1, Could you also report the curves on mathematical tasks (e.g., MATH 500)? \n\n2, How did you select the hyper parameters in Table 3?\n\n3, Do you have any hypothesis for whether combining advantage reweighing and Lopti did not improve for math problems?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1, This work rigorously describes how the gradient norm is decided by token probability, during online RL training. This provides valuable insights to the community, and encourages exploration on token-level reward design.\n\n2, The two proposed methods are sound, and straightforward. According to empirical results, they can effectively improve performance over the original GRPO.\n\n3, I especially appreciate the in-depth ablation studies. They demonstrate: 1, how different tokens with different probs affect the training dynamics; 2, different hyper-parameters affect the training dynamics compared to GRPO.", "weaknesses": "1, The first concern is with the hyper parameters. According to Figure 6, the choices of $\\alpha$ and $\\eta$ significantly affect the training dynamics. But considering that the training and test curves are similar, introducing a validation mechanism would be very helpful. Would you like to discuss such mechanism? \n\n2, On a philosophical level, I am not sure if such fixing (improving the effect of high-probability grad norms) is necessary, especially on positive samples. Specifically, those tokens are already of high probability so that improving their grad norm can only marginally change their distribution. Figure (6) is helpful in demonstrating the effects of low and high probability tokens. But those experiments are different from gradient update. Specifically, inclusions of different probability tokens in figure (6) is hard: tokens are either updated or not updated; but in gradient update, all tokens are updated, but the update magnitudes (gradient norm) are different. \n\n3, I do appreciate the bounds over the gradient norm. But I am also concerned that $c_l$ and d_l$ could be sufficiently large, dominating the effect of token probability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)