Overthinking Reduction with Decoupled Rewards and Curriculum Data Scheduling
Authors: Shuyang Jiang
Yusheng Liao
Ya Zhang
Yanfeng Wang
Yu Wang
Venue: ICLR 2026 Oral (2026)
Keywords: efficient reasoning; curriculum sampling with decoupled reward
Primary area: foundation or frontier models, including LLMs
TL;DR: theoretical unveil the underlying limitations of length reward and propose D$^2$yOR to achieve supreme efficiency without performance degradation

Abstract: While large reasoning models trained with critic-free reinforcement learning and verifiable rewards (RLVR) represent the state-of-the-art, their practical utility is hampered by ``overthinking'', a critical issue where models generate excessively long reasoning paths without any performance benefit. Existing solutions that penalize length often fail, inducing performance degradation due to a fundamental misalignment between trajectory-level rewards and token-level optimization. In this work, we introduce a novel framework, DECS, built on our theoretical discovery of two previously unaddressed flaws in current length rewards: (1) the erroneous penalization of essential exploratory tokens and (2) the inadvertent rewarding of partial redundancy. Our framework's innovations include (i) a first-of-its-kind decoupled token-level reward mechanism that surgically distinguishes and penalizes redundant tokens, and (ii) a novel curriculum batch scheduling strategy to master the efficiency-efficacy equilibrium. Experimental results show DECS can achieve a dramatic reduction in reasoning tokens by over 50\% across seven benchmarks while simultaneously maintaining or even improving performance. It demonstrates conclusively that substantial gains in reasoning efficiency can be achieved without compromising a model's underlying reasoning power. Code is available at \url{https://github.com/pixas/DECS}.

Published as a conference paper at ICLR 2026
OVERTHINKING REDUCTION WITH DECOUPLED RE- WARDS AND CURRICULUM DATA SCHEDULING
Shuyang Jiang Yusheng Liao YYa Zhang 9., Yanfeng Wang Yu Wang
Fudan University
School of Artificial Intelligence, Shanghai Jiao Tong University
Shanghai Artificial Intelligence Laboratory
shuyangjiang22 @m fudan. edu. cn
[liao20160907, ya.zhang, wangyanfeng622, ywwp:/ennneeeinnnnnnninanaseeeee yuwangs eew.etuuuuedeeeee..e
ABSTRACT
While large reasoning models trained with critic-free reinforcement learning and verifiable rewards (RLVR) represent the ttate- their practical utility is hampered by "overthinking". critical issue where models generate excessively long reasoning paths without any performance benefit. Existing solutions that pe- nalize length often fail, inducing performance degradation due to a fundamental misalignment between trajectory-level rewards and token-level optimization. this work, introduce novel framework, DECS, built covery of two previously unaddressed flaws in current length rewards: (1) the erroneous penalization of essential exploratory tokens and (2) the inadvertent re- warding of partial redundancy. Our framework's innovations include (i) first-of- its-kind decoupled token-level reward mechanism that surgically distinguishes and penalizes redundant tokens, and (ii) novel curriculum batch scheduling strategy master the sateiincy--fficage equilibrium. Experimental results show DECS can achieve a dramatic reduction in reasoning tokens by over 50% across seven benchmarks while simultaneously maintaining or even improving performance. It demonstrates conclusively that substantial gains in reasoning efficiency can be achieved without compromising model's underlying reasoning power. Code is available at htttps:: //t::///ggtttt/// com/ htt/ppixaassss DECSS
theoretical dis-
Many Penalized Correct High Tokens:
Rewarded Redundont Tokens:
AMM22024
Performance Seven Benchmaris
Optimal Curve Base
idoal PURK
Penalize Exploration
Reward Redundoncy
Previous Method
47
45
24
44
IddprTNigg UC1001
Tokens
Tokens
Necessory Reasoning Prefix Tokens
</thinlo token
Redundont Tokens
High Entropy Tokens
Reward Correctness
Penolize Correctness
(-th response of group Correct Response
Incorrect Response
Figure 1: Left: Two major flaws of prior practice apply sequence-level length reward without trol of training data. Negative advantage values penalize correct high entropy tokens from long sequences while positive ones reward redundant tokens from short sequences; Middle: Flaws of length rewards lead to inferior performance and suboptimal efficiency gains on AIME2024 dataset; Right: DECS improves pass@ I of base models while reducing 60% token costs compared the base model across benchmarks. Experimental details are presented in Appendix G.5
"Corresponding Author

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent large reasoning models (LRM; Guo et al. 2025), OpenAl 2025 : Owen 2025 trained with critic-free reinforcement learning (RL) algorithms, such as GRPO (Shao al. 2024). DAPO (Yu) cet al. 2025), and REINFORCE++ (Hu et al. 2025a), have demonstrated impressive reasoning capa- bilities through verifiable outcome rewards. hallmark of such models their increased propensity to generate high-entropy tokens (e.g., "wait", "however". "alternatively"). which serve to bridge logical transitions between reasoning steps (Wang et al. 20256). While these tokens reflect ac- tive reasoning mechanisms that enhance performance, the propagation of trajectory-level rewards to all tokens can inadvertently encourage prolonged generation led by high-entrope tokens even after reaching correct answer, phenomenon known as "overthinking" (Ji al. (002). To address this inefficiency without sacrificing reasoning quality, recent approaches incorporate a small length penalty into the correctness reward (Hou et al. 2025
Su & Cardie [022]] Aggarwal & Welleck, 2023 Zhang et al.] 2025d; Kimi et al., [222 Wu et al. 2025), using critic-free RL frameworks like
GRPO to promote concise yet effective reasoning.
Despite these advancements, we find that existing methods still fall short of achieving the optimal salciency-performanne trade-off: improvements in reasoning speed often come at the expense of degraded reasoning fidelity. This suboptimality raises a fundamental question: why do current re- ward designs fail effectively balance brevity and capability? To investigate this, we conduct theoretical analysis of the logit dynamics of two key groups of tokens within the GRPO framework: (i) high-entropy tokens that initiate exploratory reasoning paths, and (ii) those belonging the Nec- essary Reasoning Prefix (NRP), defined as the minimal prefix of a reasoning trajectory that suffices to justify the final correct answer. Our analysis reveals two critical limitations arising from the mis- alignment between sequence-level length regularization and token-level policy updates (depicted in Fig. Left)) revealing inherent tensions in how efficiency is incentivized during training.
First, sequence-level length penalties inherently suppress high-entropy tokens, even when they con- tribute to valid reasoning ($3.2). Specifically, in GRPO, overlong (yet correct) trajectories receive uniformly negative advantages across all tokens from length penalties. Consequently. when all re- sponses to a given prompt are correct but differ in length, shorter trajectories yield positive advan- tages while longer ones receive negative ones. This leads to reduction in the logits of high-entropy tokens through policy gradient updates. When easy prompts dominate the batch and response lengths vary significantly, this negative gradient becomes dominant over iterations, causing the policy to avoid generating these tokens, even if they are essential for productive exploration (Theorem [[] This leads premature convergence and deviation from the optimal efficiency-efficacy trade-off.
Second, training convergence is impeded by misaligned incentives (..3) Without explicitly de- coupling the NRP serving as the minimal sufficient reasoning prefix from subsequent generations, tokens produced after the NRP in shorter trajectories may still receive positive advantages. This falsely reinforces redundant steps, encouraging the model to continue generating beyond logical ne- cessity. These spurious rewards not only distort the learning signal but also slow down convergence, limiting the extent of achievable efficiency gains under finite training updates.
Building on these insights, we propose DECS, novel framework with Decoupled token-level re- wards and Curriculum data Scheduling for overthinking reduction ($4). To enable precise inter- vention, we fine-tune lightweight judge model to identify NRP boundaries. Based on this, we design a decoupled reward function that ensures redundant tokens generated after the NRP are con- sistently penalized, thereby suppressing overthinking during autoregressive decoding. Meanwhile, we introduce a curriculum batching strategy that adaptively balances the proportion of easy prompts according to the average NRP ratio in the current batch, mitigating undue suppression of exploratory behavior. Experimental results on two base models show that DECS reduces reasoning tokens by over 50%, while maintaining or surpassing performance on both deterministic (Pass @ I; Table I and exploratory (Pass K; Fig. 3c) metrics. In summary, we conclude our contributions as follows:
1. Misalignment Analysis: We identify a fundamental misalignment between sequence-level length penalties and token-level policy optimization in critic-free RL. Our theoretical analysis demonstrates that this misalignment not only inhibits the generation of high-entropy tokens, which are essential for valid reasoning, but also hampers efficiency improvements due to mis- guided gradient signals.
2

---

Published as a conference paper at ICLR 2026
2. Adaptive Sampling with Decoupled Reward: We introduce DECS, novel method that em- ploys a decoupled reward system to consistently penalize redundancy. Coupled with a dynamic batching strategy, this approach mitigates the over-penalization of exploration by incorporating adaptive curriculum control.
3. Comprehensive Evaluation: We perform extensive evaluations across two model scales and seven benchmarks, showing that DECS consistently reduces over 50% thinking tokens without sacrificing base models' capability boundary.
2 PRELIMINARY
2.1
REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS (RLVR)
The RL objective for the policy Te is to maximize the cumulative rewards r received from the verifier. Specifically, Policy Gradient WWilliams 1992 gives the following objective function:
( Eg D.onoffa
V log (o (ze) A<oe)((=)).)),
(1)
where D is the training distribution, q is an input prompt, is output sequence consisting of tokens for, .2...., .r), ; and A(o ji is the advantage the j-th token given the state ocj- Recently, DeepSeck-RI (Guo et al. 2025) boosted large language models' reasoning ability via the Group Relative Policy Optimization (GRPO; Shao et al. (2024)) algorithm. Each rollout is labeled with verifiable reward r(.), while its advantage is estimated using the group average and standard deviation values of rewards from group of G trajectories O = {o:) generated based on the same prompt qF
A r(o;) - mc r(oG))
(2)
std(r(o))...,r((()))
GRPO optimizes the policy using the PPO surrogate loss (Schulman et al. 2017):
G
lo.
E Bg (-(())
min (pijA, clippp., e, e)
(3)
Σ
0i,4j- ②) is the importance sampling ratio, |o,| is the sequence length. The KL term reduced align with Hu al. (2025b). Models are incentivized to explore new trials, cross-verifying temporary results using diverse approaches, and correct existing results, based on high-entropy decisive tokens (Wang al. 20256). However, although the high fre- quency of generating high-entropy triggers does boost the model for challenging problems (Muen-) nighoff et al. (025),, such improvements are not consistent (Ghosal et al. (2025. and introduce great verbosity and "over-thinking" for vanilla queries (Ji et al. 2025).
where Pi.j = (os. oi 4) Tola
2.2 EFFICIENT REASONING WITH LENGTH PENALTIES
One of the most straightforward methods to add length- based reward along with the fundamental correctness reward encourage shorter yet correct responses (Hou et al. [025] Su & Cardie [2025]
Aggarwal & Welleck 2025). Specifically, if adopting a monotonically decreasing length reward
Tunction f(1) -y accepting the sequence length 1 input, the combined reward is defined
(o,) (((.)) - 7/00 (((.))
o; correct
otherwise
where 7 is small factor prevent the length reward from leading the overall reward, which could
be adaptively computed (Zhang et al. 2025d) or preset as a hyperparameter (Kimi et al. 2025
3 ON THE LIMITATIONS OF LENGTH-GUIDEE REASONING OPTIMIZATION
In this section, we formally reveal two significant limitations of current length-reward driven ef- ficiency reasoning under the representative critic-free RLVR algorithm, GRPO, by analyzing the
3

---

Published as a conference paper at ICLR 2026
misalignment between the trajectory-level advantage and the token-level optimization objec- tive for redundant thinking tokens. Through an analysis of logit dynamics, we demonstrate that
this misalignment degrades reasoning performance ($3.2) and fails to reduce early redundancies, thereby limiting potential gains in efficiency ($3.) The concepts for each involved notation and abbreviation are illustrated in Table
3.1
LOGIT DYNAMICS UNDER POLICY GRADIENT
The LRM policy at step softmax policy, is parameterized by
exp(2oct.o))
m (ot Occ)) Edeivi exp Eogera
(5)
where Foci,or the output logit of token or given prefix Oct and or To (- Oce). Under the learning objective of the policy gradient, we have the following lemma (Cui et al. 2025):
Lemma 1 (Difference of policy logits in vanilla policy gradient). Let the actor policy To be a
tabular softmax policy and updated using Eq. 7 with learning rate n. the difference of 2ogr. between two consecutive steps m and m + saltsfies
m++1 O0C
m = To Oct) A(0
3.2 OPTIMIZATION WITH POSED EFFICIENCY
GRPO estimates advantage with intra-group relative reward by sampling G rollouts repeatedly for a prompt. When G rollouts contain both correct and incorrect trajectories, correct sequences always receive positive advantages, differing only in their magnitude and contributing little to effi- ciency optimization. contrast, when rollouts generated by the policy To on an easy prompt qa.G all correct, the correctness reward becomes constant across trajectories, leaving length as the sole discriminative signal. As a result, correct yet overlong trajectories receive negative advantage estimates through the GRPO algorithm, which activates efficiency optimization.
Recently, Wang et al. (022bb observes that the superior performance of LRMs is driven by high- entropy tokens, which lead the policy to conduct exploration and reflection. However, trajectory- level negative advantages would back-propagat to all tokens in Eq. 3 including the essential high- entropy tokens. Under Lemma the negative advantages will cause the decline of probability for generating high-entropy tokens, and thereby the optimization process shifts from its intended goal, i.e. improving efficiency while preserving performance, to one that trades correctness for shorter trajectories. Formally, we could derive the following lemma:
Lemma 2 (Decreased logits for correct high-entropy tokens). (Proof in Appendix A.7y For f defined Eq. [ the expected change of logit for high-entropy tokens [Onigh } from G correct rollouts {o.) ~ πe(- - ge,g) sampled from qe,G between two consecutive optimization steps m and m++11 is strictly negative:
EoE (osigh))
mm++1
20
In the above lemma, the correctly generated high-entropy tokens produced by ga.G have their genera- tion probabilities reduced, which may disrupt or even distort the learning direction an entire batch with respect to high-entropy tokens, subject to the constraints specified by the following theorem:
Theorem 1 (Maintenance of High-entropy Tokens Under Batch Learning). (Proof in Ap- pendix AA2) Let the ratio of prompts qa.G be k. Assume that the length reward is defined as Eq. F and T. is the standard deviation of response lengths of qa,a a average, the condition for which the expected logit change for correct high-eniropy tokens among a batch is greater than O as follows:
RG <C,
where C' is a constant with respect to the rollout tokens generated during a mini-batch.
This theorem implies the condition under which the policy would suffer from performance degrada- tion when applying length reward with GRPO. When RO becomes too large, the policy longer follows the serformance-effeieenn efficiency come the cost of the proactivity of high-entropy tokens, thereby degrading performance.
trade-off frontier. Instead, it shifts into a regime where gains in
4

---

Published as a conference paper at ICLR 2026
3.3 INSUFFICIENT EFFICIENCY
In addition to the decreased performance, current length-based reward methods also fail to achieve sufficient reduction of overthinking. Specifically, differentiate the redundant tokens to be duced by formally defining the necessary reasoning prefix as the most compact thinking process that supports deriving correct answer for the first time:
Definition 1 (Necessary Reasoning Prefix). Let q be input prompt, y² be the ground truth answer, and o = (01, 02..... -,OL) be generated response sequence on 4 where I. = [o]. The necessary reasoning prefi (NRP) of with respect q is the shortest prefix O1:K* such that ANSWER(OL.K-) = =y" and vk < K* either ANSWER(01.4 = null or ANSWER(O1.)) + y²
As the correct answer is logically justified at position K the token set {oz | > K2} is considered redundant by many works (Dai al. 2025 Yue al. 2023). To prohibit the policy from contin- ually generating further tokens after the already generated NRP tokens, we convert the objective to
minimizing the probability of generating the first thinking token after the NRP. which functions on the reduction of holistic redundancy due to the autoregressive generation of LRMs:
T min E Booxoo--1o Pog
m POgk"
8.1. j j= j=k* K +1
(6)
Applying Lemma I this objective could be converted into a policy weighted expectation advan- tages, which is shown to be positive:
Theorem 2 (Suboptimal Reduction of Redundant Tokens). (Proof in Appendix A.3) Let the
ward function f be defined as Eq. f Let j = K + denote the position of the first redundant token beyond the NRP in correct rollout Let A(o) be the group-relative advantage computed via Eq. 2 Then, the expected policy gradient signal for the first overthinking token, denoted as TT(lo Aco) 1 = K 1] satisfies:
J(A:J K + 1)
J(A:j K* + +
This theorem tells us that although the policy would reduce thinking length by penalizing tokens far from the end of NRP from overlong responses, the policy cannot learn to stop at the end of NRP the first redundant token. This undesired property keeps partial overthink-
given no penalization ing tokens, leading to suboptimal reduction of redundancies.
4 DECS
Given the above analysis, we propose DECS, which contains three main designs to achieve the high- est efficacy-efficiene tradeoff. First, ensure that redundant tokens are penalized deterministically. we train small module that precisely identifies necessary reasoning prefix (NRP) components for correct trajectories ($4.T). After that, we design a decoupled token-level reward and differentiate the reward scale for necessary and redundant tokens, to ensure enhanced efficiency without com- promising performance ($4.2). Based on the conception NRP, we propose to prevent aggressive penalization high-entropy tokens following NRP by refactoring the data distribution of batch according to the current levels of redundancy incrementally ($4.3) 3 algorithm.
Fig. illustrates the overall
4.1 DETECTION OF NRP
It is common practice to train token-level classification model to annotate NRP components. How- ever, it requires the same tokenizer the policy, which hinders adaptation to other policies. To this end, we implement this detector reasoning chunk contains the correct answer to a given problem. Specifically, given a correct rollout o, first extract the reasoning tokens Othink (01,... ... O</thhnk:>). ). Using pre-defined separator tokens, the reasoning process is segmented into multiple chunks: S = {s1, $2. , , where 8. is the e-th chunk of Othink The judgment js, E {yes, no} for substep s, is generated by prompting Mjudge given the problem q and corresponding ground truth y" as:
lightweight generator model Mjudge, determining whether
js. Mjudge 14, 8c,
5

---

Published as a conference paper at ICLR 2026
Generated Outputs
Split 6 Outputs
Outpun-1
Outpat-1
O easy prompts ?
What is 2-3?
five
Wet.
Curpan-2
Gutaur-2
al together, Walt,
ditagether:
Wot: check.
Output-(6-1)
Output (S-1)
.oqasl neosp
equlle
Just recop.
Output-G
Detector
Output-6
Palt docar' sourd.
Annwvrr
Decoupled Reword Group Advantege
NaY sokens Alvoys nen- negetive
ale /haaer
Concluaion hokone Always nen-negative Adv.
howe
prompts
Update with Token-level PPO Surrogate Loss
aly/ban
Leading Redundom tokena: regetivv
altogether 5
Anower
aedandart toiene Large
F,
alg 1/beasx
Update ratio 40. of m-th th step with Km
equale
Just
Arswer
Resindont Always
aly Phani
Km clip((m-1 1 (R. Rm-1). kQ)
Propertion
Seat
that 1905" Soarl.
Ansaer
Figure 2: Overview of the DECS training pipeline. (I) Decoupled Token-level Reward: We fine- tune small language model to detect the necessary reasoning prefix (NRP) from other redundancy, which are separately rewarded to penalize overthinking consistently while maintaining the proba- bility for generating necessary reasoning steps. As the running example "What is 2+3?" shows, the NRP contains the reasoning chunks from the starting token to the first time the model generates the correct answer "5". After that, any leading redundant token like "Wait" receives negative advan- tages, and thereby discourage any redundant tokens to be generated via autoregressive generation.
a= rー ro. (2) Curriculum Prompt Schedule: The number of easy prompts qe.G grows in step with the progressive decline in remaining redundancy.
The NRP spans all reasoning chunks from the start through the first chunk whose judgment is "yes":
NRP
8iy
where c" min fc e [1, [S]] Js. yes}
Here, + denotes the concatenation of reasoning chunks, and the c*-th reasoning chunk the first to entail the correct answer y". The training details are illustrated in Appendix G.6
4.2 DECOUPLED REWARD ASSIGNMENT
For a group of rollouts {o generated based on a given prompt g, we design a token-level reward which ensures maximum reward for NRP tokens and preferences for short yet correct responses:
r+ 1o; correct
S K V oj & Othink V oi.j 0
Ts.j (ro (r4-ru)LL)) 1 correct
j > K No E Othink
T.max
where r+ and ru are respectively the maximum and minimum positive rewards, K is the last NRP token index of o and ( denotes a padded token. Since the inverse proportional function enforces a far lower reward for redundant tokens compared to r+ any token followed by the NRP would consistently receive negative advantages. Such penalization, as a result, helps to reduce verbosity via the autoregressive generation property of LRM regardless of sequence lengths. Besides, only redundant thinking tokens possible to receive negative advantages, which prevents biased penalty on essential reasoning tokens and answer conclusion tokens, and sustains the policy during the RL training. Finally, the teven-level advantage is computed similarly to GRPO and updated with Eq.
Ti.j mcan mean(r1.).....PC
APESS
(10)
std(ri, FG.))
Appendix Ci detail explains the functionality of Eq. for penalizing any leading redundant tokens.

---

Published as a conference paper at ICLR 2026
4.3 CURRICULUM PROMPT SCHEDULE
After identifying NRP tokens, penalization of high-entropy tokens occurs only in redundant tokens following the NRP. Therefore, we schedule Fm based on the proportion of NRP Rm in correct sequences within a batch, which reflects how many correct high-entropy tokens would be penalized:
Km cll=cippmmmm-- + 3(Rm Rm- kin))
(11)
where o is the ratio of 9a,G among the current sampled batch and / is hyperparameter to con- trol the learning progress. As trajectories with zero advantages would not provide any learning signal, we follow Yu et al. 2025) to filter prompts whose G rollouts are all incorrect and fill the batch by over-sampling. This curriculum strategy, designed to be bounded and monotonic, en- ables smooth adjustment in response to the observed NRP ratio, which aligns with the principle of curriculum learning (Bengio et al. 2009). By setting moderate value 3 with grid search (see Ap- pendix H.T). DECS can satisty the condition elucidated in Theorem to maintain unbiased learning of high-entropy tokens throughout the whole training process. This yields stable convergence with no observed training instability performance degradation, which is reflected in Fig. 6 and
5 EXPERIMENTS
5.1 EXPERIMENT SETUPS
Olympiad- Bench (He et al.] (2024). AIME2024 (Mathematical Association of Americal (002a)) and AIME2025 (Mathematical Association of Americal (025b) as in-domain testbeds, GPQA- Diamond (GPQA-D; Rein et al. (202)) and LiveCodeBench-v6 (LCB; Jain et al. (2025) as held-out testbeds, covering math, coding, and science tasks with diverse complexity. We choose ThinkPrune (Hou et al., 2025), TLMRE (Arora & Zanette 2025), AdaptThink (Zhang et al.) 2025b), LC-RI (Cheng et al. 2025) as baselines, and also include GRPO to serve as performance reference. For fair comparison, we set the temperature 0.6, top_p as 0.95, and use maximum token limit 16384 suggested by Guo et al. (2025). We conduct 128 rollouts for AIME2024, AIME2025, and AMC23, 16 rollouts for OlynpiadBench, MATH500 and GPQA-D, and 10 roll- outs for LCB to compute pass @1 We also compute the Average Efficiency Score (AES; Luo et al.) (2025)) for comprehensive assessment of efficiency and efficacy. The details of both metrics presented in Appendix G.4
Evaluation We MATH500 Lightmmnn al. 2023), AMC23 (AL-MO 2024).
Training We adopt DeepScaleR (Luo et al. 2025b) the training set and choose DeepSeck-RI R1- Distill-1.5B (DS-1.5B) DeepSeek-RT-Dise Ri-titlll--------- -- (DS-7B) as base policies. We perform 16 rollouts per prompt and use veRI. (Sheng al. (025) as the training framework. r+ in Eq, Faree set 1.1 and 1.0, respectively, while 3 in Eq. 11 is set to 0.2 with grid-search. Additional hyperparameters are presented Table g
5.2 RESULTS
As shown in Table I DECS reduces average reasoming length by 57.17% on the 1.5B model while improving pass@ accuracy by +2.48 points, demonstrating simultaneous gains in efficiency and performance. On the 7B model, which exhibits less overthinking, DECS still cuts thinking tokens by 49.50%, outperforming all baselines, with a +0.8 point accuracy gain. Compared the previous best, DECS improves the AES score by 0.12 and 0.14 on the 1.5B and 7B backbones, respectively, establishing a superior prrrieecyy-erffommncce trade-off that compresses the computation without sacrificing output quality. Meanwhile, although the NRP detector is specialized for math reasoning and the training data only cover the math corpus, such superiority of efficiency generalizes robustly to oft-oa-domain tasks (56.33% fewer tokens in GPQA-D and 33.52% fewer tokens in LCB), con- firming DFCS 's strong transferability and practical value for broader reasoning tasks.
5.3 ABLATION STUDY
In this section, conduct ablation study on the DS-1.5B base policy, to reveal the critical complementary relationship between the schedule prompt scheduling (CS) and decoupled token- level reward (DR). We show the results in Table 3 and plot the comparison in Fig. 3a We observe

---

Published as a conference paper at ICLR 2026
Table 1: Pass @ (Acc) and the number of tokens (#Tok.) used across seven benchmarks. "LCB." denotes LiveCodeBench-eee "OlympiadB." " denotes the OlympiadBench, and "GPQA-D" denotes GPQA-Diamond. The best performing score is marked in bold and the recond-best underlined.
AIME2024
AIME2025
AMC23
MATH500
OlyuppiddB #Tak.
GPQA-D
LCB
Model
#Tak.
FTok.
#Tok.
Ace:
FTok.
#Tok.
RTok.
#Tok.
DS-1.50
22.94
7875
05.55
4847
53.78
8540
10560
45.21
9340
GGRPO
32:76
53.34
25.91
8451
77.09
5722
87.34
3577
53.73
6425
35.76
3953
26.45
5759
49.15
6814
0.55
AdaptThink
81.57
1488
4093
26.98
42.78
ThinkPrune
26.91
5306
20.86
4937
72.87
2869
84.27
1879
53.04
3477
35.51
38.99
25.36
S515
45.83
3975
0.62
ILMRE
29.87
75500
22.24
7151
24.51
3943
$4.86
2576
36.08
2833
35.74
4896
26.13
7737
46.28
5498
0.52
RC
23.65
6904
19.61
6681
05.59
3715
52.02
2277
31.37
4519
30.95
5377
23. 54
69-10
12.56
5202
0.15
DOCS
31.25
55.50
23.78
4965
75.37
2588
84. 40
1817
56,10
339%
35.92
32.55
27.66
6026
47.78
4000
0.74
DS-7D
50.65
10508
36.67
11096
5764
93.25
3654
09.22
46.46
7502
25.95
1966
61.57
7857
0.00
1GRPO
52.50
38.54
9670
91.88
5205
94,21
3520
72.59
49.62
6101
47.31
3569
63.86
6929
0.23
53.31
5984
36.48
86.66
3675
91.06
1834
5746
47.09
8239
60.93
6199
0.16
ThinkPrune
2105
4498
62.17
TLMRE
3329
91.33
2073
3913
6772
01.16
5142
LC-RI
50.52
6891
32.50
7387
85.74
2802
90
1473
67.76
3983
18.58
4672
46.5
6554
60:32
4823
0.28
DECS
51.33
5277
36.43
5516
89.04
3772
92.96
1728
70.28
3283
49.27
3376
48.05
5921
62.48
3968
0.54
Table 2: Generalization to the Qwen3-4B model. DECS still achieves 0.61 AES score, with 54.80% reduction to overthinking and 1.32 pass@1 improvement.
AIME20024
A1ME22025
AMC13
MATH500
Olympiadi
GPQA-D
LCB
Average
Model
#Tiok.
STok.
#Tok.
VTok.
#Tok.
Qwen3-48
64.82
11611
56.
12870
60
7478
93.74
4839
71.07
9144
30:11
H072
62.17
9713
9104
0.00
DECS
65.38
5431
$6.96
5758
93.59
2864
93.78
1648
74.09
3646
41.00
3260
63.27
6196
69.72
4115
0.61
that without adaptive scheduling of easy problems, there is noticeable performance drop, which verifies the impacts elucidated Theorem [ Meanwhile, without decoupled rewards, the policy remains nearly 25% of overthinking tokens, verifying that the sequence-level length reward fails to fully reduce overthinking as Theorem implies.
5.4 BACKBONE GENERALIZATION
In this section, we generalize DECS to Qwen3 backbone model, where we apply DECS to Qwen3- 4B (Yang et al. 2025) with the same training hyperparameters introduced in $5.1 Results in Table demonstrates that DECS successfully extends to Qwen3-4B, with 54.80% reduction of reasoning tokens and 1.32 pass@) improvement on average. This strongly implies that DECS is backbone- robust, and remains effective on stronger base model.
6 ANALYSIS
In this section, we discuss the following research questions:
RQ1: How do the decoupled rewards help DECS achieve the highest efficiency?
RQ2: How can DECS balance the exploration and exploitaiton when compressing reasoning?
RQ3: How does DECS perform with variable token budget?
RQ4: How do representative high-entropy tokens distribute after applying DECS?
RQ5: How does compressed thinking spread over various difficulty levels?
Most of the tokens reduced by DECS stem from non-NRP tokens. To reveal the significance of decoupled learning for reducing redundancy, we compute the proportion of NRP tokens in all thinking tokens (PNRP) of correct trajectories generated on AIME2024. We plot the average token costs and the average PNRP score in Fig. 3bb Although ThinkPrune reduces similar number of thinking tokens as DECS, it achieves a relatively lower PNRP score. This inconsistency reflects that part of the reduced tokens stems from necessary reasoning tokens that contribute to the final correctness, which explains its performance drops in Table [ Compared to LC-RI remaining ~ 10% redundancy, DECS further reduces non-NRP tokens and improves the PNRP score, highlighting the utility of the decoupled reward for unified reduction of overthinking.
Response to RQ1:
8

---

Published as a conference paper at ICLR 2026
Thancrune TIMAI
LGRI
1.0
Passe1
47
0.95
0.9
0.90
46
Average PNRP
0.65
45
0.80
Average Tokens
5K
0.75
4K
+1133 Avg. Tokens
0.70
8K
0.3
02
DECS
wio CS Model
wio DR
10000
Average Tokens
K
(a)
(b)
(c)
Figure 3: (a) Ablation study with two major components of DECS the DS-1.5B base model. (b) Comparison of DECS with other baselines on the proportion of NRP tokens (PRNP) and actual reasoning tokens in the AIME2024 testbed. (c) DECS performs on par with the base policy (DS- 1.5B) in terms of Pass K scores on three challenging benchmarks.
Our
B1se
30
1.0
Averagie PHRP
Thertore.
8
S0.
161M
327ER
20 10
Leve11 Leve:
Level
LeoeT Level1 ANE
Max Response Length
Keyword Count (K)
Difficulty Level
(a)
(b)
(c)
Figure 4: (a) Average tokens and Pass@1 performance with 5 increasing generation budgets; (b) Frequency of reasoning behavior tokens after applying DECS; (c) Consistent compression rates of DECS on six difficulty levels sourced from MATH500 and AIME2024.
Response to RQ2: DECS maintains similar exploration potentials as the base model. To investigate whether DECS achieves good pass I-efficiency tradeoffs by sacrificing the
problem-solving potentials compared to base models, we compare the pass@k @k scores (k (2, 4, 16, 32, 64, 128)) on AIME2024, AIME2025 and AMC23. Results in Fig. 3d and Fig. 8c reveal that across nearly all sample numbers, the success rate on the performance curve of the model compressed by our method almost perfectly overlaps with that of the original model. This result strongly demonstrates that the model's exploration ability to find correct solution through multiple attempts not injured by DECS. It suggests that preventing high-entropy tokens from receiving negative gradients sufficiently preserves most exploratory and creative properties.
Response to RQ3: DECS consistently improves the token efficiency across diverse token bud- gets. To validate whether the protection NRP and exploratory high-entropy tokens would both improve the model's performance on token-cone loken-constrained scenarios and not impair its performance with a less-constrained token limit (Snell al. 2025). we evaluate under increasing token lim- its: [2,048, 4,096, 8,192, 16,384, 32,768]. Fig. 4a 8a and [8b demonstrate the Pass 1 scores and average token costs on AIME2024, AIME2025 and AMC23 with the 1.5B policy. After applying DECS, the policy could use far fewer tokens to achieve competitive performance across diverse to- ken limits, which holds even for the 32,768 context limit. For the 7B policy (depicted in Fig. 10), DECS performs par with the base model with negligible performance gap under the 32,768 to- ken limit, but consuming fewer than 30% tokens. This further validates that DECS achieves superior efficiency compression without sacrificing the model's capability boundary excessively.

---

Published as a conference paper at ICLR 2026
Response to RQ4: DECS reduces unnecessary reflective and conclusion tokens, but remains a consistent tendency for creative and context formulation tokens. To investigate how DECS refines the reasoning process and modulate the distribution for high-entropy decisive tokens, analyzed the frequency of representative tokens with different reasoning behaviors, including "Self- Correction & Verification", "Exploration & Alternatives", "Context Setting" and "Conclusion Draw- ing" (Wu et al. (025). Results in Fig. 4b reveal a significant shift in the tendency for correction tokens with DECS, which is the main source of overthinking. Meanwhile, the negligible change in the frequency of exploratory tokens also suggests that the shearing of tokens after NRP hardly cause degradation of creative thinking. Also, the dramatic decrease of conclusion tokens reflects that after applying DECS the policy is more confident in their reasoning intermediate results, which leads to similar even higher pass@ scores across diverse benchmarks.
Response to ROS: DECS compresses non-NRP tokens under variable input complexity. Since large reasoning models (LRMs) often overthink even easy queries, examine whether DECS consistently reduces overthinking across varying difficulty levels. We compute the PNRP score the MATH500 and AIME2024 datasets, which provide self-contained difficulty gradients across six levels. As shown in Fig. 4c PNRP scores increase with problem difficulty, and DECS con- sistently achieves scores above 90% across all levels. This trend holds for the 7B model (Fig. (b), with even higher scores on AIME2024, likely due to its improved reasoning ability and reduced inherent overthinking. These results confirm that DECS enhances reasoning efficiency in LRMs across diverse inputs, demonstrating its effectiveness and generality.
CONCLUSION
In this paper, we theoretically identify two key flaws in current critic-free RL methods for reducing the overthinking phenomenon, which stems from the misalignment between token-level overthink- ing reduction and sequence-level reward assignment. To mitigate these two drawbacks, we propose DECS, which proposes decoupled reward system for NRP and non-NRP tokens for consistent reduction of overthinking, and introduces curriculum batch scheduling for maintaining exploratory potentials. Experiments show that DECS reduces ~50% of reasoning tokens while maintaining or improving performance, achieving more efficient reasoning without sacrificing accuracy.
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their insightful comments and suggestions. This work was supported by the National Key R&D Program of China (No. 20222D0162101), National Natural Science Foundation of China (No. 62576209) and STCSM (No. 2025SHZ2DZX0025005)
REPRODUCIBILITY STATEMENT
In this section, we list any related materials that help to reproduce this paper
Datasets: The training set we used is described in $5. and evaluation sets we used are described in Appendix G.2
2. Theoretical Support: Any assumptions, lemmas, propositions, theorems and corresponding proofs are detailed in Appendix [
3. Code: The code reproduce algorithm would be put into the supplementary materials.
4. Computational Resources: We use 4xNVIDIA A100 80GB GPUs conduct all experiments.
REFERENCES
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning. In Second Conference on Language Modeling, 2025. URL https: //openrevicn ew. net/ http:/oorrum2...2420xxxxxxx
10

---

Published as a conference paper at ICLR 2026
AL-MO. Amc23 dataset. Hugging Face Dataset Repository, 2024.
URL https://
huggingface co/ dat aoche99/amces Accessed: 2025-06-26
Daman Arora and Andrea Zanette. Training language models to reason efficiently. arXiv preprint Stfi:22502.04663,, 2025.
Marthe Ballon, Andres Algaba, and Vincent Ginis. The relationship between reasoning and performance in large language models-o3 (mini) thinks harder, not longer. arXiv preprint arXiv:2502.15633 2025.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th anmual international conference on machine learning. pp. 41-48, 2009.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, al. Do not think that much for 2+ 3=? the overthinking of ol-like llms. arXiv preprint arXiv:2412.211877 2024.
Zhengxiang Cheng, Dongping Chen, Mingyang Fu, and Tianyi Zhou. Optimizing Length Compres- sion in Large Reasoning Models, 2025. htRt:/tt::/// httttpppp///...trreettteet www.wawwwwv.....o...aooooo abs/ /206. 14755
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo. Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models, 2025. /tt.attp://...i..orr/r////////// wwwporrg//bb..oooooooa/.....aaat
Muzhi Dai, Chenxu Yang, and Qingyi Si. S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models, 2025. URL httt:://aaxxiii orgg abs / 2505. 07686
Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, and Amrit Singh Bedi. Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models, 2025. URL
7/arxiv. warxiv.orq/abs.... wtt.:////abbss/sss/s/ 04210
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma. Xiao Bi, et al. Deepseek-rl incentivizes reasoning in 1lms through reinforce- ment learning. Nature, 645(808) (1178081)66333. 1):633636333 2025.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, NY, edition, 2009. ISBN 978-0-387- 84857-0. doi:
00007/978-00887-4888555
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi. Zhiyuan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scien- tific problems. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 3828-3850, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi:
URL https://aclanthh.lee orgp:///cooooy.o///222222
hel-iong.2117
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. ThinkPrune: Pruning Long Chain-of-Thought Thought of LLMs via Reinforcement Learning, 2025. URL
attt:://arxiv http::://axxlv.or//bbbsss
org abs 2504. 01296
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03262. 2025a.
Jingeheng Hu. Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Bpeen-Reasoner-Pe Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model, 2025b. URL http::ttpp//ii..t/t/aaaa wwwp:/xwwwwwwxrpox///// 24290

---

Published as a conference paper at ICLR 2026
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamina- tion free evaluation of large language models for code. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https://p//......... http://ppppreeee..c/e/ee.. ew. net. tetaat/frrrt forum? ? d= chfJJYC31L
Ke Ji, Jiahao Xu, Tian Liang. Qiuzhi Liu, Zhiwei He, Xingyu Chen, Xiaoyuan Liu, Zhijie Wang, Junying Chen, Benyou Wang, et al. The first few tokens are all you need: An efficient and effective unsupervised prefix fine-tuning method for reasoning models. arXiv preprint arXiv:2503.02875, 2025.
Team Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501. 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, and Junxian He. Learn to reason efficiently with adaptive length-based reward shaping. arXiv preprint ar arXiv:2505.15611 2025.
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. Ol-pruner: Length-harmonizing fine-tuning for ol-like reasoning pruning. In 2nd Al for Math Workshop @ ICML 2025, 2025a. URL
Eorum? iirum?iddioYybCRRy
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol- preview with a 1.5b model by scaling rl. /ttp::::/ppeettyraad.o.oooooo www.oo...oooiiinoooomom site/ Sww.o--uupsssigg---ee
Mode with ott.::/co........a.aaaooaaaaaaar otg.aa/00....a...........aa..aa.
2025b. Notion Blog.
Shangke Lyu, Linjuan Wu, Yuchen Yan, Xingyu Wu, Hao Li, Yongliang Shen, Peisheng Jiang, Weiming Lu, Jun Xiao, and Yueting Zhuang. Hierarchical Budget Policy Optimization for Adap- tive Reasoning, 2025. URL http::ttta/aaaea
org abs /2/.2/507..12
Mathematical Association of America. Aime 2024 dataset. Hugging Face Dataset Repository, 2025a. URL Itttps://huut http:::://uugggnggaceee co/ wwt.:///taataaaaaaeeee /Maxwe 1- Jia /A AME _024
Accessed: 2025-06-26.
Mathematical Association of America. Aime 2025 dataset. Hugging Face Dataset Repository,
2025b. cessed: 2025-06-26.
Ac-
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fe,, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501.19395 2025.
OpenAl. Openai o3: Most advanced reasoninn model. https://openai..com com Hans CN/ 2025. Accessed: 2025-08-20.
index/ index/innttttee
Team Qwen. Qwen3 technical report, 2025. URL Utt./ttpps///rrru...rr/////////2
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In Fürst Conference Language Modeling, 2024.
12

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference Computer Systems, pp. 1279-1297, 2025.
Charlie Victor Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 1lm test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
Jinyan Su and Claire Cardie, Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards, 2025. URL httt:: org/ abs 2505.050100 18298
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, al. Stop overthinking: A survey efficient reasoning for large language models. arXiv preprint urXiv:2503. 16419, 2025.
Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412. 2025.
Chenlong Wang, Yuanning Feng, Dongping Chen, Zhaoyang Chu, Ranjay Krishna, and Tianyi Zhou. Wait, we don't t need to" wait"! removing thinking tokens improves reasoning efficiency. 2025a.
arXiv preprint arXiv:2506.088.
Shenzhi Wang, Le Yu. Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for IIm reasoning. arXiv preprint arXiv:2506.01933 2025b.
Ronald J Williams. Simple statistical gradient-followii algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-255) 1992.
Xingyu Wu, Yuchen Yan, Shangke Lyu, Linjuan Wu, Yiwen Qiu, Yongliang Shen, Weiming Lu, Jian Shao, Jun Xiao, and Yueting Zhuang. LAPO: Internalizing Reasoning Efficiency via Length- Adaptive Policy Optimization, 2025. URL http:://a.xi..orrtt////a wttp:///wwwo..////....11100.
Hang Yan, Fangzhi Xu, Rongman Xu, Yifei Li, Jian Zhang, Haoran Luo, Xiaobao Wu, Luu Anh Tuan, Haiteng Zhao, Qika Lin, et al. Mur: Momentum uncertainty guided reasoning for large language models. arXiv preprint arXiv:2507.14958, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09:: 09388. 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An Open-Source LLM Reinforcement Learning System at Scale, 2025. URL http:
//axxiv..rr/abs/2553...
Chuhuai Yue, Chengqi Dong, Yinan Gao, Hang He, Jiajun Chai, Guojun Yin, and Wei Lin. Promot- ing Efficient Reasoning with Verifiable Stepwise Reward, 2025. URL http:/t/tp::xi/// abs/ / 2508 10293
Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Yunhua Zhou, and Xipeng Qiu. Revisiting the test-time scaling of ol-like models: Do they truly possess test-time scaling capabilities? In
13

---

Published as a conference paper at ICLR 2026
Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- Vienna, Austria, July 2025a. Association for Compu- tational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v 1/265/////222.... acl-long.232. URL https://aclaame wtw.tthhooo..ooog////ooo
ume 1: Long Papers), pp. 4651 4651-4665,
2327
Zihao Zeng, Xuyao Huang, Boxiu Li, Hao Zhang, and Zhijie Deng. Done better than perfect: Un- locking efficient reasoning by structured multi-turn decomposition. In 2nd Al for Math Workshop @ ICML 2025, 2025b. UR///ctps://opeeeeriee... wtwp://wwweeeeeetoooooommo/o.o. ctmsXMMM
Guibin Zhang. Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang. Yutao Fan, Zihu Wang, Songtao Huang, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji, Michael Littman, Jun Wang, Shuicheng Yan, Philip Torr, and Lei Bai. The landscape of agentic reinforcement learning for Imm: A survey, 2025a. URL BtRL:ttts:/// httpaaai...orgb/////500/ 02547
Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. AdaptThink: Reasoning Models Can Learn When to Think, 2025b.
13417
Shengjia Zhang, Junjie Wu, Jiawei Chen, Changwang Zhang. Xingyu Lou, Wangchunshu Zhou, Sheng Zhou, Can Wang, and Jun Wang. OThink-R1: Intrinsic Fast/Slow Thinking Mode Switch- ing for Over-Reasoning Mitigation, 2025c. URL http:://axix org/ abs 2506. 02397
Xiaoyun Zhang, Jingging Ruan, Xing Ma. Yawen Zhu, Haodong Zhao, Hao Li, Jiansong Chen, Ke Zeng, and Xunliang Cai. When to Continue Thinking: Adaptive Thinking Mode Switching for Efficient Reasoning, 2025d. htl/htttt////x...cto
15400
A THEORETICAL SUPPORT
A. PROOF OF LEMMA 2
Lemma 2 For qa.a defined as above and defined in Eq. for any correct token that belongs to the high-entropy token defined in Wang al. (2025b), the expected change of logit for high-entropy tokens { (onggh from G correct rollouts {o, J Te (- 90,G) between two consecutive optimization
steps m and is strictly negative:
Boe (onigh 120
1m++1 m
<
Proof. We assume that the high-entropy token is distributed uniformly. For any sequence with L; tokens, there will be h- L, high-entropy tokens average, where h is defined as 20% in Wang et al.
2025b). For simplicity, we denote Eoe(on)))) [ z
as E
m
Since the length
"Ohagh
Zonigh
14

---

Published as a conference paper at ICLR 2026
reward is linear function W.E. length, the logit difference expectation is computed
E
2
Ohighh
2 Fobicaia
x
T
mo(o))) A(ois) lou ((riat)
f=1
A(o))
Σ To E(oiaa)
GRPO Broadcast
A(os) hI4
f(L.) - 1F(L)) L
h
O(LL)
1=1
hG 1
-1 LEIL
a1 G
Linear transformation of length
hG
(-B[LL)+ + B[L]3)
ØL
-GGo, <0
As result, E m1+1 Fonigh
m 2onggh
0 for prompt qe,G-
A.2 PROOF OF THEOREM 1
Theorem 1 Let the ratio of prompts qe,G K. Assume that the length reward defined as Eq. and a is the standard deviation of response lengths of qa,a average, the condition for which the expected logit change for correct high-entropy tokens among a batch greater than 0 as follows:
K OL C,
where a constant with respect to the rollout tokens generated during mini-batch.
Proof. Following the proof process of Lemma
2 wc can compute the upper bound of Fhigh hhgh by computing the upper bound of D E lo.l A(o))) cerrect high,
En m mm-1
which is the of advantage values of all correct high-entropy tokens:
m EB
Fhigh
mm-1 bigh
To(o A(o"))
Icorrctt hhgh
(12)
n=1
j=1
hL,A(o)))
1rorrect
where oi. is the j-th token of i-th output within a group of responses generated based on n-th prompt. Assume that the output distribution for any qe.G is i.idd, we expand this term as follows:
B G
f(L.) - MS(L)
RB
G
ΣΣ hL,A(o)))
- ccorrctt = kBh
Σ
Li
L
1
correct
0
1=1
#=
(13)
1 p
=-xBhGoL +
L
correct
==
15

---

Published as a conference paper at ICLR 2026
where ph of is the average and variance of the correctness reward for n-th prompt. Assume that the number of correct responses for each prompt is On. pe, an and vou Van expand the second term as:
On),
AB L,
AB >Σ
ΣΣ
H. 1 correct
L.
1 On
ccorrcc CB
(14)
( an)
Therefore, the second term is always positive as an < 1 and is constant within the batch B. As result, the objective would be less than 0 if and only if the first term of Eq. [3] is sufficiently
negative. Solving the inequality that Eq. 13 is positive, we obtain the following condition for which the learning signal across the batch would not penalize correct high-entropy tokens:
CB
K K. < BhG
(15)
There are two ways to break this condition: (1) more 48,6 within a prompt, parameterized as larger H; and (2) larger range of output length, parameterized as larger values of dg-
A.3 PROOF OF THEOREM 2
Theorem Let the reward function be defined Eq. 4 Let +z+ denote the position of the first redundant token beyond the NRP in a correct rollout o. Let A(o) be the group-relative advantage computed via Eq. 2 Then, the expected policy gradient signal for the first overthinking
token, denoted as I [A; = K + 1))oonto(119..c [To(o] (oz))((o) 1 = K + 1] satisfies:
J(A:== +
Proof. Considering an arbitrary easy prompt suppose the LRM mo generates a group of G correct rollouts {o:){- with lengths L,, Iz...., Lg. The advantage for each rollout is computed via group-wise standardization (Eq 2
A(o,) f(L,) - -G
L.
where L and d are the mean and standard deviation of L within the group. Assume simplified optimal reward function Aopt = -y for all = 1,2.....CC G. Now consider the first redundant token Otj=K;++ +1 generated after the Necessary Reasoning Prefix (NRP) in rollout o;- In GRPO, this token inherits the sequence-leve advantage A(o,), and contributes to the policy gradient through the term:
<0 oz²)) A(o,).
Taking the expectation over the group:
Bowno(-1o) [m A(o))(()) j = K" +1.
Formally, Let uy == mo(0, 4) be the probability of generating the prefix up to the first redundant token and simplify J [A: K 1) as [AA). Under the typical behavior of autoregressive policies, shorter rollouts have higher generation probability: if L, < Lg, then w; 2 Wi.
We only consider the case where all rollouts are redundant, i.e., for any rollout its thinking length larger than its NRP length L, > K By the definition of conditional expectation:
J(A) 21=1 W; A(o,)
G
L L
ØL
where I... 5 web 2 is the policy weighted average length. Subtracting I.,, with L, we obtain:
- -= DE
Σ L,
Σ W/
Σg
wibe C
wi
) Li) <0,
Σ1
Therefore, L. L < 0, so J(A) 00
16

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study with two major components of DECS on the DS-1.5B base model. "CS" denotes adaptive data sampling and "DR" denotes the decoupled reward mechanism.
A1ME2004
AIME2025
AMC23
MATH500
Olympiad
GPQA-D
LCB
Avg
Model
Acc
#Tok.
Ace
#Tok.
Acc
#Tok.
Ace
#Tok.
Acc
#Tok.
*Tok.
Acc
#Tok.
Acc
#Tok.
DECS
31.25
5550
23.78
4965
75.37
2958
84,40
1817
56.10
3396
35.92
3255
27.66
6026
47.78
4000
wo CS
28.78
5676
23.28
5206
71.99
3125
83.77
1830
55,12
3477
34.44
3375
26.79
5972
46.31
4095
w/o DR
30.23
69-12
22.60
6850
73.50
3548
83.94
1917
55.85
4513
33.71
4444
27.05
7561
46,70
5111
B RELATED WORK
In this section, we introduce three categories of work to improve reasoning efficiency with training:
Methods based on length rewards As is introduced in the main text, length reward is one of the most influential strategies improve efficiency for large reasoning models (LRM). Su Cardie (202)) proposes to determine the reward scale coefficient defined in Eg. 4 by comparing the average accuracy of the current step with that of the reference model. Aggarwal & Welleck| (2025) proposes L1,, which optimizes the LRM to generate a correct reasoning path within a certain context window. Lyu et al. (2025) also proposes to use length penalties to encourage the model to answer problems with different complexity with adaptive token limits. Cheng et al. (2025) uses specialized model to separate the necessary reasoning part and the redundant tokens, and reinforce the policy to output the response with the shortest NRP length. However, although they have made great efforts to encourage correct yet short reasoning paths, they do not consider the internal logit dynamics when applying sequence-level length rewards with token-level optimization objectives, and thus suffer from performance degradation.
Methods based on Adaptive Reasoning Another line of work is to teach the policy to directly enclose the reasoning process and directly output the answer for easy problems, while conducting sufficient reasoning for difficult queries. AdaptThink (Zhang et al., (025b) proposes to generate a group of responses for a single prompt, half of which is generated with no thinking content, and thereby guides the policy for necessary thinking. Zhhng et al.] (2025d) combines the length reward and AdaprThink to teach the policy to conduct further thinking for already enclosed reasoning pro- cesses, to avoid the policy from outputting incorrect conclusions with insufficient reasoning. Zhang et al. (002c) constructs dataset containing both responses from thinking models and non-thinking models, respectively and proposes to fine-tune the policy to choose thinking modes according to the difficulty of problems. Although reducing the average output tokens successfully, these methods often erroneously adopt non-thinking modes for challenging problems, which leads to performance degradation. Meanwhile, they lack penalty for overlong responses generated in thinking mode, thereby still remaining overthinking.
Methods based on substep truncation As most overthinking contents could separated into thinking chunks with some high-entropy tokens, some methods propose to truncate the redundant chunks following the NRP to achieve efficiency optimization. MinD (Zeng et al., (025b) uses su- pervised fine-tuning to teach the policy to output its thinking contents with explicit separator tokens cold-start manner. After that, it uses GRPO to teach the policy to stop after generating the NRP part. S-GRPO (Dai et al., 2025) splits full reasoning trace into multiple segments, and manually prompts the policy to derive the final answer with a different number of chunks so that the policy could output trajectories containing only the NRP part. Yue et al. (2025) shares a similar philosophy with S-GRPO, but proposes to assign process-level rewards after each split segment to encourage the policy to stop generation on the token with max cumulative rewards. However, most of these methods either introduce additional rollouts, or are not end-to-end frameworks, which hinders their actual adaptation large-scale training or agentic applications (Zhang et al. 2025a).
C DETAILED ANALYSIS OF DECOUPLED REWARD DESIGN
While the decoupled reward formulation in Eq. 9 does not explicitly differentiate between lead- ing redundant tokens (i.c., the first token immedhately following the Necessary Reasoning Prefix, NRP) and other redundant tokens, the combination of this design with the group-relative advantage
17

---

Published as a conference paper at ICLR 2026
Table Concepts of involved mathematical notations, symbols and abbreviations.
Symbol
Description
The policy model parameterized by
A given input prompt question.
0
A reasoning trace (output sequence) generated by the policy *g-
O-j
The sub-seqeence of the output up to (but not including) the j-th token.
The j-th token the output sequence o.
T
The reward signal received from the verifier.
VJ(())
The gradient of the policy objective function with respect to parameters A.
A(ocj.J)
The advantage function at step j given the state Ozj. indicating how much better the action compared to average.
NRP
Necessary Reasoning Prefix, the minimal set of prefix tokens of reasoning trace required to reach a correct answer. Formal definition in Definition [ Proportion of the NRP tokens among a complete reasoning trace.
PNRP
The total length of the i-th rollout (total number of tokens generated). The proportion easy prompts among batch at iteration m
Fm
3
Coefficient for increasing the x value for each iteration. Maximum reward assigned to necessary reasoning tokens.
r+
Base reward value.
AES
Average Efficiency Score, a metric combining pass @ score and token cost. Concatenation operator for reasoning chunks.
The index of the first reasoning chunk that entails the correct answer.
KJ-
The cumulative token count up the end of the c"-th chunk in the i-th rollout.
mechanism in DECS ensures that all redundant tokens consistently receive negative advantages dur- ing training. This property arises from the interplay between the reward structure and the relative advantage computation:
1. Reward Structure: For any correct rollout o; tokens within the NRP receive fixed high reward r+ = 1.1. Tokens after the NRP (redundant tokens) receive length-scaled reward:
T (r+ - ro)
where I., lo,|-
Lynax
2. Group-Relative Advantage Mechanism: A token receives positive advantage only its reward exceeds the average reward that position across the group of G rollouts.
Consider the leading redundant token in sequence:
Case 1: The current sequence has the longest NRP in the group. Its reward computed by Eq, is generally beloweaverage the average reward among the token group. In this case, it receives negative advantages. This could be empirically verified across different training steps and model architectures. To be more specific, we evaluated rollouts from the DS-1.5B/DS-7B model at training steps [1, 60, 120, 180, 240] and counted how many leading redundant tokens have above-average reward computed by Eq. Bq. in Table Results demonstrate that across all batches and positions, no instance was found where a leading redundant token received a positive advantage.
Case At least one sequence in the group has an equally long or longer NRP. Then, there exists at least one token in that sequence receiving the full reward r 1.1. Meanwhile, under the condition where a longer sequence has longer NRP length, we could sort the 16 sequence lengths in descending order and assume that for sequence length Lş which is the h largest among the remaining 15 sequences. this situation, there would be at least & rewards that equal to 1.1 and the remaining rewards are all less than equal rk. Therefore,
16
1 (1.1 x
Fk - ---p=e 1.0 - 0.1 Lt/ Lmax
1. 0.1 x Lm/L))
16
18

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of erronously rewarded redundant tokens using Eq. on DS- 1.5DD.B
Table 6: Human evaluation results on the classification accuracy of math-specialized NRP detector on other domains, including sci- ence and coding.
7B.
Step
60
120
180
240
Dataset
Human1
Human2
Human3
DS-1.5B
0
0
0
N/A
DS-7B
0
0
GPQA-D
99.18 97.88
98.96
99.23
LiveCodeBench
97.63
97.49
Table 7: Timing consumption (seconds) for the NRP detector within a full training step.
Step
60
120
180
240
Total time
3791
3038
2972
2368
2583
NRP detect time
130
121
101
121
94
Ratio
0.034
0.040
0.034
0.051
0.036
where Vi, L, S L3 by above conditions. Simplifying the right term, we could obtain:
Th - / S 0.1
L.k Liax
0.1
(16 - k) Ln
Lemax
16
k
Lx
0.1 (1+
16
Limxx
< 0
This holds for any leading redundant tokens for any sequence that does have longest NRP length, which also represents negative advantage value.
In both cases, the reward for the leading redundant token strictly below the group average, result- ing in negative advantage. This guarantees it will be penalized by the policy gradient update. Thus, the decoupled reward design, in conjunction with the advantage estimator of GRPO, inherently pe- nalizes leading redundant tokens, thereby penalizing all redundancies by autoregressiveness.
D CORRELATION TO RELATIVE OVERGENERALIZATIOO IN MULTI-AGENT RL
In this section, we discuss the parallels between the "erroneous penalization" issue presented in this paper and concepts like relative overgeneralization in Multi-Agent RL (MARL). The core problem, the misalignment between global reward signals and local token-level updates, resonates with broader challenges in the RL literature.
The Parallel: Relative overgeneralization in MARL occurs when agents learn suboptimal joint behaviors due to misleading credit assignment from shared rewards. Analogously, in single- agent sequence setting, the coarse-graine sequence-level length penalty acts as blunt signal. This signal fails accurately attribute cost specific redundant tokens, leading to the erroneous suppression of useful, high-entropy tokens. This is essentially form of token-level overgen- eralization.
The Solution: This parallel highlights a fundamental challenge policy gradient methods: the need for fine-grained, temporally precise feedback to avoid spurious credit assignment. Our pro- posed decoupled reward mechanism (Eq. 9 can be viewed as an instance of structured credit assignment. By explicitly disentangling necessary and redundant reasoning steps, ensure that only truly redundant tokens are penalized, effectively mitigating this loken-level overgeneraliza- tion.
19

---

Published as a conference paper at ICLR 2026
Table 8: Hyperparameters for DECS training.
Hyperparameter
RI-Distill-Qwen-l..
RI-Distill-Qwen-7B
max response length
16384
8192
batch szie rollout batch size learning rate
128
128
128
128
2.0c-06
2.0c-06
total training epochs
2
3
rollout number
16
16
0.2
0.2
0.2
0.2
E LIMITATION & FUTURE WORK
Our approach effectively mitigates the cenioormaannefffcciinno trade-off in length-rewarded GRPO by dynamically separating necessary reasoning from redundant tokens. That said, two practical considerations remain.
First, the NRP detector is implemented as small auxiliary model (1.5B parameters). While this adds minor component to the pipeline, it incurs only 5.1 training overhead (Table and achieves near-perfect accuracy (Fig. 6c). making a lightweight and reliable proxy. Integrating NRP detec- tion directly into the policy, e.g., via confidence (Yan al. (025) entropy signals (Cui et al. 2025 is a promising future direction but not required for the current solution to work well.
Second, evaluate DECS models up to 7B due to resource constraints. However, since method is aggnos- and controls learning solely through the curriculum schedule with Eq. expect it to scale smoothly larger architectures with adequate compute.
Importantly, neither limitation affects the validity or effectiveness of our core contribution: simple, low-overhead strategy that achieves strong efficiency gains without sacrificing performance across both in-domain and out-of-domain tasks.
F USE OF LARGE LANGUAGE MODELS
We mainly use large language models for proofreading and polishing of this paper.
G EXPERIMENTAL DETAILS
In this section, we provide the details of each experiment conducted throughout this paper. We provide detailed descriptions of the training hyperparameters, the test sets and prompts used for evaluation, the metrics employed to evaluate each method, and detailed procedures for reproducing important experiments.
G.1 TRAINING HYPERPARAMETERS
We present the other hyperparameters adopted during training in Table Since we schedule prompts during batch in $4.3 and use over-sampling complement full batch, the number of total training steps is half-reduced. To align with a similar number of training updates with other baselines, we train the base model for epochs for the 1.5B model. Note that we train one more epoch for the 7B base model, as we set max response length to 8192 and hence many responses exceeding this limit would be filtered out. As result, to achieve a similar number of prompts participating in the training process, we extend the training process by allowing for training one more epoch.
G.2 DESCRIPTIONS OF TESTBEDS
We present the detailed description of the evaluation datasets as follows:
20

---

Published as a conference paper at ICLR 2026
1. AIME2024, AIME2025 (Mathematical Association of Americal 2025a by: These two datasets
contain High school Olympiad-leve assessment from American Invitational Mathematics Exam- ination in 2024 and 2025. Each dataset contains 30 challenging problems covering Algebra/Ge-
ometry/Number theory.
AMC23 (AI-MO 2024): This dataset is sourced from American Mathematics Competitions system in 2023, which contains 40 problems with hybrid question types.
3. OlympiadBench (He et al. 2024]: This dataset contains comprehensive math Olympiad prob- lems from various nations. We only select the English version related to Math and keep the problems that require an answer with a number, leaving 581 problems for evaluation in total.
x This dataset is an advanced mathematics evaluation set curated by OpenAl containing 500 problems with formal mathematical notations.
MATH500 (Lightman et al.
2023):
5. GPQA-Diamond (Rein et al. 2024): This dataset is a subset of the GPQA (Graduate-
Google- Prolee-rrooffoo Q&A) dataset, which contains 198 challenging multiple-choice questions authored and verified by domain experts in biology, physics, and chemistry.
LiveCudeBench (Jain et al., 2025): This dataset is designed to evaluate the live code generation capabilities of large language models, focusing on immediate correctness and practical coding skills. We use its v version, containing 1,055 problems in total.
G.3 EVALUATION PROMPTS
For AIME2024, AIME2025, AMC23, OlympiadBench, and MATH500, we prompt the LRM with "Please reason step by step and output the final answer within [boxed(]" and use Math- Verify to evaluate the correctness. For GPQA-Diamond, we prompt the LRM with "Please reason step by step and put the answer index after ANSWER: " For LiveCodeBench, we prompt the LRM with "You will be given question (problem speci fication and will generate a correct Python program that matches the specificatisn Cquestion/ln/nRee
and passes the inputs from stdin
all tests. cests./p/nouestion solve the problem and write the answer to stdout (do not directly test on the sample inputs). Enclose your code within delimiters as follows. Ensure that when the python program runs, it reads the inputs, runs the algorithm and writes output to STDOUT."
G.4 COMPUTATION OF METRICS
AES The AES score (Luo et al. 2025a is computed by comprehensively comparing the pass @ass@) score and average token costs of the tuned policy and the base policy.
AES-
Lbase
L
3
pass passel hase
pass I base
pass@1
pass
base
(16)
Lhase
passit base -passt01
passe base
pass@1
pass@
base
This metric incorporates both the ratio of tokens reduced and the impact on model performance: penalizes methods that degrade performance while rewarding those that improve upon the baseline.
Pass@ K The pass@K (Chen al. 2021 scores are computed as below:
pass @K
(17)
K
where n is the number of samples and is the number of correct samples. When K metric is reduced to the average accuracy among the samples.
1, this
G.5 DETAILS OF EXPERIMENTS OF FIGURE
To compute the optimal curve, we use the results obtained in Fig. 4a to serve as the points to be fitted. Specifically, when setting the maximum context length 0048, 4096, 8192, 16384, 32768),
21

---

Published as a conference paper at ICLR 2026
record the actual output tokens of DECS [2010, 2504, 4985, 5518, 5808], and corresponding pass@1 1 scores as [15.42, 25.00, 30.57, 31.25, 31.77]. After that, we use the well-established log-
linear scaling law function y logg + (Muennighoff et al. 2025 Zeng et al. 2025a Ballon
et al. 2025) to fit these data points, and obtain the fitter function as y = O. 1083 logg r- 1 1.0306 where
represents the average tokens and y represents the pass 1 score, with an R 0.9936 (Hastie et al., 2 After that, we plot the base model's performance (labeled as 'Base") " and LC-RI's performance (labeled as 'Previous Method') to show that previous length-penalty based methods fail to drive the policy towards the optimal trade-off between token efficiency and model expressiveness.
G.6 DETAILS OF TRAINING NRP DETECTOR
Training Details We build the training data using 1penRR-MMth-2200 with the large language model Qwen2.5-72B (Team 2025), which demonstrates strong mathematical reasoning capabil- ities and friendly deployment requirements. Specifically, we split each model response using predefined list of discourse markers, macludingWait But, Alternatively, Hmm, However Let. which prior work has shown to signal reasoning transitions or overthinking behaviors (Chen et al. 2024] Wang et al. 2025aa Sui et al., 2025). These markers naturally segment the reasoning trace into semantically coherent chunks. For each chunk, we prompt Qwen2.5-72B with the original prob- lem, ground-truit answer, and the chunk itself (using the prompt in Fig. 5 to judge whether the chunk semantically contains the correct answer. We filter out responses that violate the expected output format and collect valid annotations until reaching 5,000 unique problems. To assess an- notation quality, we manually verify 100 randomly sampled (chunk, judgment) pairs and find no clear misclassifications, indicating high reliability of the teacher model's labels. We then tnn-ttune a Qwen2.5-11. I.Struntt model on this dataset for 2 epochs via supervised learning, constraining its output to booxed (yes no} } for efficient online inference. The model is served via vLLM (K won cct al. 2023) using the same prompt during training. As shown in Fig. [c] the detector achieves high accuracy (>99%) on the development set and stabilizes quickly during training. Given its strong agreement with human judgment and consistent performance, we treat its predictions as a reliable proxy for the ground-truth NRP position in our downstream training pipeline.
Generalization Tests To test its generalization, we evaluated the NRP detector's predictions on model generations from two out-of-domain benchmarks: GPQA-Diamond (science) and Live- CodeBench (coding), under the PeepSeek-RI-Dstilllee pendently assessed the correctness of predictions from the NRP detector. The overall evaluation protocol is the same as illustrated above, where the NRP detector classifies whether a reasoning chunk contains the correct final answer and the human expert judges whether such classification is correct. The prediction is incorrect only if (1) a chunk containing a correct answer is classified as "Falsc" and (2) chunk without a correct answer is classified as "True". We compute the classifica- tion accuracy as the proportion of chunks correctly labeled across 100 correct responses from each dataset in Table [6] These results confirm that the math-trained NRP detector exhibits near- pear-ct reliability on science questions and very high (>97%) accuracy on coding tasks. This high classi- fication accuracy provides a solid foundation for the observed zero-shot transfer success of the full DECS framework.
model. Three expert annotators inde-
H ADDITIONAL EXPERIMENTS
H.1 DETERMINATION OF 8 IN EQ.
We conduct grid search on the AIME2024 dev set on both base models. We search the 3 in the following range: [0.0, 0.1, 0.2, 0.3, 0.5] and the results in Tableeiindicate that the value 0.2 achieves a great trade-off of efficiency gains and performance maintenance, where 0.0 represents that we only takes the decoupled-rewe reward and follows Yu et al. (2025) filter out extremely easy or hard prompts. Meanwhile, the optimal value of 3 obtained in the I.5 scale model transfers to the 7B model, which demonstrates that the value 0.2 is robust. We deem that as the policy's initial ratio of NRP
Ro is approximately 0.5, value of 0.2 guarantees that there will be at most (100- 50) 0.2 = 10
percent of casy prompts among a batch. This quantity ensures that the condition in Theorem [ is
22

---

Published as a conference paper at ICLR 2026
NRP Detector Prompt
Given a math problem and segment of long reasoning process solve the problem, your task is to identify whether this segment has presented a correct final answer. If the segment contains information that can serve as the final answer to the problem and the answer is semantically correct when referring to the ground truth, simply explain the reason and output \boxed[yes). Otherwise, directly output \boxed(no).
**Problem*x:
(problem)
**Raasoning segment**:
(segment)
**Ground Truth**:
(answer)
Figure 5: Prompt for the training and inference with the NRP detector
Table 9: Grid search result on AIME2024 for different ,3 values.
Model
B
0.00
0.10
0.20
0.30
0.50
Base
Pass @1
32.98
31.77
31.25
29.23
25.42
27.99
1.5B
#Tokens
7960
6876
5550
5497
5019
12202
Pass 57.29
54.16
51.33
50.00
43.96
50.65
7B
#Tokens
8277
7525
5277
5114
4905
10508
hardly satisfied during the training progress. Although a more fine-grained search value like 0.25 may bring a better trade-off, leave for future research.
H.2 TRAINING LOGS
In this section, demonstrate the training curves of DECS on the 1.5B model and 7B model on Fig. 6 and Fig. respectively. We select AIME2024 as a representative evaluation set, and plot the average reward and response length every 5 steps. Moreover, we also plot the average response length and the proportion of NRP (PNRP) during training to show that DECS achieves superior efficiency gains by reducing large amount of non-NRP tokens in the thinking process.
H.3 ABLATION WITH OTHER RL ALGORIHTM
Apart from GRPO, REINFORCE+ (Hu et al. 2025a) is also another strong algorithm for RLVR. Therefore, we also experiment DECS by taking REINFORCE++ (R++) to estimate the advantage value and use the same update formula Eq. 3 Specifically, Eq. but change the advantage estimator as below:
the same reward design as
A = Fi.j mean(r).g. "'. (T..)
Ar A mcan(A) (A)
(td
B G lo
(A) = BB G
4
10"!
1==1
==1
ΣB
D G
Σ |00
(A mean (
std(A) =
1=
j=
E Σ 10%
23

---

Published as a conference paper at ICLR 2026
Reward
PMRP
12000
5.20
0.995
0.32
33000
6000
0.990
0.32
7030
0.30
4000
0.7
8020
2.20
0.25
7000
1000
5036
9.05
0.995
0.9970
0.28
100
150
5000
120
250
5000
2.00
253
105
450
835
0.995
Step
SieD
(a)
(b)
(c)
Figure 6: (a) AIME2024 reward and response length during evaluation for training DeepSeek-RI. Distill-1.5B base model with DECS and (b) Proportion of NRP (PNRP) and response length during training for training DeepSeck-RI- --ieskRR--isstl-1.11 base model with DECS. (c) The training log and accuracy on the dev set of the trained NRP detector.
Performance an Seven Benchmarks
PNRP Score
Ressionse Longth
5000
DECS
8000
Lenght
Pazs81
Daskg
0.433
LNNRE
8.400
pisang
0.45
61.0
L000c2
0.359
5000
0.33
1500
60.5
4.0
5.0
6.0
7.0
8.0
5.0
Stop
Stop
Average Tokens (K)
(a)
(b)
(c)
Figure 7: (a) AIME2024 reward and response length during evaluation for training DeepSeek-RI- Distill-7B base model with DECS; (b) Proportion of NRP (PNRP) and response length during train- ing for training Bi--eckk----istlll.. base model with DECS; (c) DECS improves pass @ I of base models while reducing ~ 50% tokens compared to the 7B hase model across 7 benchmarks.
where AT is the advantage for the j-th token of i-th rollout generated based on n-thh prompt among batch size of B. Results Table [O]illstttt that there is no significant difference between GRPO and R++ which verifies the robustness of DECS.
H.4 COMPARED TO MORE EFFICIENT REASONING BASELINES
Apart from the four baselines presented in Table I this section, we compare more baselines that adopt the other approaches, different from length rewards, to improve the reasoning efficiency. We select S-GRPO (Dai et al. 2025). VSPO (Yue et al. (025). MinD (Zeng et al. 2025b), LASER (Liu al. 2025) and LAPO (Wu et al. 2025). We also include the over-sampling baseline that filters prompts whose rollouts are all correct or incorrect. Table IT demonstrates that DECS outperforms other baselines on the seven benchmarks at both efficiency and efficacy, which further validates the effectiveness of DECS.
Table 10: Ablation the other strong algorithm: REINFORCE++ with DECS on the DeepSeek- RI-Distill-I.5B base model. The REINFORCE+ variant achieves similar performance and effi- ciency improvements compared using GRPO, validating the generality of DECS.
AIME2024
AIME2202
AMC23
MATH500
Olympiad
GPQA-D
LCB
Model
Acc
#Tok.
Acc
#lok.
Acc
#Tok
Acc
#Tok.
Acc
#Tok.
Acc
#Tok.
Acc
#Tok.
Base
27.99
12202
22.94
12138
69.84
7875
84,55
4847
53.78
9217
32.86
8540
24.53
10560
DECS w/ GRPO
31.25
23.78
75.37
2988
84.40
1817
56.10
3396
35.92
27.66
DECS w/ R++
30.75
5595
24.40
4978
75.68
2912
84.50
1770
$6.23
3381
35.95
3333
27.85
6043
24

---

Published as a conference paper at ICLR 2026
Our
Der
1.0
20
Paxcs
10
8
Pass
Avg. Tokenss 180
15
1.000
0.6
0.5
0.4
0
16
32
04
128
Max Response Longth
Max Response Length
(a)
(b)
(c)
Figure 8: The Pass@1 I score and average token counts (a) AIME2025 and (b) AMC23 datasets
under diverse token limits with the SrepSeek-RI-Dsitill- base policy; (c) Models applying
DECS are par with the base policy (DS-7B) in terms of Pass @ K scores three challenging benchmarks.
Table 11: Comparison with more methods targeted efficient reasoning. DECS outperforms other baselines consistently across seven benchmarks.
AIME2024
AIME2025
AMC23
MATH500
OlympiadB
GPQA-D
LCB
Model
Acc
#Tok.
Ace
#Tok.
Acc
#Tok.
Ace
#Tok.
Ace
#Tok.
Acc
#Tok.
Acc
#Tok.
DS-L.5B
Base
27.99
12202
22.94
12138
69.84
7875
84.55
4847
53.78
92.17
32.86
8540
24.53
10560
MinD
27.14
6172
21.46
3573
4690
25.95
7217
VSRM-R++
29.38
6954
22.58
6671
72.41
3633
84.10
2241
54.77
4388
33.44
4413
26.58
7377
LASER-D
28.83
5966
22.20
5584
73.55
3058
84.20
1872
55.28
3676
34.29
3863
26.42
6493
DS-7B
Base
50.65
10508
36.67
11096
88.77
5764
93.25
3654
69.22
7507
46.46
7502
45.95
8966
MinD
49.57
9441
35.16
9997
87.22
4833
91.60
28599
68.17
6457
46.18
6528
45.63
8293
VSRM-R++
47.68
6773
32.56
6953
84.66
3704
89.80
2044
66. 13
5470
45.16
$764
44.89
7525
GRRO
50.93
7371
36.01
7908
88.20
3605
92.40
2252
69.74
4746
47.87
4938
47.34
7316
LASER-D
50.89
6423
35.61
6935
87.87
2949
92.20
1836
69. 74
3914
48.18 18.18
4205
47.71
6789
DOCS
51.33
5277
36.43
5516
89.04
2772
1728
70.28
3283
49.27
3276
48.05
8921
I CASE STUDY
We here present the comparison between DECS and three baselines, LC-R1, ThinkPrune and Adapt- Think, on MATH500, GPQA-Diamond, and LiveCodeBench, respectively. To conduct a compre-
Thévchuns
G8P0
Baxa
0.95
Our
Base
0.9
Average PNRP
0.3
0.8
Pass@k
0.85
0.6
0.80
0.5
0.3
6000
Average Tokens
8600
10000
0.0 Lewel
Lewel
Difficuity
Level
Level Lewels AME
0.2
K
16
64
128
(a)
(b)
(c)
Figure 9: (a) The comparison between PNRP score token consts in AIME2024 dataset for methods applied to the DS-7B model. (b) The PNRP scores for the six levels of difficulty math problems for the SeepSeck-RI-Dstilll Ours (DECS) and GRPO in DS-1.5B backbone.
base policy. (c) The Pass K comparison between Base,
25

---

Published as a conference paper at ICLR 2026
Our
8 88
Paases
Pass-83
15
Aug. Tokens (1)
Avg: Thkeene (K)
2010
8192
TRIR
1276n
2040
B192
1618C
2048
4036
8192
163M
Max
(a)
(b)
(c)
Figure 10: The Pass @ score and average token counts on (a) AIME2024, (b) AIME2025 and (c)
AMC23 datasets under diverse token limits with the Deepseek-RI--sitill-7 base policy;
Figure 11: Illustrative example for advantage assignment under vanilla length penalty (left), GRPO (middle) and DECS (right) given a simple question "What is 2+3?". When applying vanilla length penalty, the algorithm would penalize the whole sequence for longer sequences (the second se- quence), while rewarding redundant tokens for short sequences (the first response). However, DECS always penalizes unnecessary reasoning parts following the necessary reasoning prefix. matter how short the whole sequence is (the first and last response). Additionally, the vanilla length penalty would penalize high entropy tokens in longer sequences, e.g., 2nd and 4th response, which deterio- rates the model's exploration potentials throughout the training process.
hensive evaluation, we compare the outputs with LC-RI taking DeepSeek-RI--istilly base policy on MATH500,, compare the outputs with ThinkPrune taking DeepSeek-R 1-Distill-7B as the base policy on GPQA-Diamond, and comare the outputs with AdaprThink taking DeepSeek-R1- Distill-7B the base policy on LiveCodeBench. The comparisons are illustrated in Fig. 13 Fig. 14
as the
and Fig. [ respectively.
26

---

Published as a conference paper at ICLR 2026
Token Entropy
Temperature: 0.6 Cache Samples: 6 Analyzed tokens: 494
Mean H (bits): 0.805
Min: 0.000
Max: 5.031
Prompt
reasen step step put your final. within \boxed
Highlighted By Entropy (full text)
Okay, so need to figure out what 2 plus 3 is. Hmm, let me think about this. remember learning about addition schoo, but want make understand it properly. Addition is when you combine numbers to find the total, So, if have apples and someone gives me more apples, how many apples have altogether?
me visua ize this. Imagine have set objects, like blocks or maybe my toys. take blocks and then add more blocks them, how many blocks will have? count them out loud. Starting with 2, if add that makes 3, then 4, and then 5. Wait, that seems right because 2 plus should be 5. just make sure, maybe use my fingers something.
Alternatively, think of on 5, Yeah, that makes sense.
of number Starting
steps the right, land
So. yeah. plus 3 feel confident about that
Fiinn Answer**
The result of plus 3 is \boxed(5), </think>
To determine what 2 plus 3 is, we can use mu tiple methods to verify the result)
*Counting** Starting with 2, and adding 3, we count up to 5,
*Number Line**; Starting and moving steps the right lands
Real Objects** Having objects and adding more results in a total of objects.
Basic Addition Facts** equals
5. VVisual Methods Using fingers, tally marks, or money all confirm the result 5.
"Algebraic Representation Letting x be 2 and be 3, we have x + 5.
* Number System Consistency * In binary, 2 10 and 11, Adding them gives 101, which in
decimal.
All methods confirm that 2 plus 3 equals
### Final Answer
The result of plus \boxed(S),
Illustrative example showing high-entropy forking tokens. The distribution is similar to Wang et al.] (2025b), where uncertainty-baser tokens including "Wait", "but". and "maybe" have much larger entropy values than deterministic tokens like "5" which is the final answer.
Figure 12:
27

---

Published as a conference paper at ICLR 2026
Question
Define = 2k=1 k2
and q = Σk=1 k3*
Find way to write -j=1 kk=1 (f+R)a
Ours Longest: 1724 Tokens
Ground Truth Answer: p-q
NRP: 1191 Tokens
Okay,, so have this problem where nood to find an expression for the double sum Visum (j 1) ssum (k 13minn (frnc IÚ kyr3/9 terms of Kpl) and (ig), where Vpp 'summ [k - 1 1liinfty Vfrac (2-2))) and Vi 'sum (k 1/ (frnc(]) (k-3/)) Hmm, okny, let's break down step by step First,, know that (p') known Basel problem, which sums to (ffrac ('pi'2 (6])) but don't know if that's relevant here. Maybe not directly, but it's good to remember. The other sum, (qy), is the Riemann zeta function at 3, so it's (ireta( 310, but again, not sure if that's useful yet main goal is to express the double sum Csum (j-1y "infly 'sum [k-1) Minnly 'faa [1) [G ky3))) in terns (pl) and (qt). need find clever way to relate this double sum the single sums ((p) /m remember that sometimes, for sums like this, changing variables or using symmetry
well-
help. Let me
visualize the double sum.
It's over all positive integers ((')) and (K1), and the term is Kfrrae (1] fG k))))). So, for each puir X6. kJ), we're adding up the reciprocal of the cube of their sum. innl wonder if can express this double sum as single sum by changing the order summation by some substitution. Maybe if let in ] ki, then for each (nn))
over all pairs 10. kjj such that let me try that. Let me fix (nn) and consider all pairs V. kl) where Vjj k n').
'G + k ni). That scoms promising./nine For each ((n)), (i) can range from to i(n 19, with 1(K n 11. Therefore, the double sum can he rewritten astaini/ (n=2)"innfyy Isum (j-1)/nn Vffcc [1] (n/3) h Xi 13--0 ni) implics VIk both 160 and (Kl
JÚ, and since 2n'noo simplifying the inner sum, it's just Qouu [j-1) Mrrac (11))) Since Virin [1] (n"3)) is constant with respect to 160, the inner sum is just Xfrrac in 13 (00 miithherrr the entire double sum becumestatat /frnc/n (1n3/mmjnin mnj'm can split this vnstiien/n/miinnnnnnnn
least ((n) starts from 2 (since the smallest (0 and (K3) be is I, 1I1
(n=2] (n-2) leeff Virec (n'3)- Mfrac (m"3] 'right) "sum mn-flii Veft (1) (122) [1] (n'3) tiihhtyni/ww/nee
1 can split this into two separate sumszlarlnl ln [irssmmmmmmm (n-2)"innlly (dree[]] (n/2) - bum [s (n=2l"innly Mrac [1] (n33/n/jmmn know that the original definitions of (pl) and (44) start from Vn 13), so can write these sums in terms of (pl) and (iji) by subtracting the Kn-11) terms. SnnSSo Vsuu (n=2)) (ine (u"2)
'ftu (1) (1-2) p 19, similarly. ((sum Minfly 'dracc (1) (n/3) =q Vinee (1)(3)3) =-- 19) ThThercfore, substituting back into expression:aaa
1)- (q 1) p-- jjai)'nlnSo. the double sum simplifies to Vpp q'nin'al think that makes sense. So, the key steps were
1 1- q 1 changing variables to in =j j+ ki) then rewriting the double sum as a single sum over Xnn), evaluating the inner sum, then expressing the result and then using the definitions of Vpl) and Vqy) to relate it in'k don't see any mistakes in the steps, so I'm pretty confident the answer is Xp q()/n'a" Final Answer*" Thee double sum can be writlen as Viboxed (p- 9/9).
of (pl) and (ig).. The important part was recognizing that inner sum simplifies nicely
Remaining: None
NRP Ratio: 100%
LC-R1 Longest: 4287 Tokens
NRP: 3283 Tokens
Okay, so need to find way to express the double sum Vsuu 1j 1) 1pliffty ssm [k Minffyy (frec (G ky33/) in terms of (pp) and (4t), where Vp 'sum_ (k 1ininn (frac[]] (K"2)) Vyg 'smm
13nlin [rac[]] (K-3))) Hmm, let me think about how approach iniFrrs, know that Vip)) is the well-known Rasel problem, which Kfrre ('pt2)) (6))) but I don't think I need the exact value here, Simmlarly. Yqy) is the sum of reciprocals of cubes, which converges to Apery's 'ss constant, approximately 1.202... but again, exact value isn't necessary here, hhn'Th double sum given is over all positive integers KJi and VKk). and the term inside is (ffrac [1) (j ky33)) So, double series where each term depends which are straightforward WiyMaybe can change variables or find a way to express this double sum in terms of Kp') and 'qq'. Let me consider switching the order summation or using some symmetry Another iden: perhaps express V 3)) integral. Since ((L/n33)) can be written as integral, maybe
the sum of two variables. This seems bit tricky because it's two-dimensional sum, unlike single sums
use that to interehange the and the integmal.nnn Recall that for positive integers ((n),, we huve: [1] (n'3) :ffac|] [2) lint 0*1 lint 001 (xy) I] dx dy'n Wait, is that correct? Let me check. Actually fomitted... So, that gives us i(p qi).
Remaining: 492 Tokens
Hmm, that seems too stnightforward tonniniiimlllff 'summ [mm11 1 jinnty (frcc!) [mm2]- bright )- Veftí Isum [m-1] minni 'frac [m*3] 'right )- (p- 1)- (g- e--1 1) p- qimijuinSo,, that correct? Hmm. Let me verify fonffy/fomttee...1 which indeed Vpp Theerfff,,,,,, the double *
but is that correct? [omitted.... WhWhic is equal
sum is equal to kp 4nn//n the answer is ([boxed (p- q]').'ma"" "Final Answer ui boxed (p qq'n
NRP Ratio: 87%
Figure 13: Case study of the comparison of DECS and LC-R1 in MATH500.
28

---

Published as a conference paper at ICLR 2026
Question
A spin-half partiele is in a linear superposition 0.5[1) + v3/21 1)of its spin-up and spin-down states. If [)and | !)are the eigenstates of z then what is the expectation value up to one decimal place, of the operator 100, + 5 Here, symbols have their usual meaningsininAA) -1.4knB) -0.7knC) 1.65inD) 0.85
Ground Truth Answer:
Ours Longest: 2766 Tokens
NRP: 2294 Tokens
Okay, have this quantum mechanies problem here, and need to find the expectation value of the operator 10az Sox for a given spin-% particle state. Hmm. let me iry to break this down step by step. insi' the state of the particle is given us linear combination of the spin-up up and spin-down states: 0.5 1) (x33221)). remember that (t) and [I] are eigenstates of Pauli
az acting If) gives times I?), and acting on II) gives -1 times [l), right/ainnNow operator we're dealing with 10az Sox. need find the expectation value of this operator, which is (wl (10oz Sax) (v). Since expectation value is lincar, can separate this into (0(yloz)y) S(ufox))). That should make things casier Lnn me handlo each term soparately. Starting with the first term, 10(w know that oz diagonal in the oz basis, which is the the and basis. So, the expectation value of just the weighted avernge of the eigenvalues, with the weights given by the square of the amplitudes in the state (V). Ihe state (v) is 0.5(1) - (V3322L)) So, the coefficient for (1) is 0.5, and for (1) is V3/2. The square of these gives the probahilities. Therefore, the expectation value of az (0.5)-2((+1) (1312 Calculating that:
(3/4)(-1) 0.25 0.75 -n' times that would be 10%(-0.5) 5 S/ninOkay, that's the first part. Now, the second term is 5(wlox/y)) Hmm, the Pauli X matris, which is not diagonal in the oz basis remember that ax has eigenstates (+x) and HX), which are superpositions of [1) and [t). Mayhe I need to express (4) in the ax basis to compute this expeciation value, or perhups there's another way hay.lmmnlllemmiivelyyy recall that write (v) a column vector, then ax acting (v) be computed, and then the inner product (v oxlu) can be found i'nne''s represent (p) vector. Since (1) is [1: and [0; 1]. then (v) is [15*[I: 0] + (Y3)2*(0; 1] [0.5; 13/2). So, the vector is (0.5, (n ox is [[0.1][[1.0
can written matrix form as [[O.1]][I.O]]
So, oxly)
is [0.5*1 + (v3/2)"0, 0.5*0 ('3/ which is [0.5, V3/2]"T Wait. that's the same as (y)? Hmm, so oxly) lyy? [omitted] Since and real this (the coefficients are real), this becomes 2ab In'nSo for state, 0.5, 13/2. Therefore, (ox) (332)) (v3y2 0,866 T'Thrrefore, 5*(xx) = 5*(333 = 5*0.66 = 4,33.
4.33. n'nSo, adding the two terms together: -5 4.33 -0.67. So, approximately -0.7 when rounded to one decimal place/ninAM okay, so that makes sense now. My initial mistake was probably in the way computed axlv) and then the inner product. But using formula for expectation value terms of coefficients, it's much simpler. So, the expectation value of is 2ab, which (0055))())322) = v3/2 = 0.866. Therefore, 5 times that is approximately 4.33, and 10 rimes the expectation value of az is -5. So, adding them gives spproximately 0.67, which rounds to -077, So, the answer should be oplion B) -0.7.
Remaining: None
NRP Ratio: 100%
ThinkPrune Longest: 8098 Tokens
NRP: 1535 Tokens
Okay, so have this quantum mechanics problem here about a spin-half partiele. Hmm, let's soo. The partiele is in lincar superposition given by 0.5(f) + (VRy21L),
need to find the expectation value of the opentor 100, + 5a x. answer choices A) -1.4, B) -0.7, 1.65, 0.85, Alriiggh first, lot me recall what the expectation value of operator is. For quantum state (v). the expectation value (O) is given by (v O (v). So, I need to compute (vl (10n z + Sa x) (v). minninc expectation value is linear, can sepanae this into 10/y/r 2/1) S(yla xiy). So, I need to find the expectation values a a_x separately and
combine themn in me write down the given state (v). is 0.5(t) (V3)2 1). Let me note that (f) and (1) the eigenstates a_z, which the Pauli z-matrix So, f_x +17) and >1)) -1) Fiinstt compute (ar 4). That is. (ulo_zv)) zz)) win.nii 2_z diagonal in the =_ busis, which is the same as the basis in which (t) and II) are defined, this should be straightforward. [ommiteed...I..m Loookigg at the options, B is -0.7, which would be the unswer n't wait, let verify the calculations again because sometimes might have made mistake. inniiisst (a 2): probabilitiss 0.25 and 0.75, 0.25 0.75 - -0.5. That seems right. 'in Then, for (a_x): the expectation value/ninAnn minnnothe way compute (a x) to note that 6_x |-X--|- |-X-] but maybe that's more complicated. iltemmtttivvllly for a general state (v) ait) bll). (a x) ann2 [b/22 Wait, no, that can't Waii, a_x is [[O.I].[1.0]] (1.) so when you compute (vio xlv)., it's terms. Wait,ait let's compute again step by step/ninjyr
[el xif) + [b/2"(1]a x(I)
[0.5; (v3)2). So, in /v) (33322 ; 0.5 Then, (vi (V) [0.5, (13)2) multiplied by (x3y2; ; 0.5 SoinSo, 0,5*(v3)-2 + (v3y2 *0.5 = (33)4 (v3y4 v3 2 = 0. n0, that seoms correct inin TheTherefrre, 5(a x) 5*(V3y2 as 4,3301, 10(a_z) x) -5, total expectation
matrix form, it's column voctor (0.5; (x3y2)).n'ne
4,3301 .0.6699, which is approximately- 0.67, so -0.7 when rounded to d ThhTeerrerr,,, the answer should he -0.7.
Remaining: 5990 Tokens
Wain but let me think again. Is there another way compute this? Maybe using the properties of states. [omitted...] So. in conclusion, (o_x) is approximately 0.866, which is va 2/wntbereeore 5(a x) is 5*313 2) =: 5 5*0.866 = 4.33mn'nThen, 10(a z) -5, as computed earlier ThhThrrffre, total expectation value is 4.33 = -0.67, which is approximately -0.7 when rounded to one decimal place Therhereo answer is B) -0.7.
NRP Ratio: 20%
Figure 14: Case study of the comparison of DECS and ThinkPrune in GPQA-Diamond.
29

---

Published as a conference paper at ICLR 2026
Question
Slavic preparing present for a friend's birthday. He has an array a of n digits and the present will be the product of all these digits. Because Slavic is good kid who wants to make the biggest product possible, he wants to add to exactly one of his digits. Wh'n is the maximum product Slavic can make? wnw.iuttnnnn line contains a single integer ((1 S t S 10*) the number of test cases. in'n The first line of each test case contains a single integer n(1 S 9)- the number of digits. The second line of each test case contains space-separated integers d; (0 S a, s 9) the digits in the array. array..n'nOOtpurtes Ont.nnnnpuunnnnnnnoooo For each test case, output a single integer- the maximum product Slavic can make, by adding 1 to exactly one of his Sigipe Input 1::mm'mmmm44m'm22 2in'n33mn'n0) 2'n'm5unnnn432 3 4'n''n9'n'n 9 9 9in'n'n'nnSample Output
first
Ours Longest: 2932 Tokens
NRP: 2471 Tokens
Okay, I need to solve this problem where Slavic can add exactly one digit in amay digits maximize the product. Hmmm, let's think about how to approach this, l''aSo the problem is: given amay of digits (each between and 9). 1 can choose one digit to add to. Them, compute the product of the digits. need find the maximum possible product after this operation. in'aFisst, need
2n'nSo, the ama [2.2.12]. For each nn'nLe'ss see''n'nOption 1: add first - [3.2.1.2]. Product is 3*2*1"2=12 innnnption 2: add to sccend 2 - same as option 1. product 12.in'nOption 3: add to [2.2,2.2] Product is 2-4=16. wntptttonot add to last - [2.2,1.3]. Product 2"2*1 331222mn maximum [1,1,2] product 1*1*2=2, innnAdd to 2 [0.1.3] 331*3=0 maximum is 2. Which matches sample Anotherr the 2: makes it 3, so the array hecomes [4,3,3,3, Product is 443*303"4 m1334444433 4*3"3 4927 432 452.hhhWhhchh matches sample Annoohhrr sample: 9 Adding
position, adding I and compute -Snitedd..JSammpl Input Laaaa
Which matches the sample
output Anothhe samiie:innnainnnn 2annnossible opddons/uinndd
[0.2.2] - 0*2*2=0
sample: 5 digits
the best option?ulnAddinng
them gives others So product is 10 * Cnnoouut 9*8: 9*9-81, 81*9-729, 729-9-6561, 6561 -9-59049, Multiply by 10: 430467210. Which matches the cample//nniS the code should handle all cases correclly inmellly'niaNow, in Python, the code can be writlen as follows./ t'm'nt Intrinput())allafor
59049*9-531441, 53144199-478200
178296999-4304677
range(t)án' int(iaput())ain a list(map(int, input().pliic)))aa max max_prod prod O'a'n for i in range(u))laia new a:lm'n print(max prodja'nYes, that's the code.
a a.copyO'ain new_a(ij +- Tiann product Timin for num max max__rree prod productin'n
product numinin if product max proddproddnnn
Remaining: None
NRP Ratio: 100%
AdaptThink Longest: 3797 Tokens
NRP: 1349 Takens
Okay, I've got this prublem sulve. Let's read carefully. The problem is about Slavic who wants maximize the product of digits in an arry after adding to exactly one of the digits, So, for each test case, I need to find which digit to increment by such that the product becomes us large as possible inlimm, let's think about the approach. For each digit in consider increusing by and then calculate the product. Then, aray,, Wait, but the amay could have zero. Oh right. because if digit 9, adding makes it 10. But if digit is O, possibilities hnaibiiiiiess/aaaaiii
the maximum among
these
adding makes it which might be better in some cases. iinLet me outline the cups:'n'n.. For each test case'n Read the amay of digirs/nn adding would make 10. since we're treating each digit single digit, mayhe I should represent them integers. for each clement j, Fll compute a I, but only if a i 9, becomes 10, [omitted...] -'n'no the code structure would be something like:'ninfor in range(t)/mn n int(input())n range(n):ln
each digic in the array, create copy where that digit is increased by
wait, the digit
- list(map(int, input()-pliitO))n mav mav_prood Oln for new afi]
new a.copy()'n
current prod I'm (eurrent_proot prod _aaua_pod:ln max_pred current parreent_prod_a print(mux__ _pod)
num new
current prod *- numin
Remaining: 2026 Tokens
Wait, but wait. What if the original arey has a zero, and adding I to it? (emitted//w'nNoos for the code'ninRead timinFor each test caseriain Read nlnin Read the list as integers.lu'n Initialize max prod to zero.ln'n For each in O..n- I:inin a a.copy(kmin current max prod. update.in'n print max pnnd.'n'Buut wait, in the case where all digits are exeept one. Iono the codeinint int(inpat()infor in range(t);in n int(input()in a list(map(int, input(..plit())) max prll-prrouuuttte 0tn product * numin
+=iinnn compute product new a.innn
product larger than
range(n))'n new a a.copyy0in product:in
new alll += I'm product
product productin print(max productjiniWaiit but wait. What if the ammy is [0, 0, 0], the original product 0, Adding [omitted]'ninSo the code should be correct'ninThus. the code is as written above.ln
I'm for num in
aćin
product
NRP Ratio: 40%
Figure 15: Case study of the comparison of DECS and AdaptThink in LiveCodeBench.
30


Reviews:
[{"review_id": "aKHyrnqHgI", "reviewer": "Reviewer_XB3N", "summary": "This paper targets the “overthinking” phenomenon on LRMs. Based on the fundamental misalignment between trajectory-level rewards and token-level optimization. The authors propose a DeCS framework based on theoretical discoveries that are erroneous penalization of essential exploratory tokens and  inadvertently rewarding partial redundancy. The DeCS framework includes (1) decoupled token-level reward mechanism and (2) curriculum batch scheduling strategy. Through comprehensive experiments the authors demonstrate effectiveness on their proposed framework", "questions": "**\\[Q1\\]** In the RQ2 experiment, I understand that the exploration potential maintains similar with the base model. I'm curious about how exploration compares between DeCS  and GRPO-trained models. What differences or similarities exist in the exploration behavior between DeCS and models trained with GRPO?\n\n**\\[Q2\\]** Table 1 indicates that the proposed DeCS framework demonstrates lower average performance metrics compared to GRPO-trained models. Furthermore, the manuscript predominantly focuses on comparisons with baseline methods and analyses of individual DeCS components, while direct comparative analysis with GRPO appears somewhat limited. Given these observations, are there any theoretical reasons why DeCS shows a lower performance compared to GRPO?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "**\\[S1\\] Solid Theoretical Foundation**  \nThe proposed framework is developed through strong theoretical foundation that enhance its novelty and provide convincing justification for its effectiveness. Also the paper's clear definition of concepts like NRP will likely provide significant value to the field by establishing precise terminology\n\n**\\[S2\\] Comprehensive Experimentation and Analysis**  \nThe paper presents a satisfying experimental structure organized around research questions, including thorough comparisons with baseline methods and detailed ablation studies. This approach effectively addresses potential questions about the proposed framework while convincingly demonstrating its effectiveness.", "weaknesses": "**\\[W1\\] Readability Issues**  \nWhile the logical flow of the paper is solid, it was difficult to understand due to the extensive use of abbreviations and mathematical notations. Additionally, the token length comparison in Table 1 would have been more effectively conveyed through visualization. \n\n**\\[W2\\] Limited Model Generalization**  \nThe experiments are confined exclusively to the DeepSeek model family. The paper would be strengthened by extending experimentation to model families with different exploration capabilities to demonstrate broader generalization of the proposed framework.\n\n**\\[W3\\] Domain Dependency of the NRP Classification Model.**   \nWhile the NRP classifier trained on OpenR1-Math-220K demonstrates strong reliability through human validation within the mathematical domain, the paper lacks analysis of its performance in non-mathematical domains (e.g., coding, science, etc.). It will be great for the authors to additionally verify whether the NRP classification remains robust and consistent across these domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "badFFPtmxH", "reviewer": "Reviewer_fsTY", "summary": "This paper introduces DECS, a novel framework designed to mitigate \"overthinking\", the generation of excessively long reasoning paths, in large reasoning models (LRMs) trained with reinforcement learning. The authors first provide a theoretical analysis identifying two key flaws in current length-penalty methods: the erroneous penalization of essential exploratory tokens and the inadvertent rewarding of redundant tokens. To address these, DECS incorporates three main components: (1) a lightweight judge model to detect the Necessary Reasoning Prefix (NRP), (2) a decoupled token-level reward that specifically penalizes tokens generated after the NRP, and (3) a scheduling strategy to balance efficiency and performance during training. The authors conduct extensive experiments, demonstrating that DECS can reduce the number of reasoning tokens while maintaining or even improving performance.", "questions": "1. I am intrigued by the parallels between the \"erroneous penalization\" issue and concepts like relative overgeneralization in MARL; a brief comment on this could enrich the paper's context within the broader RL literature.\n2. Could the authors elaborate on the reward design shown in Fig.2 and Eq.9?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper provides a clear theoretical analysis of why existing length-reward mechanisms are suboptimal.\n2. The DECS offers a carefully designed approach grounded in theoretical reasoning.\n3. The authors evaluate their method across diverse benchmarks, use two different model scales, and compare against multiple relevant and strong baselines.\n4. DECS achieves a reduction in token count while simultaneously improving performance.", "weaknesses": "1. The framework's reliance on a separate, fine-tuned NRP detector adds complexity to the training pipeline. A deeper analysis of its robustness would strengthen the paper.\n2. The decoupled reward is used to differentiate between \"Leading Redundant tokens\" and other redundant tokens, but this does not appear to be reflected in Eq.9.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nr4M4vlgK8", "reviewer": "Reviewer_YPs8", "summary": "The paper addresses the \"overthinking\" problem in large reasoning models trained with reinforcement learning, where models generate excessively long reasoning paths without performance benefits. The authors propose DECS (Decoupled rewards and Curriculum data Scheduling) to reduce the reasoning length while not harming the performance.", "questions": "See the weaknesses.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The studied problem is important;\n- The writing is easy to follow;", "weaknesses": "- The proposed reward and curriculum designs are quite artificial. In particular, the decoupled reward depends on pre-training an effective NRP detection model, which introduces practical challenges given that the definition of NRP is ambiguous, let alone the difficulty of reliably detecting it with another model.\n- The method is not straightforward to implement, as it requires pre-training a separate detection model whose accuracy cannot be guaranteed or easily verified.\n- The curriculum design component lacks novelty and, as such, does not constitute a substantial contribution to the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sV4IhrkWTG", "reviewer": "Reviewer_tgZx", "summary": "Handling RLVR model’s overthinking behavior by separating tokens according to their role and penalizing redundant tokens, and scheduling batch with curriculum.  \n–\nPrior approaches incorporating length penalties (e.g., Eq. 4) broadcast trajectory-level rewards to all tokens via GRPO's advantage estimation (Eq. 2). When all responses in a group are correct but vary in length, shorter sequences receive positive advantages while longer ones receive negative advantages uniformly across every token. Consequently, high-entropy tokens like \"Wait\" or \"However\"—which initiate productive exploration—are suppressed simply because they appear in longer trajectories, even when they are part of the necessary reasoning process (Lemma 2, Theorem 1). DECS addresses this by decoupling rewards at the token level: it assigns maximum reward (r₊) to all NRP tokens including high-entropy exploratory tokens, while inversely penalizing only the redundant tokens generated after the NRP (Eq. 9). This surgical approach maintains exploratory capacity (preserving Pass@K performance, Fig. 3c) while achieving superior compression (>50% token reduction, Table 1).", "questions": "* Q1: The paper presents strong technical contributions, but accessibility could be improved with concrete worked examples (e.g., token-by-token walkthrough of the \"2+3\" problem showing advantages under baseline vs DECS). Given the paper is already dense, where do you envision such pedagogical content fitting best?\n* Q2: Table 1 shows 7B has less overthinking than 1.5B initially. Does this trend continue? Would larger models need DECS?  (Scalability to 30B+ models unverified)", "limitations": "", "rating": 10, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The idea is backed by theoretical claims, and they are logically correct. The two flaws revealed by Lemma 2 and Theorem 2 (suppression of high-entropy tokens + failure to penalize redundant tokens) are addressed by DECS through curriculum scheduling (Theorem 1) and decoupled rewards.\n- In addition, the paper make each claim irrefutable through quantitative verification at every level - from theory to core analysis (Fig 1, 3), then validate broadly with 7 benchmarks x 2 models x 5+ baselines, with robustness checks (Tables 5-7)\n- The paper organized their contents very nicely, well distributed content between main text and appendix. \n- Good generalization to out-of-domain tasks (LiveCodeBench, GPQA-D)", "weaknesses": "Limited accessibility for non-expert readers: While the paper presents rigorous theoretical analysis and strong empirical results, the presentation could be more accessible. Figures 1-2, which are central to understanding the core contributions, are dense with notation that is not immediately intuitive without careful reading of Section 3. For a paper introducing novel concepts like \"Necessary Reasoning Prefix\" and \"decoupled token-level rewards,\" more pedagogical support would strengthen impact.\n\nSuggestions for improvement:\n- Add a concrete, annotated example in Appendix showing token-by-token reward assignment for a simple problem (e.g., \"What is 2+3?\"). Compare how (a) base GRPO, (b) length penalty, and (c) DECS assign advantages to specific tokens including high-entropy ones.\n- Figure 2, while comprehensive, relies heavily on abstract labels (\"NRP\", \"Redundancy\", \"Correct Thinking Process\") that may not be \nimmediately intuitive to readers unfamiliar with the concepts. Consider replacing these placeholders with concrete text snippets from an actual model response (e.g., showing \"Wait, let me verify... 2+3=5\" as NRP vs \"Let me triple-check... yes, 5\" as redundancy) within the same figure space. This would make the core contribution—that DECS protects exploratory tokens while pruning redundancy—immediately graspable without requiring back-reference to Section 3.\n- Include an intuitive explanation of why high-entropy tokens matter, possibly with a probability distribution visualization showing how entropy differs between \"5\" (low H) vs \"Wait\" (high H).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)