S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models
Authors: Mz Dai
Chenxu Yang
Qingyi Si
Venue: NeurIPS 2025 poster ()
Keywords: Large language model
Reasoning model
Reinforcement learning
Post-training
Primary area: deep_learning

Abstract: As Test-Time Scaling emerges as an active research focus in the large language model community, advanced post-training methods increasingly emphasize extending chain-of-thought (CoT) generation length, thereby enhancing reasoning capabilities to approach Deepseek R1-like reasoning models. 
However, recent studies reveal that reasoning models (even Qwen3) consistently exhibit excessive thought redundancy in CoT generation. This overthinking issue arises from the inherent limitations of conventional outcome-reward reinforcement learning, which systematically overlooks the regulation of intermediate reasoning processes. This paper introduces Serial-Group Decaying-Reward Policy Optimization (S-GRPO), a novel reinforcement learning paradigm that enables models to implicitly evaluate the sufficiency of intermediate reasoning steps, thereby facilitating early exit in CoT generation.
Unlike GRPO, which samples multiple possible reasoning paths in parallel (parallel group), S-GRPO only samples one reasoning path and serially selects multiple temporal positions from the path to exit thinking and directly generate answers (serial group). For correct answers within a serial group, rewards gradually decrease based on the exit positions along the reasoning path from front to back. This design encourages the model to produce more accurate and concise thoughts, while also incentivizing early thinking termination when appropriate. Empirical evaluations demonstrate that S-GRPO is compatible with state-of-the-art reasoning models, including Qwen3 and Deepseek-distill. Across diverse benchmarks such as GSM8K, AIME 2024, AMC 2023, MATH-500, and GPQA Diamond, S-GRPO achieves a substantial reduction in sequence length (40.4%～61.1%) while simultaneously improving accuracy (absolute 0.72%～3.92%).

S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models
Muzhi Dail: Chenxu Yang", Qingyi Si
'Huawei Technologies Co., Ltd. Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
mzda 6668gmmi1 il com, yangchenxu@ ie ac. en, siqingyi@huawer com
GRPO
S-GRPO
Parallel group
0/1 reward
Serial group
Decaying reward
Thhinic
1
2
T
aothinics
othinic
p=1
ohhinkc
thinnk
ethinkk
Baanoh
Branch,
Thini
Chiinkk
complete reasoning paths
complete reasoning path early-exit
Figure 1: Comparison of parallel-group-relatiy GRPO and our serial-group-relative S-GRPO.
Abstract
As Test-Time Scaling emerges active research focus in the large language model community, advanced post-training methods increasingly emphasize ex- tending tfaii--ff-thougee (CoT) generation length, thereby enhancing reasoning capabilities to approach Deepseek RI-like reasoning models. However, recent studies reveal that reasoning models (even Qwen3) consistently exhibit excessive thought redundancy CoT generation. This overthinking issue arises from the inherent limitations of conventional silcomeerewartt reinforcement learning, which systematically overlooks the regulation of intermediate reasoning processes. This paper introduces Serial-Group Decaying- Reward Policy Optimization (S-GRPO), a novel reinforcement learning paradigm that enables models to implicitly evaluate the sufficiency of intermediate reasoning steps, thereby facilitating early exit in CoT generation. Unlike GRPO, which samples multiple possible reasoning paths in par- allel (parallel group), S-GRPO only samples one reasoning path and serially selects multiple temporal positions from the path to exit thinking and directly generate answers (serial group). For correct answers within serial group, rewards gradually decrease based on the exit positions along the reasoning path from front to back. This design encourages the model produce more accurate and concise thoughts, while also incentivizing early thinking termination when appropriate. Empirical evaluations demonstrate that S-GRPO compatible with state-of-the-art reason- ing models, including Qwen3 and Deepseek-disse Across diverse benchmarks such as GSM8K, AIME 2024, AMC 2023, MATH-500, and GPQA Diamond, S- GRPO achieves a substantial reduction in sequence length 400.%%-6..11% while simultaneously improving accuracy (absolute 0.72%~3..92%%
Equal contribution. The work was done when Muzhi and Chenxu were interns at Huawei. Corresponding Author.
t
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Test-Time Scaling T demonstrates strong correlation between extended chain-of-thougle (CoT) and enhanced reasoning capabilities in Large Language Models, which is more effective than scaling model parameters
( and GPT-ol 4 has further sparked the 2 The success of DeepSeek-R2
research interest in reasoning models [ within the LLM community. Furthermore, reinforcement (RL) in post-training has demonstrated significant potential stimulating long chain-of- tfhooughht generation and strengthening deep-thinking capabilities.
learning 66
10
However, recent studies [M] [2] have identified a critical limitation in reasoning models: their tendency to engage in redundant thought processes, a phenomenon referred to as Overthinking [I] 13. They frequently generate unnecessarily lengthy reasoning sequences 13 14 including irrelevant information and superfluous reasoning steps. This redundant thought inflates computational overhead and even undermines reasoning accuracy by diverting the model from valid reasoning pathways to incorrect ones [[]] We attribute this issue to the inherent limitations of 0/1 outcome-rewant RL (e.g.. GRPO ()) where reliance on final outcome rewards fails identify when intermediate reasoning steps are sufficient.
| (left), standard outcome-reward RL, such GRPO, tends to sample query multiple times in parallel to obtain a parallel group. All reasoning chains and corresponding conclusions within the parallel group receive 0/1 outcome rewards, which reinforces correct outcomes but overlooks the presence of overthinking or inefficiencies in intermediate reasoning steps. While this approach successfully aggregates pass@k reasoning capability into pass@ 1, the neglect of regulating intermediate reasoning results in inefficient inference. Conversely, as illustrated in Figure [ (right), we construct a serial group by sequentially generating multiple completions during a single CoT process. By prioritizing rewards for earlier correct completions, this approach encourages models to demonstrate complete reasoning capabilities the initial phases, enabling early exit and preventing overthinking.
As shown in Figure
Motivated by this, we propose Serial-Group Decaying-Rewd Reward Policy Optimization (S-GRPO), a simple yet effective modification to address standard outcome-reward RL (GRPO)'s inability to regulate intermediate reasoning processes. During S-GRPO training, we construct a serial group for each query using a two-phase rollout process, In the first phase, a complete reasoning path is generated. In the second phase, subsequent rollouts introduce early-exit interventions at different positions along the reasoning path generated in the first phase, producing intermediate answers. Finally, the serial group formed by combining the complete reasoning path from the first phase with the intermediate answers appended to its corresponding truncated reasoning path. On this basis, we assign the rewards that decay according to their order of early exit for the correct ones within the serial group S-GRPO culminates in computing serial-group relative advantages and using their policy gradient to update model parameters.
Our contributions are threefold:
We pioneer serial-group RL paradigm that overcomes the critical limitation of outcome-reward RL. in regulating intermediate reasoning processes, accompanied by an open-sourced training framework (released once accepted).
The proposed S-GRPO algorithm enables models to produce higher-quality reasoning paths during the early stages of CoT generation, and implicitly early-exit once sufficiency is achieved. S-GRPO preserves the integrity of the original reasoning process via two-stage rollout procedure, ensuring that the model's 's pre-existing reasoning abilities are not compromised, which is well-suited as the final stage of post-training.
Extensive experiments across GSM8K [16] AIME 2024 17). AMC 2023 18. MATH-500 19).
and GPQA (20] benchmarks with Qwen3 21 and Deepseek-seriee reasoning models, demonstrate 0.72%~3.920 absolute accuracy improvement alongside 40.4% 1.11% % average token reduction, establishing an efficiency- accurray synergistic improvement.
2
Method
The proposed Serial-Group Decaying-Rewt Reward Policy Optimization (S-GRPO) is a novel rein- forcement learning mechanism that innovatively leverages rule-based outcome rewards to regulate

---

Early
Serial group
athinko>
A'
T
CoT
T
othinka
0
Decaying
reward
Query
Pollcy model
-..
T,
CoT
...
Policy modal
Group
T
computation
othinkk>
0
yr
Ar
T ethinic
Co
AP
Full Thought Rollout
Early-exit Thought Rollout
O: Output
7: Thought
C: Conclusion
r: Reward
A Advantage
Figure 2: The framework of S-GRPO. The complete answer inducer omitted in the figure and represented by </think> instead. The complete answer inducer is "Time is 1imi ted, stop thinking and start answering. \n</think> \n\,n"
intermediate reasoning processes and improve reasoning efficiency. During S-GRPO training, the LLM exits early and generates answers different positions within a complete CoT, forming serial group. Rewards for correct answers are assigned based on position, with earlier thinking exits receiving relatively higher rewards. This strategy encourages the model produce high-quality reasoning earlier and terminate once sufficient reasoning is achieved. In particular, the framework divided into three stages: Gerial-Group Generation, Decaying Reward Strategy, and Advantage Computation and Parameter Update.
2.1 Serial-Group Generation
GRPO is originally designed to enable reasoning models to achieve their pass@k k potential under pass @1 settings. To achieve this, GRPO generates multiple CoTs in parallel for each query, forming a parallel group, and rewards only those with correct answers. contrast, S-GRPO aims to achieve efficient reasoning through sufficient-reasoninn early exit. Specifically, ventions at different positions within single CoT to construct a serial group, thereby allowing the RL training to compare the thought sufficiency in different positions along the reasoning path. It consists of two stages: Full Thought Rollout and Early-exit Thought Rollout.
perform early-exit inter-
2.1.1 Full Thought Rollout
In the Full Thought Rollout stage, the model generates complete reasoning path sequentially for each query, represented as {O" = Ti, .,,...,, Tn, / /hink >, J.). To expose the model to diverse reasoning sufficiency scenarios and enhance its ability handle early exits effectively, we adopt random-length truncation during training. Specifically, the reasoning sequence is truncated at m randomly selected temporal positions P.. where P: = and ~ Uniform(L, n). ensuring that the truncation points are uniformly distributed across the reasoning path. This randomness allows the model implicitly evaluate reasoning sufficiency across varying depths and prevents overfitting to specific reasoning path lengths.
2.1.2 Exily-exit Thought Rollout
In the Early-exit Thought Rollout stage, the policy model extracts several early-exit reasoning paths {CoT', CoT² .... " } {O%[ Pi), OºI: Pal..... OO P.,]), representing truncated segments of the full reasoning path at different positions. For each early-exit path, the model generates corresponding answers C₁ C2...., C.-
In particular, we insert the prompt "Time is limited, stop thinking and sta rt answering. \n</think> \n 'n" at each truncated position P.. This prompt explicitly in-

---

structs the model to halt further reasoning and produce answers C³. Details of this process are illustrated in Figure
To ensure that the serial group contains early exit samples with correct answers (obtained by Early-exit Thought Rollout), sample more queries than the training batch required (over-sampling) for data filtering, such as DAPO [22].
2.2 Decaying Reward Strategy
To encourage models produce adequate and correct reasoning steps earlier stages of CoT
generation for accurate problem-solving, we propose a Decaying Reward Strategy. This mecha- nism assigns rewards based on the correctness of answers generated during the two-time rollouts ( C1, Cr,....... Cm, and Co ), while decaying rewards according to their order of early exits. For the answer C* of each response O', the reward r² is defined follows:
if C' is correct,
if C' is incorrect.
Where Night refers the accumulated number of correct answers up to and including the current position. The Decaying Reward Strategy is designed with dual objectives: (1) Exponential decay for correct answers: The strategy applies exponentially diminishing rewards to enhance the quality of earlier reasoning steps, which is overlooked by binary 0/1 outcome rewards. (2) Zero reward for
incorrect answers: The strategy enforces correctness-firs optimization strategy, ensuring the model maintains robust core reasoning capabilities.
This dual-objective design strikes balance between reasoning sufficiency and efficiency, guiding the model to produce reasoning sequences that are both accurate and concise.
2.3 Advantage Computation and Parameter Update
After computing the decaying rewards, S-GRPO calculates the advantage for each response in the serial group. Specifically, the advantage calculated by subtracting the mean reward of the group from its corresponding reward, as defined by the formula: A, r; mean(r:). Here, for training stability, the standard deviation is removed from the advantage computation compared to the GRPO. Subsequently, the computed advantage for each sample is broadcast to all corresponding response tokens. Finally, parameter updates performed based on the advantage values of each sample (Algorithm]) D].
The optimization objective is as follows:
(s) ElgnP(QQ) ~o (01g)]
lo.
f.1 "g
min
A clip(
e, 1 1+ ++)) (2)
o.f Tga
where = | g, Oi,et) denotes the conditional probability of the token at position L. The models ne. iiaa correspond the training model and sampling model, respectively. q represents the full and early-exit thought rollout responses generated by the model. The advantage A, for each response computed as Å, = r; mean(r:). where r; is the decaying reward assigned to the response. For token-level advantage A, it is defined to be equal to the corresponding sequence-level advantage Å,. The hyperparameter e is used to bound the importance sampling ratio.
the input query, and (o:)
Experiments
3.1 Experimental Setup
Training datasets. We selected problems from DeepMath-103k 23 to build our training set. This dataset is a large-scale and challenging collection of mathematics problems, focusing on difficulty levels ranging from grade to grade 10. addresses the lack sufficient complexity present in existing datascts by offering approximately 103,000 carefully curated problems. The dataset is

---

Algorithm Serial-Group Decaying-Rewe Reward Policy Optimization (S-GRPO)
Require: Query Q, Policy model no. Number of positions sample m
Ensure: Updated policy parameters 0
1: Full Thought Rollout:
2: Generate complete reasoning path O = (Ti, 22.......... ......TTTT T, </think>, C²) using Te for query Q
3: Sample m temporal positions P. E (Pi, P2, ..,,,., Pmn where P. Uniform(1,n)
4: Early-exit Thought Rollout:
5: Tethink> tokenizer.encode("Tie An</think>\n'nn)
is limited, stop thinking and start answering.
6: for each position P. do
7: Append Temiink after position P. to form CoT = (Ti, (2..72..... TT, T.P,,Ter Tothinks)
8: Generate answer C' using policy model my conditioned on CoT
9: Form output O' = (CoT", C*)
end for
11: Rule-based Dynamic Decaying-reward:
Nright +
b Counter for correct answers
for each output O' where E 21, m, 0}
order of position do
14: if C'i is correct then
Nright + Night + 1
y² -
Exponentially decaying reward
17: else
18:
nd +-
Zero reward for incorrect answers
19: end
20: end for
21: Group Computation:
22: Compute advantages A for each output based rewards yf
23: Update policy parameters using policy gradient with advantages A
constructed through an extensive data cleaning process applied to known benchmarks, ensuring no overlap with widely used benchmarks.
Benchmarks. To comprehensively assess the models' capabilities across range of reasoning tasks, we select five popular benchmarks that reflect diverse levels of difficulty: GSM8K [66 is well-curated collection of 1,319 problems in elementary mathematics. This benchmark is specifically designed to evaluate multi-step reasoning in foundational math tasks. Problems typically involve two eight sequential operations, relying primarily on basic arithmetic performed over multiple intermediate steps. AIME 2024 [17] includes 30 advanced problems selected from the 2024 edition of the American Invitational Mathematics Examination. This highly regarded competition tests participants' ability to reason mathematically across broad range of topics, including arithmetic, algebra, combinatorics, geometry, number theory, and probability core components of secondary- level mathematics. AMC 2023 [18] consists of 40 questions spanning key areas such as algebra,
geometry, number theory, and combinatorics. As part of the American Mathematics Competitions organized by the Mathematical Association of America (MAA), AMC aims to foster problem-solving skills and identify promising young mathematicians. MATH-500 [19] presents a set of challenging problems drawn from high seheol-level mathematical competitions. To ensure comparability with
prior studies, we use the same subset 500 problems originally compiled by OpenAl for evaluation purposes [24]. In addition to these mathematical evaluations, we also examine performance scientific reasoning tasks, using the following benchmark: GPQA 200 is rigorously constructed dataset containing graduate-level questions in physics, chemistry, and biology. Notably, even domain experts with PhDs achieve only 69.7% accuracy on this benchmark (25). For our experiments, we focus on the highest-quality subset, GPQA Diamond, which contains 198 carefully chosen questions.
Baselines. We compare S-GRPO with various existing efficient reasoning methods, including the fraining-free, based approach DEER [15), the off-policy optimization method COnCISE [26). and on-policy RL-based approaches such as original GRPO, RI + Length Penalty [27]. and ShorterBetter [28]. DEER makes early-exit decisions during the inference phase, guided by the confidence scores of intermediate answers. ConCISE generates more concise chains of thought by
5

---

Table I: Experimental results on four large reasoning models.
GSMSK
AIME
AMC 2023 Tokens
MATH-500
GPQA
Overall
Method
Asc
Tokens
Ave
Tokens
Acc
Aze
Tokens
Acc
Tokens
Acc
Tokens
DeepSook-R--Disii----
Vanilla
92.4
1,833
55,4
13,232
77.2
9,693
85.8
5,590
50.1
15,385
72.18
9.147
DEER
88.8
917
53.3
10,971
87,5
4,142
91.8
2,431
47.5
5,280
73.78_1.66
4,748.48.14
ConCISEgPT
92.9
9.751
92.0
2,244
50.0
5,892
ConCISEsimPO
92.1
715
48.3
7,745
91.0
1,946
48.0
4,859
GRPO
93.2
1,767
55.0
13,451
87.5
9,887
93.6
5,317
50.7
15,817
76.00_3.82
1.14%
RL Length Penalty
92.4
1,062
51.9
7,464
86.9
3,540
92.2
2,451
49.1
3,984
74.30.2.32
3,700 59.5%
ShorterBetter
53.3
5,288
75.9
2,580
S-GRPO
93.8
56.0
7,377
87.5
3,494
92.4
2,252
30.8
3,751
76.10.3.92
3,556.61.1%
Vanilla
94.2
2,129
64.4
11,099
90.5
5,527
93.5
3,844
59.2
6,034
80.36
5.727
DEER
93.3
982
70.0
10.335
90.0
4,349
91,4
2.753
57.1
4.767
80,36-000
4,637 19.0%
GRPO
95.3
2,120
65.8
13,504
91.9
6,595
94.0
4,471
58.9
7,354
81.18
6.809 8.99%
RL Length Penalty
94.7
775
7,950
88.1
3,396
92.4
1,993
56.0
4,380
77.24-3.12
3,699.35.44 4%%
S-GRPO
96.2
724
64.4
6,712
91.9
3,352
93.6
2,146
59.3
3,334
81.08.0.72
3,2544.4..2%
Qwen3-SB
Vanitla
95.4
2.370
74.1
15,326
91.3
9,452
93.4
5,577
55.6
8,741
81,96
8,293
DEER
95.5
981
76.7
11,287
95.0
6,198
93.4
3,208
52.5
3,104
82.62
2.660
4,956.14.
GRPO
95.8
2,358
72.7
15,154
92.8
8,983
94.4
5,440
55.8
8,819
82.30
8,150.1.79
RL + Lengtl Penalty
95.4
1,323
13.8
9,666
93.4
5,042
94,2
3,247
56.2
5,293
82,60 00.60-0.66
4.914.40.7%
S-GRPO
1,292
8,810
5,962
95.2
3,166
57.7
5,271
84.26.22.3
4,900.40.95
Qwen3-14B
Vinilla
95.5
1,909
75.4
14,116
96.9
7,576
95.2
5,078
58.8
7,576
84.36
7,251
DEER
95.5
908
76.7
10,333
95.0
5,099
94.8
2,987
57.1
2,435
83.82.0.8
4,352.40
GRPO
96.1
1,956
77.7
14,544
98.4
8,000
95.8
5,140
39.3
7,966
85.46-1.1
7.521+3.7%
Length Penalty
95.8
1,090
9,056
5,059
95.8
2,866
59.4
4,949
84.48 0.112
4.604. 4.609.36.5%
S-GRPO
96.3
952
77.9
8,932
97.8
4.537
96.4
2,652
60.6
4,537
85.80+1.44 1.44
4,322.40.44
integrating specialized prompt tokens and applying early-exit mechanisms during inference, followed by SFT/SimPO to further encourage succinct reasoning. RL + Length Penalty assigns reward values based on the deviation of each correct response length from the mean, penalizing longer correct responses. ShorterBetter promotes shorter yet accurate reasoning paths by assigning higher rewards to compact chains that yield correct answers based on GRPO.
Models. We evaluate S-GRPO on four large reasoning models, including CrepSeek-RI--istiiee Qwen-7B,
R1- 14B [3). Qwen3-8B, and Qwen3-14B [21]. Despite achieving state-of the-art performance reasoning tasks, these models tend to generate overly verbose reason- ing processes that contain excessive redundant information.
Metrics. S-GRPO is designed to improve correctness while minimizing inference length, thereby enabling more efficient reasoning. To evaluate its performance, we adopted two key metrics: Accuracy (i.e., pass @1 I) and Token Count (Tokens). Due to the inherent instability of generating long sequences in reasoning models and the limited sample sizes of certain benchmarks, we conducted multiple evaluation runs and reported the averaged results in the tables. Specifically, we performed 16 trials on AIME 2024 and AMC 2023, 8 trials on MATH-500 and GPQA Diamond, and trials on GSM8K.
Training details. For S-GRPO, we use a learning rate of 1 x 10-6 and randomly select 8 temporal positions for each query. Since we adopt an on-policy mode, the generation batch size and training batch size are both set to 128 x 8. For GRPO, we use the same learning rate and batch size settings. For RL + Length Penalty, we follow the settings described in its original paper (27) and set the scalar parameter to 0.2. Across all experiments, we employ Adam 29 the standard optimizer.
3.2 Experimental Results
Main results. The experimental results in Table D demonstrate that S-GRPO consistently out- performs existing baselines across five benchmark datasets and four reasoning models, achieving significant improvements. Compared to vanilla reasoning models, S-GRPO achieves an average accuracy (absolute) improvement of 0.72% to 3.92%, while reducing the generated sequence length by 40.4% to 61.1% 1%. S-GRPO achieves notable improvements on in-domain mathematical reasoning benchmarks (e.g., GSM8K, AIME 2024, AMC 2023, MATH-500) and out-of-domain scientific reasoning tasks (e.g., GPQA), demonstrating its effectiveness and robustness. Specifically, when applied to R---sssii-------llll------------
S-GRPO achieves a 10.3-point increase in accuracy on

---

GSMBK
AIME 2024
95
Mean response length
80
response length
2200
70
90
2000
60
12000
Pass@1
85-
1800
50
10000
80
1600
40
8000
75
1400
30
70
1200
20
6000
65
1000
10
60
800
2000
5
15
20
25 30
15
25
5
15
25
10
15
26
Max Length (K tokens)
Max Length (K takens)
Max Length (K tokens)
Max Length (K tokens)
DeepSeek-R1- Qwen-7B
DeepSeek- 1i--Diiil-Qwen-77B Qwen-BB S-GRPO
Qwen3-8B
Owen3-88 S-GAPO
Figure 3: Performance of DeepSeek-Re DeepSeek-R1-Distille R---issill---ee--7 and Qwen3-8B without or with S-GRPO
training on GSM8K and AIME 2024 under different generation-lnnth budgets.
AMC 2023 while utilizing only 36% of the original reasoning budget. Similarly, it obtains 0.6-point improvement on AIME 2024 with just 56% of the original inference length.
Comparison with SOTAs. We compare S-GRPO with several approaches designed for efficient reasoning. For example, DEER reduces reasoning length by only 7%%~27% on AIME 2024 and some- times sacrifices accuracy, and its effectiveness is mainly limited to simple tasks. In contrast, S-GRPO consistently reduces reasoning length across both simple and complex tasks while simultaneously improving reasoning accuracy. Compared to the original GRPO. S-GRPO achieves comparable or even better accuracy while significantly shortening the inference trajectory, indicating that the proposed Serial-Group Generation mechanism does not hinder the model's exploration capability in reinforcement learning. Moreover, the reward shaping based on serialized intermediate outputs more effectively guides the model toward efficient reasoning.
Compared with the most recent training- based efficient reasoning methods, S-GRPO achieves the best in both reasoning length reduction and accuracy. This can be explained by the fact that ConCISE's off- off-policy optimization forces the model to fit into a new data distribution, and ShorterBetter and RL + LLngth Penalty overly emphasize length reduction, at the expense of generalization and accuracy- driven optimization. In contrast, S-GRPO preserves the integrity of the original reasoning process via two-stage rollout procedure, ensuring that the model's pre-existing reasoning abilities are not compromised, which is well-suited as the final stage of post-traun post-training.
Performance with different token budget. We vary the generation- length budget during inference from short to long, and compare the change in accuracy and actual reasoning length of S-GRPO and vanilla CoT GSM8K (representing simple problems) and AIME 2024 (representing complex problems). The results in Figure 3 show that, across all budgets tested, S-GRPO consistently achieves higher accuracy while generating shorter sequences compared to vanilla CoT, further highlighting the effectiveness of our method.
Moreover, we observe that under tight length budgets, S-GRPO achieves significantly higher accuracy than vanilla CoT while generating sequences with comparable length. Differently, under larger length budgets, S-GRPO produces significantly shorter reasoning paths with slightly better accuracy, compared with vanilla COT. The trends in both accuracy and actual generation length of S-GRPO are smoother than vanilla COT, indicating its greater robustness to variations in length budget. Overall, S-GRPO achieves high accuracy under a low length budget, suggesting that our method is capable of generating concise yet accurate reasoning paths.
Ablation results. To verify the effectiveness of each design component in S-GRPO, we conduct ablation studies under three different settings. w/o. Decaying (Shortest 1) denotes the setting where only the shortest correct response in the serial group is assigned reward of 1, while all other responses receive a reward of 0. w/o. Decaying (All 1) refers the configuration where all correct responses in the serial group are assigned a reward of 1, while incorrect ones receive a reward of O. w/o. Serial indicates that we further remove the Serial-Group Generation mechanism based on the w/o, Decaying (All 1) configuration
7

---

Table 2: Ablation results on Qwen3-8B.
GSM8K
ATME 2024
AMC 2023 Tokens
MATH-500
GPQA
Overall
Method
Acc
Tokens
Acc
Tokens
Acc
Acc
Tokens
Acc
Tokens
Acc
Tokens
Qwen3-8B
GGRPO
96.1
1,292
77.3
8,810
95.0
5,962
95.2
3,166
57.7
5,271
84,26
4,900
w/o. Decaying (Shortea 1)
95.9
1,175
69.6
8,721
92.5
4.58
94.8
2,740
55.7
4,734
81.70 2.56
4,790.10.499
Decaying (All
96.0
2,385
74.4
14,940
94.7
9,000
95.0
5,614
54,9
8,955
83.00.1.26
6,179.66 9%
- wlo, Serial
95.8
2,355
72.7
15.154
92.8
8,983
94.4
5,440
55.8
8,819
82.30.1.96
8,150,66.34
The results in Table Table@iiniion iindicate that rewarding only the shortest correct response imposes an overly strict constraint. Although this setting leads to further reductions in reasoning length, it comes at the cost of accuracy. And removing the design that assigns higher rewards to shorter outputs, i... w/o. Decaying (All J. results in lengthy reasoning. This is because long correct answers also receive high rewards, and the model does not shift in favor of generating short Co'Ss. When the Serial- Group Generation mechanism removed, our method degenerates to GRPO, achieving performance comparable to w/o. Decaying (All /) in both accuracy and reasoning length. This demonstrates that Serial-Group Generation, as an essential component of S-GRPO, does not compromise the model's exploration capability in reinforcement learning.
Case study. Figure in the Appendix visually illustrates how S-GRPO generates answers at different early-exit positions and also shows the decaying reward assignments. For the first early-exit position, where the model produces an incorrect intermediate answer, we the reward to Ó. For subsequent exits that yield correct answers, we apply a decaying positive reward scheme, where earlier exits are associated with higher reward values. This design incentivizes the model discover reasoning paths that are both accurate and succinct.
To more intuitively illustrate the effectiveness of our approach, representative example is shown in Figure 4 The left and right sides of the figure compare the reasoning processes of vanilla CoT and S-GRPO. Although both methods yield the correct final answer, S-GRPO achieves this using less than half of the reasoning budget, showcasing its effectiveness in mitigating the overthinking problem [TI] The central portion of the figure displays an early exit obtained by directly truncating the vanilla CoT's reasoning process using the same token budget as S-GRPO. The reasoning model fails to reach the correct conclusion based on the available partial reasoning. This demonstrates that S-GRPO effectively identifies the correct solution path and guides the model to concise and accurate reasoning, while inherently avoiding the underthinking issue of superficial exploration (30
4 Related Work
The success of OpenAl ol 31 highlights the powerful potential of reinforcement learning in post-training to enhance model reasoning capabilities. With the open source of Deepseek-RI (3] and Qwen3 [21]. reasoning models are now widely deployed locally, drawing the LLM community's 's attention to the efficiency of long chain-of-thougle generation.
Reinforcement learning for reasoning. Rule-based outcome reward RL 3 32 33 34
35
36
37
has emerged as the mainstream approach for post-training optimization of reasoning models. This paradigm simplifies reward design by employing binary 0/1 rewards determined through rule-based correctness evaluation, eliminating the need for separate reward models as required in original GRPO 16 3 38) implementations, thereby substantially reducing memory and computational overhead during RL training. Recent advances have introduced numerous RL algorithm variants focusing on training efficiency 33 40 41 value function optimization [42] 43 44], and other aspects.
However, these developments preserve GRPO's fundamental mechanism that samples several Co''s in parallel for group, consequently overlooking optimization opportunities during sequential thought generation within a single COT. Our proposed optimized version of GRPO, S-GRPO, maintains the rule-based outcome reward framework while enabling intermediate reasoning process rewards by sampling multiple early-exit thought chains in serial for one COT. We allow existing reasoning models to effectively mitigate overthinking and boost inference efficiency.
Efficient reasoning. Approaches for efficient reasoning can be categorized into two groups: training-free free and training- methods. Training-free methods typically improve reasoning effi-
8

---

Q: There are periods the day for normel student but John has take extra classes, Each class minutes long He goas to class for days a week He then spends 1/16 of his weekly minules each on Selurday and Sunday as extra learning lime. How many houre a week does he spend learning learningeePPeesee Prilleseeeeeeeeee reason step by and your finel answer within (boxed)).
Qwen3-8b
Same thinking part Qwen3-8b Hard Truncation
Qwen3-8b S-GRPO
Okay, need to figure out how many hours a week, John spends learning Let me start by braaking down the problem step by step. vinnFirst. the problem says there are 6 partods a day for John has take Sunday
Okay, need to figure out how many hours a
Okay, try to figure how many hours
week John spends learning. Let me start by braaking down the problam step by step. viniFisst.
week John spence learning. Hmm, let me start by broaking down the probiem step by stop. Ininiirstt the problem says there are perieds in day for normal student, but
the problem says there are 6 pertods a day for normal student, but John has take classeg, cominteds Then Saturday and Sunday are the other days. WninSo maybe total time regular time (5 cays) plus extra time on Saturday and Sunday. VnreBBu the problem saye 'iHie then spends 1/16 of his weekdy minutes each on learning time.
normal student,
to take extra he spends classes during week. He goes class days a wook. So, per day, classes 40 minutes. Let me
classes. comilled= Then Saturday are the other days. ininSo maybe total time regular time (s cays) plus extra time on Saturday and Sunday. IrinBut the problem saye 'eHee then spends 1/16 of his weekdy minutes each on leamning time.
vomilleds dassee
dassse
compute that. vriui classes 40 minutes = 320 minunes per day. Then, multiplled by 5 days
Saturday and Sunday
Saturday and Sunday since = regular time extra time.
week: 320 15, Let calculate that. 320
somitted> since = regular time oxtra time.
somitted>
that's time spent classes during the week. Then, there's the extra learning time on Sanurday and Sunday, The problem says he spends 1/16 of his weekly minutes each on Saturdey and Sunday extra learning comittedd Vnethinoaireetoo determine how many
1800 minutes.
and the extra lime is 1/18" then I regular time
and the extra time is 1/18"T. then T reguler time
+ 1/16"1 kmnWWhchh gives us 1600 (1 1/16]
+ 1:16"1 knnWWhihh gives us 1600 (1 1/16)
1800 / (15/16) 1600 16 115 1705.57
1800 / (15/16) 1600 16 /15 1706 complicttd. 'nin limited slop thinking and start answering ing Invthhossstte To determine how many hours per
minutes, BrinBut Allematively, maybe problem leamiin time 1/16 of sime spends an regular clesses, comitted ieinTherelore the anewer is 30 hours Vinel think this is the correct interpretation, "Fina
complicated Inin
minutes, BrinBut
the extra
hours per weak John spends learnnng.
week, John spends learning we break the problem
the problem into clear steps and compute the total time spent learning, then convert il lo hours. Vinn -- -i-inndde "SStpp I: Calculate Regular Class Time
down into lwo parts: regular class time and "exira leamiin time on wookands " inle--linndde "SStop
<thhnlo/rinn determine how week John spends learning. comilleds convert
1: Regular Class "vie- John
classes commilleds 25rog256000015 wimes 60)
""rinJohn hastp- classes per
extra/er> Each class
minutes"
minutes to hours, divide by Suitteotittletooollll hours per week)- ldrral (800))0) - 30 Wesl| hourstynSSirie--iilo
W5a0025600x9000 = (2aag Napprox 28
allends comineds S5it 600 200 1800
Woevd hours)yn$Sittn--ill ANl ""Finel "reweer"ieeSSiiiiii virsoerd 20.44/48
Mexxtmmnutts//nn Snnn 1800)600) - 30
Wextthoure//nSSS Sin
KNM "Finel Anewer 300)
ninJohn spends approximately 28. 44 hours per
440 "Final Answerr"nturrllle 3/tr/rlooxxedd30y/
Right
learning
x
Wrong
Right
Whole token number: 10088 Thinking Budget: 9641
Whole token number: 4940
Whole token number; 4858
Conclusion: 447
Thinking Budget: 4418
Conclusion: 522
Same budget
Thinking Budget: 4418
Conclusion: 440
Minimum
Figure 4: Comparison of a generated content sample on GSM8K.
ciency through dynamic prompting strategies 45 46
48
49
50
51 52) Best-of-N sampling
15 mechanisms during reasoning. 15 and our method employ a similar operation by implementing early exit during CoT generation and producing intermediate answers. The key distinction lies in that DEER directly makes early-exit decisions during inference based on the confidence of intermediate answers, whereas our method leverages RL to reinforce models' behavior of correct early exit, which requires no additional compute overhead during inference.
pruning [53 54) and optimizations [55] 56 57] and early-exit [58
DEER
Training-based methods primarily supervised fine-tuning models with variable-length CoT data
(59 60 61 62
63
64
45). or training with length-rewards RL 12
65
66
67
68
69
70 71 72).
Recently, COnCISE 26 constructs concise CoTT data by inserting prompt tokens and employing
carly-exit during inference, then enhances the model's 's reasoning conciseness through SFT/SimPO
77 74] (27) and (28) upweight rewards for the shorter CoTs, having correct answers, in parallel- sampled reasoning chains. However, they inherit GRPO's parallel sampling paradigm that neglects the reward function's 's attention to the intermediate serial reasoning processes.
5 Conclusion
This paper introduces Serial-Group Decaying-Rewart Policy Optimization (S-GRPO), which in- novatively leverages rule-based outcome rewards to regulate intermediate reasoning processes. By incentivizing the LLM to produce high-quality thoughts earlier and exit promptly when the generated thought is sufficient, S-GRPO enhances reasoning efficiency and maintains accuracy. Empirical evaluations demonstrate that S-GRPO achieves optimal synergy between efficiency and accuracy, significantly outperforming existing efficient reasoning methods. Specifically, S-GRPO demon- strates compatibility with state-of-the-art reasoning models, including Qwen3 and Deepseek-diste achieving a 40.4%><61. 1% reduction in sequence length while improving accuracy (absolute) by 0. 0.72%~3.992 3.92% across five benchmark datasets: GSM8K, AIME 2024, AMC 2023, MATH-500, and GPQA Diamond. S-GRPO is well-suited for deployment as the final optimization stage in post-training pipelines, offering a practical solution for enhancing reasoning efficiency.

---

References
[1] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters, 2024.
[2] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[3] DeepSeek-AT, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li. Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jinn Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen. Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang. W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie, Xin Cheng. Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He. Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang. Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zchui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie. Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu. Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
[4] OpenAL. Learning to reason with 1mms. https: /tpp///ppaaa//aa/////// wwtinnaa..oooaaammmmmmmmmmmmaa research/ 2025. Accessed: 15 March 2025.
[5] Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang. Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang. Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng. Chen Gao, and Yong Li. Towards large reasoning models: A survey of reinforced reasoning with large language models, 2025.
[6] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
[7] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05800 2022.
[8] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
10

---

[9] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
[10] Shyam Sundhar Ramesh, Yifan Hu, lason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, and Ilija Bogunovic. Group robust preference optimization in reward- RLHF. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[11] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like 1lms, 2025.
[12] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12550 12599, 2025.
[13] Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXi::2502.08235, 2025.
[14] Yuyang Wu, Yifei Wang. Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms. arXiv preprint arXiv:2502.0721 2502.0726, 2025.
[15] Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Zheng Lin, Li Cao, and Weiping Wang. Dynamic early exit in reasoning models, 2025.
[16] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenyy Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021.
[17] MAA Committees. Aime problems and solutions. com/wiki//ndexx Phpp/AiME_Problems ems lacemm__md__oottiiiio
[18] AI-MO. Ame 2023, 2024.
[19] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[20] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023.
[21] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
[22] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system scale. arXiv preprint arXiv:2503. 144766 2025.
[23] Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song. Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath- 103k: 103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. 2025.
[24] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. 2023.
[25] OpenAl. Learning reason with 1lms, September 2024.
[26] Ziqing Qiao, Yongheng Deng, Jiali Zeng, Dong Wang, Lai Wei, Fandong Meng, Jie Zhou, Ju Ren, and Yaoxue Zhang. Concise: Confidence-guider compression step-by-step efficient reasoning, 2025.

---

Scaling reinforcement learning with [27] Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025.
[28] Jingyang Yi and Jiazheng Wang. Shorterbetter: Guiding reasoning models to find optimal inference length for efficient reasoning. 2025.
[29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988, 2014.
[30] Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song. Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts are all over the place: On the underthinking of ol-like Ilms, 2025.
[31] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
[32] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao. Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihho Gao, Weimin Xiong, Weiran He, Weixiao Huang. Wenhao Wu, Wenyang He, Xianghui Wei. Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Ilms, 2025.
[33] Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward training time for Ilm reasoning, 2024.
[34] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model prsi-nraining, 2025.
[35] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025.
[36] Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. Curriculum sft, dpo and for long cot from scratch and beyond, 2025.
Light-rl:
[37] Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with progressive context extension for efficient training rl-like reasoning models, 2025.
[38] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[39] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wen-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An ooen-source Ilm reinforcement learning system at scale, 2025.
12

---

[40] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective, 2025.
[41] Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang. Yinghan Cui, Chao Wang, Junyi Peng, Shimiao Jiang, Shiqi Kuang, Shouyu Yin, Chaohang Wen, Haotian Zhang, Bin Chen, and Bing Yu. Srpo: A cross-domain implementation of large-scale reinforcement learning on Ilm, 2025.
[42] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment, 2024.
[43] Yufeng Yuan, Yu Yue, Ruofei Zhu, Trantian Fan, and Lin Yan. What's behind ppo's collapse in long-cot? value optimization holds the secret, 2025.
[44] Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang. TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu. Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang. Yonghui Wu, and Lin Yan. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025.
[45] Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget-awann Ilm reasoning. arXiv preprint arXiv:2412.185// 2024.
[46] Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less. arXiv preprint (tt:::5022./600) 2025.
[47] Ayeong Lee, Ethan Che, and Tianyi Peng. How well do Ilms compress their own chain-of- thought? a token complexity approach. arXiv preprint arXiv:2503.0f14) 2025.
[48] Matthew Renze and Erhan Guven. The benefits of concise chain of thought on problem- solving in large language models. In 2024 2nd International Conference on Foundation and Large Language Models (FLLM), pages 476-483, IEEE, 2024.
[49] Qiguang Chen, Libo Qin, Jiaqi Wang, Jingxuan Zhou, and Wanxiang Che. Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of- thought. Advances in Neural Information Processing Systems, 37:54872-54904 2024.
[50] Simon A Aytes, Jinheon Back, and Sung Ju Hwang. Sketch-of-thought Efficient 11m reasoning with adaptive cognitive-inspired sketching. arXiv preprint arXiv:2503.05179, 2025.
[51] Yu-Neng Chuang, Helen Zhou, Prathusha Sarma, Parikshit Gopalan, John Boccio, Sara Bolouki, and Xia Hu. Learning to route ilms with confidence tokens. arXiv preprint arXiv, 2410, 2024.
[52] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and lon Stoica. Routellm: Learning to route llms with preference data, 2024.
URL https://arxiv. org/abs/2406.1866
[53] Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36:41618 416500 2023.
[54] Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. Reward-guided speculative decoding for efficient Il reasoning. arXiv preprint arXiv:2501. 19324, 2025.
[55] Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Xinglin Wang, Bin Sun, Heda Wang, and Kan Li. Escape sky-high cost: Early-stopping self-consistency for multi-step reasoning. arXiv preprini arXiv:2401. 14000 2024.
[56] Rohin Manvi, Anikait Singh, and Stefano Ermon. Adaptive inference-time compute: Lims can predict if they can do better, even mid-generation. arXiv preprint arXiv:2410.0277 2024.
13

---

[57] Pranjal Aggarwal, Aman Madaan, Yiming Yang, et al. Let's sample step by step: Adaptive- consistency for efficient reasoning and coding with llms. arXiv preprint arXiv:2305.11860 2023.
[58] Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking. arXiv preprint arXiv:2504.00 09858,, 2025.
[59] Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system into system arXiv preprint arXiv:2407.06022, 2024.
[60] Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter chain-of- thought without compromising effectiveness. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24312-24333 2025.
[61] Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in Ilms. arXiv preprint arXiv:2502. /2067, 2025.
[62] Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Col-valve: Length-compressibe chain-of-thoughy tuning. arXiv preprint 1rXiv:250209901. 2025.
[63] Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang, Yujin Kim, and Se- Young Yun. Self-training elicits concise reasoning in large language models. arXiv preprint arXiv:2502.20122, 2025.
[64] Tengxiao Liu, Qipeng Guo, Xiangkun Hu, Cheng Jiayang, Yue Zhang, Xipeng Qiu, and Zheng Zhang. Can language models learn to skip steps? arXiv preprint arXiv:2411.01855 2024.
[65] Haotian Luo, Li Shen, Haiying He, Yibo Wang. Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-prune:: pruning. arXiv preprint arXiv:2501. 12570, 2025.
fine-tuning for ol-like reasoning
[66] Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning, arXiv preprint arXiv:2503 2025.
[67] Daman Arora and Andrea Zanette. Training language models to reason efficiently. arXiv preprint 24440225002.04433, 2025.
[68] Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long cfain-of-thought reasoning in llms. arXiv preprint 2xii:25522.03373, 2025.
[69] Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang. Kai Wang, and Shiguo Lian. Dast: aefficulty-adapiii wow-thinking for large reasoning models. arXiv preprint 2XXiv:5503.04477 2025.
[70] Yuxiao Qu, Matthew YR Yang. Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine-tuning, arXiv preprint arXiv:2503.07572 2025.
[71] Yingqian Cui, Pengfei He, Jingying Zeng. Hui Liu, Xianfeng Tang, Zhenwei Dai, Yan Han, Chen Luo, Jing Huang, Zhen Li, et al. Stepwise perplexity-guide refinement for efficient tfain-of-thought reasoning in large language models. arXiv preprint arXiv:2502. /3260, 2025.
[72] Muzhi Dai, Shixuan Liu, and Qingyi Si. Stable reinforcement learning for efficient reasoning. arXiv preprint arXiv:2505. /8086, 2025.
[73] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024.
[74] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward, 2024.
[75] Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.
14

---

[76] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: survey. IEEE transactions on pattern analysis and machine intelligence. 44(1 1):7436-7456, 2021.
[77] Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12622-12642. 2024.
[78] Wangbo Zhao, Yizeng Han, Jiasheng Tang. Zhikai Li, Yibing Song, Kai Wang. Zhangyang Wang, and Yang You. A stitch in time saves nine: Small vlm is precise guidance for accelerating large vlms. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19814-19824 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction have already claimed our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: There is a limitation section in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not propose theoretical type methods.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 3.1 details the base model, data, and parameters used for training.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer:
[No]
Justification:
We will release code once accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.er cc/
public celiii/uuiieee quw.g/udeeeeeeeeeeo CodeeSui ss onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips. //npsscc/pubbll wttpcccppppppppcteeeeeecccccccep cetii///uie///eeeeeeeeoe.sscssee ies CodeeSume ss cioooooliio oooiccyy icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Section 3.1 details the base model, data, and parameters used for training. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We performed multiple samplings during evaluation and took the average as the result (Section 3.1).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We wrote the details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited the paper of the assets used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers thcode com dataaetteett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Section 3.1 details the base model used for training.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
/atpap/rpppp.cc/ooofoeeeeee///// 2025/ LLM) for what should or should not be described.
Please refer to our LLM policy
22

---

A
An example of Training Data
Here an example that shows the training data truncation and decaying reward assignment.
Find the smaless integer SrS such that the cyclic group $C (12 subgroup the symmetric group $8 15. Provide single integer.
Full Thought Rollout:
Okay, so need to find the smallest integer n such that the cyclic group C, is subgroup of the symmetric group S, Hmm, let me think. wirniitst, I remember that every group can be embadded into some symmetric group Thinking Exit!
@ liiited, stop thinking starf answering knottinksinin
Truncated Thought Rollout:
.../nelhhnnko//// To determine the "ssmalisst integer" $ $ such that comitedd includes an element of onder 12 must invoive
elements (since single 12-oycle provides S
order 12), Hence, C_(12) S "canmot" embeedee
Si0io-iotestss Final AnsserrniinSeriloooeeee 12josss
x
Reward: 0
by Gayley's thaorem, which says that any group is komorppic to subgrouu the aymmetric gr pup on own elamants, So Cy that would mean It's subgroup ol S. Bul the <omilees> Wat, but the cycle are disjoint, so they ast on different elements. So, the total number elements is 6 4 10.
Thinking Exit!
O Time liiited, stop thinking and start answeriing ioclthinkaiin Truncated Thought Rollout:
...fhethhinko///i To determine the smallast & such that the eyclie group $ C. (12) $ comitteds combinations, the smallest sum consisting a 8-cyda and a 4-cydle in S.S.7 has order 12, and theratora, the cyclic group (_()1)) 5 can be embedoed Into S $_7 -iyio--- Final Answer/wn www.::/wweooooeee//t......
Reward: 1
Therefore, parmutation exsats vinAtemnaiivlly could batter? Let think. comittdd 12 2 LCM which LCM winAlterralivell. 12 as *3 Same thing. Vin
winnltematteely S. Same thing. WinAltmmttilyy 12 which we saw has LCM
Thinking Exit!
+ Time Immited stop thinking and start answering inothinkzlein Truncated Thought Rollout:
.../n-othinkk//m// To determine the smallest integer $ S such that the cyclic group C_(12) 8 is vomitteds 33 7 Sinin Honoa, the smallast $ $ for which S S.nn contains a permutation of order 12 (and thus contains 5 C_((2)) 5 as subgroup) istini $Svillboxee77)nn5
Reward: 1/2
Is there get LCM 12 with smale sum? For example, <omitted> Same thing, Vntowltermatively 2 33 which sow LCM www.teeaaaveyy 12 3*3*2
thing. there get 12 with smallar axample,
oycles of length
Thinking Exit!
@ Time limited, stop thinking and start answering inothinkokein Truncated Thought Rollout:
.../nstthinks/iii determine smallest integer such that cyclic group C_(12) S subgroup <omitted> possible value -nteee Final httpwa//nsSrlbbxxeee/eoe
?
Reward: 1/4
4 7. ls there way to get LGM 12 with a totel cycle length les than 7Pvinl et me think. For example, 2 and 67 LGM is Not enough. 3 and 4: LCM 12, sum 7. If try to use more cycles. For somiltes> minimal number of elements needed to represent a permulation of order Vieesllemstivveyy for
group C_m, the minimal
Thinking
Time limiled, stop thinking start answering vnoltinksiite
Truncated Thought Rollout:
w./nehhhnkko/niin To determine the smallest integer i such that the eyclic group G (12) S subgroup the symmettic conciude that the ""malless"
comitted smaller symmatric group indveesSSillboxxee7 2 Reward: 1/8
C_(12) $ subgroup
the minimal number such that the LCM of some integers whose sum n. wiinTherefrre, to find minimal n, we can think of <omitted How about a 2-cyclo and 4-cydl?? LGMI2 4jm4, Still not s. How about 8-cycla? Thinking Exit !
& Time limited, stop thinking and start answering Insothinickein
Truncated Thought Rollout:
//mothhinkk/////n To determine the emallest integer n such that the comitteds that can contain an element of order 12th, and thus "can contain the cycll group & C_(12) $ -nin---rindde Final www.wrr.wwnsSS/lbooeeeeeeeee
+ Reward: 1/16
possible n-8 WinTherefore, comiled-eum indeed 7. Thherhhr,,or,, the Wulnuututttt just be thorough, nu7 noB,
Thinking Exit
O Time limited, stop thinking and start answering Vnottinksiiin Truncated Thought Rollout:
...neethinnc///ii determine smallest intager 5 such comilteds Final Answer wwwinn/iibooodd(1e1
element of order 12ieato---niea
Reward: 1/32
can have permutatin of order 12. in S, the maximum order of an element <amittec- differentway to get order 1n S, with fewer
eloments? ? For exammple, by having single cyde? But no. because a single cycle in 8, can have maximum
Thinking Exit! !
@ Time limited, stop thinking and start answering vnothinkslein
Truncated Thought Rollout:
... .../hehhinko/n/// To determine the emailest integer S such that the comilteds total hew.lleeeeeeenn-eiiiiieeee win--yiinn ConclusionieieTTe smalleet number n for which C (12) S subgroup of S S SsswvsSvibboeee()4665
Reward: 1/64
longth which has order which less than comitteds www.hinicliiinooooooooooooooooo detormine the smallost -omittocs for which 5 C_(12) ombedded subgroup in S_nn Seiteteeleatee Final Anwerriidsribbooeeee/vs
Reward: 1/128
Figure 5: An example shows the training data truncation and decaying reward assignment.
23

---

B Computation Resource
In our experiments, 64 80g memory used train the models.
C Robustness to Prompt Template Variations
We evaluated the sensitivity of our thinking path truncation mechanism different prompt templates. Three variations were tested: (1) explicit time constraint prompt, (2) detailed explanation prompt, and (3) a minimal delimiter-only prompt. All variants included the \n</thiie nkk in in delimiter.
Table 3: Performance across different prompt templates for thinking path truncation.
AIME 2024
MATH-500 Tokens
Prompt Template
Acc
Tokens
Acc
Qwen3-8B with S-GRPO
Time is limited, stop thinking and start answering. \n</think>> In
77.3
8,810
95.2
3,166
Considering the limited time by the user, have to give the solution based 77.1 on the thinking directly now. \n</think>>>> \nin \n</think> \n/n
8,780
95.0
3,202
76.9
8,823
95.3
3,154
The results demonstrate remarkable consistency across all prompt variations. Accuracy differences are negligible, and token usage remains stable. This indicates that the \n</think> \n in delimiter, rather than the specific prompt wording, is the key factor in controlling thinking path truncation. Such robustness to prompt engineering choices enhances the practical applicability of our approach.
D Comparison of Truncation Strategies
We investigated different strategies for truncating thinking paths beyond random truncation. Specifi- cally, we explored semantic-based truncation at specific markers that indicate thought shifts, such as "Wait" and "Alternativel"" tokens, following similar approaches in prior work [75].
Table 4: Performance comparison of different truncation strategies.
AIME 2024
MATH-500 Ace Tokens
Truncation Strategy
Acc
Tokens
Qwen3-8B with S-GRPO
Random selection "Wait" "Alternatively" position
77.3
8,810
95.2
3,166
76.7
8,930
95.2
3,378
While semantic- baased based truncation achieved comparable performance random truncation, it exhibited critical stability issues during training. As the model improves through training iterations, reasoning chains become increasingly compressed and streamlined, resulting in significant reduction of transition tokens like "Wait" and "Alternatively". This progressive decrease creates a practical limitation: for certain queries, insufficient semantic markers are available to meet the sampling requirements for the second rollout phase (requiring samples). Consequently, semantic-based truncation becomes unreliable as training progresses.
In contrast, random truncation maintains consistent applicability throughout training regardless of how compressed the reasoning becomes. The combination of stable training dynamics and comparable performance validates our choice of random truncation as the primary strategy in the implementation of S-GRPO.
E Computational Efficiency of Dual-Rollout Training
We measured the actual time cost of S-GRPO's dual-rollout approach compared to standard GRPO on Qwen3-8B training with rollout n=8.
Despite requiring two inference passes, S-GRPO incurs only 20% additional time overhead rather than the 2x increase. This efficiency comes from two factors: (1) The second rollout only generates the
24

---

Table 5:
Time cost comparison between GRPO and S-GRPO.
Method
Average Rollout Time (s)
Average Time Increase
Relative Overhead
Qwen3-8B Training
GRPO (single rollout)
457
1.00x
S-GRPO (dual rollout)
550
+938
1 1.20x
shorter conclusion portion, not the full response, reducing decoding time proportionally sequence length: (2) The truncated thinking path uses efficient prefill computation rather than sequential decoding.
F Effectiveness Across Different Domains
Beyond scientific and mathematical reasoning, we extended our evaluation to MMLU-Pro (general knowledge) and LiveCodeBench v5 (code generation, 10-2020--02))0).
Table 6: Performance of S-GRPO across different task domains.
MMLU-Pro
LiveCodeBench v$
Model
Acc
Tokens
Acc
Tokens
DeepSeek-RI-
ReepSeek-RI-Distille
Qweeill--wee--B
Vanilla
52.4
3,646
35.8
12,255
S-GRPO
52.4
1,597
35.9
7,692
Qwen3-8B
Vanilla
75.1
4,422
56.2
14,794
S-GRPO 75.2
2,313
56.2
10,203
The results demonstrate that S-GRPO also maintains accuracy while achieving substantial token reduction across other domains. On MMLU-Pro, token usage decreased by 56.2% and 47.7% for the two models respectively, with accuracy preserved or slightly improved. Similarly, on LiveCodeBench, S-GRPO reduced tokens by 37.2% and 31.0% while maintaining code generation performance.
G Limitations and Future works
In the later stages of training, S-GRPO encounters bottleneck: the average reward begins to decline as the response length shortens. This is because the randomly truncated early stopping points become increasingly short, and excessively short CoTs are less likely to generate correct answers. In future work, optimizing the truncation strategy to replace the uniform sampling truncation points could provide improvements.
H Response Length and Accuracy Curve
The Figure 6 below illustrates the response length and accuracy trends on the AIME 2024 dataset different training steps for models trained with the RL with length penalty method and the S-GRPO method. The base model used is
Both methods demonstrate similar downward trend in response length. However, the RL with length penalty method exhibits an earlier decline in accuracy, highlighting the superior performance of the S-GRPO approach.
25

---

AIME 2024
Mean response length
13000
54
12000
Accuracy
52
11000
50
10000
48
9000
RL Length Penalty
8000
RL Length Penalyy
SRGRPO
S-GRPO
0
10
20
30
40
50
60
20
40
60
Step
Step
Figure 6: The response length and accuracy trends the AIME 2024 dataset across different training steps for models trained with the RL with length penalty method and the S-GRPO method.
I More Case Study
To further demonstrate the effectiveness of S-GRPO in comparison to vanilla CoT and its truncated variant, we present two additional examples from the AIME 2024 benchmark in Figure 7 These cases highlight the advantages of S-GRPO in solving complex reasoning tasks with a significantly reduced token budget while maintaining accuracy.
In the first example (top row), all three methods attempt to find the greatest real number satisfying a geometric constraint. The vanilla CoT model produces the correct answer but requires a substantial thinking budget (12,015 tokens). When truncated to match the thinking budget S-GRPO 167 tokens), the vanilla CoT fails reach the correct conclusion. In contrast, S-GRPO successfully identifies the correct solution path and arrives at the correct answer using only 167 thinking budget (7,214 total token budget), demonstrating its ability to achieve efficiency without sacrificing accuracy.
In the second example (bottom row), the task involves calculating the length of geometric segment based on given constraints. Vanilla CoT again fails when truncated to the same thinking budget as S-GRPO (9,333 tokens), producing an incorrect answer. Meanwhile, S-GRPO reaches the correct solution with minimal total token budget (10,836 tokens), significantly outperforming vanilla CoT's full reasoning process, which consumes 28,171 171 tokens without arriving at the correct solution.
These examples further validate that S-GRPO effectively balances reasoning depth and efficiency, avoiding overthinking. By guiding the model concise and accurate reasoning paths, S-GRPO consistently outperforms baseline methods in challenging reasoning tasks.
J More Related Work
Dynamic and early-exit architectures. There are works exploring dynamic neural networks and early-exit mechanisms to improve inference efficiency. Dynamic neural networks [76] provide general framework for adaptively adjusting computation based on input complexity, enabling selective activation of subnetworks or layers during inference. Building upon this concept, LayerSkip [77] introduces inter-layer carly-exit strategies that allow large language models to skip redundant transformer layers dynamically, achieving faster decoding without substantial performance loss. While LayerSkip focuses on inter-layer early-exit, S-GRPO operates at the reasoning-pon pathingg--ttthhht level.
More recently, hybrid approaches such as A Stitch in Time Saves Nine 778 leverage small vi- langulan models guide the early termination of reasoning in larger models, serving as precise external controllers that accelerate computation. These studies collectively demonstrate the potential of adaptive computation and early-exit mechanisms to balance efficiency and accuracy large-seale models, inspiring further exploration into reasoning-level early stopping and adaptive generation. S- GRPO eliminates the need for small model assistance by internalizing early-exit capabilities directly into the reasoning model through reinforcement learning.
26

---

S88, SCS, and be point the hyporbola S18 20000)220) Vihacly (21/04) 18 such that BABCDS mhombss diagonals interpct greateed
by step. purt your
docowel))
Okay, need Owen3-8b Same thinking part Qwen3-8b a Hard Truncation
Qwen3-8b S-GRPO
greatest number
Okay need
the grentedt number
so ined find greanest real number thst lusa than BDr for all themss ABCO on the
that losu than BDr for all such shamss ABCD
thi loss than BD loral such thomts ABCO
the hyperbbla ahot yrok in The monobus has diegonsts internecting at the orgin me try
thh hoperto i sh200 y124 in The if itombu has
topurtolla v120 2/24 m1, where the d digonals imternec the crinn Himm.
diagomals interwecting at the oriin me try to
shart by
visua loe corelltion befone. Sic this condition is cquio oott to diagonals is being
has sldos
visu lze frrs mombus
sides
ling some proper los mombi
regusl twlore So,
given
Tharslor..
C. Thwelloww
dleuur ORf, both and
this condition cqu cquiiilent to diagonals being
hyporbo So, pomaps can paromotito
papertiaatie. Tharefooe, the tont Bhat ABCD
peppestizalee Thersfore, the cored fun that ASCD
paonra A and it euch that OA O8 lrin cantlladoo
as mombus equivteet to as bd C, given
is mombus is equtvalont to as bd O given
venruut let me chock with speatto volues. me
somiited- Therftore the domain ol (ggE))))
connited- Thersfore
agoroaching
compute
/20011
(SH9) (ogrl/54)) aort) into since Sik) oven a (sinos reptacing
(a-y (00 sortic intttut since
opporaches Then
480
doesn't charge
oven in janos roplacing by a docsn't change
Therofore, you, 400 is the infiuum Theraforo,
B00n we can considor a so (Sq (sqn))a)),
apii), wo can considrr k D. so E (901))15))
the answer 460. Therooree liink the answer is Vapawe)/en)) FFrol Arrovw (00 problem, are given ww-thiniooein' Sariirae "2gptt Wieely bas SSASCCOSS ivecribed this hyporto with diagonals intersetiing at
aontittt Toeeore on need minim zn Sik I 12007 1918 fonks (5411566), sertes sie), werle0 then (506 Thaewlore 5i0)
an need mimamiee
Time immicc
miunn and
lng.
180tt 1155t- 5) womitodi that les than BDr for all sach stmmbs Thee The arrense is (oored-440)) leta "Final Arsesse onlosee(eaatli ethinibertet hyper wioeeiit 221(20)
the hyporboia a 57t 21)2)) (2)4))
S. with deagaree a staeowciing at the cam tacio
the origie, We ure is find the greates com dlub--
25st tooks (1so0/1111 in -at
Since 3880 can get artirnrrly doow 480
pioblem are glven 22104) comthet number leas reeggaa pga
Firsi Annworete sni which lehntess
the greatest real
but never actuniyy reaches
the "merim SBD*2S
stricll greater Stiree
number than SanDD195
for all such thobrr aoweeelleoose eodjon
wilbosed 120)
euch momb innerotibossdd 4000/n
Right
X Wrong
Right
Whole token number: 13877
Whole token number: 7624
Whole token number; 7214
Thinking Budget: 12015
Thinking Budget: 6167
Thinking Budget: 6167 Minimum
Conclusion: 1862
Conclusion: 1457
Same budget
Conclusion: 1047
Redaangls glus SABCES and SEFCHI drawn such sD CSFS coliinat, Abo, 917 1945. what length SCITS Prafteee
BBC=166 1066,5AB-1075 5F0
fins whth Wibnsed)
Owen3-8b
Same thinking part Qwen3-8b e Hard Truncation
Qwen3-8b S-GRPO
Okey, Fneed lird the length
gown
Chay need lind the
Gill giowr
Okey, so inedd find lenghs of GI glenn
these rectangles and some on lineer paints. Let
these gloe and some colinwer no nhs. Let
these rectangler and same coll inner points Lat
mc by visualize this list. There are two reclungles les ABCD and
mo my to visualze to this fizz. There are two
me try 30 vitua loc this stst There ale too
The pointa
restanghees ABCD and GHt points C.
restangles, ASCD and dt sight Aloo, points A, sides
The poinse
on same ara gitt Also points
on same dhaight hlso po
the
10r, FG and EF 186. Homm boo lut
10r, FG
BF 186. Horm Fiee
107,
and EF 194, lnad to lind GE wFrrd lel me seetch thls mentally Let me recal
me seatch thls mentally Let mee recal contted
me stetch this mentally Let recal com hect-
E is at 184,00 than F at 0000, which
EE at 184.00 then at 00, witch is
that ructangles have opposite sidea equal and all
just loft of Djo.d, So, the order would E. D,
of Djadd, the would be
Plug Hia, 17% 289
sonisogo
sont
tntthhergfore
1200) 12 1900 top404- 104000 2 P
1201 12 1300 wertl 150044 120100)) /2:
561 G (eat d81)/2)) vnleoom
sn (pp000004) 2 pan 184, mejj Which
se a wy oy 2 (2on- 184, majo. Wiich
beino are hav anedd vnd iee whenees
glves two solution wi 184, 186) - 588.
gives hwo oobutoonteeiitee 184, 165)2 - E6C.
the case above. we had impiional numbors.
105/2
i 2011 220
10512 218.115 rin.
CE 106. Vilo Ansoee (ilbanedd) 104je
10788055 another condition For
CE
orw
tiininn
westthnicsiiiie SS5 SEFGIS
prostem
given
choose herethat
problem analyznng the guomatic contiuaation
to
neall
lolowing propert Vine S&D C. FS5
ont Anower of (SB- Yeg
inviking too gerangiee SASCIIS and SHiGHS
collinerro to- SSA 0GS in on diele to- Skie non tode places point S9558 betweon SSD65
molhimettale solee the problem we are givan bed soctangles. BEABCSSS and BEFG mods-
with the folloing na srown valuestrin tnin SAB 1073,
glven MACCDs cominoo reasonable
SEC
and SCS, and BSP88 boyond SCS, is
givent lengthe
1son1848 (1185 mn winter Findl Answern
1ioot1848 tynsSrinn vinede Final
incorty the
Willboeed 99 (oar1(84011)in
wwlssniaooed 199 (8881(48)))
X Wrong
X Wrong
Right
Whole token number: 28171 Thinking Budget: 26555
Whole token number: 11239
Whole token numberÉ 10836
Thinking Budget: 9333
Thinking Budget: 9833 Minmium Conclusion: 1503
Conclusion: 1616
Conclusion: 1906
budget
Figure 7: Comparison of two generated content samples on AIME 2024.
27


Reviews:
[{"review_id": "CAxaH7n3b2", "reviewer": "Reviewer_6wUr", "summary": "This paper proposes Serial-Group Decaying-Reward Policy Optimization (S-GRPO), a novel reinforcement learning paradigm that incentivizes reasoning models to exit early during chain-of-thought (CoT) generation, addressing the overthinking problem in large language models. By introducing a two-phase rollout procedure and a decaying reward strategy, S-GRPO improves reasoning efficiency and accuracy. Empirical results show significant accuracy gains (0.72%–6.08%) and sequence length reductions (35.4%–61.1%) across diverse benchmarks. The method is compatible with state-of-the-art reasoning models and offers a practical solution for enhancing reasoning efficiency in post-training pipelines.", "questions": "see weakness", "limitations": "see weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths：\nConsistent accuracy improvements \nAblation studies and detailed comparisons with existing methods \n\nWeaknesses:\nIs it possible to directly apply a penalty for output length during the original GRPO training? Does the proposed method demonstrate superiority over this approach?\n\n\nEarly exiting is a common technique in dynamic neural networks and will be discussed in the related works.\nFor example, \n[1]  Dynamic neural networks: A survey (small modles) \n[2]LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (language models)\n[3] A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (multi-model mdoels)\n\nHow many GPUs were used for training?", "was_revised": true, "final_justification": "After reviewing the rebuttal, I will be keeping my rating the same.\n\nThe authors have acknowledged the questions raised in the initial review. However, the rebuttal primarily points to existing results or promises future revisions to the paper rather than providing new evidence or experiments that would address the core concerns.\n\nSpecifically:\n\nW1 (Length Penalties): The authors correctly point out that they have compared their method to length penalty approaches. While they argue their method is more nuanced, the fundamental concern about a simpler baseline remains.\n\nW2 (Related Works): The commitment to expand the related works section is appreciated, but this does not change the substance of the paper as it stands.\n\nW3 (Computational Resources): The clarification on GPU usage is helpful.\n\nOverall, the rebuttal clarifies the authors' positions but does not substantially alter the assessment of the paper's contributions or its limitations. Therefore, my initial evaluation remains unchanged.", "rebuttal": "Thanks for your suggestions. \n## W1\nThe performance of directly applying length penalties for RL.\n## Reply to W1:\nWe sincerely appreciate your question. Actually, the approach of applying a length penalty during RL training has already been included in our comparative experiments (Table 1 in the main text), denoted as \"RL+length penalty\" and \"ShorterBetter\".  Our results demonstrate that while these methods can reduce reasoning length to some extent, they often degrade accuracy, especially on challenging tasks, consistent with findings in papers [4] and [5].\n\nFor instance (Table 1 in the main text), on the DeepSeek-R1-Distill-Qwen-7B model, the accuracy (Acc) of \"RL + Length Penalty\" and \"ShorterBetter\" drops below that of the Vanilla (51.9 vs. 53.3 vs. 55.4) on AIME 2024. In contrast, our proposed S-GRPO not only preserves accuracy but even improves it over the Vanilla (56.0 vs. 55.4), while still achieving a significant reduction in reasoning length.\nThese results suggest that directly applying a length penalty is a relatively coarse control mechanism and may lead to underthinking on complex queries. In contrast, S-GRPO enables more nuanced control, achieving a better balance between efficiency and accuracy.\n\n## W2:\nMore discussion on early exiting in Related Works.\n## Reply to W2:\nWe sincerely appreciate the reviewer's suggestion to discuss the recommended works [1,2,3]. We will incorporate a detailed comparison in the Related Work section. These works indeed share the high-level concept of early-exit, but with fundamental differences in implementation: Compared with [2], while they focus on inter-layer early-exit (e.g., skipping transformer layers), our method operates at the reasoning-path level. Compared with [3], our approach eliminates the need for small model assistance by internalizing early-exit capabilities directly into the reasoning model through reinforcement learning.\n## W3:\nHow many GPUs were used for training?\n## Reply to W3:\nEach training run was conducted on 64 GPUs with 80GB of memory. We have already provided details about the computational resources in Appendix B. We will further clarify this information in the main text.\n\n&nbsp;\n&nbsp;\n\n[1] Dynamic neural networks: A survey\n\n[2] LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding\n\n[3] A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs\n\n[4] Efficient RL Training for Reasoning Models via Length-Aware Optimization\n\n[5] Stable Reinforcement Learning for Efficient Reasoning"}, {"review_id": "6hxO3tQzUq", "reviewer": "Reviewer_65Nr", "summary": "The paper aims to implement an RL algorithm to make the LLM conduct reasoning efficiently. Based on the original group relative policy optimization (GRPO) algorithm, this paper proposes to form the trajectory group along the sequence length dimension, which only samples one reasoning path and serially selects multiple temporal positions from the path to exit thinking and directly generate answers. For correct answers within a serial group, rewards are gradually decreased when the sequence becomes longer. With S-GRPO, the reasoning models exhibit competitive reasoning performance while being much more efficient.", "questions": "From Figure 4, it seems that the model trained after S-GRPO will not output sentence like 'Time is limited, stop thinking and start answering.\\n</think>\\n\\nTo' why does this not happen if the model is trained with every correct trajectories that ended with this sentence during thinking?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The method is simple yet effective.  The method simply conducts multiple early exits along one reasoning path, and uses exponential decay reward for correct answers to make it work. The experimental results show that the accuracies do not drop significantly with reasoning tokens are significantly decreased.\n2. The experimental\n\n## Weaknesses\n1. Inefficiency in implementation. For each query, S-GRPO uses the model to infer one response first, then conduct early exit from different parts of the trajectory. Intuitively, this will roughly double the inference time at each iteration step during training.\n2. The ablation study is a bit too simple. Intuitively, simply adding \"Time is limited, stop thinking and start answering.\\n</think>\\n\\n\" would make the model learns to output this sentence at each trajectory, and the truncation is conducted randomly, is there any more elegant strategy that worth study?", "was_revised": true, "final_justification": "The author's response has resolved my concerns; according to other review comments and their corresponding responses, I tend to keep my score.", "rebuttal": "Thanks for your suggestions. \n## W1:\nDouble inference time during training？\n## Reply to W1:\nWe measure the actual time cost of GRPO and S-GRPO with \"rollout n=8\" on Qwen3-8B. It shows that GRPO's single rollout takes an average of approximately 457s, while S-GRPO's two rollouts take around 550s in total, representing only about a 20% increase in inference time, far below the intuitive estimate of \"doubling\".\nS-GRPO's two rollouts do not double inference time because:\n- The decoding time is proportional to the response length, as it is a length-bounded process. The second rollout only regenerates the conclusion portion, which is substantially shorter than generating the full thinking path + conclusion from scratch. Since decoding time scales with sequence length, this second phase incurs much lower computational cost.\n- The query + thinking path portion uses prefill (rather than decode), which is computationally efficient and doesn't significantly impact overall timing.\n\n## W2:\nInfluence of different prompt templates. & More discussion on truncation ways.\n\n## Reply to W2:\n1. We experiment with different prompt templates for truncating the thinking path. Our results show that as long as the `\\n</think>\\n\\n` is present, the model terminates its reasoning and proceeds to generate a conclusion. Moreover, the specific wording of the prompt hardly affects the results, as shown in the table below.\n\n| Model | Method | Prompt | AIME 2024 Acc | AIME 2024 Tokens | MATH-500 Acc | MATH-500 Tokens |\n| ----- | ------ | ------ | ------------- | ---------------- | ------------ | --------------- |\n| Qwen3-8B | S-GRPO | Time is limited, stop thinking and start answering.`\\n</think>\\n\\n` | 77.3 | 8,810 | 95.2 | 3,166 |\n| Qwen3-8B | S-GRPO | Considering the limited time by the user, I have to give the solution based on the thinking directly now.`\\n</think>\\n\\n` | 77.1 | 8,780 | 95.0 | 3,202 |\n| Qwen3-8B | S-GRPO | `\\n</think>\\n\\n` | 76.9 | 8,823 | 95.3 | 3,154 |\n\n2. Beyond random truncation, we have also explored truncation at specific semantic markers, such as tokens indicating thought shifts like \"Wait\" and \"Alternatively\" [1]. As shown in the table below, these semantic-based truncation strategies achieve similar final performance to random truncation. However, the strategy of truncation at specific semantic markers is less stable during training: As training progresses, the reasoning chains become more compressed and streamlined, leading to a significant reduction in transition tokens like \"Wait\" and \"Alternatively\". This creates a practical limitation where, for certain queries, there are insufficient semantic markers to meet the sampling requirements for the second rollout phase (n samples), making this approach less stable for training. The stability advantage of random truncation, combined with its comparable performance, supports our choice of this strategy in the final S-GRPO implementation.\n\n| Model | Truncation position | AIME 2024 Acc | AIME 2024 Tokens | MATH-500 Acc | MATH-500 Tokens |\n| ----- | ------ | ------------- | ---------------- | ------------ | --------------- |\n| Qwen3-8B | Random selection | 77.3 | 8,810 | 95.2 | 3,166 |\n| Qwen3-8B | \"Wait\" or \"Alternatively\" position | 76.7 | 8,930 | 95.2 | 3,378 |\n\n## Q1:\nWhy are the specific prompts used during training (e.g., \"Time is limited...`\\n</think>\\n\\n`\") not generated during inference?\n## Reply to Q1:\n1. One possible explanation for this phenomenon lies in the nature of reinforcement learning. We have experimented with training using truncated data via supervised fine-tuning (SFT), and under SFT, the model does learn to generate the sentence \"Time is limited, stop thinking and start answering.\" However, this behavior does not emerge under reinforcement learning (RL). During RL training, both positive and negative samples (regarding adv > 0 is positive, adv < 0 is negative) are present. The sentence \"Time is limited, stop thinking and start answering.\" may appear in both types of samples, leading to a gradient tradeoff that prevents the model from learning this pattern.\n\n2. Another possible explanation is that S-GRPO is applied only in the final RL stage on a model that has already been extensively trained. This final RL stage is relatively short and primarily targets optimizing the model's overthinking behavior, rather than inducing significant shifts in its internal generation patterns.\n\n&nbsp;\n&nbsp;\n\n[1] s1: Simple test-time scaling"}, {"review_id": "8ptcNagaSx", "reviewer": "Reviewer_Rkde", "summary": "The paper introduces Serial-Group Decaying-Reward Policy Optimization (S-GRPO), a new RL training paradigm aimed at addressing the overthinking problem in current reasoning models during inference. \n\nInnovations of the Paper\n\n- Serial-Group Generation: Unlike GRPO, which parallelly samples multiple reasoning paths, S-GRPO forms a complete reasoning path. It then truncates this path at different positions to generate a series of intermediate results, constituting a \"serial group.\"\n\n- Decaying Reward Strategy: Rewards are assigned to the answers in the serial group based on their truncation positions, encouraging the model to exit as early as possible.\n\nExperimental results show that when combined with models such as Qwen3 and Deepseek-distill, S-GRPO achieves a sequence length reduction of 35.4% to 61.1% while increasing accuracy by 0.72% to 6.08% on benchmarks including GSM8K, AIME 2024, AMC 2023, MATH-500, and GPQA.", "questions": "none", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n- Innovative Serial-Group and Decaying Reward Mechanism: By serializing intermediate reasoning steps and using decaying rewards, the model is guided to exit early when reasoning is sufficient.\n\n- Improved Efficiency and Accuracy: It significantly reduces reasoning length while enhancing accuracy, outperforming baseline methods (e.g., DEER, GRPO), with notable advantages in complex reasoning tasks (e.g, AIME 2024).\n\n- Adequate Experiments: Effectiveness is validated across multiple mainstream reasoning models (e.g, Qwen3, Deepseek-distill) and in both mathematical and scientific reasoning tasks.\n\nWeaknesses\n\n- Task Domain Limitation: Validation experiments are primarily focused on mathematical and scientific reasoning tasks; effectiveness in other domains remains untested.\n\n- Complex Hyperparameter Tuning: The reward function formula and sampling strategy for truncation positions are relatively complex, requiring separate tuning for different tasks.", "was_revised": true, "final_justification": "The author's response has successfully resolved my concerns. I will keep my score and recommend for acceptance.", "rebuttal": "Thanks for your suggestions. \n## W1:\nEffectiveness in other domains.\n## Reply to W1:\nFollowing the works of [1] and [2], most of the optimization work of the R1-like model uses math and science as the main benchmark. Considering the space of paper, the evaluation of more scenarios was not shown in the main text. We have further evaluated two models, DeepSeek-R1-Distill-Qwen-7B and Qwen3-8B, on both general knowledge tasks (MMLU-Pro) and code-related tasks (LiveCodeBench v5, 2024.10-2025.02), as shown in the table below.\n\n| Model                  | MMLU-Pro Acc | MMLU-Pro Tokens | LiveCodeBench (v5, 2024.10-2025.02) Acc | LiveCodeBench (v5, 2024.10-2025.02) Tokens |\n| ---------------------- | -------- | ----------- | ------- | ---------- |\n| **DeepSeek-R1-Distill-Qwen-7B (Vanilla)**       | 52.4    | 3,646       | 35.8   | 12,255     |\n| **DeepSeek-R1-Distill-Qwen-7B (S-GRPO)** | 52.4    | 1,597       | 35.9   | 7,692      |\n| **Qwen3-8B (Vanilla)** | 75.1    | 4,422       | 56.2   | 14,794     |\n| **Qwen3-8B-SGRPO (S-GRPO)**     | 75.2    | 2,313       | 56.2   | 10,203     |\n\nOur results show that on both the general knowledge task (MMLU-Pro) and the code-related task (LiveCodeBench v5, 2024.10-2025.02), S-GRPO significantly reduces output length while maintaining comparable accuracy, demonstrating effective compression.\n\n## W2:\nNeed complex hyperparameter tuning?\n## Reply to W2:\n1. Compared with GRPO, our S-GRPO introduces minimal modifications in terms of hyperparameters. The main change lies in shifting from the traditional one rollout n setting to a two rollout n setting. In practice, we recommend fixing the first rollout number to 1 (focusing solely on the second rollout n), as the training goal of S-GRPO is to compress the thinking content. We experiment with different second rollout n, as shown in the table below. The results show that varying this hyperparameter (e.g., 8 or 16) yields stable performance across benchmarks, indicating that S-GRPO is not sensitive to this hyperparameter and does not require fine-grained tuning.\n\n| Model | Second rollout n | AIME 2024 Acc | AIME 2024 Tokens | MATH-500 Acc | MATH-500 Tokens |\n| ----- | ------ | ------------- | ---------------- | ------------ | --------------- |\n| Qwen3-8B | 8 | 77.3 | 8,810 | 95.2 | 3,166 |\n| Qwen3-8B | 16 | 76.7 | 8,730 | 95.4 | 3,198 |\n\n2. S-GRPO inherits the core structure of GRPO and introduces only a lightweight extension through a decay-based reward function along the reasoning path. This design is general as the original GRPO and does not require task-specific tuning, making it applicable to a wide range of reasoning compression tasks.\n\n&nbsp;\n&nbsp;\n\n[1] ConCISE: Confidence-guided Compression in Step-by-step Efficient Reasoning\n\n[2] Training language models to reason efficiently"}, {"review_id": "Snj2ze5nnq", "reviewer": "Reviewer_N5Z5", "summary": "The authors propose two techniques to solve the long-answer problem in reasoning models: short responses generation and decaying reward assignment. The method enjoys shorter response length and better result.", "questions": "See Weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The method is novel as far as I know. Inserting a termination sentence in the middle of the response is interesting.\n2. The writing is crystal clear. I enjoy reading the paper.\n3. The method is significant. Long response is a fatal problem for reasoning models.\n4. The quality is good. The authors did solid ablation study to verify the contribution of each part. And the experimental result looks solid.\n\nWeaknesses:\n1. There is one more ablation study the authors may need to do. The whole process in the paper sounds generating the response twice: first getting the reasoning path and truncating it to insert the early termination sentence. This may have twice of the inference cost in training. More generations do have benefit in training.The authors should compare the results with double generation of the responses in training. \n2. It's unclear whether the prompt in termination sentence matter \"Time is limited, stop thinking and start answering.\" Do the authors try different templates? What if the authors add this sentence to the end of normal training of reasoning models and ask the model to generate again?\n3. Math is a relatively easy task. What's the method's performance on code related tasks?", "was_revised": true, "final_justification": "The author's response addressed my concerns. I will keep my score.", "rebuttal": "Thanks for your suggestions. \n## W1:\nDouble inference time during training? & Comparison with double completions.\n## Reply to W1:\n1. We measure the actual time cost of GRPO and S-GRPO with \"rollout n=8\" on Qwen3-8B. It shows that GRPO's single rollout takes an average of approximately 457s, while S-GRPO's two rollouts take around 550s in total, representing only about a 20% increase in inference time, far below the intuitive estimate of \"doubling\". S-GRPO's two rollouts do not double inference time because:\n- The decoding time is proportional to the response length, as it is a length-bounded process. The second rollout only regenerates the conclusion portion, which is substantially shorter than generating the full thinking path + conclusion from scratch. Since decoding time scales with sequence length, this second phase incurs much lower computational cost.\n- The query + thinking path portion uses prefill (rather than decode), which is computationally efficient and doesn't significantly impact overall timing.\n\n2. We conduct the suggested ablation by doubling the rollout number (n→2n) for the \"RL + Length Penalty\" baseline from Table 1 in the main text:\n\n| Model | Method | Rollout n | Generation Time | AIME 2024 Acc | MATH-500 Acc |\n|-------|--------|---|----------------|-----------|----------|\n| Qwen3-8B | RL + Length Penalty | 8 | 450s | 73.8 | 94.2 |\n| Qwen3-8B | RL + Length Penalty | 16 | 763s | 73.5 | 94.3 |\n| Qwen3-8B | S-GRPO | First rollout n = 1; Second rollout n = 8 | 550s | 77.3 | 95.2 |\n\nIt shows that doubling n for \"RL + Length Penalty\" increases generation time by about 70%, but the performance remains similar, still failing to maintain accuracy with length degradation. This demonstrates that S-GRPO's two-phase generation is more efficient and effective than simple scaling of generation quantity.\n\n## W2:\nTry different prompt templates. & Influence of adding \"Time is limited...`\\n</think>\\n\\n`\" to the end of normal training of reasoning models.\n\n## Reply to W2:\nWe experiment with different prompt templates for truncating the thinking path. Our results show that as long as the `\\n</think>\\n\\n` is present, the model terminates its reasoning and proceeds to generate a conclusion. Moreover, the specific wording of the prompt hardly affects the results, as shown in the table below.\n\n| Model | Method | Prompt | AIME 2024 Acc | AIME 2024 Tokens | MATH-500 Acc | MATH-500 Tokens |\n| ----- | ------ | ------ | ------------- | ---------------- | ------------ | --------------- |\n| Qwen3-8B | S-GRPO | Time is limited, stop thinking and start answering.`\\n</think>\\n\\n` | 77.3 | 8,810 | 95.2 | 3,166 |\n| Qwen3-8B | S-GRPO | Considering the limited time by the user, I have to give the solution based on the thinking directly now.`\\n</think>\\n\\n` | 77.1 | 8,780 | 95.0 | 3,202 |\n| Qwen3-8B | S-GRPO | `\\n</think>\\n\\n` | 76.9 | 8,823 | 95.3 | 3,154 |\n\nFor the question of “add this sentence to the end of normal training of reasoning models and ask the model to generate again”:\n1. When using this way during inference,  it is conceptually similar to the think budget truncation method in \"Qwen3 Technical Report\". It could prompt the model to stop thinking and generate a conclusion, but often at the cost of degraded accuracy. As shown in Figure 4 of the main paper and Figure 7 of Appendix E, adding this prompt to a normally trained model can affect the performance, especially when the reasoning process is not explicitly trained to accommodate such abrupt termination.\n2. When using this way during normal GRPO training, it is equivalent to \"S-GRPO without reward decay\", which is shown in the ablation study in the main text (Table 2, Row \"w/o. Decaying (All 1)\"). We also show the results in the table below. The results show that \"w/o. Decaying (All 1)\" results in lengthy reasoning, and our complete S-GRPO is better than \"w/o. Decaying (All 1)\" on all five benchmarks.\n\n| Method | GSM8K |  | AIME 2024 |  | AMC 2023 |  | MATH-500 |  | GPQA |  | Overall |  |\n|--------|-------|-------|-----------|-------|----------|-------|----------|-------|------|-------|---------|-------|\n|        | Acc | Tokens | Acc | Tokens | Acc | Tokens | Acc | Tokens | Acc | Tokens | Acc | Tokens |\n| Qwen3-8B (S-GRPO) | 96.1 | 1,292 | 77.3 | 8,810 | 95.0 | 5,962 | 95.2 | 3,166 | 57.7 | 5,271 | 84.26 | 4,922 |\n| Qwen3-8B (S-GRPO w/o. Decaying (All 1)) | 96.0 | 2,385 | 74.4 | 14,940 | 94.7 | 9,000 | 95.0 | 5,614 | 54.9 | 8,955 | 83.00 | 8,179 |\n\n## W3:\nMath is a relatively easy task. What's the method's performance on code-related tasks?\n## Reply to W3:\nBesides math tasks, we have evaluated science tasks such as the GPQA benchmark in the main text. And we have further evaluated two models, DeepSeek-R1-Distill-Qwen-7B and Qwen3-8B, on both general knowledge tasks (MMLU-Pro) and code-related tasks (LiveCodeBench v5, 2024.10-2025.02), as shown in the table below.\n\n| Model                  | MMLU-Pro Acc | MMLU-Pro Tokens | LiveCodeBench (v5, 2024.10-2025.02) Acc | LiveCodeBench (v5, 2024.10-2025.02) Tokens |\n| ---------------------- | -------- | ----------- | ------- | ---------- |\n| **DeepSeek-R1-Distill-Qwen-7B (Vanilla)**       | 52.4    | 3,646       | 35.8   | 12,255     |\n| **DeepSeek-R1-Distill-Qwen-7B (S-GRPO)** | 52.4    | 1,597       | 35.9   | 7,692      |\n| **Qwen3-8B (Vanilla)** | 75.1    | 4,422       | 56.2   | 14,794     |\n| **Qwen3-8B-SGRPO (S-GRPO)**     | 75.2    | 2,313       | 56.2   | 10,203     |\n\n\nOur results show that on both the general knowledge task (MMLU-Pro) and the code-related task (LiveCodeBench v5, 2024.10-2025.02), S-GRPO significantly reduces output length while maintaining comparable accuracy, demonstrating effective compression."}]

Author rebuttal:

Meta-review: The paper introduces S-GRPO, an RL method that encourages early exit in chain-of-thought generation via serial truncation with decaying rewards. Compared to GRPO, it reduces reasoning length while maintaining or improving accuracy on multiple math/science benchmarks, and, per rebuttal, also on MMLU-Pro (general knowledge) and LiveCodeBench (code).

**Strengths:**

- Clear, simple extension to GRPO with practical impact.

- Consistent reductions in output length with preserved or improved accuracy.

- Solid experiments and ablations; results hold across models and tasks.

- Rebuttal provided broader evaluation and efficiency measurements (~20% overhead, not double).

**Weaknesses**

- Some reviewers felt simpler baselines (length penalties, prompts) could suffice; **novelty** seen as incremental.

- Main text focused on math/science; broader tasks only briefly covered.

- Added complexity in rollout design vs. simpler approaches.

--

The method is practically useful, well-validated, and integrates easily with existing GRPO pipelines. While not a fundamental paradigm shift or a clearly novel methodology, it delivers consistent, meaningful efficiency gains for reasoning LLMs. I recommend poster acceptance (not spotlight) given the incremental nature of the contribution.

**My quick note on discussions and rebuttal:**

- Training cost concern: Authors showed overhead is ~20%, not 2x.

- Prompt/truncation sensitivity: Tested multiple templates.

- Baseline adequacy: Authors compared against length penalties and showed accuracy drops; some reviewers remained unconvinced.

- Domain breadth: Added MMLU-Pro and LiveCodeBench, showing similar benefits.

- Two reviewers moved to clear acceptance, two stayed borderline; overall evidence supports acceptance.
Decision: Accept (poster)