Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty
Authors: Zewei Yu
Lirong Gao
Yuke Zhu
Bo Zheng
Junbo Zhao
Sheng Guo
Haobo Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Efficient Reasoning
Reinforcement Learning
Adaptive Coordinated Penalty
Primary area: reinforcement learning

Abstract: Large Reasoning Models (LRMs) have demonstrated remarkable performance on complex reasoning tasks by employing test-time scaling. However, they often generate over-long chains-of-thought that, driven by substantial reflections such as repetitive self-questioning and circular reasoning, lead to high token consumption, substantial computational overhead, and increased latency without improving accuracy, particularly in smaller models. Our observation reveals that increasing problem complexity induces more excessive and unnecessary reflection, which in turn reduces accuracy and increases token overhead. To address this challenge, we propose Adaptive Reflection and Length Coordinated Penalty (ARLCP), a novel reinforcement learning framework designed to dynamically balance reasoning efficiency and solution accuracy. ARLCP introduces two key innovations: (1) a reflection penalty that adaptively curtails unnecessary reflective steps while preserving essential reasoning, and (2) a length penalty calibrated to the estimated complexity of the problem. By coordinating these penalties, ARLCP encourages the model to generate more concise and effective reasoning paths. We evaluate our method on five mathematical reasoning benchmarks using  DeepSeek-R1-Distill-Qwen-1.5B and  DeepSeek-R1-Distill-Qwen-7B models. Experimental results show that ARLCP achieves a superior efficiency-accuracy trade-off compared to existing approaches. For the 1.5B model, it reduces the average response length by 53.1% while simultaneously improving accuracy by 5.8%. For the 7B model, it achieves a 35.0% reduction in length with a 2.7% accuracy gain. The code is released at https://github.com/ZeweiYu1/ARLCP.

Published as a conference paper at ICLR 2026
STOP UNNECESSARY REFLECTION: TRAINING LRMs FOR EFFICIENT REASONING WITH ADAPTIVE REFLEC- TION AND LENGTH COORDINATED PENALTY
Zewei Yu Lirong Gao
Yuke Zhu?
Bo Zheng* Junbo Zhao!
Sheng G Haobo Wang
1 State Key Laboratory of Blockchain and Data Security, Zhejiang University
2 MYbank, Ant Group
3 Innovation and Management Center, School of Software Technology (Ningbo), Zhejiang University
(yuzewe ju, gaol i rong, j. zhao, wanghaobo) j.j..edu edu.u............
fel ix. yk bbababaabae -nc. nc. com guargyuandenttgroup. com
guosheng1 001 gma com
ABSTRACT
Large Reasoning Models (LRMss) have demonstrated remarkable performance on complex reasoning tasks by employing test-time scaling. However, they often generate over-long chains-of-thought that, driven by substantial reflections such repetitive self-questioning and circular reasoning, lead to high token consump- tion, substantial computational overhead, and increased latency without improving accuracy, particularly in smaller models. Our observation reveals that increasing problem complexity induces more excessive and unnecessary reflection, which turn reduces accuracy and increases token overhead. To address this challenge, we propose Adaptive Reflection and Length Coordinated Penalty (ARLCP), a novel reinforcement learning framework designed to dynamically balance reason- ing efficiency and solution accuracy. ARLCP introduces two key innovations: (I) a reflection penalty that adaptively curtails unnecessary reflective steps while pre- serving essential reasoning, and (2) length penalty calibrated to the estimated complexity of the problem. By coordinating these penalties, ARLCP encourages the model to generate more concise and effective reasoning paths. We evalu- ate our method on five mathematical reasoning benchmarks using DeepSeek-R1- Distill-Qwen-Il 1.5 and DeepSeek-R1 -in-SSek-----iiiill------- --iistill--wen--71 models. Experimental re- trade-off com-
sults show that ARLCP achieves a superior efficiency-accuraa pared to existing approaches. For the 1.5B model, it reduces the average response length by 53.1% while simultancously improving accuracy by 5.8%. For the 7B model, it achieves 35.0% reduction in length with a 2.7% accuracy gain. The code is released at https//ggthuu.com///eewwuuuuuuue http://ewwiYYl//R.....
INTRODUCTION
Large Reasoning Models (LRMs). such as OpenAl ol (OpenAl 2024), QwQ (Qwen Team 2025
and DeepSeek-R1 (Guo et al., 2025). have demonstrated exceptional capabilities in complex rea- soning tasks. When tackling challenging problems, these models employ long chain-of-thought rea- soning with self-reflective mechanisms, systematically exploring multiple solution pathways while generating extensive reflective reasoning traces. This iterative reasoning framework enables them to perform significantly better than conventional LLMs. Typically, the reasoning process in these LRMs explicitly organized with <th i and </think> tags, which separate internal thinking from final outputs, offering transparent insight into its multi-step reasoning process.
Although overlong reasoning improves the performance of LRMs, it also brings severe efficiency challenge with substantial token usage and computational overhead, limiting practical deployment
in real-time or resource-constraine settings (Qu et al. 2025 Yue et al. 2025 Yang et al. 2025c).
Coorreponning Authors

---

Published as a conference paper at ICLR 2026
Existing research has mainly explored two directions address this efficiency challenge. The first is training-fie free inference-stage optimization, such as Early Exit (Yang et al. 2025b; Qiao et al,
2025 Xu et al., 2025) and Model Switch (Liao et al. 2025 Ong al. 2025 Yang al.) (025c) These methods do not change the model's inference capability or distribution, but only optimize the generation process with early stopping pruning. thus offering limited efficiency gains for
redundant models or complex tasks. The second direction is training LRMs under length-penalg, guidance using supervised fine-tuning (Jiang et al. 2025 Xia al.)
2025
Huang et al. (025) or
Reinforcement Learning (RL) (Arora & Zanette| 2025] Zhang et al., 2025
Aggarwal & Welleckj
2025). While such methods enhance control over reasoning length, they often sacrifice reasoning
quality-for for example, by suppressing reflection or discarding the entire thinking process-which negatively affects answer accuracy. Therefore, dynamically balancing length and accuracy remains valuable direction for achieving efficient reasoning.
Based on detailed analysis of the reasoning process, we find that LRMs, particularly smaller ones, frequently produce redundant reasoning steps, such as repetitive self-questioning loops ("wait"") unproductive hesitations ("hmm"), and circular reflections that fail to advance task resolution, as also evidenced in (Yang et al. 2025c Ghosal al.. 2025). Furthermore, analysis of the reflective behaviors of LRMs during the reasoning process. Defining complexity as a model-aware measure of problem difficulty, we observed (as shown in Figure [and and Figure 2 that as the complexity of the problem increases, this over-reflection phenomenon becomes more severe, incurring significant costs in inference time and computation. More critically, even after prolonged over-reflection during the reasoning process, LLMs still fail to yield correct answers.
conduct an in-depth
Motivated by these observations, we propose Adaptive Reflection and Length Coordinated Penalty (ARLCP), a dynamic reinforcement learning method that enables LRMs to balance accuracy and efficiency in reasoning. The core of ARLCP reduce unnecessary reflection behaviors of LRMs during reasoning while preserving accuracy, thereby reducing token consumption and improving efficiency. Unlike prior approaches that naively truncate reasoning steps or apply static penalties, ARI.P introduces two key innovations: (1) a reflection penalty that adaptively stops unnecessary reflection while preserving critical reflection processes, and (2) a length penalty that reduces the out- put tokens, calibrated to problem complexity. By dynamically adjusting reflection tokens, ARLCP automatically adjusts penalty weights to maintain accuracy while minimizing token overhead.
We evaluate ARLCP on five mathematical reasoning benchmarks using DeepSeek- R1 R--Distill-- -wenl- 1.5B and DeepSeck- RI-itstilll Prtistill-Qeee--17 7B as the base models. The experimental results demonstrate that ARLCP achieves significant efficiency-accurage improvements: average response length by 53.1 1% while improving accuracy by 5.8%, and on the 7B model, it achieves 35.0% response-length reduction with 2.7% accuracy improvement. This consistent performance across different model scales validates the effectiveness of our approach in balancing reasoning efficiency and solution quality.
the 1.5B model, it reduces
In summary, our key contributions are as follows:
We identify the phenomenon of over-reflection in reasoning models, where redundant unpro- ductive reasoning steps degrade inference efficiency and repurpose such signals inversely to ter- minate unnecessary reasoning proactively.
We introduce ARLCP, reinforcement learning method that dynamically adjusts reflection and length penalties based online complexity estimates.
We evaluate ARLCP across multiple math datasets, demonstrating superior performance- efficiency trade-off': our method significantly reduces token consumption while preserving or improving accuracy, outperforming existing approaches for efficient reasoning.
REALATED WORK
Large Reasoning Models. Following Open-ol (OpenAl 2024), researchers have developed ad-
vanced reasoning models through detailed rewards and bearch-based methods (Qwen Team 2025) Notable approaches include mutual learning between models (Qi et al., 2024). example-gun guided search (Zhang et al.] 2024),, and MCTS--nteeraan self-play for self-correcting reasoning (Zhao et al.
2024). The release of DeepSeek-R (Guo et al.) 2025) further popularized "RI-style" models that achieve multi-step reasoning and self-reflection using only simple rule-based rewards (Team et al.
2

---

Published as a conference paper at ICLR 2026
2025
Yang et al. 2025a). However, overthinking behaviors significantly increase computational
costs, driving active research into efficiency reasoning.
Efficient Reasoning for LRMs.
Most existing methods improve the efficiency of LRM fo- cus on reducing response tokens. Training-free approaches include prompting with token budgets [Muennighoff al.
2025 Aytes et al. 2025), model switching (Liao al. 2025 Fan al. 2025
Yang et al. 2025c). and early exit mechanisms Yang et al. |025b; et al. 2025). Other strate-
gies use Supervised Fine-Tuming (SFT) with compressed Chain-of-Thouugh (CoT) data (Jiang al.
2025) or length- data from sampling and post- processsn (Shen et al. 2025 2025 Yu et al.
Rafailov et al. 2023). Furthermore, Reinforcement Learning (RI. techniques often incorporate
length-based rewards (Arora & Zanette 2025 Luo et al. 2025a Liu et al., 2025a) adapt other
rewards (Zhang et al., 2025] Aggarwal & Welleck 2025). While different, these methods fail to
dynamically adapt response length to problem's 's intrinsic complexity mitigate model's over- reflection tendencies. Our work addresses this gap by introducing a synergistic penalty mechanism that incorporates reflection and length penalties achieve efficient reasoning.
MOTIVATION AND OBSERVATION
This section examines common patterns that regularly appear during models' reasoning processes. By carefully studying these patterns, we hope to find practical mechanisms to improve the efficiency of the models.
Reflection correlates with problem complexity. During inference, observe that reasoning models frequently generate specific reasoning- supportive tokens such "wait", "hmm", and "alternatively". which are intrinsically linked to the model's internal behavior. To quantitatively analyze these patterns, we conduct study mea- suring average reflection token counts across Drepseck-Distilled--eeee 1.5B and 7B on mul- tiple math datasets with an increased complexity level, as shown in Figure T Notably, the num- ber of reflection tokens positively correlates with dataset complexity, demonstrating their sensitiv- ity to problem complexity.
140
1.58
120-
Average Reflection Token
100
80
60
40-
20-
Level
Level
Level
Level
Level
Figure
Average reflection token counts statis-
tics.
18000
180
100
L4000
16000
2 2.4x
14.525
Average Reffecttion Count
1.0
-.2x
135.86
333-25
78
1.58
12000
Accuracy (50)
60-
10000
1.00
800D
6000
5.15
80
40
4000
xo
41.00
2000
ro
156
78
1.58
78
1200
22-2000
DeepSeek-IL1 (0l1--iitii---- models
DeepSeelk Si--eek--L--Distll---wwwe
Reflection Token Co Count Interval
Aug Tokens
Aug Tokens
Reflection Count
Roffection Count
Figure 2: Output statistics of two models on the AIME 2024-2025 datasets. The reported metrics include average response tokens (left), average reflection token count (medium) for both correct and incorrect answers, and accuracy trend with different reflection token count intervals.
Over-reflection leads to incorrect responses and inefficiency. To further investigate the relation- ship between reflection tokens and model performance, we analyze the performance of DeepSeek- RI Sirillled--wen---.. and 7B in terms of response length and reflection tokens for both correct answers and incorrect answers and the accuracy trend with different reflection token count intervals AIME 2024-2025 datasets, as shown in Figure 2 The results indicate that correct responses

---

Published as a conference paper at ICLR 2026
exhibit significantly shorter average token lengths and fewer reflection tokens than incorrect ones. And smaller reasoning models (1.5B) require substantially longer responses while exhibiting lower accuracy than larger models (7B). Additionally, the number of reflection tokens increases, model accuracy declines. These results indicate that while some reflection is necessary, unproductive or ex- cessive reflection-a phenomenon term "over-reffection" traps models in inefficient and often incorrect exploration of the solution space.
Key insights and motivation. Our observations highlight three key insights:
Rellection complexity indicator: Higher reflective token counts correlate with increased prob- lem complexity, enabling dynamic assessment and adaptive reward design.
Over-reflection risks: Incorrect responses often exhibit excessive reflection, trapping models in inefficient solution-spaat space exploration.
Balanced intervention needed: Low-complexity tasks still require length penalties for their low- reflection behaviors, while complex tasks require strategies to stop unnecessary reflections.
Motivated by these observations, we propose ARLCP, a reinforcement learning method that com- bines two key components: (I) dynamic reflection penalty adjusted based on problem complexity and (2) a length penalty to ensure an overall penalty. By balancing these two penalties, our ap- proach systematically reduces token usage while maintaining improving model accuracy through optimized reasoning processes.
4 METHODOL.OGY
In this section, we introduce Adaptive Reflection and Length Coordinated Penalty (ARLCP) algo- rithm, which consists two essential components: (1) an adaptive reflection penalty based on the complexity of problems. (2) length penalty ensures the overall penalty in low-reflection situation.
4.1 PRELIMINARY
Given an LRM M, an input prompt x = [......, Tn. <think>>, where [æı...., 22]] represents the problem context and <think> serves as the special token initiating the reasoning process, M generates a response y= ..,..... y/, </think>, U1+2...., -mm]- Here, [Vi...., 1t] represents the thinking phase, constituting an extended chain of exploratory reasoning, reflection, and self- validation. while < </think> explicitly terminates this process. [ 3mm] is the subsequent solution segment which contains only the validated steps and final answer to the problem. Since the generation is auto-regressive, meaning that given a prompt and tokens S [y1...., Vkl generated so far, next token y*1 generated from the conditional probablity distribution M( r, yex). The whole auto-regressive steps can be decomposed as:
T
( II
M MM Veck)
The auto-regressive generation process terminates when the reasoning model M outputs the end-of- sequence (BOS) token. Therefore, in the response y = [yt..... M/, </think>, 2t42, ym]. the final token ym. always corresponds to the EOS token.
4.2 ADAPTIVE REFLECTION AND LENGTHI COORDINATED PENALTY (ARLCP)
Our proposed approach ARLCP is reinforcement learning method that adaptively imposes flection penalty according to the complexity of each problem, supplemented by length penalty, allowing the LRR flexibly reduce unnecessary reflection and thus minimize token consumption. Based on empirical observations in Section 3 we identify three key findings: (1) reflection intensity correlates with problem complexity; (2) incorrect solutions exhibit excessive reflection patterns that hinder efficient reasoning; (3) static reflection penalty is ineffective for low-reflection problems.
The complexity of each problem X. Motivated by the above findings, we first estimate the model-aware complexity of each problem based on the Reflection Token Counts (RTC) of the
4

---

Published as a conference paper at ICLR 2026
Algorithm ARLCP (Adaptive Reslection and Length Coordinated Penalty)
Input: Policy model M; dataset D; hyperparameters m, R, R2, A1, X2, As
1: Initialize: Ground truth answers o" (P.) for all P. E
for training step , .M
3: Sample a batch D3 from '>
4: For each Pi E Do. generate m candidate responses of ~ P.) using sampling
for of batch do
6: Extract LEN(of) (response token count), RTC(o,) (reflection token count via keyword matching). and (N
7:
Compute correctness C(o,)) + 1(ANS(o)) = o o'
8:
end for
Collect correct responses Veernea = [o,1C((0)
10:
Compute Pæ = mean (RTC()oomee) )) on std(RTC() Corneet
11:
Compute fz = mcan(L.EN() (oomec)))) )) 0L = std(LEN () (ocome)))
12:
for all of batch do
Normalize penalties: S(RTC) = [
S(LEN) = LENCOFD-HE
13:
A1,
if RTC((,)) S
14:
Determine aj via thresholding: aq
X2.
if mm
< RTC(o S n2
A3,
if RTC(O))) > n2
01 Calculate reward r(o)) = C(o.) - (1- on - 02 (LEN))
Compute ag
16:
17:
end for
18: Update M using policy gradient update
19:
end for
Output: Optimized policy M with adaptive penalty control
response. Specifically, for each prompt P: in the batch, the LRM generates candidate roll- outs o2 = [ot, of .e.,,"") using standard sampling. Each reasoning trajectory of consists of: (1) LE indicating the response length tokens. (2) (TC( indicating the reflection tokens count, based reflection-trigger keyword matching, shown in Appendix [A.4 The complexity of the problem À categorized into three levels: simple, moderate, and hard through threshold-based segmentation: simple with weight \ when RTC(O,)) does not exceed the lowest threshold n,, mod- erate with weight X2 when RTC(o,) falls between nı and the moderate threshold n2, and hard with weight Ag when RTC(o)) exceeds the highest threshold n12-
Reflection penalty. Then, we propose an adaptive mechanism that dynamically imposes reflec- tion penalty according to the complexity of each problem A. This mechanism allows the LRM to flexibly adjust its reasoning process, discouraging unnecessary reflection on simpler problems while permitting more extensive reasoning for complex ones. Formally, we define the reflection penalty coefficient an, which is modulated by the estimated problem complexity X, as follows:
X1, if (TRCC(O))
X2, if if < <RTC(o)) Snz;
01 =
As iF
For reasoning trajectory of its reflection penalty f(RTC(o/)) can be caleulated as:
((RTC())) =o( RTC(, mean mean0mmmaaa((CCCC (RTC(o:)))omeer
(3)
std(RTC((o)) correct
Here, MecanRTC((( o.)) connett and (RdC(((.)) conrect are the mean and standard deviation of reflection tokens whose answers are correct. respectively. or is the sigmoid function.
Length penalty. While the reflection penalty effectively discourages excessive reflective behav- iors, it may not fully constrain other forms of unnecessary verbosity that do not directly stem from reflection. Therefore, further introduce length penalty based on the total token count LEN(o,)
5

---

Published as a conference paper at ICLR 2026
of the generated output. This complementary penalty encourages the LRM generate overall more concise responses, suppressing both redundant reflection and any additional irrelevant or verbose content. The length penalty is calculated as:
(() LEN( mean (reannLEE(oo))ccco.
(4)
std(LEN((:) correct
The length penalty coefficient ag of of is defined as 0g = a - 01, where a is the overall penalty coefficient. This coefficient allows the total penalty be flexibly allocated between reflection and length penalties according to the complexity of each problem.
Building the above, for each rollout of the composite reward function jointly optimizes for accuracy and efficiency, and is formulated as follows:
r(o²)) = C(o)) (1 -1-a0 a1 ((TT((()) a2f (LEN(
(5)
where the A represents the extracted answer from the solution segment and o (px) be the ground truth of p,. Hence, the accuracy reward C(of)) is calculated by:
c(o,) = [(ANS(o)) - o" (p:)}.
(6)
The adaptive penalty strategy maintains accuracy while effectively stopping unnecessary reflec- tions and reducing the response tokens. During each training step, the LRM samples m roll- outs o; = [o!, of o! Once sampling finishes, correct responses are calculated to compute mean(RTC(o:)) correct. sc((((())
(LEN(LEN(o.)) correct. and std(LEN((:)) correct. Accuracy rewards, reflection penalties, and length penalties are then calculated for each response, while coeffi- cients a and ar are determined based on the response complexity. The overall penalty is computed accordingly. Finally, the advantage of each response is estimated using online RL algorithms (lever- aging multiple responses) for policy gradient update.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
LRMs. We conduct all experiments on DeepSeek-RR-Distill--eeesss Qwen-1.5B and DeepSeek-R1- Distill- Distill-Qwen-7B Qwen-7B, two popular reasoning models obtained through supervised fine-tuning on large- scale high quality distillation.
Datasets. The training dataset we use is DeepScaleR (Luo et al. 2025b) dataset, which is a col- lection of 40K unique math problem-answer pairs compiled from: AIME problems (1984-2023), AMC (prior to 2023), Omni-MATH (Gao et al. 2025), and Still dataset (Min et al. 2024). The training dataset does not overlap with our evaluation benchmark dataset. For evaluation, we use five widely recognized math benchmarks with increasing complexity: GSM8K (Cobbe et al. 202T) test set (1319 grade school math problems), MATH500 (Lightman et al.) 2024) (500 high-school competition math problems), AMC2023 (40 more challenging high-school level math competition problems), AIME 2024 and AIME 2025 (30 Olympiad-level math problems).
Evaluation metrics For evaluation metrics, we consider both pass@1 accuracy (Acc) and sponse length (Length). We also report the average accuracy variation, A Acc, and the average length variation rate, ALength(%% across all test datasets. Considering the limited size of AMC2023, AIME2024, and AIME2025, we repeatedly sample 16 responses for each case and report the aver- age results. For all models, we set the evaluation context size to 16K, and set the temperature 0.6 as default in DeepSeek's 's models.
Implementation Details. Our implementation integrates the VeRL framework (Sheng et al. 2025 with REINFORCE Leave One Out (RLOO) policy optimization method (Ahmadian et al. 2024) While GRPO (Sao et al. 2024) has gained popularity as a policy optimization approach, we avoid its due to two critical limitations: (1) GRPO demonstrates sensitivity in non-standard settings involving length penalties in the objective function, and (2) this sensitivity can trigger abrupt policy collapses, empirically observed in recent studies Arora & Zanette 2025 Dai et al. 2025). These

---

Published as a conference paper at ICLR 2026
Table I: Performance comparison of different methods multiple reasoning benchmarks. Metrics include accuracy (Acc), response tokens (Length), and relative changes (AAcc, ALength), The best and second results are bolded and underlined, respectively.
AMC 2023
AIME 2024
AIME 2025
GSM8K Acc Length
MATHI-500
Overall
Method
Acc
Length
Acc
Length
Acc Length
Acc Length
AAcc ALength(S)
DeepSeek RI Distilll-Quii Distill-Qwen-I.5B
Vanilla
66.72
7742
30.00
12256
21.40
12201
78.46
1009
80,20
4582
NoThinking
49.22
1377
14.38
4266
9.79
3630
69.98
278
69.20
908
-12.84
-81.04%
SFT
67.66
7657
26.04
12181
21.46
12258
79.38
928
81.00
4531
-0.25
-2.07%
DPOsheres
69.53
7091
29.60
11721
23.13
11879
77.10
989
84,20
4185
1.36
-5.21%
Ol-Pruner
70.47
7046
27.71
11937
20.83
11855
78.39
913
82.60
4375
0.64
-5.69%
TLMRE
72.10
2798
25.80
5386
19.60
4581
84.30
530
82.10
1800
1.42
-58.10%
AdupiThinkk
67.19
3342
30.83
6864
22.50
6626
84.23
488
83.20
1869
2.23
-51,47%
LASER
75.94
3899
28.75
7629
25.42
7084
82.26
850
84.60
2365
4.04
-38.69%)
ARLCP(ours))
73.28
3037
34.17
5951
26.46
5196
87.34
658
84.60
1812
5.81
-53.05%
DeepSeek- RI RI-Distill- Ri-Disilll-ween-2
Vanilla
87.50
5980
51.46
10547
35.00
11427
87.34
694
91.20
3673
Nothinking
61.25
1179
23.96
2671
16.25
2499
84.68
284
80.60
706
-17.21
-74.59%
SFT Showar
85,80
600S
49.17
10575
38.96
11384
86,43
606
89.00
3575
-0.69
-3,09%
88.90
5295
50.42
9899
39.58
10692
87.87
624
90.80
3340
0.95
-8.64%
O1-Pruner
87.66
5509
51.46
10034
38.33
10793
86.80
625
91.20
3422
0.53
-7.01%
TLMRE
88.12
3560
52.08
6393
35.63
6584
88.78
866
91.20
2419
0.60
-26.32%
AdaptThink
86.88
3641
56.25
8784
37.71
9622
90.29
330
91.00
1862
1.87
-34.68%
LASER
89.45
2894
56.45
5995
39.38
6410
86.73
818
91.00
1871
2.04
-33.97%
ARLCP(ourr))
89.69
3260
56.67
6795
39.58
7560
89.31
578
91.00
2086
2.69
-34.96%
instability risks motivate choice of RLOO, which maintains robustness under length-penalizer objectives. The details of selecting RLOO are shown in Appendix A.2 All experiments are con- ducted 8 NVIDIA A100 GPUs with prompts using model-specific templates for problems (see Appendix A.6). context length of 16K tokens, batch size 128, learning rate 2e-6, and ARLCP pa- rameters m 16,
0.05, A2 0.1, A3 0.15, n 40, n2 80, 0.2. The details of selecting 11 and n2 are shown in Appendix A.7
5.2 BASELINES
We compare ARLCP with the following representative methods for efficient reasoning:
1. Nothinking (M et al. 2025) enables reasoning models to bypass long thinking and directly generate the final solution by prompting with </think>, We use this method to determine the lower bound of the response token for each dataset and do not include it in the comparisons.
2. SFT Shortes constructs training data by sampling multiple responses for each problem and selecting the two shortest correct responses, followed by the standard SFT pipeline for model fine-tuning.
3. DPO generates preference data by pairing the shortest correct response with the longest responses per problem through multiple sampling, then applies DPO (Rafailov et al. 2023 for fine-tuning.
4. O1-Pruner (Luo et al., 2025a) employs pre-sampling to estimate reference model performance, followed by off-policy RL-style fine-tuning to optimize shorter reasoning processes while main- taining accuracy constraints.
5. TLMRE (Arora & Zanette 2025 introduces length- based penalty during on-policy RL training to explicitly incentivize shorter response generation.
6. AdaptThink (Zhang et al. 2025) enables reasoning models to adaptively select optimal thinking mode based on the problem complexity by on-policy RI.
7. LASER (Liu et al. 2025a) propose unified view for RL-based CoT compression, unifying vari- reward shaping and truncation methods. Building on this view, they introduce new approaches with adaptive length-based reward shaping.
All baselines are re-implemented using DeepScaleR dataset ensure fair comparative evaluation.

---

Published as a conference paper at ICLR 2026
2:5 2224
CANDE
NOTA N22
$500
1000
1350
4590
1400
1630
1300
600
422
3500
1000
7530
THAN Co0T1
Tranding Kinge
Tuaniig Blegs
Trcking Stngs.
Kingo
Thetning Steps
Figure 3: The analysis of accuracy, length, and reflection for model responses five benchmarks (AMC 2023, AIME 2024, AIME 2025, GSM8K, and MATH 500) across different training steps.
5.3
MAIN RESULTS
the evaluation results of various methods on five mathematical reasoning bench- marks (AMC 2023, AIME 2024, AIME 2025, GSM8K and MATH500), Compared the vanilla and SeepSeek-R1-Ditiinn -1----sill----ee--- 7B models, ARLCP achieves re- markable improvements by reducing average response lengths by 53.1% 1% and 35.0%. respectively, while enhancing average accuracy by 5.8% and 2.7%. On -eapsSee-RR---siil--------eee ARI.C achieves the highest overall accuracy (AAcc 5.81) while reducing the average response -i-isee-----siisi----ee-eeeee
ARLCP maintains significant accuracy gain (AAce = 2.69) and the greatest reduction in response length (34.96%) %).. The exper- imental results of ARLCP across both DeepSeek-RR RI--RI--DD -Dsstill-Qeent
length by 53.05 %. Similarly, for
and DeepSeck-RI-Diste Qwen-7B demonstrate a clear trade-off between reasoning accuracy and response length.
Furthermore, ARLCP excels particularly in complex tasks, attaining best results competition- level benchmarks (AMC 2023 (73.28%), AIME 2024 (34.17% %). AIME 2025 (26.46%) )) for the unnec-
1.5B model. This suggests that, compared simpler problems, LRMss tend generate essary reflective behaviors when dealing with complex tasks. Thus, the key reason for the significant improvement lies in adaptive reflection penalty mechanism, which dynamically adjusts penalty strength according to problem difficulty, enabling more effective suppression of unnecessary reflec- tions in complex tasks. ARLCP allows efficient reasoning without compromising solution quality, achieving the balance between accuracy and efficiency through coordinated penalty design.
To evaluate the effectiveness of our approach, we deeply track the changes in response length and the number of delletion-triigeringg words throughout the training process. As shown in Figure 3 both the average count of reflection words and the output length gradually decrease as training pro- gresses, while accuracy correspondingly improves. These results indicate that our proposed adaptive reflection penalty successfully encourages the LRM to eliminate unnecessary reflective behaviors and produce more concise outputs. Consequently, this not only reduces superfluous token consump- tion but also significantly lowers inference costs, underscoring the practical value of our method for efficient large-scaa scale deployment.
5.4
MORE ANALYSIS
The analysis of reflection behavior. To evaluate the efficacy of our approach in mitigating unnec- essary reflective behaviors, we conducted comparative analyses between ARLCP and the original models across multiple math datasets, presented in Figure The results demonstrate that our model significantly reduces the average reflection token count, effectively stopping unnecessary re- flective behaviors in the model.
The impact of two core penalty. To verify the cooperative effect of length penalty and reflection penalty in ARLCP, we conducted an ablation study on the 1.5B LRM by separately removing each component. When removing the reflection penalty, fixed the length penalty parameter 02
8

---

Published as a conference paper at ICLR 2026
1.58
Varilla AAM C2
1191
100
Varila
Reffacciion Takan Count
AR
88.7
120
51.9
40.0
67%
89
132
111
GOMMK
AMEEREC
0GMMX
Mams
AMO
Figure 4: Comparison of average reflection token count between vanilla models and ARLCP.
Table 2:
The impact of adaptive penalty components on the performance of LRM. The dynamic
penalty yields the best balance between improved accuracy and reduced response length in LRM.
Method
Length Penalty og
Reflection Penalty 01
AAce
A
ARLCP
Adaptive(V)
Adaptiee(
5.8
-53.1%
w/o Length Penally
x
/
5.6
-29.6%
w/o Reflection Penalty
0.05
X
1.4
-58.1% %
Fixed Coefficients
0.1
0.1
5.5
-47.5%
0.05; when removing the length penalty. we retained the original adaptive parameter 01. As shown in Table[]] removing either component led performance degradation, confirming their cooperative roles in optimizing reasoning efficiency. We further analyze the impact of the adaptive coefficients on model performance. We fix the adaptive coefficients an and 02 0.1 and train the base 1.5B model. The training results indicate that when coefficients are fixed, the model performance de- grades, demonstrating the effectiveness of the adaptive coefficients in our approach.
Table Performance of different parameter settings.
Complexity Thresholds (M,, 02)
Aaoe
ALength(S5
20, 40
3.36
-00.2%
60, 100
2.67
-19.7%
40, 100
3.36
-20.0%
40, (ours)
3.58
-21.7%
Penalty Weights (Ai, de, 23,
Aace
ALengthi %)
Different a
0.1
3.48
-9.5%
0.3 0.2 (ours)
2.26
-30.7%
3.58
-21.7%
Different À
0.025. 0.05, 0.075 0.1, 0.15,
3.39
-18.2%
3.08
-18.8%
0.05, 0.1, 0.15 (ours)
3,58
-21.7%
The sensitivity analysis. To explore the impact of the complexity thresholds (n1, 122) and penalty weights (2,,A2,4 Ag.a) a) on model performance, we evaluate various parameter configurations of model (trained for 100 steps). The results,
ARLCP applied to the summarized in Table demonstrate that:
Impact of n1 and n2: Increasing n, leads slight initial improvement in performance, fol- lowed by a decline; the best performance is achieved at n, = 40. In contrast, varying n2 have a relatively minor effect performance, with the optimal setting found n2 80.

---

Published as a conference paper at ICLR 2026
Impact of a:: A smaller leads to lower reduction in output length and lower accuracy, whereas a larger a yields greater length compression but at the expense of more noticeable drop in accuracy. Our chosen strikes good balance between the length compression ratio and accuracy. Impact of Åي X2, A3: Furthermore, with a fixed, either increasing or decreasing the overall magnitude of the À parameters causes only minor fluctuations in performance, and the overall performance remains at stable level.
The performance of ARLCP on the out-of-distributtion benchmark MMLU. ARLCP achieves the best balance on MMLU, improving accuracy while significantly reducing response length compared to other baselines.
Table 4:
MMLU
Method
Ace
Length
AAce
ALength
DeepSeek-RR RIa---R--Diiiii Qwen-7B
Vanilla
63.4
1257
Nothinking
51.2
128
-12.2
-89.8%
SFT
62.8
1321
-0.6
+0.05%
AdaptThink
63.6
856
+0.2
-31.9%
TLMRE
63.9
872
+0.5
-30.6%
ARLCP
64.1
742
+0.7
-110%
The generalizability to different domains. To evaluate cross-domain generalization, we assess ARLCP MMLU, challenging benchmark comprising 14k multiple-choice questions spanning 57 diverse subjects. As shown in Table compared to the vanilla -eanSeee-RR--isiill---eeeee ARLCP reduces response length by 40% while achieving a 0.7% improvement in accuracy. These results demonstrate that ARLCP not only effectively transfers beyond mathematical reasoning to a broad range of open-domain tasks, but also consistently enhances reasoning efficiency across all question types. Importantly, the ability to maintain even improve accuracy while significantly shortening responses highlights the robustness and generality of the proposed adaptive penalty mechanisms. This suggests that ARLCP is well-suited for practical deployment in diverse, real- world settings where both efficiency and accuracy are critical.
Table 5: Performance comparison of ARLCP on multiple reasoning benchmarks with Qwen3-1.7B.
2023
AIME 2024
AIMR 2025
GSMSK
MATH-500
Overall
Method
Acc
Length
Ace
Length
Acc
Length
Length
Length
Acc ALength(%)
Qwen3-1.7B
Vanilla
75.16
8484
38.75
13138
288.54
13287
90.67
2094
87.00
5096
ARI.CP(our))
79.69
5320
42.92
9466
32.992
9755
89.99
1057
89.60
2568
3,00
-38.19%
The generalizability to different series models. To evaluate the generalizability of our approach across model families, we extended our experiments to the Qwen3 series-using Qwen3-1.7B as representative example-which lies outside the DeepSeek-Ri Boacsk----DDiiill-Oo 1-Diistill-ween family. As shown in Table 55) ARLCP consistently improves accuracy while substantially reducing output length across nearly all benchmarks, demonstrating strong transfcrability beyond the original model family.
CONCLUSION
In this work, we first identify the phenomenon over-reflection reasoning models significantly impacting model efficiency and analyze its correlation with problem complexity. Based on this in- sight, we propose ARLCP, a dynamic reinforcement learning method that teaches LRMs to stop unnecessary reflection and enhance thinking efficiency through an adaptive reward strategy condi- tioned on problem complexity. Experiments on five mathematical reasoning benchmarks demon- strate that ARLCP reduces average response length while improving accuracy.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have provided open-source code to reproduce all experiments described in this study (ARLCP). The datasets are publicly accessible, and all experimental tation of model training, evaluation protocols, and reproducibility benchmarks-can be replicated using the provided resources.
full implemen-
ACKNOWLEDGMENT
This paper is mainly supported by the NSFC under Grants (No. 62402424). This work sup- ported by MYbank, Ant Group. Haobo Wang also supported by the Fundamental Research Funds for the Central Universities (No. 226-2025-00085 & 226- 226-2024-00145 and Zhejiang Provincial Uni- versities (No. 226-2025-00065), This work is also supported by Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China.
REFERENCES
Pranjal Aggarwal and Sean Welleck. LI:
Controlling how long reasoning model thinks with reinforcement learning. In Second Conference on Language Modeling, 2025.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learn- ing from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), ACI. 2024, Bangkok, Thailand, August 11-16. 2024, pp. 12248-12267. Association for Computational Linguistics, 2024. doi: 10.18653/V 120AAALLLL2DAAAAAAALGGGG.G..
Daman Arora and Andrea Zanette. Training language models reason efficiently. CoRR, bbss2502.044463 2025, doi: 10:4555/ARXXW2200000444
Simon A. Aytes. Jinheon Back, and Sung Ju Hwang. Sketch-of-thought: Efficient LLM reasoning with adaptive cognitive-inspired sketching. In Proceedings of the 2025 Conference on Empirical Association for Computational Lin- Methods in Natural Language Processing, pp. doi: 10.18653/v 10133////222.oooo wet.:5.emlllpmmmm main. 1236. guistics. November 2025. ISBN 879-8-89176-333
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 10. 14168, 2021.
Muzhi Dai, Shixuan Liu, and Qingyi Si. Stable reinforcement learning for efficient reasoning. CoRR, abs/2505. 18086, 2025. doi: 10. 10448550/ARXI..2505..8 18086.
Siqi Fan, Peng Han. Shuo Shang, Yequan Wang, and Aixin Sun. Cothink: via instruct models guiding reasoning models. CORR, abs/2505. 220077,, 2025. doi: 10.48550/
reasoning
ARXIV.2505.22017.
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni- math: A universal olympiad level mathematic benchmark for large language models. In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, and Amrit Singh Bedi. Does thinking more always help? understanding test-time scaling in reasoning models. CoRR, 0bs/25506.42.0,, 2025. doi: 10.48550/ARXIV.25066 .04100
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu. Runxin Xu, Ruoyu Zhang, Shirong Ma. Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhi- hong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng.
11

---

Published as a conference paper at ICLR 2026
Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, Hao Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. I.. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang. Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, Tao Wang. Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nic. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi. Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yonggiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang. Yaohui Li, Yi Zheng, Yuchen Zhu, Yunx- ian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl incentivizes reasoning in Ilms through rein- forcement learning. Nature. 645(808 )):63666,, 2025. doi: 10. N0.1038/S41588 586-025-09422-Z.
Jian Hu. REINFORCE++: A simple and efficient approach for aligning large language models. CoRR, abs/2551.032665 2025, doi: 10.48550/ARXIV.5500
Chengsong Huang. Langln Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. Efficient fest-time scaling via self-calibration. In NewrIPS 2025 Workshop Efficient Reasoning, 2025.
Yuxuan Jiang, Dawei Li, and Frank Ferraro. DRP: distilled reasoning pruning with skill-aware step decomposition for efficient large reasoning models. CoRR, abs/2505.13975, 2025. doi: 10.48550/ARXIV.5500.
Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. Reward seward-guided speculative decoding for efficient LLM reasoning. In Forty- second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net, 2025.
Hunter Lightman. Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, and Junxian He. Learn to reason efficiently with adaptive length-based reward shaping. CoRR, doi: 00.48850/ARXI..2555.. 15612.
abs/2505.15612, 2025a.
Zhaowei Liu, Xin Guo, Fangqi Lou, Lingfeng Zeng, Jinyi Niu, Zixuan Wang, Jiajie Xu, Weige Cai, Ziwei Yang, Xueqian Zhao, Chao Li, Sheng Xu, Dezhi Chen, Yun Chen, Zuo Bai, and Liwen Zhang. Fin-rl: A large language model for financial reasoning through reinforcement learning. CoRR, abs/2503. 16252, 2025b. doi: 1048550/ARXI.2550.66635
Haotian Luo, Li Shen, Haiying He, Yibo Wang. Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: Length-harmonizing fine-tuning for ol-like reasoning pruning. CoRR, abs/2501. 125700, 2025a. doi: 00.48550/AXXIV2551..222
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing ol prepreview with a 1.5 model by scaling rl. Notion Blog, 2025b.
12

---

Published as a conference paper at ICLR 2026
Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking. CoRR, abs/2504.09888, 2025. doi: 10.48550/ARXIV. 2504.09858.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng. Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji- Rong Wen. Imitate, explore, and self-improve: reproduction report on slow-thinking reasoning systems. CoRR, abs/2412.09413, 2024.
doi: 10.45500ARRXIV244200993
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel J. Candès, and Tatsumori Hashimoto. sl: Simple test-time scaling. CoRR, abs/2501.19393, 2025. doi: 10, 1044550/ARXXI.250111922
Isaac Ong, Amjad Almahairi. Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025.
OpenAl. Learning to reason with 1lms. Saaming-tto-resoon with 1tthhhlmmm// 2024. Accessed: 2025-05-07.
https::/oppnnttt cww/coo///ooooo. /ooo//nddx/////
Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller Iims stronger problem-solvers. CoRR, abs/2408.06195, 2024. doi:
10.4850ARKXV24080000111
Ziqing Qiao, Yongheng Deng, Jiali Zeng, Dong Wang, Lai Wei, Fandong Meng, Jie Zhou, Ju Ren, and Yaoxue Zhang. Concise: Confidence-guidee guided compression in step-by-step efficient reasoning. CoRR, 0as/2505.04885 2025. doi: 10.45500ARXXI.2555..00 04881.
Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, Peng Li, Wei Wei, Jing Shao, Chaochao Lu, Yue Zhang, Xian- Sheng Hua, Bowen Zhou, and Yu Cheng. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. CoRR, abs/2503.21614, 2025. doi: 10.48550/ ARXIV.2503.21614.
Qwen Team.
Owq-32b-prewt
https:
gwq 32b-preview 2025. Accessed: 15 March 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1700.063477 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300. 2024. doi: 10.48550/RXXV..402..55
Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, and Shiguo Lian. DAST: difficulty- slow-thinking for large reasoning models. CoRR, abs/2503. 044472, 2025. doi: 10.45550/ARXI.22503.000
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. In Proceed- ings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 April 2025, pp. 1279-1297. ACM, 2025.
13

---

Published as a conference paper at ICLR 2026
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuming Tang. Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang. Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong. Weiran He, Weixiao Huang. Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang, Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du Yuxin Wu, Yuzhi Wang. Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang. Zhilin Yang. Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with Ilms. CoRR, 0es/2501.12999 2025. doi: 10.48550/ARXI..5500.02
Heming Xia, Chak Tou Leong, Wenjie Wang, Yongqi Li, and Wenjie Li. TokenSkip: Controllable chain-of- thothtt compression in LLMs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 3351-3363. Association for Computational Lin- guistics, November 2025. ISBN 979-8-89176-3322
doi: 1..18663/112225.mmmlp---
Yuhui Xu, Hanze Dong. Lei Wang. Doyen Sahoo, Junnan Li, and Caiming Xiong. Scalable chain of thoughts via clastic reasoning. CoRR, abs/2505.00315 2025. doi: 10.48550/ARXV..55500000
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang. Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, abs/2505 093888,, 2025a. doi: 10448550/ARXIV.55555.0 099388
Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Zheng Lin, Li Cao, and Weiping Wang. Dynamic early exit in reasoning models. CoRR, abs/2504. 15895, 2025b. doi: 10.48550/ARXIV2504.55599
Wang Yang, Xiang Yue, Vipin Chaudhary, and Xiaotian Han. Speculative thinking: Enhancing small-model reasoning with large model guidance at inference time. CoRR, his/25004.13299 2025c. doi: 10. 1044550/ARRXIV2504.222
Bin Yu, Hang Yuan, Haotian Li, Xueyin Xu, Yuliang Wei, Bailing Wang, Weizhen Qi, and Kai Chen. Long-short chain-of-thought mixture supervised fine-tuning eliciting efficient reasoning in large language models. CoRR, abs/2505.03469 2025. doi: 10.48550/ARXI..2505.055
Linan Yue, Yichao Du, Yizhi Wang, Weibo Gao, Fangzhou Yao, Li Wang, Ye Liu, Ziyu Xu, Qi Liu, Shimin Di, and Min-Ling Zhang. Don't overthink it: A survey of efficient rl-slyl large reasoning models. CoRR, 0tw/2508.0212222 2025. doi: 10. 10.48550/ARXIV.25088000 02200...0.......
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: LLM self-training via process reward guided tree search. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Process- ing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Jajje Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. AdaptThink: Reasoning models can learn when to think. In Proceedings of the 2025 Conference on Empirical Methods in Natu- ral Language Processing, pp. 3716-3730 Association for Computational Linguistics, November 2025. ISBN 979-8-89176-332-6 doi: 10. 10.1665//1120025eemlpmmmo
14

---

Published as a conference paper at ICLR 2026
Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tiaqqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. Marco-ol: Towards open reasoning models for open-ended solutions. CoRR, abs/2411.14405. 2024. doi: 1048500ARXIV.41111000.
A
APPENDIX
A.1
OPTIMIZING THE ADAPTIVE PENALTY WITH REINFORCEMENT LEARNING
The adaptive penalty in ARLCP (Equation tion/length penalties. However, autoregressive sampling during response generation renders this penalty -inddffferentible. To address this, we employ Proximal Policy Optimization (PPO) (Schul- man et al. 2017). a policy gradient method that maximizes the expected reward while constraining policy updates to avoid instability.
5
combines accuracy rewards with dynamic reflec-
PPO Objective with Density Ratio PPO optimizes the policy by maximizing the surrogate ob- jective:
min ()) A(v <k x), diplit [18 x]] Aly <k ,,),
where the density ratio is defined as:
fh (v.x)= = To no(s"10 y Told(y"< yzk)
(8)
RLOO Advantage Estimator Traditional PPO uses a value network to estimate advantages, which adds computational overhead for large language models. Instead, we adopt the REINFORCE Leave-One-Out (RLOO) estimator (Ahmadian et al. 2024), which computes advantages via Monte Carlo sampling. For a prompt P with ma generated responses
10", O".., the trajectory
advantage is computed as:
m
A P) = R(O,P)
Σ R(OiP)) 17i
where R(O", denotes the trajectory return. In ARLCP, the trajectory return corresponds directly which integrates accuracy, reflection, and length
the reward function r(ot) from Equation 5 penalties.
We approximate the laken-level advantage by directly using the sequence-leve advantage:
A(y5*,z) =:00') P).
(10)
This framework balances computational efficiency and performance by eliminating the value net- work while retaining robust policy updates through RLOO.
A.2 THE DETAILS OF SELECTING RLOO
Although RLOO can produce high-variance gradient estimates, we choose it over the more popular GRPO due to critical limitation of the latter: GRPO is prone to catastrophic collapse when com- bined with length-base based rewards (Arora & Zanette 2025 Dai al., 2025). As shown Table 6
our method also fails when using GRPO. To avoid this failure mode, we adapt RLOO and design our reward function accordingly.
To mitigate the theoretical variance of RLOO,, we use standard strategies: large batch sizes (128) and multiple rollouts per prompt (16). Additionally, our penalty function provides regularization that further stabilizes training (see Equation 3). As shown in Figure 3 training remains stable throughout, confirming that RLOO an effective choice for our setting successfully avoiding the critical failure modes of GRPO.
Moreover, RLOO is not essential to our framework. To verify this, we implemented ARLCP using REINFORCE+ (Hu 2025) alternative advantage estimator. As shown in Table 6 both the RLOO and REINFORCE++ variants achieve strong performance on DeepScek-Re -i--stlll- Qwen- 1.5B, demonstrating that our adaptive penalty mechanism is compatible with different policy
15

---

Published as a conference paper at ICLR 2026
Table 6: Performance comparison between different RI. methods.
Method
A acc
A Length( %)
ARLCP with GRPO
-2.11
-65.2%
ARLCP with REINFORCE+
4.87
-50.4%
ARLCP with RIOO (ours)
5.81
-53.1%
gradient algorithms. This flexibility enables ARLCP to effectively balance accuracy and conciseness across various RL advantage estimators, with the RLOO-based implementation yielding the best overall results.
A.3 ANALYSIS OF REFLECTION- TOKEN COUNTS VERSUS PROBLEM COMPLEXITY ACROSS OTHER REASONING MODEL FAMILIES
To assess whether the correlation between reflection and problem complexity generalizes beyond single model family, we examine the reflection behavior of GPT-OSS and Qwen3-Thinking across varying levels of problem complexity. As shown Figure both models exhibit the same trend ob- served in the DeepSeck-distilled Qwen models: the number of reflection tokens gradually increases with rising problem complexity. This result confirms that the relationship between reflection and problem complexity is not specific to one architecture, but rather holds across diverse reasoning models.
175
Average Reflection Token Count
GPT-OSS-20B
150
Qwen3-4B-Thinking
125
100
75
50
25
0
Level
Level
Level 3
Level
Level 5
Figure Average reflection token counts statistics other models.
A.4 REFLECTION TRIGGERS
We consider the following reflection-trigger keywords to calculate reflection token counts in our work, and their variants are also treated as reflection triggers to ensure comprehensive coverage. For instance, the variants of "wwit" include "wait", "Wait", wait", ".wait", and "WAIT". Similar expansions are applied to other reflection -rrigge keywords to account for capitalization, spacing, and punctuation variations. These tokens carefully curated to comprehensively capture reflection occurrences in the output, thereby ensuring both computational accuracy and analytical precision. TTe reflection-trigger keywords include:
16

---

Published as a conference paper at ICLR 2026
"wait", "alternatively". "hold on",, "another thought", "verify", "think again", "but", "how-
ever", "alternative", "cceek"", "double-check" "oh", "hmm""
A.5 THE ANALYSIS OF FIXED REFLECTION TRIGGERS
Although our reflection keyword list is fixed, we emphasize that this approach highly scalable within the current LRMs ecosystem. The list is derived through LLM-based reflection token recog- nition and statistical analysis of outputs from the DeepSeek- ---eek-------silllll DistiillQweer models (1.5B and 7B).
To validate our claim, analyzed reflection tokens in other LRM families and domains (such as finance), including Qwen3-4B- -ei--iiiiiiin--000 GPT-OSS-201 and Fin-RI-7B Liu et al. (0025b)
The most frequent reflection keywords for each model are shown follows:
Reflection keywords 20w------hiii--22555
"wait", "hold on", "maybe", "check", "think again", "another thought", "verify", "but", "hmm", "alternative".
Rellection keywords GPT-OSS-20B
"maybe", "wait", "but", "alternative", "check", "however", "hmm", "double-check"" "an- other thought".
Reflection keywords Fin Fi7-RI-7B
"wait", "alternative", "but", "check", "however", "maybe", "perhaps", "hmm".
All three sets exhibit strong overlap with predefined triggers. This cross-model consistency confirms that our reflection triggers captures universal reflection behaviors in modern LRMss. Con- sequently, using fixed set does not hinder the generalizability of our method across different models or domains.
A.6 PROMPT TEMPLATE IN EXPERIMENTS
Following prior work Zhang et al. (025)), use the following prompt for all experiments: question. To ensure consistency across different settings, this base prompt is applied uniformly every evaluation. Specifically, the complete prompt integrated with the models' templates SeepSeek-RI-Distil--PP
and DeepSeck-RI-Distee Satiill-Qen--7B0 is constructed as follows:
Prompt Template for Reasoning in Experiments
serl
(question) Assistant)
>< think \n
A.7 THE DETAILS SELECTING COMPLEXITY TURESHOLLD (n 112)
We determined the complexity thresholds and n2 through a pilot study on five benchmarks: GSM8K, MATH-500, AMC2023, AIME2024, and AIME2025, Using both DeepSeek-PR -I--stiil--- Qwen- and 7B models, we measured the average number of reflection tokens per problem and found values to lie the range [0, 120], as shown Figure Based on this empirical distribution, defined a three-tier complexity scheme:
Easy: < count < 40,
Moderate: 40 < count < 80,
Hard: count > 80.
This scheme is model-agnostic: to adapt it new model, only needs to inference a representative set problems across complexity levels, estimate its reflection token distribution, and align the thresholds accordingly.
17

---

Published as a conference paper at ICLR 2026
Question (MATH-500): Define
p
and
K3
k=1
Find a way to write
20
k+
in terms of p and q- Original Response:
<think> Alright, so have this problem here where I need to express a double sum in terms of known series, p and q. Let write down what know: [omited][...]
But I don't t think need the numerical value here; it's just another constant.
Hmm, okay, so I have manipulate this double sum somehow relate it to p and q. [omited][...]
Wait, so for each fixed n. goes from n-1, and n n- [omited][...]
Wait, is that right? [omited][...]] But wait, 1/n is independent of the inner sum is just (n I) * 1n2. [...] But I know that Σx=1 2 p. so -n N = p - 1. [omited][...]
Wait, let me double-check my steps make sure I didn't make mistake. [...[oomit 2000 tokens]
But wait, the integral [omited][...] Wait, yes, [omited][...] Wait, hold on, [omited][...] Wait, now have two different results: [omited][ [[..] Hmm, so which one is correct? Let me test with numerical compuuationooommtdllll.... [..][oomit 6000 tokens]
Therefore, think that the second method has mistake, perhaps in the substitution or in han- dling the integrals. Alternatively, maybe should accept that the first method is correct and S p-q. Therefore, the answer is S p 4. #ininal * p </think> [omited][[..] Thus, the final answer is:
(9713 tokens, 115 reflection tokens count)
ARLCP's Response (ours):
<think> Alright,, so have this problem here where I need to express a double sum in terms of known sums, p and q. Let me write down what know. [omitted][-..] but maybe don't need that exact value right now. Similarly, q the sum from k=1 to infinity of gz That's known as Apéry's constant, but again, maybe [omitted][...] Hmm, [omitted][ Wait, is that right? Let check. [omitted][...] But since 1/n is independent of the inner sum is just adding I/na (n 1) times. [omitted][[..] Therefore, the answer is p q. Let me just double-check my steps to make sure didn't make any mistakes. [omitted][[..] **Final Answer** [omitted][....] Therefore, the final answer is:
</think>
(1479 tokens, 8 reflection tokens count)
Figure 6: For a math problem from MATH-500, tokens in thinking, which contains many unnecessary reflections. In contrast, ARLCP-7B effec- tively reduces unnecessary reflections and produces concise final solution.
costs about 9000
A.8 CASE STUDY
We present a case study ARLCP in Figure demonstrate its capability in mitigating unnecessary reflections. As shown in the figure, when tackling challenging problems, the DeepSeek-B ---isttil--
18

---

Published as a conference paper at ICLR 2026
Qwen-7B model generates significant token costs during reasoning, often accompanied by redundant and repetitive reflections. contrast, our ARLCP-7B model dynamically stops unnecessary reflec- tions through adaptive mechanisms, generating concise solutions while maintaining accuracy. This comparative analysis highlights the effectiveness our approach optimizing both computational efficiency and solution quality.
A.9 LLM USAGE STATEMENT
This paper utilized Large Language Models (LLMs)) as general-purpose tool for language polish- ing and grammar refinement during writing. The role of L.MSs is limited to enhancing the clarity and readability of the manuscript, including optimizing sentence structure and correcting errors in linguistic expression. No part of the core research methodology. data analysis, or theoretical con- tributions generated or influenced by LLMs. The authors confirm that this disclosure complies with the conference's ' guidelines for transparency in LLM usage.
19


Reviews:
[{"review_id": "w7chyjSFoX", "reviewer": "Reviewer_rUtM", "summary": "This paper addresses the excessive and unnecessary reflection problem in Large Reasoning Models by proposing an adaptive reflection and length coordinated penalty (ARLCP). The method first estimates the problem's complexity by counting the Reflection Token Counts in a generated response, then assigns a penalty weight based on whether the problem is categorized as \"simple,\" \"moderate,\" or \"hard.\" During RL training, a second penalty is applied to the overall response length to discourage general verbosity. The method achieves a superior efficiency-accuracy trade-off, simultaneously reducing length and improving accuracy.", "questions": "- Have you tried directly using RLOO on the training dataset? I'm curious why this method achieves short outputs but still has the best performance.\n- Did you face any instability during training? Does RLOO truly prove to be more stable?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work directly addresses the \"over-reflection\" in Large Reasoning Models (LRMs), a key issue that leads to high token consumption, computational overhead, and latency without improving accuracy.\n- The core idea of an adaptive reflection penalty is a key innovation.\n- The thorough analysis and ablation show that the method improves both efficiency and accuracy.", "weaknesses": "- The core \"Adaptive Reflection Penalty\" relies entirely on a manually curated list of \"reflection-trigger keywords\" like \"wait\", \"hmm\", \"alternatively\", etc. This relies on a specific model response style and is not generalizable to other distributions or domains.\n- The method introduces several new and important hyperparameters: the complexity thresholds ($n_1=40, n_2=80$) and the penalty weights ($\\lambda_1=0.05, \\lambda_2=0.1, \\lambda_3=0.15, \\alpha=0.2$). The paper presents these as fixed values but does not include a sensitivity analysis.\n- The method uses the output Reflection Token Count (RTC) as a proxy for the input problem's complexity. The paper itself shows that incorrect answers (i.e., failed reasoning) lead to higher reflection. This means the model might be penalizing \"hard\" for a problem that is actually \"simple\" but on which the model got stuck and \"over-reflected.\" It's penalizing the symptom (high RTC) rather than the cause (intrinsic problem difficulty).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pMyQIH2ohL", "reviewer": "Reviewer_AdYW", "summary": "This paper identifies the problem of **over-reflection** in Large Reasoning Models, where repetitive self-questioning causes inefficiency without accuracy gain. It proposes **ARLCP (Adaptive Reflection and Length Coordinated Penalty)**, a reinforcement learning method that dynamically adjusts reflection and length penalties based on problem complexity. Experiments on several math benchmarks show that ARLCP significantly reduces reasoning length while improving accuracy, achieving a better efficiency–accuracy balance than previous length-penalized approaches.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Strengths**:  \n1. The paper is well-written and clearly structured, making it easy to follow.  \n2. It identifies a practical issue—over-reflection in large reasoning models—and provides a targeted solution.  \n3. The proposed ARLCP method is simple yet effective, adaptively balancing reflection and reasoning length.  \n4. The experiments on multiple benchmarks demonstrate clear improvements in both accuracy and efficiency.", "weaknesses": "typo line 448: analysis -> analyze\n\nWeaknesses:  \n1. The method appears potentially sensitive to the hyperparameters n₁ and n₂. It would be helpful if the authors could analyze how these values affect stability and performance.  \n2. The paper lacks clear guidance or heuristics for selecting n₁ and n₂, which seem crucial to the method’s effectiveness.  \n3. The work assumes that longer reasoning tends to produce incorrect answers, yet the analysis mainly shows correlation rather than causation. It remains possible that more difficult problems naturally require longer reasoning.  \n4. The paper could better position itself relative to prior studies on difficulty dynamics in reinforcement learning for efficient reasoning, such as [1].  \n\n[1] Learn to Reason Efficiently with Adaptive Length-based Reward Shaping. https://arxiv.org/abs/2505.15612", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UT8aBRdmtr", "reviewer": "Reviewer_dg2c", "summary": "The authors observe that reasoning models often over-reflect, producing repetitive self-questioning and circular reasoning that lead to excessive token usage and computational overhead. These reflective behaviors are typically signaled by words such as wait, hmm, or check again. Their analysis shows that as problem complexity increases, models tend to generate more unnecessary reflection.\nTo address this, the authors propose estimating problem difficulty based on the frequency of reflection keywords. They introduce a reinforcement learning framework that adaptively applies both a reflection penalty and a length penalty. This approach encourages efficient reasoning by discouraging redundant thought patterns while maintaining problem-solving effectiveness.\nExperiments using DeepSeek-distilled models demonstrate reductions in reasoning length and improved accuracy compared with existing methods.", "questions": "1. Does the proposed method also enable a non-reasoning model to acquire reasoning ability? The experiments only evaluate reductions in reasoning length for models that already possess reasoning capability.\n2. The authors demonstrate that the adaptive penalty works effectively with RLOO. Would the approach generalize to other reinforcement learning techniques?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of estimating problem difficulty through the frequency of reflection keywords is novel and interesting.\n2. The method is well-motivated, conceptually sound, and shows empirical effectiveness.\n3. The writing is clear and the presentation is good.", "weaknesses": "1. The method may not generalize well to other model families. It assumes that reflection behaviors are expressed through specific keywords, which is a strong and model-dependent assumption. It would strengthen the paper if the authors extended the analysis of reflection-token counts versus problem difficulty to other reasoning model families (GPT-OSS, Qwen 3 Thinking, etc.).\n2. All experiments are conducted on a single model family (DeepSeek-distilled Qwen), limiting the generality of the findings.\n3. The paper's contributions are incremental. Prior studies [1,2] have already established the correlation between reflection tokens and reasoning performance, and existing works [3,4] have explored the use of length penalties to encourage concise reasoning.\n\nReferences\n\n[1] Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning\n\n[2] s1: Simple test-time scaling\n\n[3] ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models\n\n[4] Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ONrWSu4nlj", "reviewer": "Reviewer_FW4W", "summary": "This paper addresses the problem of computational inefficiency in Large Reasoning Models (LRMs) by identifying and targeting a specific failure mode over reflection, which is a pattern of repetitive and unproductive reasoning. The authors propose a novel reinforcement learning framework, Adaptive Reflection and Length Coordinated Penalty (ARLCP), designed to train models to produce more concise yet accurate reasoning paths.  It proposes a method based on empirical evidence linking over-reflection to incorrect answers, simultaneously improving reasoning accuracy while significantly reducing response length.", "questions": "1. The paper states the goal is to \"permit more extensive reasoning for complex ones,\" but the implementation penalizes reflection most heavily on problems with the highest RTC. Can you clarify this apparent contradiction between the stated goal and the reward mechanism? \n\n2. The reflection penalty relies on a fixed keyword list. Have you analyzed whether the model learns to circumvent this penalty using synonyms, and how do you see this lexical approach scaling to other models with different linguistic patterns?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper offers an insightful framing of the efficiency problem by identifying over-reflection as a distinct, harmful behavior. The empirical analysis shows that incorrect answers correlate with higher reflection and token counts, providing a solid motivation for penalizing this behavior to improve both accuracy and efficiency.\n\n2. The ARLCP framework is elegant and intuitive. Using the model’s own Reflection Token Count (RTC) as a real-time complexity proxy is a smart, self-contained design that removes the need for external classifiers. The adaptive penalty that balances reflection and response length based on complexity is a well-motivated and effective idea.\n\n3. The experiments are thorough and convincing. ARLCP consistently outperforms strong baselines across multiple reasoning benchmarks and model sizes. The 1.5B model’s 5.8% accuracy gain with a 53% length reduction is a particularly impressive result.\n\n4. Ablation studies and OOD tests (e.g., MMLU) demonstrate robustness and transferability. The results suggest that over-reflection is a general issue in reasoning models and that ARLCP provides a practical, broadly applicable mitigation.", "weaknesses": "1. The reliance on RTC as a proxy for complexity is a fragile heuristic. It can misclassify simple but confusing problems and, paradoxically, penalize the most reflective (and supposedly complex) cases most heavily—undermining the intended adaptivity.\n\n2. The use of a manually curated list of ``reflection keywords'' is brittle and easily gamed. Models can paraphrase to avoid penalties, making this more of a short-term engineering fix than a scalable solution.\n\n3. The paper introduces several crucial hyperparameters (e.g., $n_1, n_2, n_3, \\lambda, \\alpha$) that drive the penalty mechanism, but no sensitivity analysis is provided. This limits confidence in the method’s robustness and reproducibility.\n\n4. The justification for using RLOO is incomplete. While alternatives are said to be unstable, the paper doesn’t discuss RLOO’s known high variance and potential training instability. Clarifying this trade-off would strengthen the technical argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)