Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
Authors: Wei Liu
Ruochen Zhou
Yiyun Deng
Yuzhen Huang
Junteng Liu
Yuntian Deng
Yizhe Zhang
Junxian He
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Large Reasoning Models
Efficient Reasoning
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Large Reasoning Models (LRMs) have shown remarkable capabilities in solving complex problems through reinforcement learning (RL), particularly by generating long reasoning traces. However, these extended outputs often exhibit substantial redundancy, which limits the efficiency of LRMs. In this paper, we investigate RL-based approaches to promote reasoning efficiency. Specifically, we first present a unified framework that formulates various efficient reasoning methods through the lens of length-based reward shaping. Building on this perspective, we propose a novel **L**ength-b**A**sed **S**t**E**p **R**eward shaping method (LASER), which employs a step function as the reward based on target length. LASER surpasses previous methods, achieving a superior trade-off between performance and efficiency. Next, we further extend LASER based on two key intuitions: (1) The reasoning behavior of the model evolves dynamically during training, necessitating reward specifications that are also adaptive and dynamic; (2) Rather than uniformly encouraging shorter or longer chains of thought (CoT), we posit that length-based reward shaping should be difficulty-aware i.e., it should penalize lengthy CoTs more for easy queries. This approach is expected to facilitate a combination of fast and slow thinking, leading to a better overall tradeoff. The resulting method is termed LASER-D (**D**ynamic and **D**ifficulty-aware). Experiments on five open-weight models from 1.5B to 32B demonstrate that our approach significantly enhances both reasoning performance and response length efficiency. For instance, LASER-D achieves a **5.3** improvement on AIME2024 while reducing token usage by **64**%. Further analysis reveals that our RL-based compression produces more concise reasoning patterns with less redundant ``self-reflections''. All resources (Models, Code, Data)
are available at https://github.com/hkust-nlp/Laser.

Published as a conference paper at ICLR 2026
LEARN TO REASON EFFICIENTLY WITH ADAPTIVE LENGTH-BASED REWARD SHAPING
Wei Liu'> Ruochen Zhou?
Yiyun Deng' Junxian He
Yuzhen Huang! Junteng Liu'
Yuntian Deng' Yizhe Zhang'
'The Hong Kong University of Science and Technology UUniversity of Waterloo
"City University of Hong Kong
*Apple
ABSTRACT
Large Reasoning Models (LRMs) have shown remarkable capabilities in solving complex problems through reinforcement learning (RL), particularly by generating long reasoning traces. However, these extended outputs often exhibit substantial redundancy, which limits the efficiency of LRMs. In this paper, we investigate RL-based approaches to promote reasoning efficiency. Specifically, we first present a unified framework that formulates various efficient reasoning methods through the lens of length-basee reward shaping. Building on this perspective, we propose a novel Length- bAsed StEp Reward shaping method (LAsER), which employs step function as the reward based on target length. LASER surpasses previous methods, achieving superior trade-off between performance and efficiency. Next, we further extend LASER based on two key intuitions: (1) The reasoning behavior the model evolves dynamically during training, necessitating reward specifications that are also adaptive and dynamic; (2) Rather than uniformly encouraging shorter or longer chains of thought (CoT), we posit that length-basee reward shaping should be lifficulty-wwar i.e., it should penalize lengthy CoTs more for easy queries. This approach is expected to facilitate a combination of fast and slow thinking, leading to a better overall tradeoff. The resulting method is termed LASER-D Experiments on five open-weight models from 1.5B to 32B demonstrate that our approach significantly enhances both reasoning performance and response length efficiency. For instance, LASER-D achieves a +5.3 improvement
(Dynamic and Difficulty-aware
AIME2024 while reducing token usage by 64%. Further analysis reveals that our RL-based compression produces more concise reasoning patterns with less redundant "self-reflectionss.. All resources (Models, Code, Data) are available at httpps//gtthub..com///kusss/////
INTRODUCTION
Recent advancements leveraging reinforcement learning (RL) (DeepSeek-AL et al. 2025 Kimi et al. 2025 OpenAl et al.) 2024 Zeng et al. 2025) demonstrate that LLMs can evolve into powerful Large Reasoning Models (LRMs), capable of producing extended chains of thought (CoT) enhance their reasoning abilities. However, these longer reasoning trajectories come at the cost of increased token usage and potentially incorporate more compounding errors. Many of the generated tokens tend be unnecessarily verbose. For example, LRMs may output thousands of tokens to solve elementary math problems that could otherwise be addressed within just a few hundred tokens, as shown in Figure (right). This phenomenon is commonly referred to as the over-thinking issue (Chen et al. 2025).
Recently, RL-based CoT compression methods have demonstrated the most favorable balance between token efficiency and accuracy (Kimi et al., 2025 Arora & Zanette 2025) Aggarwal & Welleck 2025
Hou et al. 2025) Fang et al., 2025 Xu al.) 2025). These methods can be broadly categorized
into three classes as shown in Table 1 budger ( (Aggarwal & Welleck 2025| [uu et al. 2025 Qi
ct al. 2025), adaptive (Fang et al. 20255 Tu et al. (025). and full (Kimi et al. 20255] Hao et al.
2024 Wu et al., 2025). Budget constrains the model to reason within fixed response thinking
length. Adaptive leverages a dual-mode design that switches between thinking and non-thinking (or
"Correspondence Wei Liu (wliucn @esc.ust.hk) and Junxian He (junxianh @ usk.

---

Published as a conference paper at ICLR 2026
Model Performance AIME2024
Question: What is the answer of 1+1?
38
Origiral
<think-
Origial Nidel
<thinks
35-
Tuncaden
Allight, so have this math probiem here: 1
plus Himm, okay, let me thine about how to
Okay, so need to figure out what 1=1 is. Himm, Ithikk it's poetty traighiforean remember
solee this remember from school that
arthimetic, addingg give Yeals, that's right the should be </thinke
34-
(ASER (Duro
LASER-O (Oum)
have object and add another how
many do have in total?
32-
visuplite Imagine
single friend.
The sur
ppick another Now, how many apples do have?) count them. one, two, So, that must be two apples total. Therefore, plus equals 2.
30-
(oowed(2]
tokens totall
28
Watt, is there another Mayoe using objects or fingers If hold up one finger and then add another finger how many lingers method, whether it's counting objects or using my fingers, the result the same...
think about this?
26
That's two. Yeah, that
regardless
23
3500
4500
5500
6500
7500
8500
15956
"* "Answer:""
Average Takana
[1400
Figure 1: Left:: Accuracy and response length on AIME2024. For the figure of more benchmarks, please refer Appendix [8 Each point represents single training run with different hyper-parameters, Given the high computational cost of obtaining this figure, the base model used is DeepScek-R 7B and 32B models are in $5.3 Our methods, LASER and LASER-D achieve the hest efficacy-efficieng trade-off compared to all other methods. Notably, they yield +5.3 improvement in accuracy while reducing tokens usage by 64% compared to the original model. Right: Example of reasoning process after LASER-D training. While the original model produces meaningless "self-eeflection" repeatedly for trivial questions like "I "I+1=?", LRMs after LASER-D D training recognize such questions during thinking and provide the answer directly.
Results on
long- and short- (ootttthinking) for different queries to improve efficiency. Full pursues policy models that can achieve extreme efficiency across all difficulty levels and problems, while simultaneously striving to preserve, or even improve, reasoning accuracy. These different designs naturally lead to different trade-offs. Models trained with budger-mode methods require manually defined budget, but such human-craffin budgets are inherently suboptimal- -fften too restrictive for hard questions and overly generous for easy ones. While adaptive mode teaches models to decide when to think, in practice, they tend to collapse into a degenerate pattern: skipping thinking on simple questions but still expending redundant reasoning effort on harder ones. By contrast, models trained under the full mode reason significantly more efficiently even on difficult problems. Yet achieving accuracy improvements alongside such compression remains challenging. As shown in Table [ most prior methods across all three modes struggle to simultaneously deliver substantial token reductions on hard problems and consistent accuracy gains. Some even rely on ntlte-stage pipelines with additional SFT, but still fall short of this goal.
In this paper, we follow the line of full CoT compression, aiming to improve reasoning efficiency and accuracy through concise, principled mngle-stage methods. We begin with simple baseline in full mode that yields surprisingly effective results ($2). Specifically, we further train long CoT reasoning models using RL with rule-based correctness reward, while restricting the context window size to a smaller value than the model's typical generated length, so that long responses will be truncated. This approach substantially improve token efficiency with only modest reduction in accuracy. To better understand this truncation method and to connect it with other RL approaches that incorporate length-based rewards (Kimi et al.) 2025 Arora & Zanette 2025 Aggarwal & Welleck 2025), we introduce a unified length-based reward shaping perspective that encompasses various RL strategies for mitigating overthinking ($3). Building on this reward shaping formulation, we recognize the limitations of previous length-based reward methods and propose a novel reward shaping approach ( ($3.3) that employs a step function guided by desired target length. We refer to this approach as LASER (Length-bAsse bAsed StEp Reward). LASER achieves the best trade-off between reasoning performance and efficiency among all evaluated baselines, delivering consistent gains in both reasoning efficiency and accuracy.
Next, we identify two key points that are lacking in LASER: (1) the desired reasoning length should evolve during training as the model's 's reasoning behaviors dynamically change, and (2) rather than uniformly encouraging short or long CoT, length-based reward shaping should be difficulty-wware - allowing harder questions higher token limit while constraining easier questions to fewer tokens. To this end, we propose Dynamic and Difficulty-awe which adaptively applies different length reward shaping based on problem difficulty. Notably, our algorithm is fully automated with an integrated automatic adapting module, eliminating the need for manual procedural tuning.
-aaggt--bsedd StEp Reward for RL (LASER-D),
2

---

Published as a conference paper at ICLR 2026
Table I: Comparison of RL-based CoT compression methods. Compression mode: Full aims learn policy models that improve efficiency across all questions: Budger constrains the model to follow target response or thinking length. Adaptive trains the model to switch between thinking and non- non- thinking (or long- and short- thinking) modes. Single stage indicates whether the method can he completed within single-stage RI. training pipeline without additional SFT multi-stage RL. Hard problems efficiency reflects whether the method enables models achieve more than 50% token reduction on AIME24. Accuracy improvements indicates whether the method improves reasoning accuracy. Difiicult-aware shows whether the method explicitly accounts varying reasoning effort across different question difficulty levels.
Compression Mode
Single Stage
Hard Problems
Accuracy Improvements
Difficulty Aware
Methods
Efficiency
PhunkPrune (Hou et al.
2025
Full
Kimi-k1.5 Kiml M.. 2025
Full
Emfiiintt Reasoiing Arora Zanctte
2025
Full
LI Aggarwal & Welleck 2025
Budget
X
X
al.
Budget
Anytmeekasoone
2025
Budget
Thinkless (Fang et 0. 2025
Adaptive
X
AutoThink [Tiret 2025
Adaptive
LAPO (Wu at. 2025
Full
LASER LASER-D
Full
X
Full
Table 2: Results of baseline truncation method with different context window. Ta denotes the models after RL
training with context window k. Accuracy (%) with average token usage for each datasct. "Original" denotes the original DeepSeck-Re SoasSeck-R1-Distiie RRi-Dittilll-e Distill-Qwen- Qwen-1.5B.
Accuracy (%)
Generation Length (tokens)
MATH
Olympiad Bench
MATH
Olympiad Bench
AIME
AMC
Avg.
AIME
AMC
Avg.
500
500
Original
83,9
28.9
71.6
43.3
56,9
5042
15956
8202
11510
10177
Ta192
81.8
24.8
70.9
43.9
55.3
1795
4465
2560
2841
2915
T5144
80.9
52.3
1351
1947
Tooe
77.7
19.2
62.2
38.5
49.4
1054
2481
1484
1564
1646
We conduct comprehensive experiments on five reasoning models ranging from 1.5B to 32B parame- ters, across four challenging benchmarks: MATH500, AIME2024, AMC2023, and OlympiadBench. Our extensive evaluations demonstrate that our proposed LASER series outperforms existing works, while LASER-D achieves the best -fficacy-effiiiencyy trade-off. Unlike methods that improve token efficiency at the expense of accuracy, our proposed approaches deliver substantial gains in both dimensions even on the challenging AIME2024 benchmark. For example, applying LASER-D to DeepSeek-R1 1-r--Seek-1--isiill---- -Distill-Qwen-1.555 improves accuracy by +5.3 percentage points while reducing token usage by 64% on AIME24.
2 TRUNCATION: A SIMPLE YET EFFECTIVE BASELINE
In this section, we a simple yet effective full-mode baseline, where we simply set max generation length value significantly smaller than the model's original context window during RL training (e.g., 8, 199 vs. 32,768 tokens in DeepSeek-Be -aa-Seeek-R---stiiiie --Distiiled). Responses exceeding this limit are truncated and treated as incorrect, encouraging concise yet accurate reasoning. This baseline has also been explored recently in concurrent works Hou et al. (2025): Luo et al. (2025). In our experiments, we adopt
Beees-RR---ssill--eeeeeet 1.5B as the base model and investigate the effects of truncation by limiting maximum generation lengths to 4,096, 6, 1,14, and 8,192 tokens.
Effectiveness of Truncation Table the performance of models across various benchmarks under different truncation sizes. Compared to the original model, surprisingly. RL training with context window of 8192 tokens achieves substantial 71% improvement in token efficiency, while maintaining competitive accuracy with 1.6 absolute point degradation on average. This demonstrates that truncation is a simple yet effective approach for enhancing reasoning efficiency in LRMs.
Efficacy-Efficienyy Trade-off Although truncation improves efficiency overall, it disproportion- ately harms harder tasks. On AIME, accuracy drops by 4.1 under an 8192-token limit and by 9.7 at 4096, compared to only 7% MATH500. This stems from the higher truncation ratio harder benchmarks: over 75% of AIME responses exceed 8192 tokens, versus 15% for MATH500 (Fig- ure 7b. These results highlight the efficacy-effciencyy trade-off: truncation reduces tokens effectively,
3

---

Published as a conference paper at ICLR 2026
but the accuracy cost is amplified complex tasks that demand longer reasoning. Next, formulate the truncation baseline from the reward shaping perspective and connect it with related works.
3 A UNIFIED VIEW ON TRUNCATION AND LENGTH-BASED REWARDS
In this section, aim to understand the truncation baseline and other single-stage RL-based efficient reasoning approaches of the full compression mode through a unified perspective. We first connect them together via length- based reward shaping, and then derive new alternatives with this view.
3.1 THE UNIFIED FORMULATION
Here we first present unified formulation, and then we show how the truncation baseline and other works fit into this formulation. Specifically, we define the reward function with two parts: correctness term Cly) and a length-based term S(y) controlled by control variable A(y):
R(x, v) = =C() - A(v) S(y)
(I)
In most length reward methods, C(v) = R(x, representing the original rule-based reward for correctness. However, in truncation-based approaches, C(y) = 0 we discuss below. The term S(y) denotes the length reward, which varies across different methods.
Formulating the Truncation Baseline As shown in Table 3 truncation is special case of the length reward with C(y) = 0, where the target length LT is enforced by the context window. ThinkPrume (Hou et al. 2025) is another truncation-based approach, which extends vanilla truncation by introducing adaptive target length LA replace fixed target length Lr. They iteratively choose LA and separate their training into three stages. Table 3 also outlines other formulations, we will introduce them individually in the following sections.
3.2 CONNECTING PREVIOUS EFFORTS TOGETHEE
In this part, build on our unified formulation in Eq. T and aim to connect previous approaches, below we describe several main categories of them. Table 3 formulates different nength-baeed reward shaping by different designs for C'yy), X(y) and (gy). Parameter a hyperparameter coefficient that controls the magnitude of the length reward S(v). We provide detailed explanations for each formulation in Appendix
Group-based Reward In group-based reward, the length reward 5'y) is designed to encourage brevity by assigning higher scores to shorter responses within a rollout group, such as Efficient Reasoning (Arora & Zanette 2025) and Kimi-k1.5 (Kimi et al. (025) as formulated in Table 3 However, this comparison-butt
approach can lead to reward hacking. Models tend to exploit Sgy) by generating overly concise responses, particularly for simpler questions. We demonstrate this phenomenon for the Efficient Reasoning baseline in Figure Sa and Figure Appendix [E, where training accuracy initially decreases while total reward increases. Additionally, Table 4 shows more significant drop in MATH500 accuracy compared to other methods, further supporting this observation.
Budget-based Reward Although budget-based reward methods are typically categorized under the budger compression mode rather than full, some concise single-stage approaches (Aggarwal Welleck (025) can still be unified into our formulation since they are also implemented through length-based reward shaping. Budget-based rewards use query-specific target lengths (budgets) and penalize responses that deviate from these instructions. While this mitigates reward hacking seen in based schemes, it can destabilize training. Models require exposure to diverse budgets, but in large context windows (e.g., 16.384 tokens), these targets become sparsely distributed, causing reward fluctuations. Figure 8b illusttaas this instability. Using LI-Max as a representative method, observe that with smaller contexts (4,096 tokens), it achieves stable reward increases compa- rable to other methods. However, with 16,384-token contexts, rewards fluctuate significantly and underperform alternative approaches.
4

---

Published as a conference paper at ICLR 2026
Table Formulation of different approaches based Eq. | C(y) is mainly for correctness, Svy) is the length reward, and Myy) is control variable to control how length reward is applied. ((H)) stands for 1 [[xx y) = 1) and I(-) is an indicator function. is the negative reward given for incorrect responses. L(y) is the length of the generated response. is the coefficient that controls the magnitude of the length reward. The shapes of different rewards are shown in the visualization, where axis is the length of the response. Blue represents the curve for correct responses, while Red represents the curve for incorrect responses. For approaches, ThinkPrune, LASER-D, there are different lines with similar colors indicate that the reward is dynamic which is realized by different LA values. The details of visualization are available in the Appendix M
Stz)
Visnalization
Truncation Method
Vanilla Truncation
0,1
Min v)
it Dia) Si for Lt
ThinkPrune Hou al 2025
Mis
Lig) ii Aa L(a) Au
Group-base based Reward
Fificient Reasoning
2055
Rinapt ((R)
Menils Srmteyy
U..S Swaw
( -1
Kimi-kl Kimi 2025
Rln.gh
min (0. 0.5
(T 0
Badget-hasd Reward
LI-Exet Agguraal & Walleck 2025 2025
-a [k(p) Azl
LI-Max Aggarwal & Welleek, 2025
0,8(A)
clip( (A(v)) -2))--6.)...
Leneth-Based Step Reward and Variants
LASER
T0K0o s kx)
LASER-D>
3.3 BRIDGING THE GAP: LENGTH-BASED STEP REWARD
As shown in Eq. and visualized in Table 3 a key limitation of the truncation method is that it assigns the same penalties to overlong responses as it does to incorrect ones, which may over- over-penalize long but correct explorations. To address this issue, we extend it as novel reward shaping approach called Length-bA StEp Reward (LASER), which adopts step reward function guided by target length, rather than performing hard truncation.
Specifically, we design the length reward term S(y) as an indicator function based on a target length LT. This function assigns a length-based bonus to responses shorter than Lt. We also set the context window significantly larger than the target length Lyp (e.g.. 16,384 vs. 4,096) where truncation rarely happens. And the length reward term S(y) is only activated when responses are correct, thereby improving the efficacy-efficiens vanilla truncation approach; the only difference is that, instead of truncating long responses, it awards bonus rewards to correct responses that do not exceed the target length. To balance the correctness reward ((y) and length reward S(y), we follow a typical setting and set 0.5. We also find that LASER is robust across different values of (see $5.2).
trade-off. As visualized in Table LASER closely resembles the
Empirical results are demonstrated in Figure I and Table 4 training with the LASER reward achieves improved efficacy-efficiency trade-off compared to all previous methods. Notably, it is the first approach to simultaneously deliver significant improvements in both accuracy and token efficiency on the challenging AIME24 benchmark. These results establish LASER as a promising reward design for enhancing the balance between efficacy and efficiency.
5

---

Published as a conference paper at ICLR 2026
4 ADAPTIVE LENGTH-BASED STEP REWARD SHAPING
4.1 DESIGN PRINCIPLES
We highlight two key limitations not addressed in the design of LASER: (1) LASER requires fixed target length, whereas the optimal length should be changed dynamically during training; (2) Additionally, different questions demand reasoning traces of varying lengths, with simple ones solvable by shorter reasoning and harder ones requiring longer deliberation.
Therefore, we extend LASER to be Dynamic and Difficulty-
which we term LASER- D. Rather than using a single fixed target length, our approach dynamically adjusts the target length throughout training and tailors it to questions of varying difficulty. Concretely, LASER-D decouples the target length hyperparameter across different queries, allowing distinct target lengths to be assigned various queries. Moreover, these target length hyperparameters are dynamically adjusted throughout training.
We separate queries into three buckets of easy. medium, and hard difficulty levels, based on the correctness rates within the rollout batch - for each question, we have k rollouts and use thresholds k/3 and 2k/3 separate them. As such, we have three distinct target length hyperparameters for these three query groups. Notably, we perform difficulty assessment for the queries during real- time RL training and use the training rollout batch, thus it only incurs negligible overhead on the computation. Being dynamic and difficulty-awann aware, one challenge raised is how to set the dynamic processes of the decoupled target length hyperparameters. Next, we introduce an automatic adapting mechanism, to adapt them without any manual intervention.
4.2
AUTOMATIC ADAPTING MECHANISM
LASER-D is driven by an automatic adapting mechanism that periodically evaluates and adjusts the target length parameters (L in Table 3) for each difficulty level. Specifically, we first extract small monitoring dataset DM (e.g., 500 samples) from training data that mirrors the distribution of the training data. Every N training steps (e.g., 20), our approach searches and sets the target length hyperparameters based on this monitoring dataset.
Each query in DM is assigned difficulty label d E (casy, medium, hard} based on the number of correct rollouts among its K sampled rollouts[Since queries in different difficulty levels exhibit distinct length distributions, we compute the target length separately for each difficulty level. To do so, we introduce the Expected Correct Responses metric, ECRg, which estimates how many complete, correct responses can be expected for queries of difficulty d under length limit i. Formally, for each candidate length 1, we compute
FCRa(()) Pi.d Ca
where Pt.a is the empirical coverage ratio, the fraction of all rollouts (computed over all monitoring queries with difficulty d) whose lengths do not exceed l. The constant Ca is the minimum number of
correct rollouts used to assign difficulty d (e.g., G6 Cuediuu 3,, Chard=1 for K K8 here).
For each difficulty level d, the monitoring module enumerates candidate target lengths from the lower bound target length Lr tokens up to the maximum context window (16,384 tokens) with an interval of T, computing coverage ratios Pi.a- We select the smallest target length LA satisfying ECR, ' 1 as the adaptive target length for each difficulty, ensuring at least one complete and correct response.
Intuitively, this mechanism sets the target length as the minimal generation length such that least one rollout response is expected to be correct. This approach is reasonable because generating sequences shorter than this length would likely be detrimental, as correct responses are less probable. Conversely, generating longer sequences may be redundant, since correct responses can already be obtained with a shorter generation length.
Selecting the Adaptive Target Length. For each difficulty level d, we enumerate candidate lengths from the lower bound Lr up to the context limit (16,384 tokens) with step size I. We select the smallest length / satisfying ECR,(!) 2 1 as the adaptive target length LA for that difficulty.
Practically, K is set to he the same the rollout size used during training, to maintain consistency with the training scenario.

---

Published as a conference paper at ICLR 2026
Intuitively, this is the minimal length at which at least one correct response is expected; shorter limits harm correctness, while longer limits are redundant.
Dynamic and Difficulty-Aware Reward During training, we apply these senittorin-derivee parameters to implement dynamic and difficulty-awan rewards. Each training question's 's difficulty level is determined using the same classification method described earlier. Easier questions receive smaller target lengths LA while harder questions receive larger ones. Since monitoring runs every N steps, the difficulty-dependen target lengths are automatically adapted to the evolving policy model.
Computational Efficiency This automatic adapting mechanism adds minimal computational overhead. By using small monitoring dataset and evaluating only periodically, our method increases computation by just 3.5% in our experiments.
5 EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Setup We experiment with five capable and representative LRMs across four different sizes known for their overthinking tendencies: DeepSeek-RI -tt-Disslll-Qwee-..BBB DeepSeek-R1 Ri-issilll 11--Dstill-Qwee--BB DeepSeek-R RI-Distill-Qwen-B33
-wr--Seee-RR--isisl---laaaaa We adhere to the original prompt from DeepSeck-R1 (DeepSeek-Al et al. 2025). with the full prompt available in Appendix [.I]] We train these models using the DeepScaleR--reviie Preview-Datase (Luo et al.. 2025), high-quality mathematics dataset containing 40K competition-level
Openneessoinng-Nemotrooon
iresston-answwe pairs. We leverage GRPO (Shao et al./ 2024) as our online RL algorithm. We evaluate the models on four benchmarks of varying difficulty: MATH500 (Hendrycks et al., 2021b). OlympiadBench (He et al., 2024). AIME 2024, and AMC 2023. We set a 0.5 for our methods in all experiments balance the trade-off between correctness rewards and solution length penalties. LT is tlper-parameter for our approaches because the automatic adapting mechanism will enumerate the target length from Lr to the context window size to select the adaptive target lengths LA, described in $4.2 Parameter settings for baseline methods are provided in Appendix I.3] and full details of our training procedure and evaluation methodology can be found in Appendix
1.2
Baselines According to Table 3 we train models using different types of length rewards design and compare our LASER, LASER-D to previous works. Considering the high computational cost of RL training, we select Efficient Reason (Arora & Zanette (2025) and L1-Max (Aggarwal & Welleck 2025) as the representatives, since they perform better accuracy compared to other methods inside same group and are more close to our settings, For ThinkPrune (Hou et | (025). we re-evaluate their models. We also compare with the recent mode method LAPO (Wu et al. (2025) and list two adaptive mode methods, Thinkless (Fang et al. 2025) and AutoThink (Tu et al. 2025) for reference.
5.2
TRADE-OFF
Since there is a trade-off between accuracy and response length, one of the best ways to evaluate different methods is to compare the efficacy-effcienccy trade-off. We start with the DeepSeek-RI - Distill-Qwen-1.3 1.55 model, as its small size allows us to run multiple experiments to investigate the trade-off of different approaches. To fully evaluate the potential of each method, we adjust key parameters (a for group-based reward, LT for other methods) to explore different tradeoff's along the accuracy-length trade-off curve. The full details of different hypee-parameters for different methods can be found in Table 13 As a result, each point in Figure 1 corresponds to fully trained model under a distinct hyperparameter setting. Benchmark- level results are reported in Table 4 with additional truncation results in Appendix L
As shown in Figure I LASER-D achieves the best efficacy-effciinne trade-off. On AIME2024 reaches 34.2% accuracy with just 5, 5,750 tokens (a 64% reduction), and across benchmarks (Figure 5) it obtains 60.3% accuracy with 3,520 tokens on average, compared to 10, for the original model. Compared to LASER, LASER-D benefits from its dynamic and difficulty-aware design, yielding consistent gains. LASER itself already outperforms prior baselines, while adaptive methods such as Think and Thinkless collapse into extreme patterns, saving tokens easy tasks (MATH) but remaining verbose on hard ones (AIME). Although these methods reduce tokens relative to their original models, but still consume around 10, 000 tokens on AIME. In contrast, both LASER and
7

---

Published as a conference paper at ICLR 2026
Table Accuracy (%) with average token usage for each dataset and different methods. Most important results this table are visualized in Figure I and Figure in Appendix [ The base model is DeepSeek-RR PeySSeek-R--isaee R--Dstilll- Qwen-1.5B. "Original" denotes the original model. I's is the truncation method with context window k. L1 is budget-mode compression. Thinkless and Auto AutoThink Think are based on adaprive. Other methods, including ours, are full mode. "Group" denotes the Efficient Reasoning (Arora & Zanette (025) with different o. We only show the thrce most representative results here. For the full results, please refer to Tabel 14 in Appendix L
Accuracy (%)
Generation Length (lokens)
MATH 500
AIME
AMC
Olympiad Bench
Avg.
MATH
500
AIME
AMC
Olympiad Bench
Avg.
Original
83.9
28.9
71.6
43.3
56.9
5042
15956
8202
11510
10177
L1-Max-1024
76.4
15.0
59.4
39.1
47.5
661
1303
933
938
959
4096
79.7
20.0
65.0
41.0
1718
11.99
1229
1245
Thinkless
82.0
27.5
68.4
46.5
56.1
3702
10680
7244
7751
7344
AutoThink
84.0
34.6
67.0
44.8
57.6
2195
9514
5059
5559
5581
Ta192
81.8
24.8
70.9
43.9
55.3
1795
4465
2560
2841
2915
Ta144
80.9
20.2
66.2
42.1
2821
1917
1947
2009
Time
77.7
19.2
62.2
38.5
49.4
1054
2481
1484
1564
1646
GrouD
74.6
25.0
69.2
43.1
53.0
1069
4747
2162
2536
2629
Group
78.1
28.1
68.0
44.4
54.7
1135
5628
2635
2944
3085
Group
77.0
29.0
69.5
44.9
55.1
1228
6301
2808
3271
3402
Group Group
74.4
30.2
65.5
43.1
53,3
1193
4839
2457
2703
2798
LAPO
84.3
29.3
71.2
51.7
39.1
2354
8318
3568
4863
5581
LASER1, by 2010
29.2
71.6
4815
2895
LASER
83.9
31.0
74.1
45.7
58.7
1914
5915
3136
3579
3636
LASER 5100
85.6
31.5
75.9
47.7
2736
6589
4162
4547
4509
LASER- D 1004
83.0
30.6
72.8
43.7
57.5
1362
4991
2560
2837
2862
LASER 2244
82.2
31.0
73.3
46.2
58.2
1623
5158
2572
2960
3059
LASER Doys
34.2
75.3
47.3
5750
3474
3520
LASER-D improve accuracy while reducing token usage across difficulties, advancing the frontier of full-mode methods.
Ablation Studies We conduct ablation studies focusing two aspects: the effectiveness of the proposed difficulty-aware component and the robustness of our methods under different hyperpa- rameter settings. We assess the necessity of the difficulty-aware mechanism by removing it from LASER-D. AASER-D.TYE The results in (show that removing the difficulty-awar component leads to con- sistent accuracy drops across benchmarks, highlighting that dynamically adapting the target length to question difficulty is essential for achieving better securacy-efffiieng trade-off. We examine the stability of our framework with respect to key hyperparameters, including the length reward coefficient (Table[), bucket thresholds TO). and ECR R thresholds (Table IT]. Across all these settings, our methods remain stable and maintain strong performance, confirming that our design is robust to hyperparameter choices. This also illustrates the robustness of our in-batch difficulty estimation strategy. Please refer to Appendix F. and Appendix F.2 for detailed results.
5.3 EXPERIMENTS ON LARGER MODELS AND MORE MODEL FAMILIES
We conduct experiments on larger models and different model families. Given the computational cost of larger models, we set key hyperparameters for each method to achieve appropriate trade- off. Specifically, for DeepSeek-RRl RI-DistilllOO -welll--wen-BB... we set a = 0.2 for the group-based reward, Tor = 8192 for the truncation method in LASER, L 4096 for LASER-I.. Notably, do not tune a with fixed value 0.5 in all experiments of our methods. As shown in Table LASER-D achieves the best trade-off with better accuracy and significantly fewer tokens. On the AIME dataset, it achieves an accuracy of 58.3 88,3, representing a gain of +5.2 points, while using only 5, 379 tukens--uusstatiaall fewer than the 13, 414 tokens used by the original model. Compared to other methods, LASER and LASER- D also attain better trade-offs on most benchmarks, particularly on the more challenging ones. For the 32B model, due to computational constraints, we compare the the original baseline under this larger setting and set L 8192. LASER-D achieves comparable accuracy, while still significantly reducing output length. Notably, the accuracy of DeepSeek-Ri Distill-Qwen-32B on training dataset is already very high-over 76%, leaving little room for further improvement. We speculate that with more challenging and diverse training data, LASER-D could yield further accuracy gains.
model with
Beyond scaling within the DeepSeck-Rl-Distie REl-Distill-wwer family, we conducted experiments on three different LRMs: DeepSeek-RI -1--istill-Llamma 8B (based on LLaMA-3.1), OpenRessoning-Neeotro 1.5B (based on Qwen- Ins-rust) and DeepScaleR-1.555 We apply LASER-D with L, = 4096 for DeepSeek-R1-Distie -a-iisttll--aamm--00 Jam DeepScaleR-1, 1.5B and Lr 8192 for waw.Reaasonnnn-eemorroo
8

---

Published as a conference paper at ICLR 2026
Table 5: Accuracy (%) with average token usage for each dataset and different methods using 7B and 32B models. "Original" denotes original model. 7 the truncation method with context window
Accuracy
Generation Length (tokens)
MATH
Olympiad Beach
MATH
Olympind Bench
AIME
AMC
AIME
AMC
Avg.
DeepSeck RI Disstill Qwen-7R
Original
92.6
53.1
88.4
58.9
73.3
4017
13414
6433
8987
8213
Tai9o
92.0
51,9
88.3
56.4
72.2
1972
5655
3159
3606
3598
Group
89.4
48.1
82.8
53.7
68.5
780
4271
1693
2348
2273
LASER
92.2
54.4
89.7
58.1
73.6
2317
6320
3733
4262
4158
LASER-D
92.2
58.3
90.0
61.0
75.4
1836
5379
2694
3350
3315
DeppSok-Rt--isiil--weenss
Original
94.4
71.7
93.1
64.6
80.9
3553
10335
6177
7697
6941
LASER-D
70.8
4209
DoopSeck-R--Disill-Llaee
Original
89.4
49.2
90.0
55.4
71.0
4412
13760
6636
8970
8445
LASER-D
90.8
51.0
90.3
56.8
72.2
3937
8438
2768
5121
5075
OpenRensoning Nemotron 1,5B
Original LASER-D
84.5
47.7
83.9
55.5
8556
23486
13651
16493
15547
85.2
52.7
86.9
57.4
70.5
4506
14809
6670
9354
8835
DeepScaleR-
Original LASER-D
6263
88.5
42.5
83.3
52.8
66.8
2100
S802
3978
3349
3807
Model Performance Across All Benchmarks
Acc 24.8%5
Accuracy
31
Orginal
LASSh
LAbORDD
2000
3000
4200
3000
6005
9771
Average
Figure 2: Performance on OOD benchmarks including GPQA (Rein et al. 2023 LSAT (Zhong et al. 202 Wang et 2022), and MMLU (Hendrycks et al. 2021a). Please refer to Figure for details on each benchmark.
1.5B, given its much longer original output length (20K+ tokens). Table 5 shows that LASER-D consistently delivers substantial accuracy improvements while reducing token usage by 50%. demonstrating strong generalizability.
5.4
EXPERIMENTS ON OUT-OF-DOMAIN BENCHIARRSS
We evaluate whether LASER and LASER-D can generalize to domains outside the RL training distribution. We select three out-of-domain benchmarks: GPQA (Rein et al., 2023), LSAT (Wang et al. 2022), and MMLU (Hendrycks et al. 2021a), following the evaluation settings established by LI (Aggarwal & Welleck) 2025), and LivecodeBench (Jain et al.) (025)) for code generation. Figure and Table (full details in Appendix demonstrate that compared to the original model, LASER and LASER-D consistently achieve significant improvements in both accuracy and token usage, demonstrating robust generalization capabilities. And LASER-D maintain the best trade-off even compared to LASER.
ANALYSIS
We DeepSeck-RI-Disillle R.--Distill-ween--.. the backbone, conduct comprehensive analysis that includes oudget-forcing inference (Appendix[]) J]) dynamics of adaptive target lengths (Appendix K). qualitative evaluations (Appendix N. and changes of thinking patterns.

---

Published as a conference paper at ICLR 2026
Table 6: Accuracy (%) with average token usage LiveCodeBench (Code Generation). The base model is
DeepSeek-RR Boick-RR--isttlle Beaill-Qwen-115B
Accuracy (%)
Generation Lenyth (lokens)
Original
24,091
13120
LASER-DA 112-
27.955
6373
LASER 2011
LASER-DA Day mo000
28.409
6600
Truncation
LAGER
LASER-D
Keywords
114
Tokons
140
30
Figure 3: Average keyword amount and response length over RL training on AIME24, The truncation method
uses 8192 token context window. LASER and LASER-D employ target length of Lr = 2048
Backtracking
Verification
Enumeration
Subgoal Setting
Bohavtor Frequency
Frequency
Tokens
12
0.3
0.3-
0.3-
0.1-
Global Stop
Figure 4: Changes in reasoning hehaviors ratio and response length over RL. training iterations on AIME2024. The figure shows how LASER-D's thinking patterns change during training with target length of Lr 2048.
6.1 CHANGES OF THINKING PATTERNS
To better understand the changes of response length, we analyze the changes of thinking patterns over RL iterations on AIME2024 with 16 samples per question. We analyze through two approaches, keywords counts (Yoo et al. 2025) and reasoning behavior ratios (Zeng et al. 2025).
Shifts in SSelf-Reflection"s Keywords "Self-reflection" or momaem"men" reasoning has emerged as an intriguing behavior in LRMs (DeepSeek-AI et al., 2025). Following previous
work (Yeo et al. 2025), track this behavior by monitoring seven representative keywords: ("recheck", "rethink" " "try again" " "wair", "alternatively" "retry", "neery","hovv"" "however"]. Figure 3 shows that average keywords decline notably as response length decreases, suggesting RL reduces spuri- ous "self-reflection".. Interestingly, later training stages maintain shorter outputs while recovering moderate levels of "self-reflection" indicating more efficient use of this behavior.
Changes in Thinking Behaviors To more comprehensively capture shifts in thinking behavior, we employ proprietary large language models to conduct behavior-level analysis. We further an- alyze reasoning behaviors using the cognitive behavior framework of (Gandhi et al.) (025) with gpt-4. 1-mini, focusing on four representative categories: Backtracking, Verification, Subgoal Setting, and Enumeration. Implementations and the full behavior list provided in Appendix G 4 shows that Backtracking declines sharply (from 30% to just above 10%) as response length is reduced, consistent with keyword statistics above, while the other behaviors remain stable, with slight increase in Subgoal Setting These results indicate shorter reasoning preserves core reasoning behaviors while minimizing unnecessary backtracking, rather than collapsing into shallow heuristics.
CONCLUSION
In this paper, we propose a unified view for RL-based CoT compression, unifying various reward- shaping and truncation methods. Building this view, we introduce new approaches with adaptive, length- based reward shaping. Extensive experiments demonstrate that our proposed methods achieve superior efficacy-efficient efficiency. Our analysis of reasoning behaviors reveals that our RL-based CoT compression effectively encourages models to reason more concisely and productively.
trade-off and significant improvements in both accuracy and token
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Pranjal Aggarwal and Sean Welleck. LI: Controlling how long reasoning model thinks with reinforcement learning, 2025. URL https: /ttpa//awwwv.orr//////.3.. htt.ttts///axv...rrraas/////////
04697
Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025. URL
https://arxia org/ abs /ttp////2222 2502. 04463
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang. Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang. Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like 1lms, 2025. URL https .1111...11817
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. compass hatp:p//......ga.oooooa/oa///aaa
opt.compasccssss.eooo 2023.
AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjee Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai. Jiaqi Ni. Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li. Miaojun Wang, Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye. Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li. Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong. Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning, 2025.
URL
org. / 12948
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Thinkless: Llm learns when think. arXiv preprint arXiv:2505. 13379, 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self self-improving reasoners, or, four habits of highly effective stars, 2025. URL
pttps://arxxr http:/axxwwwv..ggaaaaa wwtp:///wwwrr.a////505. 01307
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space, 2024. URL https
//arxiv. www.:orggg......... 221/24122200 06769
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang. Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiaBBench: A challenging benchmark for promoting AGI with olympiad-leve bilingual multimodal scientific problems. In I Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd
11

---

Published as a conference paper at ICLR 2026
Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1..1866//.//2222.cclloomggg. URL htt.tll..RRtttaaaaaaaatt htt.aa////aaaaatttoooooooor/////
211/
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https://
arxiv. attp//0bs20000.o. 03300
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fffth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b.
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long thain-of-thought of Ilms via reinforeement learning, 2025, URL
https:// org/ abs /ww.////5000 2504 01296
Naman Jain, King Han, Alex Gu, Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Le7ama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free eval- uation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. URL beattps://operevviiie www.:/eeeeeeedeccceeeeeeeoo
Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang. Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang. Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang. Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma. Shupeng Wei, Sihan Cao. Siying Huang. Tao Jiang, Weihao Gao. Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He. Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang. Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang. Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with llms, 2025. URL /tttps://arxxr
org/ abs / 22//25511110 12599
Michael Luo, Sijun Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa. and lon Sto- ica. Deepscaler: Surpassing ol-preview with 1.5b model by scaling rl. https
r/pretty-radio-bil bow. http://e........eeeeeeeeeeeeaaaa ol
wwith-a-1-95 5--
2025. Notion Blog.
Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. URL https: / /aaxiiiiiii www.aiivvoorrr..ooo v..rg/////...........oo.o.o..... abs 2251 19393
OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Kl-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang. Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler. Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li. Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang,
12

---

Published as a conference paper at ICLR 2026
Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song. Fred von Lohmann. Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, lan O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, llya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore. Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo. Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga. Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, I Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz. Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang. Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowd- hury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk. Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang. Rhythm Garg, Robin Brown, Roshan James, Rui Shu. Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay. Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi. Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojeiech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai o1 system card, 2024. URL hnttp::://// / //tpxva.avvvv/aa///a//ea./aaaa org/ abs 2412 16720
Penghui Qi, Zichen Liu, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Optimizing anytime reasoning via budget relative policy optimization. arXiv preprint 2rX15:2505.13438 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A grad google-proof q&a benchmark, 2023. URL httpps: arxiv. orgg abs 2211 12211.12022 12022
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Songjun Tu, Jiahao Lin, Qichao Zhang, Xiangyu Tian, Linjing Li, Xiangyuan Lan, and Donghin Zhao. Learning when to think: Shaping adaptive reasoning in rl-style models via multi-stage rl, 2025. URL hhttp::p////// http://///.w.....oo///////////// 2tw.bb///255.113333
Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. From Isat: The progress and challenges of complex reasoning. IFEF/ACM Transactions on Audio, Speech, and Language Processing, 2022.
Xingyu Wu, Yuchen Yan, Shangke Lyu, Linjuan Wu, Yiwen Qiu, Yongliang Shen, Weiming Lu, Jian Shao, Jun Xiao, and Yueting Zhuang. Lapo: Intermalizing reasoming efficiency via acngtt-a policy optimization, 2025. URL
Yuhui Xu, Hanze Dong. Lei Wang, Doyen Sahoo, Jumnan Li, and Caiming Xiong. Scalable chain of thoughts via elastic reasoning. arXiv preprint arXiv:2505.053/5, 2025.
13

---

Published as a conference paper at ICLR 2026
An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-madd math technical report: Toward mathematical expert model via self-improvement, 2024. wtt.tttss///aaii/ee
abs abs//4/9..... 2409. 1211..2122
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning Ilms, 2025. URL httpp::/apaattt.... org/ abaaabbs/2550 03373
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai. Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source IIm reinforcement learning system at scale, 2025. URL htttpss//arrdii..or//a///////// 14476
Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing He. Zejun Ma, and Junxian He. Simplerl- z00: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URL httpttttts:/xxii org. abs 2503. 100000.189222 8892
Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan. Ar-lsat: Investigating analytical reasoning of text, 2021.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
For this paper, large language mLdels(LLMs) are used solely for polishing the writing. The entire research process, including ideation, experimentation, and analysis conducted independently of LLM assistance.
B SEFICACY-EFFICIENT TRADE-OFF
We illustrate the
trade-off in Figure Our proposed methods, LASER and LASER- D, demonstrate sigmificant improvements in both accuracy and token usage across all benchmarks, particularly in the most challenging ones. Notably, LASER-D achieves the best efficacy-effciinncy trade-off compared to all other methods.
Model Performance Across Benchmarks
Model Parformance on AIME2024
Original Modal
36
Groug-based
ThinkPrnne
34
LAS EA (Guns)
LAS EN-D (Durs)
Accursyy
Arc 44.27
30
56
28
Original Model Truncation
54
Gom-p-baved
ThinlFrune
24
LAGER-D (Durs)
52
1 10177
22
0500 15956
Average Tokena
Average Tokena
(a)
Figure 5: off between accuracy and response length across various methods. Each point represents single training with different parampaaaetsrss. Our methods, LASER D and LASER, achieve the best trade-off compared to all other methods. (a) Accuracy response length on benehmarks (MATH500, AIME2024, AMC2023, Olympiad Bench) (b) Accuracy and response length on AIME2024
C EXPERIMENTS ON OUT-OF-DOMAIN BENCHMARKS
We evaluate whether LASER and LASER-D generalize beyond their RL training distribution. Following the evaluation settings LI (Aggrrwal & Welleck 2025), we test on three out-of-domain benchmarks: GPQA (Rein et al. 2023). LSAT (Wang et al. 2022), and MMLU (Hendrycks et al.) 2021a). Figure shows the efficac--ffficency trade-off on GPQA and the average performance across all benchmarks. Relative the original model, both LASER and LASER-D achieve substantial improvements in accuracy and token efficiency, demonstrating strong generalization. Notably, LASER-D consistently maintains the best trade-off, outperforming LASER.
6
We further assess whether LASER-D can improve coding performance while reducing token usage. Using the LivecodeBench benchmark (Jain al., 2025) under the OpenCompass framework (Con- tributors) 2023), we find that our method on DeepSeek-BR RI-Distilll-wwaet accuracy while cutting token usage by roughly 50% (Table
enhances reasoning
D RATIO OF TRUNCATED RESPONSES DURING TRAINING WITH TRUNCATION
We analyze the ratio of truncated responses when applying an 8192 token limit during training. Our findings show that the proportion of truncated responses is initially very high-ixchh enchediin 45% and remains substantial (above 10%) even after 200 rollout steps. This high truncation rate highlights the context window constraints in training is sub-optimal.
15

---

Published as a conference paper at ICLR 2026
Parformance on
Porformance on
23
Accuracy (50
Orginna Hodel
Dep na Hedel
2000
3003
4000
5000
5000
7000
11200.8
2000
4003
5003
5900
15421
Average Takens
Average Tokens
(a) GPQA
(b) LSAT
Model Performance on MMLU
Model Performance Across All Benchmarks
Accurscy
Hetel
Turcrten USdA UIGEN
Tuncat
Geopliooe LAdEA
1020
2592.7
3300
4002
5000
6200
23722
(c) MMLU
(d) Average
Figure 6: Performance on out-of-domain benchmarks including GPQA (Rein 2023), LSAT Zhong 2021
Wang et al. 2022), and MMLU (Hendrycks et al. 2021a).
Table 7: Accuracy (%) with average token usage LiveCodeBench (Code Generation). The base model
DeepSeck-RR Baotcck-R1--- R1-Distill-Qweat
Accuracy (%)
Generation Length (tokens)
Original
24.091
13120
LASER-DL 1
27.955
6373
LASER D 2044
28.409
6420
LASER =096
28.409
6600
Truncatioe Bode
Figure 7: The ratio of truncated responses in training data with 8192 tokens limit.
16

---

Published as a conference paper at ICLR 2026
E DYNAMICS OF ACCURACY AND REWARDS THROUGHOUT TRAINING
We present the accuracy and rewards for various methods across training iterations in Figure 8a and Figure 8b) As discussed in $ group-based rewards tend to exploit the length rewards S while causing significant drop in accuracy. Budget-based rewards such L1 LI-Max- 16384 (Aggarwal & Welleck 2025) suffer from unstable training when the context window large. contrast, other methods like truncation methods, LASER and AASER-Ddemonstrr a simultaneous increase in both rewards and accuracy throughout the training process.
0.00
8.57
3.40
0.33
0.20
Namur
JAZZVD
(a)
(b)
Figure 8: (a) Accuracy training dataset training iterations for different methods (b) Rewards
training iterations different methods
F ABLATION STUDIES
F.1 NECESSITY OF THE SEFICULTY-AWARE COMPONENT
To evaluate the importance of the difficulty-aware component, we conducted an ablation study on LASER-D with LT 2048, where removed the difficulty-awan mechanism and did not divide questions into the three difficulty buckets.
As shown in the Table 8 removing the difficulty-awan design leads to accuracy drops across most benchmarks. This result highlights that dynamically adapting the target length based on question difficulty is crucial for achieving better eecuracy-efficiiee trade-off in dynamic framework.
Table Accuracy (%) with average token usage for each dataset and different methods.
Accuracy (%)
Generation Length (tokens)
MATH 500
Olympiad Bench
MATH
Olympiad Bench
AIME24
AMC23
Avg.
500
AIME24
AMC23
Avg.
LASER-D2 2.452045
82.2
31.0
73.3
46.2
58.2
1623
5158 4693
2572
2960
3059
w/o Difficult--Aaare
82,3
29.0
70.9
45.8
$7.0
1491
2437
2767
2847
F.2 ROBUSTNESS TO HYPPER-PARAMETER
To evaluate stability and sensitivity our methods, we conduct three groups of experiments on a, bucket thresholds, and the number for ECR 2
Different a We conducted four experiments on LASER LASER-Dusing different settings of the length reward coefficient
0.3, 0.7 and target lengths L 2048, 4096.
As shown in the Table 9 our method remains stable across different a values and consistently achieves the best trade-off between accuracy and token usage. Different values have only minor impact on the accuracy-token trade-off. Overall, a = 0.5 a strong choice, and other values also demonstrate sufficient robustness.
17

---

Published as a conference paper at ICLR 2026
Table 9: Accuracy (%) with average token usage for each dataset and different a settings.
Accuracy (%)
Generation Length (tokens)
MATH 500
Olympiad Bench
MATH
Olympiad Bench
AIME24
AMC23
Avg.
500
AIME24
AMC23
Avg.
LASER-D DJ
82.2
31.0
73.3
46.2
58.2
1623
5158
2572
2960
3059
84.2
47.3
1872
$750
2981
3474
3520
LASER-I BAERR-D... L., 0.33
82.7
31.7
74.2
44,9
58.4
1581
5272
2612
3049
3129
LASER-D =0=0008,0002
84.7
33,9
75.0
46.8
60.1
2061
5835
3272
3611
3694
LASER D 2005..0.0-007
82.1
30.4
72.7
45.0
57.6
1419
4982
2520
2812
2933
LASER-D D 4036.a00.7
83.3
33.2
75.5
47.8
60.0
1844
5303
2932
3318
3349
Bucket Thresholds To assess the sensitivity of our method to bucket definitions, we conducted an ablation study using an alternative set of thresholds: defining questions answered correctly 1-3 times hard, 4-6 times as medium, and 7-8 times easy-comparee to the original thresholds of 3-5, and 6-8, respectively. This experiment was performed on LASER-D with target length Lr 2048. As shown in the Table IO the new threshold configuration also leads to consistent improvements in both accuracy and efficiency. This result supports the robustness of our method and demonstrates that it is not overly sensitive the specific bucket thresholds used for difficulty estimation.
Table 10: Accuracy (%) with average token usage under different difficulty settings.
Accuracy (%)
Generation Length (tokens)
MATH
AIME24
AMC23
Olympiad
MATH
500
AIME24
AMC23
Olympiad Bench
Avg.
LASHR-DD DD, -244 (Original)
82.2
31.0
73.3
46.2
58.2
1623
5158
2572
29N1
3059
1-3 hard, 4-6 as medium, 7-8
82.7
30,0
73.1
47.1
58.2
1524
1934
2529
2825
2953
Different ECR thresholds To better illustrate the robustness of this setting, we conduct an ablation with ECR 2 on LASER-D (LT 2048). As shown in the Table [T] the model still achieves improvements in both accuracy and efficiency compared to the original model. Compared the ECR 1 setting, the ECR 2 version encourages the model generate longer out- puts consistent with our design, since a stricter constraint requires the model to choose larger LA to meet the criterion.
Overall, this experiment supports the stability of our method across different ECR thresholds, and confirms that the design is flexible yet effective. And according the experiment, ECR 2 is already reasonable choice for training.
Table 11: Accuracy (%) with average token usage under different ECR thresholds.
Accuracy (%)
Generation Length (tokens)
Olympiad Bench
MATH
Olympiad Bench
500
AIME24
AMC23
Avg.
500
AIME24
AMC23
Avg.
LASER-DA Dyy m204s. ECR2
82.2
31.0
73.3
46.2
58.2
1623
5158
2572
2960
3059
Dj,, =001k, RC22
31.0
73.0
58.2
1816
5694
3365
3496
G ANALYSIS OF REASONING BEHAVIROS
We apply the cognitive behavior framework proposed by Gandhi et al. (2025) to conduct detailed analysis of how reasoning behaviors change during our long-to-short RI. We use gpt-4. 1-mini to perform more fine-graid fine-grained analysis of cognitive behaviors throughout the training process. Following Zeng et al. (0025), we use the prompt shown in Figure I] to prompt gpt 4. 1-mini to identify and analyze reasoning behaviors. We analyze these behaviors on AIME2024 by sampling one question 16 times, resulting in 480 responses for analysis. Since we start from a LRM, reason- ing behaviors such backtracking naturally appear in every response, especially for challenging benchmarks. We specifically track four key behaviors: Backtracking. Verification. Emmmeration, and Subgoal Setting. For each behavior, we calculate its frequency ratio relative all behaviors and
18

---

Published as a conference paper at ICLR 2026
Truncation
LAGER
LASER
Takena
Keywords
2013
14.0
100
Figure 9: Average keyword amount and response length over RL training on AIME24. The truncation method uses 8192 token context window. LASER and LASER-D D employ target length of Lr 2048.
Backtracking
Subgoal Setting
Behawlor Frequencyy
Frequency
Tokens
0.4
62-
0.2-
0.2-
200
602
200
400
Global
200
400
0.0
200
400
Figure Changes in reasoning behaviors ratio response length over RI. training iterations on AIME2024. The figure shows how LASER-D's -''s thinking patterns change during training with a target length of LT 2048.
report how these ratios change throughout the training process. The complete list of all reasoning behaviors analyzed is provided in Table T2
Table 12: Complete list of reasoning behaviors
Reasoning Behavior
Subgoal Setting Enumeration
Verification Backtracking
Creative Analogy and Abstraction
Abstraction and Parametrization
Analytical Insight via Asymptotic Analysis
Creative Abstraction Coordinate Setup Use of Multiple Mathematical Tools and Identities
Creative Analogies and Insightful Generalizations
Algebraic Manipulation and Insightful Generalization Abstraction to Modular Arithmetic and Divisibility Creative Analogies and Abstractions Insightful Generalization / Alternative Modeling
H SUPPLEMENTARY DETAILS: LENGTH-BASED REWARD SHAPING FORMULATIONS
In this section, we provide additional details regarding the various formulations of length-based reward shaping presented in Table These formulations can viewed different variants of our unified framework Eq. which can be implemented by making specific design choices for three key components: C(v), A(y), and S(y) inside the framework. Here we review the formulation of Eq. to better illustrate following approaches.
R( 3) = C(y) x-(g) S(y)
H.1 TRUNCATION
Vanilla Truncation As aforementioned discussions 3 truncation is special case of the length reward with C(y) = 0, where the target length Lr is enforced by the context window, p is set as 0. follows the design:
19

---

Published as a conference paper at ICLR 2026
Prompt:
thain-of-reasoniee generated by Language Model attempting solve math problem. Evaluate chain- of of-resonninng determine whether demonstrates beneficial problem- solving behaviors deviate from typical linear, monotonic reasoning patterns commonly observed in language models.
sstart_of_reasonnii reasronnnn
(input)
<end_os hidd_oo_rrre reasaning>
Specifically, actively identify and emphasize beneficial behaviors such as:
(1) Backtracking: Explicitly revising approaches upon identifying errors or dead ends (e.g.. "This approach won't work because..").
(2) Verification: Systematically checking intermediate reasoning steps (e.g. "Let's verify this result by.."! "].
Breaking complex problems into smaller, manageable steps (e.g.. "To solve this, first need
Subgoal Setting:
(4) Enumeration: Solving problems by exhaustively considering multiple cases possibilities.
Additionally, remain attentive to and encourage the identification of other beneficial behaviors not explicitly listed here, such as creative analogies, abstraction to simpler cases, or insightful generalizations.
Important:
Clearly specify each beneficial behavior you identify.
Provide explicit examples reasoning chain.
no beneficial behaviors observed, explicitly empty list.
Provide evaluation clearly, formatted as follows:
""soon
'behaviour!:""
eexmmpee:
Figure 11: Prompt used identify and analyze reasoning behaviors with gpt-4. 1-mini
20

---

Published as a conference paper at ICLR 2026
C(v) 0 1(y) = 1
Swy) R(x,y)
if L(y) S LT if [(y) LT
Prink ThinkPrune (Hou al. 2025) another truncation-based approach, which extends vanilla truncation by introducing an adaptive target lengths LA to replace fixed target lengths Lr. set as The design follows:
C(v) = 0
X(y)
S(y) = [((,,)) v) if (y)) > LA
if L(v) < LA
Their training methodology employs progressive three-stage stage process with iterative refinement of La. Each subsequent stage initializes from the checkpoint of the previous stage while mannually reducing the value of LA- Specifically, they progressively decrease LA through values of 4096, 3072, and 2048 across the three stages.
H.2 GROUP-BASED REWARDS
In the context of group- based rewards, the length reward S(y) is specifically designed to promote brevity by assigning higher scores to shorter responses within a rollout group. This mechanism functions as a comparison-bar
group-based
reward system that inherently favors more concise responses. Most of them follow the design C(v) = R(x, y) to keep the accuracy performance of models.
Efficient Reasoning Efficient Reasoning (Arora Zanette 2025) follows the principle of group- based reward by specifically encouraging conciseness within correct responses. The mean and variance scalars are computed exclusively from the subset correct responses, ensuring appropriate statistical distributions. By selectively rewarding conciseness only when answers correct, this approach maintains higher accuracy compared to Kimi-k1.5 (Kimi er al.) 2025), which encourages wrong responses to be shorter. Considering the similarity between the approaches and the better efficacy-efficenncy trade-off, we select Efficient Reasoning as the representative group-based reward in this paper. The corresponding design can be formulated follows:
Cly) = R(x, v)
A(y) I(R)
((y) Mean(y) STD(L)
S(y) -
Kimi-k1.55 The design of Kimi-k1.5 is similar to Efficient Reasoning (Arora & Zanette 2025). with two main differences. First, the scalar factors are computed using the minimum response length and the difference between maximum response length and maximum length within rollout group. Second, Kimi-k1.5 encourages all responses be shorter, rather than focusing solely shortening correct responses. Such a design has the potential to intensify reward hacking, as models may exploit the reward function by favoring shorter responses to maximize their scores. The designs follows:
C(y) R(x, y) X(y)
(.. L(y)-Immia
if = 1
S(y) min 0, 0.5
Ianax Luwin
L(y) Lomin
if = 0
Lmnase Limin
21

---

Published as a conference paper at ICLR 2026
H.3 BUDGET-BASED REWARD
Budget-based rewards use query-specific target lengths (budgets) and penalize responses that deviate from these instructions. And the coefficient a controls the trade-off between length reward and correctness reward. They come in two flavors: exact mode and max mode. We follow same settings as L1 (Aggarwal & Welleck 2025) and set 0.0003 for exact mode, 0.01 for max mode.
Exact Mode In exact mode, the model must hit the specified target length Lr exactly, and any deviation (even shorter outputs) is penalized. The design can be formulated as:
C(y) R(x, v) X(1)
S(y) a (Lly) LTl
Max Mode In max mode, only outputs that exceed Lr incur penalty. The designs follow:
C(y) = 0
X(y) I(R)
S(y) clip(a: (T(y) - Lr) + 1,
I TRAINING CONFIGURATIONS
We leverage the prompt from DeepSeek-AI et (2055,, which is shown in Figure 12 for thinking is "<think>... </think>". "<hhink>..<htinkk>"""
And the mark
1.1 TRAINING PROMPT
list our training prompt Figure 12 al. 2025).
which follows the prompt from DeepSeek- R1 (DeepSeek-A)
Prompt
User piinput]Peease reason step by step. put your answer \boxed]]
Assistant
Figure 12: Training prompt for our training.
1.2 TRAINING EVALUATION DETAILS
We employ the verl (Sheng et al. 2024 framework for model training and Qwen-Math-Evve YYang
al.] (2024) for evaluation. During training, we set the rollout batch size 128, conduct 8 rollouts per prompt, use a temperature of 0.6, and train with mini-batch size of 64. In our preliminary experiments, we found long-to-shor RL benefits from clip-higher strategy (Yu al. 2025). So we follow DAPO (Yu et al. 2025) and set Ehigh as 0.28. For evaluation, we maintain sampling temperature of 0.6 and permit a maximum of 32,768 tokens to be generated. The number of samplings during evaluation is contingent on the dataset size: 4 samples per question for MATH500 and OlympiadBench, and 16 samples for AIME 2024 and AMC 2023.
1.3 FULL
LIST FOR DIFFERENT LENGTH- BASED REWARDS
We list the all hyper- for Lr, and LA in Table 13
J BUDGET-FORCING INFERENCE
To further analyze the impact of different length rewards, we conduct experiments using the budget- forcing setup introduced in S1 (Muennighoff et al. 2025 which restricts the model to stop reasoning
22

---

Published as a conference paper at ICLR 2026
Table 13: The details of key ppram--rrmeeerss for different methods
Methods
Hyper- Parameeas
Truncation
lr [10240, 8192, 7168, 6144, 4098, 2048)
Think-Prune
LA = [4096, 3072, 2048)
Group-Based Based Rewards
[0.4, 0.2, 0.1, 0.05)
LI-Max
a = 0.002
LASER
LT [8192, 4096, 2048
LASER-D D
L7 4496, 2048, 1024
after a fixed number of tokens We adopt their experimental setting and evaluate across 500, 1000, 2000, 4000, 8000] We follow the budget-forcing implementations of Muennighoff et al. (2025); Hou et al. (202)). Specifically, we follow their implementations and modify the codebase of Owen-Math-Eil Eval. We stop the thinking process of LRMs by appending "</think>> A An* * DrBSScc-R--Dissil-O-ee
Answer. *** Since empirically, starting with "An't In ******in Final Answer. *** We use the same settings as our evaluations where we sample responses for multiple times with temperature = 0.6.. As shown in Figures 13a and 13b despite not being explicitly trained with any budget-forcing mechanisms, LASER-D consistently achieve strong trade-offss between accuracy and token efficiency, particularly on harder questions or when inference budgets are moderately constrained.
.SB typically summarize its final answer
While LASER performs competitively on average benchmarks, it lags behind DE under strict token budgets or on more challenging examples. LI-Max, specifically trained to meet varying budget constraints during training, performs best under extremely tight budgets, demonstrating the strength of budget-specific optimization. However, its performance plateaus when more budget is available, limiting its ability improve harder tasks and resulting in suboptimal trade-off, as shown in Figure 13b Group-based methods are also effective in low-budget scenarios due their reward structure favoring shorter outputs, though this often leads overly brief responses. ThinkPrune shows comparable performance to LASER under looser budgets but inherits the limitations of truncation-baat based approaches, struggling on difficult problems even when more tokens are available.
Average chnakksrkss
Benchmarkc: AIME2024
Accurscy 0%
20
45
40
Average
ThindPunne
ThinPPunne
15
10-
LASER-D
LASER.D
500
2000
2500
1000
1050
3050
4000
salo
Average Takens
Tokens
(a)
(b)
Figure Budget-foreing inference with different methods. (a) Average accuracy with different output budget on all benchmarks (b) The accuracy of different methods on AIME2024 with different output budget.
K DYNAMICS OF ADAPTIVE TARGET LENGTHS
In this section, analyze the dynamics of adaptive target lengths during the training process of LASER-D DD. Figure s4oshows how the adaptive target length LA changes over training iterations for both methods.
As demonstrated in Figure [4, our method dynamically selects appropriate target lengths based on problem difficulty. For easy problems (left figure), the model quickly identifies that shorter target
23

---

Published as a conference paper at ICLR 2026
lengths sufficient. For ceddiimmdiffiullty problems (middle figure), the model begins with longer target lengths (10,000+) and gradually reduces them to 3000-4000 as training continues. For difficult problems (right figure), the model consistently maintains target lengths near the maximum context window size, with some fluctuations attributable to computational precision issues. This adaptive behavior highlights the effectiveness of approach in efficiently allocating computational resources based on problem complexity.
Level
Medium Level
Hard
LAFER - Oreetc
LASER r 15M
3500
LASEN Our LASER -Oaeeoo
1458% 0,
Bagget Langoh
LASER R
3200
Adigive Sarget
22000
2500
4000
4000
3305
4020
200
200
300 470
300
800
730
2000
350
230
300 420
300
620
700
320
203
300
400
5.00
600
St.p
Figure 14: Dynamies of adaptive target lengths during training process of LASER-D. The figure shows how the adaptive target length LA changes over training iterations for problems of different difficulty levels (casy, medium, hard). For easy problems, the model selects short target lengths; for medium problems, it gradually decreases from higher initial values; and for hard problems, it maintains consistently high target lengths near the context window limit. This demonstrates methods' ability to adaptively allocate computational resources based on problem complexity, unlike fixed-length approaches.
L FULL MAIN RESULTS
We list the full results of different methods in Table 14
M VISUALIZATION DETAILS
In this appendix, we provide details about the visualization of different reward functions depicted in Table These visualizations illustrate how different methods calculate rewards based response length.
M.1 VISUALIZATION PARAMETERS
Each visualization captures the relationship between response length and reward value with the following specifications:
X-axis: L(y) represents the response length, ranging from 0 to 20 tokens.
Y-axis: Reward value, with different ranges depending on the method.
Line styles: Solid lines represent rewards for correct responses (blue), while dashed lines represent rewards for incorrect responses (red).
Target length (Lr): Set to 10 tokens for all methods.
The visualizations were generated using high-resolution grid of 400 points between 0 and 20 tokens.
M.2
UNIFIED REWARD FORMULATION
Each method can be represented using the unified reward formula:
fl( y) C(v) X SSy)
We implement the specific components for each method in this simulation as follows. Note that the paramters only used for better visualization which different from the practical experiments.
24

---

Published as a conference paper at ICLR 2026
Table 14: Full results of accuracy (%) with average token usage for each dataset and different methods. Most important results in this table are visualized in Figure [ and Figure 5 in Appendix [B] The base model DeepSeek-RI-Dicn Dill-isill--Qee---- 1.5B. "Original" denotes the original model. T2 is the truncation method with context window &. L1 budget-mode compression. Thinkless and AutoTTink are based on adaptive. Other methods, including ours, are full mode. "Group" denotes the Efficient Reasoning (Arora & Zanette 2025 with different
Accuracy (%)
Generation Length (tokens)
MATH
Olympiad Bench
MATH
Olympiad Bench
500
AIME
AMC
Avg.
500
AIMR
AMC
Avg.
Original
83.9
28.9
71.6
43.3
56.9
5042
15956
8202
11510
10177
L1-Max-1024
76.4
15.0
59.4
39.1
47.5
1303
938
LI-Max-4096
79.7
20.0
65.0
51.4
875
1718
1159
1229
1245
Thinkless
82.0
27.5
68.4
46.5
56.1
3702
10680
7244
7751
7344
AutoTThink
34.6
67.0
44.8
57.6
2195
9514
5559
T10240
82.7
26.9
73.1
44.1
56.7
2056
5458
3036
3405
3489
Ts192
81.8
24.8
70.9
43,9
55.35
1795
4465
2560
2841
2915
T7108
68.6
43.0
54.18
1553
3726
2251
2323
2463
Taias
66.2
52.35
2821
Tame
77.7
62.2
49.4
1054
2481
1484
1564
1646
T2045
73.2
15.8
$6.9
35,9
45.45
721
1029
936
1084
943
Group.-0.4
74.6
25.0
69.2
43.1
$3.0
1069
4747
2162
2536
2629
Group.-0.2
78.1
28.1
68.0
44.4
54.7
1135
5628
2635
2944
3085
Group
77.0
29.0
69.5
44.9
$5.1
1228
6331
2808
3271
3402
Group.-0.05 00005
74.4
30.2
65.5
43.1
53.3
1193
4839
2457
2703
2798
LAPO
84.3
29.3
71.2
51.7
59.1
2354
8318
3568
4863
5581
LASER Fy 2048
29.2
71.6
44.1
57.1
1913
4815
2493
2767
LASER by 1006
31.0
74.1
58.7
1914
5915
3579
L, 8192
85.6
31.5
75.9
60.2
6589
4162
4547
4509
LASER 7y 1024
83.0
30.6
72.8
43.7
$7.5
1362
4991
2560
2837
2862
LASER 2z 2048
82.2
31.0
73.3
46.2
58.2
1623
5158
2572
2960
3059
LASER D Ly
84.2
34.2
75.3
47.3
60.3
1872
5750
2981
3474
3520
Vanilla Truncation
C(y) 0
A(y) = 1
S(v) R((ry)
if [(y) < LTT if L(y) > L1
where LT 10 and p ==0 0.
ThinkPrune
C(y) = 0
A(y) 1
S(y) = R(I, v)
if & LA
if L(v) > LA
where LA {10, 7.5,5).
Efficient Reasoning
Cly) = R(x, v)
A(y) I(R)
S(y) -a (Liz) MMan(y)
STD(I.)
where y 10 and
25

---

Published as a conference paper at ICLR 2026
Kimi-k1.5
C(y) R(x,0)
X(y)
0.5 L(y)-Amin
if 1
S(y)
Lmax Limon
min 0, 0.5
L(0) Linnn
if 0
Lumas Luiin
where Lmnn = 2.5 and Lmax 20.
L1-Exact
C(y) R(Z. v)
X(y) = 1
S(y) (T(y) Trrl
where a 0.03 and LT 10.
L1-Max
C(y) = 0
X(y) I(R)
S(y) = clip(a (L(y) - LT) + 0,
where 0.03 and LT 10.
LASER
C(y) R(x,8)
A(y) I(R)
S(y) = a I(L()) < LT)
where Ly = 10.
LASER-D
C(y) R(x,))
A(y) = 1(R)
S(y) ((((g)) LA)
where LA {10, 7.5,5).
For methods with multiple adaptive target lengths LA values (ThinkPrune, LASER-D D, and LASER- DE), different shades of the base colors were used:
Correct responses (blue): RGB(26,71,1422) 161( RGB(12255222))
Incorrect responses (red): RGB(139,0,0), RGB(183,50,40), RGB(224,93,86)
N QUALITATIVE ANALYSIS ON EFFICIENT REASONING
We conduct a qualitative analysis on the trivial question "1+1=?" and the MATH500 dataset to understand how RL improves reasoning efficiency. Comparing the original ScepSeek-RI-Distiine Qwen-1.5B model with the MASHR-D-trringg version, generates repetitive "self-reflection" even for trivial questions, while the trained model directly provides the answer. Our analysis of MATH500 (detailed Figure 15 Figure [6 and Figure 17) reveals that the original model tends towards verbose, redundant explanations of single ideas. In contrast, the LASER-D-traines D-trained model expresses the same concepts more succinetly using structured formulas, significantly improving token efficiency. This suggests RL-based approach not only reduces unproductive backtracking but also encourages a shift towards more concise and direct expression.
how the original model
26

---

Published as a conference paper at ICLR 2026
Quastion: It $100
Mrac 2/0c 215, what the value of 2) +it RJJ)7 Express your answer fraction
<thint
<think-
Okey so need find the distance between too rts
need find the
and 4, Himm, loommmbr from school that therds formula this kind probiom mu think & someetinng with the distance formua?? Healt thikk its led the discant formuls it's used find the stance benween two points in coodinete plane
points
coo plane something square
oquares coo me wiite that down.
Let me try to rocall how that form works. think it's somothinn llue takinng the differonce in the cooncinates squoring it taking the df flore soe in the cocrdhates, squering mat aditing them togethes and then saking the equare Yeah soundh
The formula d sertiss -ctyn2 byd y1112) can plug coerdinanes
formula.
Let aesgg 14,, y1u as 12, -5] tet, yot ae 1-4, 30 Then the difee ences would be x2 xi 22 and y y a 3- (-6) 9.
poina, say vt) jull yúz, distance d beowoor them caloulitedd by formula
d sogtlk2 wit-2 92 y11-2)
So now square those di I I 6)- - and 5+2 a1. Then add those togather: 36 - Bt 117.
F mailyy the distance the aquaro noor of 112 should check mik oan tow simpitee The factire of 117 are 5133, so upstpe13g (hoogett 3)
Fist ined di coordinates poirts. yl is the
point that and
second
So the destance is 3813 units.
Now, insed to find the difierances theik and coordinates So subtruct an from which is 20, Let caloulate that Himms 4 miniss 2 S. that is 6 Nest, subtract yf from 52, which is (3 I Wark. sustracting requtive line acding the poirtiee that would 3 which St
"Fia Answert
the distance betonen the points is (penst unts.. <ithincs
find distanoe betwoon points 6f) and formulo.
So now hare the differnncos 6 for the coo coordinato and 9 for the coordinats ats. Buc hoe wetre poing to square them, reganie gn doeent muther because squaning nequtive number makes pouitive Si. (-4)12 2h2 a
Nov, lined to aquared difieroncss together
Subostucing the given points 1.y 110 and W. 800 WKAR 12. caliulate the differences
Therefore, the
2-x
noad radicall
factorn (imm, 117 dissted by
M (
perfect is numbees
x
y.1
sgrt)137 sarage since sqrt20 a becomnss 27
V
Sqguring there diffrrences and witling them
Whet, let me mabe sureil dis that coreetly so, timee 13 117, and eqyaat 3, you, that wou ld make semna So, putting et together the destanee oid tamns the squuane root of 13.
V
1-002 2 56 51 117
V
Les me ass vertly the stops aqain make sure iokvvit make Coondimmeen 12, --i and c-t, compute you yit 1-60 Sqquare theme 1yn2 sha summ att Square 1q1t)110), fector: 117
Taking the moct:
sqrtten1. 1.3)) 3hsert112)
Yeat: correct
plotting way approach Pythagorean graps
distance Wilooad unit..
hopotaruss
(550 Takens]
diffewenee conrril rates. the horoovte would the di eistance bebween and inceed urnts, vertical leg would betwear and serch it un then, uain Pythagoraa the Inpoteruse would be sqptid-22 i 2-1) spribi + R13 sQriE117) foy, oame result ss bwfone, wh in neweaating
distance
Arother ghtt Could have mosd up the onder sustruction? Like. iil nadi doneict ic -st instaed of att would that change anythred of bet me sen, do oil
that conedd (-ft 7 4 come as beitow And yt- oit -1 wiich squared in also So, oome mestt and + (19012 +81 312, So, that's conustent
[2145 Tokens]
Figure 15: The full example of Figure
27

---

Published as a conference paper at ICLR 2026
Up 1/ veectendt qu = 1/Muntty Moeer (5-0 (find a way Meum (j 1)*iinlly toum 11"iinlyy Viar terms 5p5 and 5
<thiikk
Dayy, no hown this penblen needto repernc pencolust
have proslem whene
double of and which known sume of pepppeca on parne nupes sespectivll Las mo writ downiwhel moad findl
know
Fist is defieed surs from = enfinl 1,6: menember enovn buse probiem,, andits
over all and intinty Mi
durty i sum from 1 to limit of t,k0 and this mork ralled Kan conzaint in soo't again naghe
And 3. 199, Euikwigen 19t
compuree
The mai gosl in tiss rxprension for the diubille sure
Thanefors
ZuIli 1pth gudkeigh tag ket
this
S E.tmaap on 10her 2.0mi2pp (Umn- 1em%..
from otly So, need to wercower 5. virich a doub cun moer incii saa ring pe earh twrnis ownr ()
Wrich can within ax
neeseer that montoonedline iito cingle eun changing vortes or using porno eity Muplic cas find veteii sutiun that dimpl thes
E. 2ear
the doulliees
Siee E inwtter tark so prodge trint -1
symmetric do par can make sutbettation
setin e and than aou many timwa each appears the doute sum
Theethon, 10 piig
ki tn-n for ecarople,
exprest
of mfamimn onlyy IVTT aedd (1ul2 und rnd,
11.0 12.2s (ST)) stu.
10
The finil croona
Recausn each these sue ja- ( lc whhne iu each these sent botas 1d91
Voocedto of
Sona can vetttenee
[Total Tokens: 1114]
S Jejengen to- 1ee!
mesimpplyy expression
tisst whit 1eit Lhi- Lit..
Theeffree 2 (wu2)=1200> 1899
E.lengen thr: "nannige tra
Butwolt origina i 2.ln tp- which peaeee
joetper
So, it subnoct from ggnt dhwzyan tpot
furly sestrucing
Thandion sublltuting buck
s dp- 1h- - e Ti
when
JeJeedd cendgennrn sostt 1091 3. 2. tyst- Z.0m/)e
tXP go-1 11--i 12 pinq So. s -ig
But check Let compust
This ado Mlaiing show to rpprosiion Vo
to fa te
Thantore tial anoweris: y Phooetp--111
[Total Tokenns: 17523)
Figure 16: Additional case study demonstrating the evolution of reasoning efficiency. In this example, the original model required over 17K tokens to solve question from the MATH500 dataset, while our trained model accomplished the same task using only 1K+ tokens.
28

---

Published as a conference paper at ICLR 2026
Question: if $fxt 210x-22 what the value 56:2) -R:1))-R015 Express your answer as common fraction.
equal divided by The quastion uaking me find the wakee of fi-2 plus fi-11 plus sit and expreess ameser as a common froction himm, me break this down stop by step becausee wwnnt to madoe sume don't make any mistkkes
Ovay have this function foo
Cluy, so inedd fi-2) II-1 100 where fid given by Be 2000 2) Hmm. let' start by plugoing in each salue step by step.
1::2
6: (21(2) 2012 22 2V1-4) (-80014)
[t-1)
rirs, know that to find Ni-7), fi-1), and 90L reeed bustutte down separataly.
f:11- (3%(-1)
22- (43 22/133 (-910-3)
valuees function 156,
5/3.
(00) 2000 23 (-291-2)
Starting 5) 21:
Now, add them logether 2 5/3 Thars 5/3. Convert 3 to thirds 9/3 5/3 14/3.
Let me comp pube the numerator and denom nator repartely
to avoid mistaess The numerator is any 2 which is 6. than so whbiteet 2t no 2 is The denominettr is (-2) -
The value () + R:11) 10) (bowsdyfflicc 4//30 10:10
which 14, So putting it together en ft:2) r- (81) Hmm dividing two megatives gives positive, that's 2: Got that 01-2 is
where fidt Vrad
21 W:
Calculate V (1221 MAIR-2) (MauBr 24-2
(rad) 6 211-4) VMrci ()) 4) - 202. calcurons V M-D
Nest compute
MMM 1) -Mocg--11 20: 1 21 stact 3 20:
Mracc 50-5) (frectjunna Calculate 100 viift00
n--0 (21(1)) 21 01-10
Mracal0) 2100 VMact 2) Ioled the MiectS)33)
results togetherig Vrrc//1133
Now 90010) - (310 21 21
(frocBl1a) + strodt)331 4aangh) value is Wpbosddfffrreeeeeegg
Compute ru numerstor and centominatoo Numwotor 1'0 is minus 2 2. Denommnttor O 2 2 So 113 a 2/N 2, which So 100 Nehe newd and thase thewe together 1--2 1-11 which (5/3)-
[567 Tokens]
numbers fractions
aquel 663
ready A/d.ic 14/3
fractionn adding them uoll/3
numeratans +
on, let me doutte-check my caknibe one because 5/3 1 513, which is 14/3. Yeah that soems right.Btt let mhe make suri didh''s make any mistabes camps ping each tici Maytee should check each su bestttttonn againStarting with Numorator 3nd 21 indeod 6, minus 2
Denam inston 6-1) - is So -1/4 2. That conrectr--1n N.merttt 31
Denamin cntommnnttr 6-1)-
So
is So -21-2 CarnectAccdingg 2 5/3 + 1:2 is 513, and is 3/3 6/3 5/3 3/3 14/3. That and 2/3. Himm, I worder if can express that as mbod mumber but the quection says to express av comman fraction so 14/3 the WWtt out just to make sure eieett make mistall the inbal funation. The function tho BBr 2500 Weah loles right to
-2,
substtiuted conectyyNaat
100 plug
20/0
is (-2M-2)
Wolt
make sura didn't mass any computation emors 3n),
f minux -% dioded by (-4) 2 Cornect 1h -1, minus 2 3. disited by (:3) S/8 Correct is minus 2 12, divided by (-2) 1. So aiding them Ingether is 5/1 5/ is 57% and is a7 So 6'3 5/3 3/3 is 14/3, Yes, that sooms right Walt but just thought there chemae that = 2 problem?? Because the function undetined
this
ant
urlely compute
Therefore: sum 14/3.Bu be sbsolutely
thorough me compuse each tern againcfor fll2
N Numenstor 171-) -6, minus is Devominator (-2)-
2 4.5c 8 divddd by 4 Correc for ft
Nameraton 271-1 -3, mius 2 Demoninttor (-1)-
2 -3 So is diiddd by is 5/3 Correct 10)
Nameraton 310 is minus 2 natoomnator: D 2
is is 6/3 2:5a divided by - Connec Adding them together
18
number? maybe represent
is with remainder that's and 2/3 but the quention says ropress an common traction so 1433 tttetttttte y if were tette as a desimal It's spprosimatdly 6666.. but the qu quentin wants en woxr fraction, so 1479 is the way tnikk thar' thorough. don't soe any mistalics in my cakoulaions. so I'm confidant that 14/3 the cornect "Finall Ansow treh value 1d f6-7) 1-11) dioosss
[2095 Takens]
Figure 17: Further example demonstrating improvements in reasoning approach
29

---

Published as a conference paper at ICLR 2026
O LIMITATIONS
Despite our work's effective improvements in performance and efficiency, limitations remain. Our and most previous works focus primarily the math as it provides an excellent verification environment and testbed for validating new methodologies. We believe further validation in code generation and agentic tasks would be valuable to determine if similar favorable trade-off's can be achieved in these contexts. Importantly, our methods were not specifically designed for mathematical tasks but were developed as domain-agnostic approaches that should naturally extend to other areas. In future work, we plan to explore more realistic scenario tasks, particularly those involving agentic reasoning, to further validate our approach and improve the efficacy- -fffcccyefficcencyy trade-off in more broader areas.
30


Reviews:
[{"review_id": "JhZmZRmest", "reviewer": "Reviewer_7JFv", "summary": "This paper **(LASER)** aims at addressing the _overthinking_ phenomenon in Large Reasoning Models (LRM) through a novel RL reward shaping method. Specifically, it first draws inspiration from truncation-based baseline methods and existing length rewards in efficient reasoning by proposing a step function reward. It then takes problem difficulty and the RL training dynamics into account by further introducing an adaptive module **(LASER-D)** that controls the reward shaping hyper-parameters on-the-fly. Experiment results on various in-domain and out-of-domain benchmarks demonstrate that LASER effectively reduces reasoning length with minimal influence on the accuracy.", "questions": "Most questions are raised in the weaknesses. Other than above:\n\n1. [Question] During inference time, does LASER further uses the difficulty assessment module to evaluate the difficulty of a new query, or simply let the RL-finetuned model to generate the reasoning without any pre/post-processing?\n\n2. [Typo] **L144:** lack of a verb after \"In this section, we []\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is well motivated and shows a good understanding of existing methods in LLM efficient reasoning.\n- The experimental study seems comprehensive (applying the method to 5 LLMs and evaluating on both in-domain and out-of-domain tasks).\n- The paper is rigorous in terms of including important details like hyper-parameter choices and abundant ablation studies in the main text & appendix.", "weaknesses": "1. My main concern lies in the numerous *manual design choices* (often appearing as hyperparameters) despite the paper’s claim that most modules are *automatically adaptive*. For instance, the criterion for defining $L_A$ is set as the smallest value ensuring $ECR_d \\geq 1$, where “1” represents the threshold for “at least one complete and correct response.” Why fix this threshold as a constant rather than make it depend on the rollout size $K$? Similarly, the lower bound length $L_T$ appears to be manually chosen, and the experiments suggest that its value is adjusted for different model (e.g., 4096 for the 8B model and 8192 for the 1.5B model). Other examples of such manual tuning include the choice of **three** difficulty levels and the update interval of $N$ steps. For practitioners, what concrete guidance is provided for selecting these parameters in new settings?\n\n\n2. While the overall presentation is clear, I find the explanation in Section 4.2 (*Automatic Adapting Mechanism*) particularly difficult to follow. In particular, the term “group” is confusing, as it can refer to both the “rollout group” (i.e., the $K$ rollouts for a given prompt) and the “difficulty group.” As a result, any statement that simply mentions “group” without specification becomes ambiguous. Additionally, the concept of $ECR_d$ seems to represent a per-difficulty quantity, yet its introduction follows immediately after the sentence “we sample $K$ responses for each query in the monitoring set, and $ECR$ is computed as…,” which makes it sound like a per-query statistic instead. I recommend a thorough revision of this subsection to improve clarity and precision.\n\n3. [Minor] Based on Table 5, I notice that while LASER does work on non-Qwen models as well, the magnitude of length reduction is perceivably smaller than the two Qwen models highlighted throughout main text (on most benchmarks the reductions are $> 50$% on Qwen models, but only around $40$% for other models). Is there any intuitive reason that can explain such discrepancy?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WccG1rAs0z", "reviewer": "Reviewer_k3oM", "summary": "This paper addresses the \"over-thinking\" issue in Large Reasoning Models (LRMs), where models generate excessively long reasoning traces. The authors propose a unified framework for viewing various RL based chain-of-thought compression methods through the lens of length-based reward shaping. Building on this, they introduce two novel methods: LASER, which uses a step function to reward shorter correct responses, and LASER-D, which dynamically adapts the target length based on question difficulty and the model's evolving state during training. Experiments on several math reasoning benchmarks show that the proposed methods achieve significant improvements in token efficiency while also showing some improvements in reasoning accuracy compared to baselines.", "questions": "1. Could you clarify the choice of alpha=0.01 for the L1-Max baseline, as the original paper reports alpha=0.002? Was this value tuned, and if so, how?\n2. Can you confirm that all reproduced baselines used the identical GRPO implementation and hyperparameters (including the \"clip-higher\" strategy), with the reward shaping term being the only difference?\n3. How does the difficulty-aware mechanism handle the \"cold-start\" problem at the beginning of training, where many prompts may have zero correct rollouts, potentially making the difficulty estimation noisy? Maybe this is not an issue for the training dataset used, but could this be a limitation for training on harder datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents empirical results, demonstrating a strong trade-off between efficiency and performance. The proposed methods, consistently reduce token usage by a good margin across multiple models and benchmarks while often improving accuracy.\n2. The introduction of a unified framework for RL-based CoT compression is a good conceptual contribution. It provides a clear and structured approach to understanding and comparing different reward shaping strategies.", "weaknesses": "1. Framing and Claims: The framing of performance improvements and baseline comparisons could be strengthened. The paper claims adaptive methods like AutoThink and Thinkless remain \"verbose on hard ones (AIME).\" However, the data in Table 4 shows these methods do reduce token usage on AIME compared to the original model, albeit less than LASER-D. Furthermore, AutoThink achieves higher accuracy on AIME than several LASER and LASER-D variants, which complicates the claim. \n2. Deepseek-r1 distilled models are typically known to be very verbose shown in several recent works. Results on Qwen3 or even DeepScaleR would make the results more impactful.\n3. For fair comparison, results should also be compared against officially released, downloadable models from prior work when available (e.g., L1-Max) that have the same starting model, since there may be small changes in training setup.    \n4. The unification framework is a strong point, but its coverage doesn't seem to be exhaustive. For instance, It is unclear how it would accommodate methods where a budget is enforced structurally, for example, by inserting an \"end think\" token after a certain number of steps (eg, in case of Qwen3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9CXOkK821I", "reviewer": "Reviewer_MWVL", "summary": "This paper proposes a simple framework, LASER, and an advanced version of i,t named LASER, which use reward shaping and dynamic reward shaping methods to do reinforcement learning, respectively, to improve the large reasoning models' efficiency and reasoning ability. The paper starts from a bunch of findings, which motivate their method contribution with difficulty dynamic reward shaping.", "questions": "1. Is it possible to include experiments with other model families, such as Qwen3 and LLama\n2. Is it possible to include some experiments in other reasoning tasks, such as coding or commonsense reasoning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper conducts a lot of evaluation experiments, which point out the deficiency of the current reward shaping methods for efficient LRM training.\n2. The design of the method, especially how to monitor the difficulty dymanics, and the logic of the paper are reasonable. \n3. The experiment section includes a lot of baselines and different length settings, and different sizes of models.", "weaknesses": "1. While the method includes some novelty from the length penalty findings, it seems like an improved version of the previous method.\n2. The task in the experiment is limited, which only contains mathematical reasoning.\n3. The model family is limited, which only contains 2 versions of DeepSeek models.\n4. While the tables in the experiment section show the effectiveness of this method, it is still a little bit hard to compare methods because sometimes other methods are better at token usage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DTUl1D2Tk6", "reviewer": "Reviewer_2Nyo", "summary": "This paper addresses the problem of inefficiency in Large Reasoning Models (LRMs) caused by over-thinking, where models generate unnecessarily long reasoning chains. The authors propose LASER and LASER-D, two adaptive length-based reward shaping methods that dynamically adjust reasoning length according to task difficulty. The approach provides a unified framework that formalizes existing CoT compression techniques and introduces a fully automated RL method that balances conciseness and reasoning depth. Experiments show significant improvements in both accuracy and token efficiency across multiple models and benchmarks.", "questions": "1. How stable is the in-batch difficulty estimation across training, and have you tried combining it with more static difficulty signals?\n\n2. Did you observe cases where the model learned to bypass the reflection keyword filter using alternative phrasing?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents a unified conceptual framework that reformulates various length-control methods as instances of length-based reward shaping. This brings theoretical clarity to prior work and establishes a strong foundation for the proposed methods.\n\n2. The design of LASER and LASER-D is elegant and well-motivated. The difficulty-aware mechanism that adjusts reasoning budget based on in-batch correctness is intuitive and largely automated, reducing the need for manual hyperparameter tuning.\n\n3. Empirical results are very strong. LASER-D achieves large gains in both accuracy and efficiency (e.g., +5.3% accuracy with a 64% token reduction on AIME2024), demonstrating a clear improvement in the accuracy-efficiency trade-off.\n\n4. The evaluation is thorough and convincing. The authors show consistent gains across model scales and domains, and the behavioral analysis offers meaningful insights into how the method reduces unproductive reasoning.", "weaknesses": "1. The difficulty estimation based on in-batch correctness is a noisy heuristic. It may misclassify easy problems as hard, leading to inefficient allocation of reasoning budget and potential instability during training.\n\n2. The “automatic” mechanism introduces new hyperparameters (e.g., monitoring size, update frequency) whose tuning is not well discussed. This adds complexity and may limit practical applicability to new domains.\n\n3. The analysis of reasoning behavior relies on keyword counting, which is a coarse proxy for reflective thinking. The model could learn to express similar unproductive reasoning with different phrasing, weakening the interpretability of this analysis.\n\n4. The step-function reward creates a sharp incentive boundary at the target length. This may encourage minimal valid responses rather than richer reasoning, potentially sacrificing explanation quality in borderline cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)