ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning
Authors: Jingyang Yi
Justin Wang
Sida Li
Venue: NeurIPS 2025 poster ()
Keywords: Reasoning
Efficient Inference Methods
Reinforcement Learning
Primary area: deep_learning
TL;DR: We propose ShorterBetter, a reinforcement learning method that trains reasoning models to generate concise yet accurate Chain-of-Thought traces by rewarding the shortest correct response among sampled outputs.

Abstract: Recent models such as OpenAI o1 and DeepSeek-R1 have demonstrated strong performance on reasoning-intensive tasks by generating extended Chain-of-Thought (CoT) traces. While longer reasoning helps with thorough exploration of solution paths for complex problems, it also often leads to inefficient and redundant outputs—a phenomenon commonly described as $\textit{overthinking}$.  In this paper, we propose $\texttt{ShorterBetter}$, a simple yet effective reinforcement learning method that enables reasoning models to learn their own optimal CoT lengths without manual supervision. We define the $\textit{Sample Optimal Length}$ (SOL) as the length of the shortest correct response among multiple generations, which serves as a dynamic reward signal to guide the model toward efficient reasoning. Applied to DeepSeek-Distill-Qwen-1.5B/7B as base models, $\texttt{ShorterBetter}$ achieves 50\%-80\% reduction in output lengths in both in-domain and out-of-domain reasoning tasks while maintaining accuracy. Our reasoning trace analysis shows that $\texttt{ShorterBetter}$ refines the structure of the reasoning traces by reducing unnecessary repetition, excessive self-verification, and over-exploration of alternatives.

Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning
Jingyang Yi' Jiazheng Wang"
Sida Li
Data Science Institute, The University of Chicago
Chicago, IL 60637 fjingyang22, jiazhengw, (sstar2 edu
Abstract
Recent models such as OpenAI ol and DeepSeek-RI have demonstrated strong per- formance on reasoning-intensive tasks by generating extended Chain-of-Thouge (CoT) traces. While longer reasoning helps with thorough exploration of solu- tion paths for complex problems, also often leads to inefficient and redundant outputs a phenomenon commonly described overthinking. In this paper, we propose ShorterBetter, a simple yet effective reinforcement learning method that enables reasoning models to learn their own optimal CoT lengths without manual supervision. We define the Sample Optimal Length (SOL) as the length of the shortest correct response among multiple generations, which serves as dynamic reward signal to guide the model toward efficient reasoning. Applied Birkk-iiiil---ee-----
base models, ShorterBetter achieves 50%-80% reduction in output lengths in both in-domain and out-of-domain rea- soning tasks while maintaining accuracy. Our reasoning trace analysis shows that heerterBetter refines the structure of the reasoning traces by reducing unneces- sary repetition, excessive serificifi,,,onn, and over-exploration of alternatives.
Introduction
Improving the reasoning capabilities of Large Language Models (LLMs) has recently led to sub- stantial performance gains in highly structured and complex tasks, such as mathematical problem- solving (Shao et al., 2024) and competitive programming (Jiang et al., 2025). Exemplified by OpenAl ol (OpenAI, 2024) and DeepSeck-RI (Guo et al., 2025), these reasoning models are incentivized to generate extended Chain-of-Thour
(CoT) steps during inference. Such extended reasoning often includes detailed intermediate steps, reflective self-verifications, and exploration of alternative problem-solving strategies. These advancements are widely believed to enable LLMs to exhibit human-likk like System-2 slow thinking (Evans, 2003; Qu al., 2025: Li et al., 2025), wherein significant test-time computational resources are deliberately allocated in exchange for improved accuracy and robustness challenging reasoning benchmarks (Jain al., 2025: Rein et al., 2024).
However, recent studies have shown that excessively long reasoning processes can be counter- productive, leading not only to computational inefficiencies but also to the so-called overthinking phenomenon (Chen et al., 2025; Luo et al., 2025a), where additional reasoning steps yield diminishing returns or even negatively impact model performance (Su et al., 2025). For example, identify
Equual contribution
'The source code, model weights, and analysis pipelines are available at: https://jjaaaeeee wang wea.github.ii
yea.ge gi titbhubii chub.io//ttei //horterBBetter. SotorterBettrr_PPoeeeaPPaeee
*We provide an experiment in Appendix B.3 demonstrating that incorrect reasoning traces often have longer
output lengths.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

AIME
NUYMPIAD-BENCE
AMC
MINERVA
0.5
Accuracy
0.4
0.2
0.0
2500
5000
7500
10030
2000
4000
5000
8000
2000
4000
5000
3000
1500
3000
4500
6000
MATHQA
BBH
MMLU
LIVECODEBENCH
0.5
Accuracy
0.4
0.2
500
1000
1500
2000
300
600
910)
1200
1500
250
7:0
1000
1250
2500 5000
7900
10000
Avg length (tokens)
Avg length (tokens)
Avg length (tokens)
Avg length (tokens)
DeepSeck-RI-Din RI-Disstll-Qwee--..5 Qwen2.5-1158. Instrrcct
ShorterBetter-1.58 (a=1)
Training Efficient-1.SB
ShorterBetter--.55 (u-2)
Figure 1: ShorterBetter exhibits strong length reduction performance (1.SB models). We report accuracy and output length of five sets of models on various tasks. Models include DeepSeek-R1- Sollll-ween-111.1 Qwen2.5-1 1.5--------- Instruct (non-reasoning model), ShorterBetter- and a =2) and Training Efficent- 1.5B (Arora and Zanette, 2025). Our method achieves significant
(a =1 length reduction compared to all baseline reasoning models while maintaining accuracy.
several common problematic patterns in lengthy reasoning traces: (1) verbose repetitions without new insights, (2) early stopping of promising paths. (3) continued exploration of alternative solutions despite having arrived at correct answers, and (4) collapse of reasoning coherence following very long outputs." These observations indicate that lengthy reasoning traces are highly compressible, mo- tivating techniques producing more concise reasoning chains that improve computational efficiency while preserving overall performance.
Existing approaches targeting efficient LLM reasoning primarily focus on explicitly controlling reasoning length, for instance, by imposing length budgel based on user specifications (LI; Aggarwal and Welleck (2025)), or by incentivizing shorter reasoning paths through comparisons with reference models (O1-Pruner; Luo al. (2025a)). Further discussions in Section 2. In contrast to these explicit control strategies, our work is based on the hypothesis that reasoning models implicitly possess an optimal reasoning length (OL), which is dependent on model capacity and problem difficulty. OL thus represents an in be-beteeen "sweet spot": reasoning that is shorter might contain insufficient analysis, while reasoning that is longer risks redundancy and overthinking. This hypothesis has recently undergone theoretical formulation (Wu et al., 2025) and empirical support (Su et al., 2025). Since OL is intrinsic and not directly observable, we approximate it with the Sample Optimal Length (SOL): for each problem, we generate multiple outputs and define its SOL as the length of the shortest correct response. Leveraging SOL as a practical proxy for OL, we propose ShorterBetter, reinforcement learning method that teaches the model to dynamically identify and steer towards efficient reasoning lengths during inference:
related approaches are provided
In summary, our contributions are as follows.
1. We propose ShorterBetter, a novel reinforcement learning method that guides reasoning model autonomously learn its optimal reasoning lengths by defining and using Sample Optimal Length as a dynamic reward signal.
2. We empirically demonstrate that ShorterBetter, when applied to 1.5B/7B base models. achieves substantial (50%-80%) reductions in output token length on both in-domain mathe- matical reasoning tasks and diverse off-o--domain benchmarks, while maintaining accuracy.
3. We introduce novel reasoning trace analysis frameworks to demonstrate how ShorterBetter enhances reasoning efficiency. Specifically, the model reduces verbosity after finding the correct
We include concrete examples for each of the aforementioned patterns in Appendix D.

---

answer and refactors the overall reasoning traces to adopt more confident and targeted reasoning
strategies.
2 Related Work
Reasoning in LLMs Since Wei et al. (2022) demonstrated that CoT prompting enhances LLM reasoning, following research has explored ways to scale test-time computation- as parallel sampling (Wang et al., 2023) and iterative refinement (Madaan et al., 2023) to improve performance. This led to Large Reasoning Models (LRMs) that autonomously generate extended CoT's through reinforcement learning-basee based post-training (Zhang et al., 2025). Examples include OpenAl's ol (OpenAI, 2024), DeepSeek-RI (Guo et al., 2025), and Qwen-QwQ (Yang et al., 2024). These models internalize the ability to deliberate over multiple steps before producing final answers, achieving significant performance gains reasoning -ontoiinnnnittnvveee intensive benchmarks (Jimenez et al., 2024; Jain et al., 2025; Gao et al., 2024). Our work addresses a complementary challenge: how to dynamically optimize reasoning efficiency without sacrificing accuracy?
The Overthinking Phenomenon This challenge, which was initially observed as verbosity and increased computational costs in CoT prompting (Nayab al., 2024; Han al., 2024), has become more pronounced with the advent of LRMs LRMs-following the identification and understanding of the overthinking phenomenon (Chen et al., 2025; Su et al., 2025). This phenomenon is typically understood through two primary lenses. Firstly, overthinking is recognized as a significant source of computational inefficiency. In this view, LRMSs expend excessive resources generating overly lengthy and often redundant reasoning traces, especially for simple problems (Lee et al., 2025). This inefficiency manifests as increased token usage, higher latency, and greater computational expenditure (Aytes et al., 2025), and might be exploited adversarially to induce slowdowns (Kumar et al., 2025).
Secondly, overthinking increasingly understood as factor that can directly impair practical performance. Fatemi et al. (2025) find that excessive reasoning can degrade logical correctness and introduce irrelevant information; Su al. (2025) reveal that beyond certain threshold. longer reasoning does not necessarily improve and can even degrade accuracy, as models may misjudge problem difficulty and introduce compounding errors; similarly, Wu et al. (2025) empirically and theoretically demonstrate U-shaped performance curve with CoT length, showing that excessively long reasoning paths are susceptible to noise and error accumulation, ultimately reducing correctness. This performance degradation has also been observed in agentic tasks where prolonged internal deliberation correlates with lower task success rates (Cuadron et al., 2025).
Efficient Reasoning with RL Many recent and contemporaneous methods have been proposed to control the reasoning length in LRMs. Here, we specifically highlight RL-based post-training strategies directly relevant to method, leaving readers to the survey by Qu et al. (2025) for an extensive overview. One category of methods uses an explicit length budget. For instance, L1 (Aggarwal and Welleck, 2025) treats the budget as user-specified parameter, while DAST (Shen et al., 2025) adjusts it adaptively based on problem difficulty. These methods then use RL to reward generations that adhere closely to this budget. Other methods design length rewards differently. 01- Pruner (Luo et al., 2025a), for example, compares generated lengths against those from a reference model. More line with our proposal of sampling multiple responses, Kimi 1.5 (Team, 2025) and Training Efficient (Arora and Zanette, 2025) utilize the shortest and average lengths, respectively, among all sampled responses as reward basclines. In contrast, ShorterBetter innovatively anchors its reward function to the length of the shortest correct response- design inspired by recent studies on the existence of optimal reasoning lengths LRMs (Wu al., 2025; Lee al., 2025; Su et al., 2025). We include extended comparisons with related and contemporary works in Appendix C.
3 Method
In this work, we aim to guide the reasoning model to think for the right amount of time, which requires knowing its optimal reasoning length (OL)a prompt-dependent quantity we formally define in Section 3.1. However, calculating OL in practice is challenging due to the absence of precise measures for both problem difficulty and model capability. To address this, we propose in Section 3.2 a sample- based approximation of OL. yielding practical length reward. Finally,
3

---

Prowpt
Re can use the Eornule wi/fl firal
Correctness Length Gap
Reward
degrse
F2E saken
by stopes
Here concise find the engle... Sinally giver (r55
Verify and Count
G
1.0
B
-0.4
0.6
Receive proept
2
0.0
-0.3
0.3
(3) deverata roloute
Enkers
+
:
Let es recall the trict solve this problen type...
1.0
0,0
1.0
500 Sokers
O
obtsin result: 45"
somple optimal length
langth revard
LLM MM (6) Updata poroeaters
GRPO
Advantage astimate
coloulate loss
Figure 2: Flowchart for the ShorterBetter pipeline. We detail the reward design in Section 3.2 and the GRPO algorithm Section 3.3. Reward values in display are rescaled for better illustration.
integrate this reward design with Group Relative Policy Optimization (GRPO; Shao et al. (2024)) in Section 3.3, resulting in simple yet effective reinforcement learning method that encourages the model to autonomously discover its optimal reasoning lengths through on-policy learning.
3.1 Optimal Reasoning Length (OL): Formal Definition
Problem Setup Let Ma be our policy model (i.e. base LRM) with parameters 0, and D be the training dataset. Each problem (2t, E 'D consists of prompt æ; and a reference response y Given I4, for each rollout the model generates a response y = (v' ........' y²) autoregressively from pa(- xz)), where y² denotes its k-th token and ((y) = is its length. For reasoning tasks, typically includes long CoT tokens (often enclosed in </think> tags) followed by final answer.
To evaluate response quality, we assume access to a verifier that assigns a normalized correctness score (y y E [0, 1]. where higher scores indicate better alignment with y". For fixed correctness threshold e 2 O and probability threshold 6 2 0, the optimal reasoning length is defined as
(z, 0) min (l()) y E V (e,e
where
(1, 0) = (v po (y b.)) 2 A sT(N. c..
Thus, (x;; represents the minimal length at which the model reliably (probability above e) produces sufficiently correct (score above c) response. Intuitively, OL is longer for more difficult problems, as the feasible set (x,; f) shrinks due to fewer correct responses; conversely, OL becomes shorter when the model Ma is capable of generating concise and correct responses more consistently. This intuition aligns with theoretical insights presented in Wu et al. (2025), despite differences in their specific formulation.
Remark
The above formulation applies broadly to both closed-ended and open-ended reasoning tasks. For the remainder of this paper, we narrow our focus to tasks with verifiable answers, such as mathematical and coding problems. These tasks naturally come with binary verifier that outputs
T(y = vt) == (v. E 10,
indicating perfect correctness (1) or incorrectness (0). Consequently, we adopt the trivial threshold choice 1 and the shorthand notation I y yf to denote correctness.
3.2 Sample Optimal Length (SOL): A Practical Approximation
Directly computing the optimal reasoning length defined the previous subsection practically infeasible, as it requires complete knowledge of the model's output distribution. To address this, we introduce the Sample Optimal Length (SOL)-a practical, sample-based approximation of OL. Given prompt 2;, method generates rollouts of candidate responses G(x,) {v1, v2.... ... yn from

---

the policy po(-(z4). We then define the SOL for the group G(z)) as:
min (yEG(=:):Nu===? {(U)), if at least one response is correct,
sson (G(x))
1 カ {(v))
otherwise.
Given the SOI,, we define the following reward function for each response E G(z;):
r(u)) = = a....-11 IIu v -B- (e(,) (SO))G
(1)
where B > are hyperparameters balancing correctness reward and length penalty, respectively. Our SOL-based reward has several attractive properties:
V No human intervention required: Unlike explicit length-budget approaches, SOL is entirely data-driven and does not require manual tuning of problem-specific length thresholds.
Adaptive difficulty and model capability: SOL naturally becomes longer for harder problems and shorter for easier ones, as it dynamically adjusts based on the sampled responses; on the other hand, as the model improves during training, SOL automatically updates reflect the shortest correct reasoning length achievable by the current policy.
Balanced handling of failure cases: When no correct responses exist within a sampled group, set SOL to the average response length neutral baseline. Our approach avoids biasing the model toward either overthinking or underthinking when there is no correct signal.
We defer the detailed discussion on specific hyperparameter choices (a, (,, ablation studies on reward design Section 5.3. To the training dynamics of ShorterBetter, please refer to Appendix B.2.
m) to Section 4, and the
3.3 Policy Optimization via GRPO
To optimize the model under the SOL-based reward function, we adopt the Group Relative Policy Optimization (GRPO) algorithm (Shao et al., 2024). GRPO operates groups of sampled trajectories (in our case, a group G(x;) has n samples) and assigns within-group advantage estimates based on normalizing the rewards [r(v1),- rr(y)] calculated in Eq. (1). Specifically, have
Ti mean(r)
std
as the advantage for the k-th step (token) of the j-th sample, where 'mean' and 'std' calculate sample reward average and standard deviation. Let Me,r be frozen reference policy (usually copy of the current model before few gradient updates), and P Po( v .x,)) denote the conditional probability on the first k tokens, the GRPO loss is
EGRPO (0)
W (A pt, r) - -y DKL [pt [pšllpt..,]]] []
(2)
n
where W denotes a clipped policy improvement objective incorporating advantage Á and likelihood ratio between current and reference policies; DxL. a KL divergence approximator (Schulman, 2020); and are hyperparameters for the clipping threshold and KL penalty, respectively. We provide the full expression and details in Appendix A.1.
Policy updates are then performed by minimizing LGRrO(#) via gradient descent, encouraging the model to increase the likelihood of high-reward samples while constraining deviation from the reference policy. We choose GRPO for its compatibility with group-based reward designs and its empirical success training LRMs (Guo et al., 2025; Face, 2025). Finally, we include flowchart illustration of our full method in Fig.
5

---

Experimental Setup
Datasets We fine-tune our base model on DeepScaleR-previeww (Luo et al., 2025b), dataset which comprises 40K mathematical problems drawn from AIME, AMC, Omni-MATH (Gao al., 2024), and Still dataset (Min et al., 2024). This mixed dataset covers tasks from high-school math problems Olympiad-level challenges. For in-domain evaluation, we holdout subsets of the training math datasets, containing 1K problems in total. Additionally, we assess the out-of-domaan generalization of our method on a set reasoning-intensive benchmarks with total of 6K problems, including MathQA (Amini et al., 2019) MMLU (Hendrycks et al., 2021), Big-Bench Hard (BBH; Suzgun et al. (2023)) and coding benchmarks including LiveCodeBench (Jain al., 2025), MBPP (Austin et al., 2021) and HvmanEval (Chen et al., 2021).
Base Models We use DeepSeek-RI --r-Seek-R--issilllle Ret--isiill--wnn--... and 7B (Guo et al., 2025) base reasoning models and train our models from them. We will refer to our trained model as ShorterBetter-1.55 and ShorterBetter-7B in the subsequent sections.
Evaluations For evaluation, we generate one response for each problem. For each benchmark, we consider 3 metries: (1) Accuracy, defined as the number of correct responses divided by the total number of responses; (2) Output Length, measured as the average number of tokens per benchmark; and (3) Accuracy-Efficiene (AE) Score, composite metric balancing length reduction against accuracy preservation (introduced by Luo et al. (2025a)). AE Score rewards models that shorten outputs without sacrificing correctness, with higher values indicating better efficiency. For the full formulation and hyperparameter choices, see Appendix A.2.
Implementation Details We used Verl (Sheng et al., 2025), open-source RL training library, for post-training. We fine-tune our 1.5B base model with batch size of N 64 and rollout group size of n = 8. Training was conducted for 300 steps with fixed learning rate of 1 x 10-6. Since the 7B variant converges faster due it's better model capacity, we trained for only 200 steps. Due to computational constraints, we set maximum inference length of 6K tokens for the 1.5B model and 5K tokens for the 7B model during training. (We set 16K maximum tokens budgets during evaluation) These constraints only affect the early stages of training, as the mean inference lengths quickly drop below the limits. To study the effect of hyperparameters, we compared two 1.5B variants with a and a = 2, while keeping = 0.001 fixed. For the GRPO training configuration, we set 7 0.001 and 0.2 in Eq. (2).
Baselines To assess the efficiency of our method, we compare our trained models against baselines:
DeepSeek-RI--D RI-DDissill-Q-ee---BBB7 (Guo et al., 2025): Models obtained by distillation from the Qwen series using larger DeepSeek-R1 models. We refer to these models Distill-1 1-1.5B and Distill-7B in the following sections.
1ven2255-..5B77Bnnstttr (Team, 2024): Instruction-tuned models from the Qwen 2.5 family of large language models. We include these non-reasoning models as contrast, i.e., fast thinking without deliberate reasoning. We denote them Instruct-1.55 and Instruct-7B hereafter.
Training Efficient (Arora and Zanette, 2025): We evaluated the 1.5B/7B open-source models released in this work, which have the same base models as ours.
O1-Pruner (Luo et al., 2025a): We evaluate the 7B open-source model proposed in this work, which is fine-tuned from the Marco-ol base model (Zhao et al., 2024).
5 Results and Discussion
In this section, we present and analyze the effectiveness of our proposed method. We compare the performance our trained models against the baselines across a range in-domain math tasks
Since this work does provide 1.5B model, we will only 7B version the baseline.

---

Distill Len.
Instruct Acc.
ShorterBetter 2) AE Scoret
Training Efficient
O1-pruner
Ace.
Len.
Ace. Len.
Ace. Len.
AE Score
Acc.
Len.
AE Scoret
In-donain Tasks
AIME
36.7
11382
16.7
1172
5288
50.0
8821
1.31
6.7
4843
0.28
Olympiad
52.0
7762
38.2
944
50.7
3410
0.44
53.9
5544
0.40
29.6
3884
0.05
AMC
66.3
7414
51.8
1084
75.9
2580
1.09
72.3 4596
0.65
30.1
4304
0,31
Minerva
40.8
5034
38.2
1367
44.1
1341
0.98
40.4
2796
0.40
23.5
5277
0.55
Out-of-domain Tasks
MathQA
83.3
3442
68.9
2585
85.4
980
0.79
78.8
1350
0.34
32.8 6321
0.14
BBH
68.3
2126
67.3
953
63.1
535
0.37
62.1
798
0.17
55.4
5431
-0.35
MMLU
66.3
1528
76.0
269
68.6
567
0.73
69.3
891
0.55
72.3
5979
0.31
LiveCodeBench
50.6
8683
24.7
189
49.5
5228
0.29
49.7
7589
0.04
21.6
9726
0.20
Table 1: Performance comparison of 7B models across diverse tasks. For each task, we highlight the best-performing results across all evaluated models. The AE Scores are computed using hyperpa- rameters & = 1, ?) = 3, and 5. The proposed ShorterBetter model consistently achieves the highest AE Scores across both in-domain and out-of-domain benchmarks. For experimental details and additional results on coding tasks, refer to Appendix A and Appendix B.1.
as well as out-of-domain benchmarks. 5 We then provide an in in-depth analysis of how our method changed the structure of the reasoning traces.
5.1 Experimental Results
ShorterBetter significantly reduces inference length while maintaining performance In Fig. 1, we compare the performance of the ShorterBetter-1.55 against Distill-1.5B. cestruct-1.5B and Training Efficient-1.5B. We report results for two variants of ShorterBetter-1. 5B, with hyperparameters a = 1 and a = 2, respectively. In in-domain math tasks, averaging over all benchmarks, the two models achieve inference length reductions of 77.6% and 79.2%, with corresponding changes in accuracy of +2.5 % and +3.2 %, respectively. In out-of- domain tasks, averaging over all benchmarks, the two variants achieve average length reductions of 77.0 % and 75.5 % with corresponding changes in accuracy of -0.8 % and -1.6 %, respectively." These results also indicate that our method achieves similar performance in terms of length reduction and accuracy preservation with different values of correctness reward hyperparameter a.
In Table 1, we provide comprehensive comparison between ShorterBetter-75 (a 2) and four 7B baseline models.? Consistent with our 1.5B model evaluation, we assess all 7B models on both in-domain and out-of-domain tasks. For Distill-7B and Instruct-7B, we only report accuracy and output length as primary evaluation metrics since they didn't undergo length reduction training. For models trained with length-reduction techniques, additionally report the AE Score, which quantifies performance improvements relative to their respective base models. (See a formal definition of AE Score Appendix A.2)
Our 7B model achieves substantial reductions in output length, up to 70%, while maintaining competitive accuracy compared to the Distill-7B baseline. In in-domain math tasks, averaging over all benchmarks, ShorterBetter-788 attains a length reduction 62.1% alongside an accuracy improvement of +7.1%. In out-of-domain tasks, averaging over all benchmarks, the model achieves a length reduction of 62.3% with an accuracy change of -0.48%. Moreover, ShorterBetters attains the highest AE Scores in all evaluated tasks, outperforming Training Efficient and O1-Pruner.
Compared to prior length-reduction methods, ShorterBetter achieves stronger length-reduution capability. We attribute this performance to the design of SOL, which gives clear signals to guide the model to learn concise and robust reasoning traces during reinforcement learning. For more
In--domain and out-of-domain tasks defined relative ShorterBetter's training dataset. See discussions Appendix A.3
'Due to space constraint, only present LiveCodeBench here and place other coding benchmarks in Fig. 6 and Table 4 in the appendix
FFor a fair comparison, we compute the AE Scores of O1-Pruner using its corresponding base model, Marco-ol-785 whereas the base models for ShorterBetter and Training Etficient are both Distill-7B. Due to space limitations, the performance of Marco-ol-75 is placed in Appendix B.4
7

---

Disral-3
SherterBateer
ShorterBetler-T1
DistilI-7B
0.59
15.0%
5.6%
0.54
Percentage
0.47
0.45
23 3.35%
0.4
0.39
0.37
0.32
10.7%
Aug: mooos
Aug 994 tokons
Verification Self-Carretion
Productive Elaberation & Calcuiation
Non- Substantive Statement
AMC
Math
Olympiad
Minerve
Exploring, Alternatives
Figure 3: Reasoning trace structure analysis. Left: Percentage of remaining output after first appearance of the correct answer; Right: Token-level distribution of reasoning categories on the MATH benchmark.
discussions on the effectiveness of SOL,, please refer to Appendix B.2. In the next section, we show that our method refines the structure of the reasoning traces relative the base model.
5.2 Reasoning Structure Analysis
In this section, we present two novel frameworks for analyzing reasoning structure, which illus- trate how ShorterBetter compresses its reasoning traces and complement the accuracy and token-length metrics introduced earlier. We employ Gemini 2.5 Flash Preview (Google, 2025) as an automated judge (details in Appendix E.1) to breakdown and compare the reasoning traces of ShorterBetter-7B and Distill-7B, shedding lights on how ShorterBetter optimizes for reasoning efficiency.
Output Length after First Appearance of Correct Answer We use LLM- auduuee to iden- tify the first occurrence of the correct answer in the reasoning trace for questions where both ShorterBetter-78 and Distill-7B provide correct answers. We conduct this analysis across four benchmarks: AMC, MATH, Olympiad, and Minerva. For each model, percentage of its total output (in tokens) appears after the first time that the correct answer is stated. The results shown in Fig. 3 (Left), where higher percentage represents more reasoning outputs after the model finds the correct answer. We observe that the percentages of tokens appearing after the first correct answer decreases by 15%, 20%, 17% and 19% for AMC, MATH, Olympiad, and Minerva benchmarks, respectively. This suggests that ShorterBetters efficiently. using fewer tokens on extra reasoning once reaches the correct answer, and will explore more detailed reasoning structure refinement in the next section.
calculate what
reaches conclusions more
Breakdown of Reasoning Traces efficiency, we conduct fine-grained analysis of the reasoning trace structure. Specifically, we develop an LLM-as-a-judge framework to assign functional categories to each sentence in the generated traces from ShorterBetter-7B and Distill-7B and compute the token-level proportion of each reasoning category. We begin by carefully inspecting common patterns Chain-of-Thonn Thought reasoning and curating a set of mutually exclusive categories, each assigned at the sentence level:
To better understand how ShorterBetter improves reasoning
* Pivotal Reasoning: Steps that directly correspond to specific part of the final solution.
Productive Elaboration & Calculation: Necessary calculations, deductions, or planning that support a pivotal step but aren't 't restated the final summary.
Exploring Alternatives: Attempts to try different approaches check other methods, even they aren't used in the final solution.
Verification & Self-Correction: Checks and corrections of earlier results to catch errors.
Non-Substantive Statement: Redundant comments, filler, or trivial rephrasing that doesn't. advance the solution.

---

We then use LLM-as-a-judge to label each sentence with the most appropriate category." we compute the proportion of each category by counting tokens using the tested model's native tokenizer, enabling a consistent token-level comparison across models. In Fig. 3 (Right), we show the distribution of reasoning categories for both models on the MATH benchmark. Compared to Distill-7B, ShorterBetter-77 exhibits notable structural shifts. Specifically, the proportions of Pivotal Reasoning and Productive Elaboration & Calculation increase by 15.6% and 6.8%, respec- tively-indiicating denser concentration of meaningful, solution-orienter reasoning. Simultancously, the share of Sunsaavattee Statements decreases by 4.8%, suggesting that ShorterBetter-778 pro- duces fewer filler or redundant tokens. We also observe reductions in Verification & Self- Correction (-14.1%) %) and Exploring Alternatives (-3.9%), implying that the model is more decisive and less prone to unnecessary detours. Altogether, these shifts support the conclusion that ShorterBetter promotes more confident and streamlined reasoning trajectories. This finding aligns with our analysis in the previous section and may help address the early stopping of promising paths issue mentioned in Section
8 Finally,
Human Verification of LLM-as-a-judgg In the reasoning-tras -trace analysis above, we employed Gemini 2.5 Flash (Google, 2025) as highly capable LLM judge, guided by carefully designed, unambiguous evaluation prompt (see the prompts in Appendix E). To assess the reliability of its judgments, we conducted a comprehensive human verification study on the reasoning traces generated by ShorterBetter-77 for the AMC dataset. Specifically, we manually annotated 2,064 sentences across 49 problems (65,329 tokens in total), achieving 94.23% (1,945/2,066 agreement at the sentence level and 93.01% % (60,765/65,329) at the token level with the LLM judge.
Functional Category
LLM Judge
Human Judge
Pivotal Reasoning
22.37%
23.42%
Productive Elaboration & Calculation 53.32%
50.79%
Exploring Alternativss
10.50%
10.79%
Verification & Self-Correction
8.69%
10.16%
Non-Substantive Statement
5.13%
4.84%
Table 2: Token-level distribution of reasoning categories judged by LLM versus human. The result is aggregated on the reasoning traces generated by Shorterbetter-78 on the AMC dataset.
As shown in Table 2. inter-annonn inter-annotator agreement remains consistently high across all five categories, reinforcing the reliability of our LLM judge's categorizations. Thus, the reliability of our LLM judge's categorizations is strongly supported. Most disagreements occur between the categories "Productive Flaboration & Calculation" and "Pivotal Reasoning" - both regarded positively in our analysis. Consequently, our main conclusion that ShorterBetter jointly increases the proportion of these meaningful reasoning categories - remains robust despite minor distributional discrepancies. We selected the AMC dataset for this validation study because it offers a balanced testbed: the problems require nontrivial mathematical reasoning while remaining tractable for exhaustive manual annotation. For transparency and to encourage reprodueibility in LLM reasoning research, we have also released our annotation tool and numan-labelel data in our open-source repository.
Overall, while our reasoning structure analysis in Section 5.2 provides valuable insights, we encourage future works to develop more rigorous quantitative metrics for evaluating reasoning structure, which could further validate and extend our findings.
5.3 Ablation Studies
Naively adopting the shortest response as target length leads to training collapses We con- ducted three ablation experiments: (1) No Correctness Reward. Shortest Length Target: remove the correctness term in the reward function and always choose the length of the shortest response as the reward signal. (2) Correctness Reward + Always Shortest Length Target: we keep the correctness reward but always set the length target to the shortest response in a group. (3) Correctness Reward + "Partial" Shortest Length Targer: when there is no correct response, select the length of the shortest
For transparency and reproducibility, our prompt and examples of each category are provided in Ap-
pendix E.2

---

Median Output Length
5K
Median Length (or-))
-06
Accuracy (a=0)
4K
3K-
0.5
2K
8.3
Training Steps
Figure 4: Sample optimal length inherently combines correctness and conciseness. Remove the correctness rewarding term in the reward function but keep the entire design of SOL, i.e., set 0.
SK
Median Length (Only Shortest Length)
0.4
Median Output Length
Length (Corectness Reward
RK-
Partal Shortost Longth)
0.3
Median (Correcteess Reward Shortest Laugth)
SK-
Accurcyy (Only Shortest Longth)
Accuracy
Accincy (C (Comrctneas
Partill Shortest Longth)
2K-
Accuncy (Corrctnnsss
Langth)
0.1
1K-
a/s
50
100
12s
150
17s
200
Training Steps
Figure 5: Naively adopting the shortest response as target length leads to training collapses. Ablation results show that replacing partially replacing SOL with the shortest response length, with or without correctness reward, leads to training collapse within 100 steps.
response as the SOL,, instead of using the average length as in Eq. (1). In Fig. 5, model trainings under the three ablation experiment settings quickly collapse within 100 steps, suggesting that naively choosing the shortest response length as a reward can easily destabilize the training.
Sample optimal length inherently combines correctness and conciseness We conducted another experiment that keeps the entire design of SOL while discarding the correctness reward term in the reward function, i.e., set 0. As shown in Fig. 4, although there are drastic fluctuations in accuracy due to the absence of explicit correct rewards, the training still runs effectively. This result suggests that the design of SOL inherently combines correctness and conciseness. Nonetheless, incorporating a correctness reward term remains essential to prevent degradation in model accuracy.
6 Conclusion and Future Directions
This paper presents ShorterBetter, reinforcement learning method that enables reasoning models to dynamically optimize their CoT lengths without manual supervision. By introducing Sample Optimal Length as a self-supervised reward signal, our method achieves significant reductions in output length while maintaining accuracy across in-domain and oft-of-domain benchmarks. We also present innovative analytical framework show ShorterBetter reasoning structure improvement.
While ShorterBetter is currently showing strong empirical results on tasks with verifiable answers, we note that our general formulation in Section 3.1 naturally extends open-ended settings with non-binary correctness scores. Another future direction is to apply and evaluate our methods on larger-scale reasoning models (Team, 2024). Finally, our reasoning trace analysis shows that different categories of reasoning behavior are compressed to different extents, which may inform the design of more fine- -na---eraed and behavior- aware reasoning optimization strategies.
Note that in experiment (1), replace SOL with sample minimal length and discard the correctness term in the reward function. The reward function is rewritten as rwy)) - minn, L(v.) E(v;)) While in experiment (2) and (3), still keep the correctness term the reward function and set
10

---

Acknowledgment
We thank Data Science Institute at the University of Chicago for supporting our computational resources. We also thank Siyang Wu and Honglin Bao for their valuable feedbacks and suggestions on our earlier manuscript.
References
Pranjal Aggarwal and Sean Welleck. LI: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503. 04697, 2025.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedzions
Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation- based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pages 2357-2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653//1/119. 1245. URL https: http///aaaaaaaaogoyoaeooooooolla ort//1991////// 1245.
Daman Arora and Andrea Zanette. Training language models to reason efficiently. arXiv preprint arXiv:2502.04463, 2025.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint arXhv:2108.0775:: 07732, 2021.
Simon A Aytes, Jinheon Back, and Sung Ju Hwang. Sketch-of-thought: Efficient I1m reasoning with adaptive cognitive-inspined sketching. arXiv preprint arXiv:2503.05179, 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint vrXiv:2107.03374 2021.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song. Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of long reasoning models. In forty-second International Conference Machine Learning, 2025.
Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang. Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.08235, 2025.
Jonathan St BT Evans. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences, 7(10):454-459, 2003.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: ///github.com//ugggg.... wtt. www.///gggiggaaee..o.ooooooo/ooe ocenn.
Mehdi Fatemi, Banafsheh Rafiee, Mingiie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint wxXv:2504.05188. 2025.
Bofei Gao, Feifan Song, Zhe Yang. Zefan Cai, Yibo Miao Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. In The Thirteenth International Conference on Learning Representations, 2024.
Google. Gemini 2.5 flash. preview/, April 2025. URI. https: http:///wwggggggggooooooa//////a flash-previeu/. Accessed: 2025-05-15.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang. Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in llms through reinforce- ment learning. Nature, 645(8081))633-638 2025.

---

Tingxu Han, Zhenting Wang, Chumrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware IIm reasoning. arXiv preprint arXhv:2412.18547, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025.
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey large language models for code generation. ACM Trans. Softw. Eng. Methodol, July 2025. ISSN 1049-331X. doi: 10.1 145/77487747 URL https: /ttp:///oooo////ooo/oooooo/ooool 10.111//37711888...
Carlos E. Jimemnez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
Abhinav Kumar, Jaechul Roh, Ali Naseh, Marzena Karpinska, Mohit lyyer, Amir Houmansadr, and Eugene Bagdasarian. Overthink: Slowdown attacks on reasoning llss. arXiv e-prints, pages arXiv-2502, 2025.
Ayeong Lee, Ethan Che, and Tianyi Peng. How well do LLMs compress their own thain-of-thoughtt token complexity approach. In ES-FoMo III: 3rd Workshop Efficient Systems for Foundation Models, 2025. URL https:/ /ttps/////ppeeeeeeeeeeeee www.eeewweeeeeeeeeeeeeeeeeeeeeee htwp://ooouuunnuuuuoo.....
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system to system 2: A survey of reasoning large language models. arXiv preprint 2rxiv:2502.17411 2025.
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, fine-tuning for ol-like reasoning pruning. In 2nd Al for Math Workshop @ ICML 2025, 2025a. URL https: http:///pwwwwweeeeeteeeeeeeeeeee fotoommmm7d ioYybCRcyw.
and Dacheng Tao. O1-pruner:
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Sto- ica. Deepscaler: Surpassing ol ol-preview with a 1.5b model by scaling rl, 2025b. URL https: p b75 with http::///pootttaaaaaooootatttttt httpee//DDeeppaaaeeeeeeeeeeeeee Scaallll-------inng--- SLtpresaaaaaaa--------1919111110 1tt.a0000000000000000eeea0000000
Notion
Blog.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534 2023.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng. Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint wxXiv22412.09413. 2024.
Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. Concise thoughts: Impact of output length on Ilm reasoning and cost. arXiv preprint arXiv:2407.19835 2024.
OpenAI. Learning to reason with language models, 2024. URL htttp:://opemaa..cm//neee learning-te Learning-to-reason
Accessed: 2025-04-21.
Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, al. survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. arXiv preprint 2rX1v:2503.22611, 2025.
12

---

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: graduate-level google-prood proof q&a benchmark. In First Conference Language Modeling, 2024.
John Schulman. Approximating KL divergence, 2020. URL http://jjaach..nt//blggkk
approx.html. Blog post.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint urXiv:2402.03300 2024.
Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, and Shiguo Lian. Dast: aiffiiulty-adaptiv slow-thinking for large reasoning models. arXiv preprint AxXi::2503.04422 2025.
Guangming Sheng, Chi Zhang. Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In EuroSys, 2025.
Mingyang Song and Mao Zheng. Walk before you run! concise IIm reasoning via reinforcement learning. arXiv preprint arXiv:2505.21178, 2025.
Jinyan Su, Jennifer Healey, Preslav Nakov, and Claire Cardie. Between underthinking and thinking: An empirical study of reasoning length and correctness in llms. arXiv preprint arXiv:2505.00127, 2025.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging big. bench tasks and whether chain-of soann--f--huugge can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003-13051, 2023.
Kimi Team, Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 2025.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://
quenlm.g witthu..oo/bbli //v.o.////e.../....
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Under- standing chain-of ofcin-of-ttee of-thought length in LLMs. In Workshop Reasoning and Planning for Large Language Models, 2025. URL https: : http://pppeeereeeeeeeeee www.eeenneetttoomotdddddaaaaa WhBkk.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:24/2.15115 2024.
Chong Zhang, Yue Deng, Xiang Lin, Bin Wang, Dianwen Ng. Hai Ye, Xingxuan Li, Yao Xiao, Zhanfeng Mo, Qi Zhang, al. 100 days after deepseek-rl: survey on replication studies and more directions for reasoning language models. arXiv preprint arXiv:2505.0055/ 2025.
Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. Marco-ol: Towards open reasoning models for open-ended solutions. arXiv preprint arXiv:2411.14405 2024.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: the claims made in abstractiintroduction have been elaborated in the rest of the paper (esp. Sections and 5) and in the appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
we have detailed the limitations and future directions in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: this paper does not involve formal theorems with assumptions and proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: we have provided the method pipeline Section and flowchart in Fig. Experiment details are specified in Section 4.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have provided the link to data and code in the first page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ca
Tiblic/guides/oodeSutee seabbbmissioooocccr) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //1 //nips
/eaee//odeeuaaaa ssi aeiiionnoliiy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we have provided sufficient experimental details in Section 4.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: due the computational cost associated with reasoning models RL training, we could not get enough replicates to report error bars (similar to other works).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: we have explicitly documented our computational resources in Appendix A.5.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

httppppccppppllcc/tccccotooioooe The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: this paper closely follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: we have discussed the broader impact of our paper in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: we finetune our models based on the base LLMs described in Section 4, which already have safeguards during pretraining. We manually inspect that our datasets (mainly math and coding benchmarks) do not pose such risks.
18

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we have properly credited the base models and benchmarks used in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi whrodeewttccec...mm com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: we open-sourced the trained models and made it public via the link we provided in the first page.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer: [NA]
Justification: this paper does not involve crowdsourcing or research with human subjects. Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: this paper do not involve study participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: in Section 5.2, we have explicitly declared that we have used LLM-as-a-judge part of analysis pipeline. We provide further details the judge LLM, including complete prompts and example outputs, in Appendix E.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: : for what should or should not be described.
/LLM)
20

---

Appendix: Table of Contents
A. Technical and Implementation Details
Page 21
B. Additional Experiment Results
Page 23
C. Examples of Unwanted Reasoning Behaviors
Page 27
D. Analyzing Reasoning Traces with LLM-as-a-judge
Page 30
Broader Impact
This work contributes the growing body of research efficient reasoning in LLMs. By reducing redundant reasoning steps while maintaining accuracy, our method could help lower computational costs and improve the practicality of deploying LLMs in resource-constrained settings. Like many advancements in ML, this research may have indirect societal implications- efficient models could reduce energy consumption enable faster inference in applications such as education or automated decision making. However, our focus is narrowly technical, and we do not make strong claims about specific real-world benefits risks. As with any optimization technique. the broader impact will depend on how is integrated into larger systems and deployed in practice.
instance, more
A Technical and Experimental Details
A.1 Detailed Specification of GRPO Algorithm
Here we provide the complete expressions and details of the GRPO (Guo et al., 2025) objective (loss function) mentioned in Section 3.3:
CGRPO (0) = -
W (A pě, v) -7 DD [pš lellr...r]
In particular, W is clipped policy improvement objective with full expression
W (Ay,,s Ph. pb.... 5) min pb pi...
A clip
pg
:1 E, Ay, A1x)
po
where the clipping function clip(z; b) == (b, max(a, a, x)) prevents the updated parameters from deviating too much from the reference policy. For KI. divergence, Schulman (2020) proposes following approximator
Pd..: pb
P p
DKL Pp) [pp
log
Finally, while the above formulation considers multiple rollouts for single prompt T; in practice we compute the GRPO loss and update the policy parameters over batch N prompts at each step.
A.2 Detailed Formulation for Accuracy-Elttiiency (AE) Score
The Accuracy-Efficiency (AE) Score, introduced by Luo et al. (2025a), provides a composite metric to evaluate whether model improves inference efficiency- iie., reduces output fnngt--wthhor sacrificing accuracy. It is defined as follows:
AE Score = ALength + 17 AAc,, if AAcc 2 0
ALength 0- AAcc, if AAcc < 0
Here, ALength and Ace denote percentage changes output length accuracy, respectively, of the evaluated model relative to its base model. Positive AE Scores indicate that the model generates shorter outputs while maintaining or improving accuracy, while negative scores penalize accuracy degradation. In our experiments, we use the same hyperparameters in Luo et al. (2025a):
21

---

p 1 (weight on length reduction)
// 3 (bonus for accuracy gains)
0 5 (penalty for accuracy drops)
The asymmetric design (with 0 > 1i) penalizes accuracy drops more heavily than gains, reflecting the practical preference for avoiding performance degradation.
A.3 Experimental Details
For evaluation, we categorize tasks either in in-domain or out-of-domain, based on the training data used for ShorterBetter Our method is trained on the SeepScaleR-Previéc dataset (Luo et al., 2025b), a composite benchmark that includes AIME, AMC, Olympiad- Benchh MATH, Minerva, etc. Accordingly, held-out subsets from these datasets are considered in-domain. In contrast, tasks such as MathQA, BBH, MMLU, LiveCodeBench, MBPP, and HumanEval which are not included in the training corpus are treated as out-of-domain.
Other length-reduction baselines are trained on different datasets. Specifically, Training Effi- cient (Arora and Zanette, 2025) is trained on MATH, CN-K12, AIME, AoPS, and Syympad--Bnnch while O1-Pruner (Luo et al., 2025a) is trained solely on MATH. There exists partial overlap training data between these baselines and ShorterBetter. Nevertheless, the out-of-domaal benchmarks defined in this work are held from all models' training data and thus serve as fair test of generalization.
For model comparisons in Fig. and Table 1, we directly downloaded the open-source models of Training Efficient 1.5B/7B and O1-Pruner 7B from HuggingFace for evaluation. For Ol-Pruner, since the paper does not provide a 1.5B version, we didn include it in Fig. 1. For Training Efficient, we used the 1.5B and 7B versions both with hyperparameters a 0.4 (following the original paper). During all evaluations, the temperature was set to 0.9. All problems answered once, and the accuracy is calculated as the fraction of correct answers among all answers.
A.4 On the Practical Choice of Hyperparameters
Choice of a and | The hyperparameters and / are selected as a pair. Since reinforcement learning frameworks such GRPO use relative rewards rather than absolute rewards, fix B3 0.001 and focus on analyzing the choice of a.
The hyperparameter controls the trade-off between accuracy and efficiency. Its value relative the length penalty (kept at 0.001) is the key factor. We tested new a values to illustrate its impact.
Table 3: Performance of ShorterBetter-1.5B with varying values.
Dataset
0.1 Ace
0.1
2.0 Acc
2.0
Ace
5.0 Lon
AMC
0.43
2503
0.57
1946
0.58
3077
Minerva
1785
0.24
Olympiad
0.30
2903
0.38
1814
0.35
3919
AIME
5395
2703
5274
Our results (shown in Table 3) provide an intuition for selecting a:
A high (e.g.. 5.0) places strong emphasis correctness. As seen in the training dynamics and the table above, this leads stable accuracy gains but weaker length reduction, resulting in longer outputs compared to the more balanced setting.
A low a (e.g., 0.1) makes the length penalty term negatively influential. This can degrade the model's reasoning capability, as the reward for correctness may not be sufficient to maintain performance. This instability can lead to worse length reduction because the model fails to learn effective, concise reasoning paths.
A balanced (e.g., 1.0 or 2.0) provides strong enough correctness signal to preserve or improve accuracy while still exerting significant pressure to reduce output length. This achieves the best overall trade-off demonstrated in our paper.
22

---

In practice the optimal choice depends the specific goal (e.g., prioritizing efficiency accuracy) and the specific model family. Based on our results, we recommend practitioners start with in the range of [2, 5] and tune as needed.
A.5
Computational Resources
We trained our ShorterBetter-1 with A100 GPUs, using approximately 16 hours. We trained ShorterBetter-7B with 8 A100 GPUs, using less than 12 hours.
B
Additional Experiment Results
B.1
Coding Benchmarks
1.SB MBPP
1.5B HvmanEvall
1.SB LiveCodeBench
0.5
0.8
0.4
Acouray
0.55
0.50
06
645
02
7B MBPP
7B Human Eval
LiveChnfeBench
0.4
Acourace
Accuracy
02
0.
2000
5000
RI-Distill
ShorterBetter Train Efficient
Instruct
Figure Model performance on coding benchmarks.
report the result of the coding experiment Table and Fig.
1.5B Models
Models
Task
RI-Distill
Instruct
ShoricrBeterr
Training Ellicient
RI-Distil
Instract
StoorierBeticr Training Ellicient
Ol
MBPP
0.544 (4151) 0.409 (124) 0.436 (116)
0,506 (2402)
0.659 (2928) 0.609 (150) 0.632 (1055)
0.607 (2113)
0.494 (495)
HumanExal
0.720 (5123)
0.463
0.628 (1551)
0.665 (3280)
(3841) (398)
0.799
0.866 (2601)
0.829 (254)
LiveCok-Bench
0.260 (1020H) 0.089 (262)
0,260 (4961)
0.244 (7792)
0.506 (8683) 0.247 (189)
0.495 (5228)
0.216 (9726)
0.483 (1847)
Table 4: Coding performance across model sizes and training methods. Each entry reports the accuracy rate and the average inference length in tokens (shown in parentheses). Both ShorterBetter models were fine-tuned with a = 2. Baseline models are from Arora and Zanette (2025). Bold numbers indicate best accuracy for each model size category.
B.2 Training dynamics of ShorterBetter
In Fig. we show the output length (in tokens) trends and accuracy trends of ShorterBetter-1 and ShorterBetter-7B during GRPO training. For clearer illustration, we present the first 200 steps of the two models.
23

---

Modian
1.38 Accuricy
Modian Lengi
Median Output Length
111
159
Truining Steps
Figure 7: Training dynamics of ShorterBetter-1.5B and ShorterBetter-1.55 models.
In Fig. 8, we illustrate how the model learns to identify optimal inference lengths throughout the training. For each sample generation (n 8) of given problem, compute the SOL and take the batch-wise average. We also track the mean deviation individual output from their respective SOL,
(1n))) For instance, a deviation of 0.2 at given training step indicates that the generated outputs deviate from the optimal length by 20% on average. The plots show that both the SOL and the mean deviation steadily decrease over training, particularly within the first 100 steps. This trend suggests that reinforcement learning successfully guiding the model to produce outputs that cluster towards the optimal length.
defined as dę (GoL()((.)
1.5 model (x
7B model (slpha-2)
SC1,
3300
-066
4000
3100
-05
3000
2500
2000
mode
1990
1000
Figure 8: SOL trends and mean deviations during training
B.3 Incorrect reasoning traces are significantly longer
In Fig. 9, we conducted an experiment using 150 problems (30 each from AIME, AMC, MATH, Olympiad-bench, and Minerva). For each problem, we generated 10 responses with the Distill- 1.5B/7B models, then filtered out problems where all responses were correct or all responses were wrong. We then computed the difference between the average lengths of correct and incorrect responses to form length gap distribution. The results show that, on average, for the same problem, incorrect responnssss are significantly longer than correct ones, by 5,541 tokens on DeepSeek-RR R---stiil- Qwen-1.5B and 5,910 tokens on DeepSeek-R1-D --Dtsti-l--it--weeee between reasoning errors and unnecessarily long CoT traces in Large Reasoning Models.
This suggests clear correlation
B.4
Performance of Marco-ol
Here we report the performance of Marco-ol on various benchmarks, which is used to calculate the AE Scores of O1-Pruner in Table 1.
B.5
Results
To address model architecture bias, we evaluated ShorterBetter on a model with a different archi- tectural backbone: SeepSeck-RI--istil---iieee This allows us to test method's generalization
24

---

Length Differences Between Correct and Wrong Outputs
Distilled 1.5B
Distilled 7B
-5540.98
Density
-10K
5K
-15K
-12K
-10K
-8K
-2K
Length Differences
Length Differences
Figure 9: Length gap distributions for the Distill-1.5B and 7B models.
Dataset
Accuracy
Length
AIME
0,067
6711
Olympiad
0.317
6325
AMC
0.301
6210
Minerva
0.217
7583
MathQA
0.331
7730
BBH
0.623
6829
MMLU
0.701
7650
LiveCodeBench
0.225
12856
HumanEval
0.841
317
MBPP
0.454
877
Table 5: Marco-ol performance across datasets
beyond the Qwen series. We trained this 8B Llama-based model for 100 steps using our framework. The results are shown Table
The results show that ShorterBetter successfully generalizes to different model architecture. Here we choose hyperparameter reduction. However, the out-of-distne -itttt--distiittiion performance has more deduction compared to Qwen series model, which suggests that fewer training steps or larger a might be needed.
5 since achieves better balance between accuracy and length
C Extended Discussion on Related Work
Recent works such as (Agg (Aggarwal and Welleck, 2025) ConciseR (Song and Zheng, 2025) share the goal of improving efficiency in reinforcement learning for reasoning, yet differ fundamentally from ShorterBetter their problem formulations and design principles.
L1 focuses on external controllability: a user provides an explicit reasoning-lenge budget, and the model is trained adhere this target. Consequently, the optimization objective and prompt structure are explicitly guided by numan-specifeer constraints. In contrast, ShorterBett.s introduces an autonomous mechanism that allows the model to discover its own optimal reasoning length based on the interplay between task difficulty and model capacity. This self self-supervised setting removes need for user-defined budgets, which may be unrealistic scenarios where the efficient solution lenghh is unknown a priori. Furthermore, while L1 relies on prompt-level cues (e.g., "think for tokens")), ShorterBetter remains completely prompt-agnostic, enabling seamless application across diverse reasoning contexts.
25

---

8B-Distill Acc.
ShorterBetter (a 5)
Len.
Acc.
Len.
In-domain Tasks
AIME
0.33
9949
0.37
5634
Olympiad
0.45
7234
0.41
3944
AMC
0.67
6995
0.62
3384
Minerva
0.33
5555
0.33
2669
Out-of domaai- Tasks
MathQA
0.77
3002
0.71
899
BBH
0.71
1860
0.60
699
LiveCodeBench
0.54
8101
0.37
1665
Table 6: Performance comparison of 8B Llama-based models across diverse tasks.
ConciseR, developed concurrently, employs a two-stage gated optimization framework: the first stage strengthens reasoning ability, and the second stage compresses reasoning length, with the conciseness reward activated only when all sampled responses are correct. This gating mechanism ensures correctness but limits sample efficiency, full-batch correctness is rare in complex domains. By contrast, ShorterBetter adopts single unified learning process that leverages the Sample Optimal Length (SOL)- the shortest correct reasoning trajectory among multiple generations- as continuous, self-supervised reward signal. This design allows effective gradient propagation even when correctness is partial, substantially improving training efficiency and stability.
In summary, while all three methods operate within the emerging paradigm reinforcement learning for efficient reasoning, ShorterBetter is distinguished by its integrated, self-supervised formula- tion and its introduction of SOL dynamic intrinsic reward. This contribution moves beyond explicit control (L1) and staged compression (ConciseR) toward an autonomous discovery of effi- ciatur-capturing an intrinsic property of reasoning models rather than imposing it externally.
capturing
26

---

D Examples of Unwanted Reasoning Behaviors
D.1 Verbose Repetition and Self- Verification in Intermediate Steps
Prompt
Denali and Nate work for a dog walking business and are paid for each dog they walk. Denali is responsible for 16 dogs and Nate is responsible for 12 dogs. Under the company's new policy, they will be assigned unassigned new dogs in groups of dogs. The ratio of Denali's 's pay to Nate's pay would be the same if Denali started walking 4x more dogs and Nate stayed 12 dogs if of Nate's dogs were reassigned to Denali. Find I if x #
Response (DeepSeek-R1-Dsttil)
Wait, perhaps the problem is not about the same pay ratio, but the same total number dogs? But the problem says "the ratio
Denali's pay to Nate's pay would be the same." So it's about the pay ratio, not the number dogs.
Wait, another thought: Maybe the ratio is in terms of the number of dogs walked, so if the ratio of their pay is same, and pay is per dog, then perhaps the number of dogs walked should be proportional.
Wait, that is, if Denali walks 16 dogs and Nate walks 12, the ratio of their pay is (16)/(12) if they
paid per dog.
Wait, so regardless of how I set up the ratio, end up with x=0. But since the problem states x40, perhaps my entire approach is wrong.
Wait, 1et me think differently. Perhaps ratio their pay dogs, regardless of direction.
same when they are assigned Wait, when they are assigned x dogs in groups of x, meaning dogs are reassigned. So the ratio is same.
Perhaps it's the pay per dog ratio that's the same Wait, maybe Denali's pay per dog is same as Nate's. But that would same as each getting paid per dog.
Wait, but if they are reassigned, their pay per dog would be same. Wait, perhaps we can think in terms of average. Wait, I'm getting confused. Maybe should go back to the original problem.
**Problem S
But we вaw that this leads to x=0. Alternatively, maybe the ratio is the other way around. Wait, perhaps the ratio is the pay per dog ratio. So if the pay per dog is same, then the ratio is 1:1.
But in that case, both would have to walk same number of dogs. But the problem says that if dogs ratio.
reassigned, it results in
But in that case, the pay per dog same, then it's just the same number of dogs walked, which would imply that 16 = 16 + 4x + 55
0 x 0, which is again invalid.
Wait, perhaps I need to approach this problem by assuming that the pay ratio remains same.
27

---

D.2 Early Stopping of Promising Reasoning Paths
Prompt
The function f satisfies the functional equation
[(x) + [(y f(x xy
for all real numbers and y. If f(1) 1, then find all integers such that f(n) = n. Enter all such integers, separated by commas.
Response (DeepSeakRR1-Distill -1.5)
Now, let's find f()):
Using the recursive formula, set
f(2) = f11 1
+ 2 4 Wait, is that right? Let me check.
Wait, actually, according to the recursive formula f(x + 1) f(x) so for =
We observe that the model starts to check for immediately after it decides to check f(2),
interrupting its own verification step.
D.3 Continued Exploration of Alternative Solutions After Reaching Correct Answer
Prompt
Find all values of that satisfy the equation x VII 2x 4.
Response (DeepSeek-Ree Ri-iistil1---ieeeeet 115B)
So, the solutions are x = and x 5. But wait, I need to check if these solutions actually satisfy the original equation because sometimes squaring both sides can introduce extraneous solutions. First, let check x 1:
Left side:
/11- 2(1) + 4 Calculate inside the square root: 11 2 9 So, square root of is 3. Then 3 4 7 But left side is so * 7. That means x 1 is not a valid solution.
Now, check == 5:
Left side: 10 Square root of 1 is 1. Then 1 4 So, 5, which is true. Therefore, x = 5 is a valid solution.
V11-2(5) 2(5) + Calculate inside the square root: 11
But wait, earlier I thought that x must be greater than or equal to Since = 5 is greater than 4, that's fine. But x = 1 is less than 4,
Wait, hold on. Let me double-check my earlier factorization and solutions.
I had x² - 6r 5 0, which factored into (x - 1) (x 5) 0, giving and x 5.
Plugging = 5 into the original equation worked, but plugging = didn't. So, x 1 extraneous.
But is there any other step I should verify? Sometimes, especially with square roots, squaring both sides can sometimes lead to other issues, but in this case, since didn't square the square root term on both sides but isolated it once, it seems fine. Let consider another method to solve the equation, ...
28

---

Same quadratic as before, so same solutions. So, same process, same result. So, But just to make sure, let me consider if squaring the equation introduced any extraneous solutions. Sometimes when square both sides, it's easy to have extra roots.
and x 5, but only x == valid.
D.4 Collapse Reasoning Coherence Following Very Long Outputs
Response [DeepSeek-R--Dss-lll---
budget 16000 tokens)
WaitWait endpoints.
about Wait.
say needWait. Maybe Wait Wait set's.
me Wait, mark,;, 's No.
the, me's number minimuButButWait. 's the,, WaitWaitWait.
ao over be alone to..
Wait.
the the the Wait Wait, 's the me's Wait Wait the. 's. with of., range's., (WaitWait nonWaitWaie me's total number integer's s theBut a asagrams, No me, Well Wait Wait a.
29

---

E Analyzing Reasoning Traces with LLM-as-a-judge
E.1 LLM-as-a-judge Setup
In order to illustrate the reasoning structure improvement of SB, used LLM-as-a-judges to help us analyze the reasoning trace because the lengthy reasoning trace is hard for human to read. We use Gemini 2.5 Flash Preview (Google, 2025) with _op_p=0 959 and temperature=O.. The prompt for reasoning trace categorization and identifying the first appearance of correct answer as follows:
Prompt (Reasoning Categorization)
You are reasoning trace analyst. Your role is to categorize each line (separated by newline, but not treat equations as an individual line) in given reasoning trace according to the function it serves within the reasoning process. The goal is to understand how different parts of the model's reasoning contribute (or not) to the final answer.
You will be given a **full reasoning trace**, which ends with a final answer after the <</think>" tag. You must:
1. *+Only analyze the content before the '</think> tag**
2. **Split the reasoning trace into individual lines**, multiple sentences might be in single line, line should be ended by newline.
3. **Assign exactly one label each line** from the following **mutually exclusive categorieg**, Choose the **most fitting** label based on the line's function in context.
### Categories:
Adopted Reasoning**
* **Definition*:: A reasoning step (e.g., setting up key equation that's s shown in the summary, stating a critical definition that's part of the summary, deriving final value for variable that's highlighted in the summary, or an intermediate result explicitly carried forward into the summary) whose content conclusion is **explicitly stated or directly corresponds to a distinct step, equation, or piece of information in the summarized solution presented after the '</think> tag.** ** These are the landmark achi achievements key building blocks in the reasoning trace that showcased in the final, condensed explanation.
*Exampl**: If the final summary after "</think>" inncuues the equation +222s-11.22
0", then the line in the reasoning trace where this equation is first derived stated this form "Pivotal Reasoning". Similarly, the line stating "s = 2.5 km/h" is "Pivotal Reasoning" if s 2.5 key reported value in the final summary.
2. E*Eploring Alternatives**
* **Definition#:: Lines that perform necessary calculations, logical deductions, initial problem interpretation (e.g., "The problem asks for...", "We are given...") variable definitions (if not themselves pivotal as per "First, I will find s, then t."), explanations of methodology that **directly contribute to reaching a "Pivotal Reasoning" step or the overall solution, but are not themselves explicitly restated distinct element in the final summarized solution.
above definition), strategic planning (e.g.,
Example*: "Hmm, I need to convert into hours because her
30

---

walking speed is in km/h." (Problem understanding leading to pivotal setup). "Multiply both sides by +(s+ 18 = 1.6 s(s + 2). (A
calculation step leading pivotal equation). "Let me compute that left side: simplification).
18/(s(s + 2))" (The process of algebraic
3. **Repetition & Rephrasing**
* **Definition**: Reasoning segments that explore solution paths, hypotheses, methods, or calculations that are *tultimately abandoned, identified by the LLM as incorrect or less viable, or otherwise do not form part of the sequence leading the adopted final solution. ** This includes trying a different formula that is then discarded, or starting a calculation that then reversed due a change in approach.
*Example*: "Perhaps I can solve for first by isolating it from equation 1... no, subtracting the equations seems more direct to eliminate L." (The statement about solving for / first by isolation would be "Alternative Path or Exploration" if that path is not taken).
4. **Vrrification Explanation** #*Definitione:: Lines that are redundant (e.g., repeating information conclusions verbatim or near-verbatim shortly after they' been stated, without adding new purpose like emphasis for next step), purely filler or conversational remarks (e.g., "Okay, .o...., "Good.", "Let me that
think...") provide overly trivial explanatiios/reppraaattt do not advance the solution or sigmificantly aid in understanding the problem-solving process (e.g., "This is quadratic equation in s, immediately after writing s² + 2s 11.25 = 0). This also includes restating parts of the problem verbatim without adding interpretation or planning elements, once the initial understanding is established. *Examplex: "So, s 2.5 km/h." (if stated immediately after "Thus, 2.5 km/h" and not serving as a premise for an immediate subsequent calculation). "Let me write them down again:" (if the equations were just presented). "Yes, that seems correct," " if it's low-effort affirmation rather than the concluding part of rigorous "Verification" step.
### Additional Instructions:
you encounter the '</think>' tag** Do not categorize anything beyond it. The content after "</think> is used the reference for "Pivotal Reasoning". or paraphrases** an earlier line without adding new value or serving clear structural purpose (like summarizing inputs before a new phase of calculation), categorize the repeated instance as ** "'NoonSSusttantvve Statement
**Stop processing
If line **repeats
if the original served different, valid purpose. * If line could arguably fit into more than one category, choose the category that best describes its **primary function or most specific contribution** in that context. For instance, a calculation that corrects previous error is "Verification & Self-Correction' rather than just "Productive Elaboration Calculation," " statement of a key equation from the final summary is "Pivotal Reasoning" even if it also elaborates. * Do **not infer logical correctness** final answer. The categorization is about the *structure and an LLM's perceived purpose* each statement within own reasoning
reasoning or the
31

---

process. Treat each line #*independently for categorization** but use *surrounding context** (preceding and succeeding lines) to understand its function, especially for identifying repetitions, planning statements, or flow of logic. If the reasoning trace starts with the <</hink>>'an tag, 1.e., ### Output Format:
Return your output as **JSON array** of objects. Each object should have:
"text": The full original line (string).
"label": One of the five category names above (string).
j'soon [ "text": "Alternatively, we can try to factor the quadratic equation, but the roots might not be integers.", "label": "Exploring Alternatives" "text": "Therefore, <span class="nath-inlime" 3</span> is the value will use.", ,, "label": "Pivotal Reasoning" "text": "So, <span
alas="nathhinllnee"xxxx Statement" Jasciigrave"
is 3.", "label": SunntSubstat
Return **only** the structured JSON list, no extra commentary or explanation.
Prompt (Identify First Appearance of Correct Answer)
You are reasoning trace analyst. Your role is to identify the first sentence where the model gets the correct answer of the problem. identify redundant overthinking process after the model has actually solved the problem.
The goal
You will be given full reasoning trace, which ends with final answer after the "</think>"tar you will also be given the problem and the correct answer. You must:
1. Only identify the sentence before the '</think> tag.
2. Identify only the **firs*** sentence where the model gets the correct answer.
3. The sentence you return should be **exactly the same** as the one in the original reasoning trace.
Return only the sentence you identify, extra commentary explanation.
32

---

E.2 Reasoning Category Examples
Pivotal Reasoning Example
First, I need to figure the ratio of the enrollments between the two grades. The 7th grade has 520 students, and the 8th grade has 650. So, the total number of students is 520 650. Let me calculate that: 520 plus 650 is 1170. So, there are 1170 students in total between the grades.
Productive Elaboration & Calculation Example
Therefore, the number of representatives for each grade should be proportional to their enrollment fractions. So, the 7th grade should have (4/9) of 18 representatives, and the 8th grade should have (5/9) of 18.
Exploring Alternatives Example
Hold on, maybe need to think about the transversal lines and corresponding angles. Since ST is parallel QR, and PR is transversal, the angle at T, which is angle STR, should correspond to angle QRP, which is angle R, 105 degrees.
Verification & Self-Correction Example
Wait, if ST is parallel to QR, then the corresponding angles when PR is the transversal would equal. So, angle S (which is angle STR) should be equal to angle QRP, which is 105 degrees.
Non- Son-Substantive Statement Example
Wait, maybe I'm confusing the angles. Let me try to think of it differently.
33


Reviews:
[{"review_id": "ZZDWu6Y8En", "reviewer": "Reviewer_iAtn", "summary": "ShorterBetter introduces a reward function that explicitly accounts for Optimal Reasoning Length (OL). In addition to answer accuracy, the function assigns a penalty to any generation whose length diverges from the shortest correct solution—or, when no correct solution appears in the roll-outs, from the batch-average length. They assess ShorterBetter on both in-distribution and out-of-distribution mathematics benchmarks and perform ablation studies to isolate and validate the contribution of the length-based penalty.", "questions": "Please see weakness. \n\nAlso, what do you mean by \"may inform the design of more fine-grained and behavior-aware reasoning optimization strategies\" (in section 6)?", "limitations": "Yes the authors mentioned future improvements such as 1) applying to larger-scale reasoning model, 2) tasks with non-binary correctness score, 3) more fine-grained reasoning optimization strategies.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strengths:**\n- The presentation is clear, and the method is easy to understand.\n- The experimental study seems comprehensive, with 4 ID and 4 OOD tasks. \n- The authors conduct a comprehensive ablation study with varying values of the hyperparameters in the reward function. \n\n### **Weakness:**\n- **Novelty:** ShorterBetter implements an augmented version of GRPO with length regularization on top of the task metrics. Previous works like L1 (Aggarwal et al., 2025) and concurrent works such as ConciseR (Song et al., 2025) have similar notions, where L1 also leverages GRPO to fine-tune LLMs. I saw the authors compare with L1:\n```\nExisting approaches targeting efficient LLM reasoning primarily focus on explicitly controlling reasoning length, for instance by imposing a length budget based on user specifications (L1; Aggarwal and Welleck, 2025) … In contrast to these explicit control strategies, our work is based on the hypothesis that reasoning models implicitly possess an optimal reasoning length (OL).\n```\nIf I understand correctly, ShorterBetter can also be viewed as a self-supervised version of L1 where the optimal length is defined by the model itself. In this context, the novelty seems insufficient.\n\n- **Robustness:** There are some interesting yet not fully explained conclusions. Specifically, Figure 4 shows that naively adopting the shortest response as the target length leads to training collapse. The resulting accuracies and output lengths are all zeros, even for “Partial.” Can the authors provide more explanation? If the training dynamics are sensitive to the length penalty, how do the authors, in practice, choose the hyperparameters? \n\n### **Minor:**\n- I don’t think the section “Output Length after First Appearance of Correct Answer” is necessary. It seems to convey the same information as comparing the total answer length. Also, why “after” and not “before”? Redundancy can also occur before the model reaches the correct answer, no?", "was_revised": true, "final_justification": "I maintain my evaluation", "rebuttal": "We thank the reviewer for finding our presentation clear and experimental study concrete. We also appreciate the thoughtful feedbacks regarding the **novelty** and **robustness** of our method. Below are our point-by-point responses:\n\n---\n## **On the Novelty of Our Method**\n\nThe reviewer has pointed out the previous `L1` (Aggarwal et al., 2025) and concurrent work `ConciseR` (Song et al., 2025), and correctly notes that all three methods operate in the space of RL for efficient reasoning. We also believe that comparing with these two methods allows us to clarify and demonstrate the novelty of `ShorterBetter`.  On a high level:\n\n- `L1` (Aggarwal et al., 2025) focuses on **external controllability**, where a user must specify a length budget.\n- `ConciseR` (Song et al., 2025) uses a **gated, two-stage process** to first improve reasoning and then separately optimize for length.\n- `ShorterBetter`, in contrast, is designed to **autonomously discover an optimal reasoning length**, based on the model's own capabilities and the problem's difficulty, within a single, unified training process.\n\n**Detailed comparison with L1:**\n\nThe reviewer suggests `ShorterBetter` can be viewed as a _\"self-supervised version of L1.\"_ While we agree our method is self-supervised, we believe this framing understates the conceptual leap. Notable differences are:\n\n- **Objective:** `L1`'s objective is adherence to a user-defined budget. The model is rewarded for matching a length ($n_{gold}$) provided in the prompt. `ShorterBetter`'s objective is to find the most efficient path to a correct solution -- which can be viewed as self-supervising the model to approximate a (hidden) optimal length. Crucially, unless the user already knows the \"most efficient solution\" to a problem a priori, the self-supervised target in our approach will also deviate from $n_{gold}$.\n- **Assumption:** In order for the method to work well, `L1` requires the user-defined budget (target) to be informative of the problem at hand, which we argue can be sometimes unrealistic (_how do you set a budget for a math question that you cannot solve?_). On the other hand, `ShorterBetter` is built on the more general hypothesis that an **Optimal Reasoning Length (OL)** exists, which the model learns to find on its own without requiring this prior knowledge.\n- **Prompt dependence:** `L1` requires appending an explicit cue—*“think for n tokens”*—to every training prompt, a design choice that can itself shape the model’s behavior; this interaction is not analyzed in detail in their paper. By contrast,  `ShorterBetter` is completely prompt-agnostic, making it broadly applicable.\n\n**Detailed comparison with ConciseR:**\n\n`ConciseR` is an excellent concurrent work, but it addresses a different problem setting with a different methodology. Notable differences are:\n\n*   **Problem Formulation:** We start with a strong but *overthinking* reasoning model and our goal is to **refine** its reasoning to be more efficient. `ConciseR` starts with a weaker base model and must first **build up** its reasoning capability before optimizing for length. These are fundamentally different starting points and objectives.\n*   **Training Paradigm & Gating:** `ConciseR` uses a **sequential two-stage** framework. Stage 1 improves reasoning, and Stage 2 enforces conciseness. Crucially, its length reward is **conservatively gated**; it only activates when *all* sampled rollouts are correct. In contrast, `ShorterBetter` uses a **single, integrated** process. Our $l_{SOL}$ provides a **flexible learning signal** as long as *at least one* correct response exists, making our method more sample-efficient, especially on challenging problems where 100% correctness per batch is rare.\n*   **Reward Signal:** `ConciseR`'s reward aims for maximum compression (rewarding $L_{Max} - l_i$). `ShorterBetter`'s $l_{SOL}$ reward targets an *optimal point*, penalizing responses that are too long *and* too short.\n\n**To summarize:**\n\nThe core novelty of `ShorterBetter` is the introduction of **Sample Optimal Length (SOL)** as a dynamic, self-supervised reward signal. This creates an autonomous and integrated framework that guides a model to discover its own optimal reasoning length, rather than simply obeying an external budget (`L1`) or following a staged compression schedule (`ConciseR`).\n\n---\n## **On the Robustness of Our Method**\n\nWe thank the reviewer for their insightful question on training robustness and hyperparameter sensitivity. We offer this explanation, supplemented by new experiments, to clarify these critical aspects.\n\n### **On Training Collapse from Naively Minimizing Length**\n\nThe reviewer correctly notes the training collapse shown in our paper (Fig. 4, Left). This collapse is caused by an unstable reward signal. Our ablation studies confirm that naively targeting the shortest length is not viable:\n\n1.  **No Correctness Reward, Shortest Length Target:** As shown in our paper, if the reward only minimizes length, the model quickly learns to produce empty outputs, causing accuracy to drop to zero.\n\n2.  **\"Partial\" Shortest Length Target:** Using the shortest length as a target only when no correct answers are found also fails (even with a reward term present). This creates a feedback loop where the model sacrifices correctness for length, leading to a training collapse.\n\n3.  **Correctness Reward + Always Shortest Length Target:** In a new ablation, we kept the correctness reward ($\\alpha=2$) but always set the length target to the shortest response in a group. This also crashed the training. Our logs show accuracy and length rapidly declining to zero. Even with a correctness reward, a single short (but incorrect) response heavily penalizes all longer (and potentially correct) responses, creating a fatal bias toward underthinking.\n\nThese experiments show that the length target design is critical. Our proposed **SOL**, anchored to the shortest *correct* response and defaulting to the *average* length in failure cases, provides the necessary stability to avoid these collapse scenarios.\n\n---\n\n### **On the Practical Choice of Hyperparameters**\n\nThe hyperparameter $\\alpha$ controls the trade-off between accuracy and efficiency. Its value relative to the length penalty $\\beta$ (kept at 0.001) is the key factor. We tested new $\\alpha$ values to illustrate its impact.\n\n**Performance of `ShorterBetter-1.5B` with Varying $\\alpha$**\n\n| Dataset  | $\\alpha$=0.1 Acc | $\\alpha$=0.1 Len | $\\alpha$=2.0 Acc | $\\alpha$=2.0 Len | $\\alpha$=5.0 Acc | $\\alpha$=5.0 Len |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| AMC      | 0.43         | 2503      | 0.57         | **1946** | **0.58** | 3077      |\n| Minerva  | 0.20         | 1785      | **0.28** | **1147** | 0.24         | 2249      |\n| Olympiad | 0.30         | 2903      | **0.38** | **1814** | 0.35         | 3919      |\n| AIME     | 0.13         | 5395      | 0.20         | **2703** | **0.23** | 5274      |\n\n\nWe present new results for $\\alpha=0.1$ and $\\alpha=5.0$ to complement the results for $\\alpha \\in \\{0, 1, 2\\}$ in the main paper.\n\nFor alpha = 0.1, the training log shows that accuracy follows a declining trend over 200 training steps. During this time, the median output length decreases rapidly.\nFor alpha = 5.0, the training log indicates a clear upward trend in accuracy throughout the training process. The median output length also trends downward, decreasing from 5000 to below 2000 tokens.\n\nOur findings provide an intuition for selecting $\\alpha$:\n\n* **A high $\\alpha$ (e.g., 5.0)** places a strong emphasis on correctness. As seen in our training dynamics log and the table above, this leads to stable accuracy gains but a weaker length-reduction, resulting in longer outputs compared to the more balanced setting. The accuracy trend is clearly positive during training.\n* **A low $\\alpha$ (e.g., 0.1)** makes the length penalty term negatively influential. This creates a risk of degrading the model's reasoning capability, as the reward for correctness may not be sufficient to maintain performance. The training dynamics log shows a downward trend in accuracy. This instability can lead to worse length reduction, as seen in the table, because the model fails to learn effective, concise reasoning paths.\n* **A balanced $\\alpha$ (e.g., 1.0 or 2.0)** provides a strong enough correctness signal to preserve or improve accuracy while still exerting significant pressure to reduce output length. This achieves the best overall trade-off demonstrated in our paper.\n\nIn practice, the optimal choice depends on the specific goal (e.g., prioritizing efficiency vs. accuracy) and specific model family. Based on our results, we recommend practitioners start with $\\alpha$ in the range of [2, 5] and tune as needed.\n\n---\n### **On the \"Length After First Appearance\" Metric**\n\nWhile we agree that our “Length after First Appearance” metric is correlated with total length,, we believe it provides a unique and valuable perspective (as pointed out by Reviewer 2UEK) on a **specific pattern of overthinking**: a model's inability to terminate its reasoning process after it has already found a correct solution.\n\nThe reviewer correctly notes that redundancy can occur both before the answer is found (\"pre-solution\") and after (\"post-solution\"). Our metrics are designed to disentangle these two effects. Total length captures the sum of both inefficiencies. In contrast, our \"Length After\" metric focuses on the post-solution component. Therefore, we believe that the total length and the \"Length After\" metrics offer complementary views. We will revise Section 5.2 to make this clear.\n\n---\n### **Clarification on Sec 6**\n\nFinally, our statement at the end of Sec 6 means that we encourage future works in efficient reasoning to also adopt similar reasoning trace analysis, which provides more fine-grained and detailed feedbacks to understand the model behavior than simply looking at overall length reduction."}, {"review_id": "3T7AiL76B1", "reviewer": "Reviewer_2UEk", "summary": "Reasoning models tend to generate redundant and inefficient chain-of-thought (CoT) traces for complex tasks, a phenomenon commonly known as overthinking. The paper proposes a simple yet effective reinforcement learning method to enable models to learn their own optimal CoT lengths without manual supervision. The length of the shortest correct response among multiple generations is used as a dynamic reward signal to guide model training. The proposed method has been shown to be able to reduce unnecessary repetitions, excessive self-verification, and over-exploration of alternatives, achieving significant reduction in output length while maintaining reasoning accuracy.", "questions": "-\tCan ShorterBetter address the issues when the sampled response is still long or encourages taking reasoning shortcuts that adversely affect out-of-distribution generalization? \n-\tAre there any tradeoffs in determining the hyperparameters of the proposed method?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n-\tThe proposed method is effective in reducing the length of reasoning traces while being able to maintain model accuracy. The obtained models perform well on both in-domain and out-of-domain benchmarks with reduced reasoning lengths.\n-\tThe proposed reasoning trace analysis frameworks, including analyzing the length of extra reasoning after reaching correct answers and fine-grained analysis of the reasoning trace structure, are useful for this line of research.\n-\tThe proposed sample optimal length serves as a dynamic and effective reward for learning efficient reasoning models. \n-\tThe paper is well-written and easy to follow.\n\n**Weaknesses**\n\n-\tThe proposed method, ShorterBetter, is motivated from the optimal reasoning length hypothesis. To approximate the optimal length, the method adopts a sample-based approach to select the shortest length of correct responses from $n$ generated rollouts. It is possible that the selected response is still long. It is also possible that the selected response is too short, so the model takes some reasoning shortcuts, resulting in reduced performance on out-of-distribution benchmarks.  It is unclear how ShorterBetter can address these issues.\n-\tThere are three important hyperparameters in  ShorterBetter, i.e., $\\alpha$, $\\beta$, and $n$. There lacks a discussion on how these parameters are chosen and what are the tradeoffs for setting these parameters.", "was_revised": true, "final_justification": "I thank the authors for the additional experiments on the hyperparameters, which are helpful, and I encourage them to include the results in the paper.\n\nWhile I still have some concerns about potential reasoning shortcuts introduced by ShorterBetter, I will retain my original rating to show my support for this work.", "rebuttal": "We thank the reviewer for highlighting the ambiguity in our interpretation of SOL and in the choice of hyperparameters. In response to your questions, our brief replies are as follows:  \n1. Our method naturally handles both long responses and avoids harmful reasoning shortcuts by adaptively balancing correctness and length.  \n2. The main trade-off lies in tuning $\\alpha$ and rollout size $n$, which balance accuracy and output length.  \n\nDetailed analysis and supporting experiments are provided below.\n\n---\n\n## **On the concerns on SOL design**\n\n**1. SOL might be too long**\n- A long SOL typically arises only on genuinely difficult problems.  \n- In such cases the shortest correct answer must necessarily be long, so an aggressive length penalty would be inappropriate.  \n- Our reward function therefore behaves *conservatively* when SOL is large; it can even penalise outputs that are *too short*.  \n- This behaviour is desirable: the model learns to respect problem difficulty and produce longer reasoning chains when required.  \n- Empirically, Fig. 1 and Table 1 confirm this dynamic balance: after training, outputs on harder benchmarks such as **AIME** and **LiveCodeBench** remain noticeably longer than on easier sets, even though overall lengths are still reduced.\n\n\n**2. SOL might be too short, causing reasoning shortcuts**\n\nWe manually inspected ~50 training examples to test whether the model was merely “guessing” the correct answer with an incorrect process (you can also refer to the last part of our reply to reviewer 5QGT). Such cases were **rare**:\n\n* All training questions are *not* multiple-choice, so random guessing is extremely unlikely.  \n* For datasets like **AIME** (answer ∈ [1, 100]) a small chance of lucky guesses exists, but its impact on training is negligible. \n* Moreover, the **α** coefficient in the correctness-reward term dampens accuracy oscillations: if an overly short SOL starts to harm reasoning quality, the α-weighted reward for correctness pulls training back on track.\n* In later training stages—once lengths have already been markedly reduced—the relative weight of **α** in the reward becomes larger, effectively stabilising any residual accuracy fluctuations. \n* However, we did observe that very large rollout sizes (*n*) occasionally produce *over-aggressive* SOLs, leading to accuracy oscillations. Hence, we should choose a moderate *n* (see detailed analysis below) to avoid this issue.\n\n---\n\n\n## **On the Practical Choice of Hyperparameters**\n\n**On the choice of $\\alpha$ and $\\beta$**\n\n$\\alpha$ and $\\beta$ are selected as a pair. Since reinforcement learning frameworks such as GRPO use **relative rewards** rather than **absolute rewards**, we fix $\\beta = 0.001$ and focus on analyzing the choice of $\\alpha$. \n\nThe hyperparameter $\\alpha$ controls the trade-off between accuracy and efficiency. Its value relative to the length penalty $\\beta$ (kept at 0.001) is the key factor. We tested new $\\alpha$ values to illustrate its impact.\n\n**Performance of `ShorterBetter-1.5B` with Varying $\\alpha$**\n\n| Dataset  | $\\alpha$=0.1 Acc | $\\alpha$=0.1 Len | $\\alpha$=2.0 Acc | $\\alpha$=2.0 Len | $\\alpha$=5.0 Acc | $\\alpha$=5.0 Len |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| AMC      | 0.43         | 2503      | 0.57         | **1946** | **0.58** | 3077      |\n| Minerva  | 0.20         | 1785      | **0.28** | **1147** | 0.24         | 2249      |\n| Olympiad | 0.30         | 2903      | **0.38** | **1814** | 0.35         | 3919      |\n| AIME     | 0.13         | 5395      | 0.20         | **2703** | **0.23** | 5274      |\n\n\nWe present new results for $\\alpha=0.1$ and $\\alpha=5.0$ to complement the results for $\\alpha \\in \\{0, 1, 2\\}$ in the main paper.\n\nFor alpha = 0.1, the training log shows that accuracy follows a declining trend over 200 training steps. During this time, the median output length decreases rapidly.\nFor alpha = 5.0, the training log indicates a clear upward trend in accuracy throughout the training process. The median output length also trends downward, decreasing from 5000 to below 2000 tokens.\n\nOur findings provide an intuition for selecting $\\alpha$:\n\n* **A high $\\alpha$ (e.g., 5.0)** places a strong emphasis on correctness. As seen in our training dynamics log and the table above, this leads to stable accuracy gains but a weaker length-reduction, resulting in longer outputs compared to the more balanced setting. The accuracy trend is clearly positive during training.\n* **A low $\\alpha$ (e.g., 0.1)** makes the length penalty term negatively influential. This creates a risk of degrading the model's reasoning capability, as the reward for correctness may not be sufficient to maintain performance. The training dynamics log shows a downward trend in accuracy. This instability can lead to worse length reduction, as seen in the table, because the model fails to learn effective, concise reasoning paths.\n* **A balanced $\\alpha$ (e.g., 1.0 or 2.0)** provides a strong enough correctness signal to preserve or improve accuracy while still exerting significant pressure to reduce output length. This achieves the best overall trade-off demonstrated in our paper.\n\nIn practice, the optimal choice depends on the specific goal (e.g., prioritizing efficiency vs. accuracy) and specific model family. Based on our results, we recommend practitioners start with $\\alpha$ in the range of [2, 5] and tune as needed.\n\n**On the choice of rollout size $n$**\n\nHere, we clarify how the rollout size *n* affects our GRPO algorithm, particularly when *n = 4, 8,* and *12* on the 1.5 B model. (Results for *n = 4* and *n = 12* are new and were not included in the original paper.)\n\n*Computational constraints*\n\n* A larger *n* increases GPU memory and wall-time requirements, especially in early training when outputs are still long.\n* With large *n* the agent may exhaust memory and trigger OOM errors; therefore we restrict experiments to modest *n* values.\n* In practice we were able to run *n = 4, 8,* and *12* reliably; higher values were infeasible on our hardware.\n\n*Training dynamics*\n\n#### All training-dynamics metrics are tested on questions unseen by the model, thereby faithfully reflecting the generalisation performance and output efficiency.\n\n| Rollout *n* | Accuracy trend                                                   | Length-reduction behaviour                                                                                        | Interpretation                                                                                                |\n| ------------ | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |\n| **4**        | Stable, no noticeable oscillations                               | Converges slowly; at step 100 the median length remains **> 1000 tokens**, and later steps rarely shorten further | Sampling space too narrow; once answers reach a moderate length the policy lacks diversity to push shorter    |\n| **8**        | Moderate, well-controlled fluctuations (see Fig. 6 in the paper) | Good balance: faster length drop than *n = 4* while maintaining accuracy                                          | Trade-off point where learning is both efficient and stable                                                   |\n| **12**       | Large oscillations with a downward accuracy trend                | Very fast: by **step 40** median length falls **< 1000 tokens**; by **step 200** it reaches **≈ 300 tokens**      | Wide sampling space produces aggressive SOL targets, accelerating length shrinkage but destabilising accuracy |\n\n* **Key observation:** Increasing *n* enlarges the candidate pool, so the chosen SOL target becomes more aggressive. This markedly accelerates length reduction but also amplifies accuracy variance.\n* **Practical recommendation:** *n = 8* offers the best balance between rapid convergence and accuracy stability and, critically, fits within the resource budget for larger models such as 7 B parameters.\n\n---"}, {"review_id": "Y9DQOgduQm", "reviewer": "Reviewer_3dRz", "summary": "This paper introduces \"ShorterBetter,\" a reinforcement learning method designed to mitigate the \"overthinking\" phenomenon in Large Reasoning Models. The core contribution is a novel, self-supervised reward mechanism based on the concept of \"Sample Optimal Length\" (SOL). For a given problem, the model generates a small batch of responses, and the SOL is defined as the token length of the shortest correct response within that batch. If no response is correct, the average length of the incorrect responses is used as a neutral baseline. This SOL-based reward is then used to fine-tune the LRM using the GRPO algorithm, guiding the model to produce more concise yet accurate reasoning traces without requiring manual length specifications. The authors demonstrate empirically that applying ShorterBetter to 1.5B and 7B parameter models results in a 50-80% reduction in output length on both in-domain and out-of-domain benchmarks, while largely maintaining task accuracy.", "questions": "refer to weakness", "limitations": "refer to weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\nThis paper presents a compelling and timely solution to a significant problem in modern LRMs. Its primary strengths lie in its originality, significance, and strong empirical results. The proposed method is novel in its use of SOL as a dynamic, data-driven reward signal, offering an elegant alternative to methods requiring explicit length budgets. The problem of computational inefficiency from \"overthinking\" is highly significant for the practical application of LRMs, and this work provides a promising path toward more efficient and deployable models. The quality of the empirical work is high; the reported length reductions are substantial and consistently demonstrated across multiple models and diverse benchmarks, supported by ablation studies that validate key design choices. The paper is also written with excellent clarity, making the methodology and results easy to follow.\n\nWeaknesses\n\nThe robustness of the core method is questionable due to its reliance on a small sample size. The entire reward signal hinges on the SOL, which is derived from a small group of n=8 rollouts per prompt during training. This small sample size can lead to a high-variance, unstable reward signal. The calculated SOL in any given step might be an artifact of a stochastic sampling outcome rather than a true reflection of the model's optimal reasoning capability for a given problem. If a truly concise and correct reasoning path is not among the 8 samples, the model will be guided toward a suboptimal target. The paper lacks a crucial sensitivity analysis exploring how the choice of n impacts training stability and final model performance. This omission undermines the quality and robustness of the proposed method, as its effectiveness may be highly dependent on this unexamined hyperparameter.\n\nA key claim of the paper is that ShorterBetter refines the structure of reasoning traces, not just their length. However, the evidence for this claim is based on an analysis performed by another LLM (Gemini 2.5 Flash Preview) acting as an automated judge. The authors use this LLM-as-a-judge to categorize sentences into functional roles like \"Pivotal Reasoning\" or \"Exploring Alternatives\" and then report shifts in their token-level proportions. The critical flaw in this evaluation is the lack of any validation for the judge's accuracy or objectivity. The paper does not report inter-annotator agreement between the LLM judge and human experts, nor does it provide any metrics to quantify the reliability of these automated categorizations. Without such validation, the conclusions about structural improvements are built on an unverified and potentially biased foundation, weakening the quality and significance of this portion of the paper's contribution.", "was_revised": false, "final_justification": "", "rebuttal": "## **On the choice of rollout size $n$**\n\nWe appreciate the reviewer for highlighting the ambiguity surrounding our choice of rollout size n. Here, we clarify how the rollout size *n* affects our GRPO algorithm, particularly when *n = 4, 8,* and *12* on the 1.5 B model. (Results for *n = 4* and *n = 12* are new and were not included in the original paper.)\n\n\n*Computational constraints*\n\n* A larger *n* increases GPU memory and wall-time requirements, especially in early training when outputs are still long.\n* With large *n* the agent may exhaust memory and trigger OOM errors; therefore we restrict experiments to modest *n* values.\n* In practice we were able to run *n = 4, 8,* and *12* reliably; higher values were infeasible on our hardware.\n\n*Training dynamics*\n\n#### Due to limited time and computational budget, we were unable to run a full evaluation suite for the rollout size variants (n = 4, 8, 12). Nonetheless, the training-dynamics already capture how each choice of n affects convergence speed, length reduction, and accuracy stability, thereby providing the key evidence needed to understand their impact. All training-dynamics metrics are tested on questions unseen by the model, thereby faithfully reflecting the generalisation performance and output efficiency.\n\nNotes: Due to NeurIPS 2025 formatting constraints, we are unable to embed the training-dynamics plots in this rebuttal.\n| Rollout *n* | Accuracy trend                                                   | Length-reduction behaviour                                                                                        | Interpretation                                                                                                |\n| ------------ | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |\n| **4**        | Stable, no noticeable oscillations                               | Converges slowly; at step 100 the median length remains **> 1000 tokens**, and later steps rarely shorten further | Sampling space too narrow; once answers reach a moderate length the policy lacks diversity to push shorter    |\n| **8**        | Moderate, well-controlled fluctuations (see Fig. 6 in the paper) | Good balance: faster length drop than *n = 4* while maintaining accuracy                                          | Trade-off point where learning is both efficient and stable                                                   |\n| **12**       | Large oscillations with a downward accuracy trend                | Very fast: by **step 40** median length falls **< 1000 tokens**; by **step 200** it reaches **≈ 300 tokens**      | Wide sampling space produces aggressive SOL targets, accelerating length shrinkage but destabilising accuracy |\n\n* **Key observation:** Increasing *n* enlarges the candidate pool, so the chosen SOL target becomes more aggressive. This markedly accelerates length reduction but also amplifies accuracy variance.\n* **Practical recommendation:** *n = 8* offers the best balance between rapid convergence and accuracy stability and, critically, fits within the resource budget for larger models such as 7 B parameters.\n\n---\n\n## **On the validation of automated judge:**\n\nWe thank the reviewer for highlighting the importance of validating our automated LLM judge. To address this, we conducted a comprehensive human-annotation study on the AMC dataset (46 problems, 2064 sentences, 65,329 tokens) using `ShorterBetter-7B`’s reasoning traces. Key inter-annotator validation results are as follows:\n\n* **Sentence-level accuracy:** 94.23% (1945/2064)\n* **Token-level accuracy:** 93.01% (60,765/65,329)\n\nThe token-level distribution comparison between the LLM and human judges is summarized below:\n\n| Functional Category                  | LLM Judge | Human Judge |\n| ------------------------------------ | --------- | ----------- |\n| Pivotal Reasoning                    | 22.37%    | 23.42%      |\n| Productive Elaboration & Calculation | 53.32%    | 50.79%      |\n| Exploring Alternatives               | 10.50%    | 10.79%      |\n| Verification & Self-Correction       | 8.69%     | 10.16%      |\n| Non-Substantive Statement            | 5.13%     | 4.84%       |\n\nQualitatively, we observe that:\n\n* The inter-annotator agreement is consistently very high across all five categories. Thus, the reliability of our LLM judge’s categorizations is strongly supported.\n* Most disagreements occur between the categories \"Productive Elaboration & Calculation\" and \"Pivotal Reasoning\"—both regarded positively in our analysis. Consequently, our main conclusion—that ShorterBetter jointly increases the proportion of these meaningful reasoning categories—remains robust despite minor distributional discrepancies.\n\nWe selected the AMC dataset as it strikes an ideal balance: it features non-trivial mathematical reasoning while remaining small enough to annotate exhaustively within our rebuttal timeline. For transparency, we plan to release our human-annotation platform and extend validation to additional benchmarks in the near future."}, {"review_id": "sRes6Z7TGG", "reviewer": "Reviewer_5QGT", "summary": "This study proposes a new method to reduce the length of the reasoning chain without sacrificing the model's performance. Specifically, in the proposed method, the shortest and correct reasoning chain is first determined via sampling, and then based on such statistics, the model reward is refined, and the model parameter is updated via Group Relative Policy Optimization. The proposed method shows more reduction in the output while maintaining the accuracy across broad benchmark sets.", "questions": "[Comparison with baselines]   \nThe proposed method and baseline methods differ in several dimensions, such as training data,  hyperparameters (Appendix A.3), and formulation details, such as GRPO. Thus, it is not clear what was truly the key to achieving improvement over baselines. The use of out-of-domain seems a weak argument to skip such concerns (ll. 483–485).\n\nParticularly, if the main claim of the paper is the use of the shortest correct response (as stated below and abstract), one should conduct clearer ablations about the definition of l^{SOL} (l.131).\n>(Luo et al., 2025a), for example, compares generated lengths against those from a reference model. More in line with our proposal of sampling multiple responses, Kimi 1.5 (Team, 2025) and Training Efficient (Arora and Zanette, 2025) utilize the shortest and average lengths, respectively, among all sampled responses as reward baselines. In contrast, ShorterBetter innovatively anchors its reward function to the length of the shortest correct response\n\nSpecifically, one can examine the method variants without the correctness condition I(y_j=y_i*)=1 or just using the absolute length of the output in defining l^{SOL}.\nSection 5.3 partially handles the concern around this point, but the analysis focuses more on orthogonal concerns about alpha and the case when there is no correct response. \n\n[Experiments with other models]  \nI’m concerned about the possibility that the success of proposed methods is biased toward Deepseek-based models. These models yield somewhat distinctive outputs with, e.g., aha-moment, and I’m interested in whether the conclusions hold even with other models.  \n\n[Quality of shortened outputs]  \nAlthough coarse analysis on the reasoning structure has been done in Section 5.2, I'm more curious about the quality (human interpretability) of the shortened reasoning chains. If the shortened chain is no longer coherent, not helpful for human users (e.g., due to substantial omission in explanation), or amplifies inappropriate properties (e.g., hurting politeness), these will be limitations in a real human-LLM interaction. Such human-involved analysis will also be informative for the readers.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n- The proposed approach achieved substantial output length reduction without compromising the accuracy\n- Broad evaluation tasks, including out-of-domain ones, are adopted, and the advantage of the proposed method is consistently observed.\n- Analysis of the reasoning structure is appreciated, which is informative about what kind of change is really made for the output\n\n\nWeakness\n- It is unclear which part of the proposed method truly contributed to the improvement over baseline methods, such as Training Inefficient (see Questions)\n- The proposed method is only applied to DeepSeek-R1-Distill-Qwen-1.5B/7B models (see Questions)\n- Further evaluation (or clarification) on the quality of the shortened reasoning chain (beyond correctness of the answer) will be encouraged (see Questions)", "was_revised": true, "final_justification": "Through the rebuttal, my concerns, particularly on ablations, are (not perfectly but minimally) handled, which makes me lean toward acceptance (3->4).", "rebuttal": "We thank the reviewer for their constructive feedback. The questions raised about controlled comparisons, ablation clarity, model generalization, and quality of shortened outputs are crucial. We have conducted new experiments to address these points directly.\n\n---\n\n## **On the Comparison with Baselines**\n\nWe agree with the reviewer that isolating the key factors for improvement is essential. To create a more controlled comparison and demonstrate the specific impact of our **Sample Optimal Length (SOL)** reward design, we conducted a new experiment.\n\nWe implemented the reward function from a key baseline, `Training Efficient`, directly within our own training framework. This ensures all other variables are held constant:\n* **Base Model:** DeepSeek-Distill-Qwen-1.5B\n* **Training Data:** The same dataset used for `ShorterBetter`\n* **Training Algorithm:** GRPO\n\nThe `Training Efficient` reward is formulated as $\\mathbb{E}\\left[\\mathbb{1}_{y = y^*} \\cdot (1 - \\alpha f(\\text{LEN}(y)))\\right]$, where $f(\\cdot)$ is a sigmoid function over the normalized length. We used $\\alpha=1$ for the most aggressive length reduction setting.\n\n**Controlled Comparison on Math Benchmarks (1.5B Models)**\n\n| Dataset | `ShorterBetter` Acc. | `ShorterBetter` Len. | `Training Efficient` Reward Acc. | `Training Efficient` Reward Len. |\n| :--- | :---: | :---: | :---: | :---: |\n| AMC | **0.57** | **1946** | 0.49 | 2032 |\n| Minerva | **0.28** | **1147** | 0.23 | 1472 |\n| Olympiad | **0.38** | **1814** | 0.35 | 2597 |\n| AIME | **0.20** | **2703** | 0.20 | 5219 |\n\nThe results show that under identical training conditions, our SOL-based reward function outperforms the baseline's formulation in both accuracy and length reduction across nearly all benchmarks. This provides strong evidence that our reward design is a key driver of the observed improvements.\n\nOf course, we acknowledge that a full \"apple-to-apple\" comparison with `Training Efficient` is unfortunately infeasible within the rebuttal window, because our settings differ in data, RL algorithm, reward design, and compute requirements. Here, we summarize the key distinctions:\n\n| Aspect               | `Training Efficient`                       | `ShorterBetter`                                              |\n| -------------------- | ---------------------------------------- | ---------------------------------------------------------- |\n| RL method            | PPO                                      | GRPO (group‑based)                                         |\n| Reward function      | Normalized length + sigmoid              | SOL                                                        |\n| Data                 | MATH, cnk12, AIME, AoPS and the Olympiad | DeepScaleR                                                 |\n| Compute requirements | 7B & 1.5B models: 8 × GB200 for 20 h     | 7B model: 8 × A100 for 10 h; 1.5B model: 4 × A100 for 16 h |\n\nIn the paper, we compared our trained model with their released checkpoints. `Training Efficient` shows weaker length reduction, which we believe is due to its conservative reward design: the length signal is normalized and passed through a sigmoid, so long but correct answers are still rewarded, just slightly less. They did not provide the model with a clear signal during RL. \n\nHowever, `ShorterBetter` defines an explicit pivot (SOL) and lets it guide the model to quickly identify the desired output length. For example:\n\n- Suppose a prompt yields 8 responses, and 2 are correct.  \n- One correct answer is **length 100**, another is **length 1000**.  \n- `Training Efficient`: Both are rewarded positively, with the longer one slightly less.  \n- `ShorterBetter`: The 1000-token answer is **penalized**, because the 100-token answer proves the task can be solved concisely.  \n\nThis design allows the model to quickly converge toward short, correct answers and achieve higher efficiency in length reduction without sacrificing accuracy.\n\n---\n\n## **On Clearer Ablation of the Reward Function**\n\nWe agree that a clear ablation on the definition of $l^{SOL}$ is essential to prove our central claim. As the reviewer suggested, we tested variants that remove or alter the correctness condition. Our ablation studies confirm that naively targeting the shortest length is not viable and that the specific design of SOL is critical.\n\n1.  **No Correctness Reward, Shortest Length Target:** As shown in Section 5.3, if the reward only minimizes length by targeting the shortest response in the group, the model quickly learns to produce empty outputs, causing accuracy to collapse to zero.\n2. **\"Partial\" Shortest Length Target:** Using the shortest length as a target *only* when no correct answers are found also fails. This creates a feedback loop where the model sacrifices correctness for length, leading to training collapse.\n3.  **Correctness Reward + Always Shortest Length Target:** In a new ablation, we kept the correctness reward ($\\alpha=2$) but always set the length target to the shortest response in a group. This also crashed the training, with logs showing accuracy and length rapidly declining to zero. Even with a reward for correctness, a single short (but incorrect) response heavily penalizes all longer (and potentially correct) responses, creating a fatal bias toward underthinking.\n\nThese experiments underscore that the length target design is paramount. Our proposed **Sample Optimal Length (SOL)**, which is robustly anchored to the shortest *correct* response and defaults to the *average* length in failure cases, provides the necessary stability to avoid these collapse scenarios.\n\n---\n\n## **On Experiments with Other Models**\n\nTo address the valid concern about model architecture bias, we evaluated ShorterBetter on a model with a different architectural backbone: **`deepseek-ai/DeepSeek-R1-Distill-Llama-8B`**. This allows us to test our method's generalization beyond the Qwen series. We trained this 8B Llama-based model for 100 steps using our framework.\n\n| Dataset | Baseline (Llama-8B) Acc. | Baseline (Llama-8B) Len. | SB alpha=5  (Llama-8B) Acc. | SB alpha=5  (Llama-8B) Len. | SB alpha=2  (Llama-8B) Acc. | SB alpha=2  (Llama-8B) Len. |\n| :--- | :---: | :---: | :---: | :---: |  :---: | :---: |\n| AMC | **0.67** | 6995 | 0.62 | 3384 | 0.57 | **2808** |\n| Minerva | **0.33** | 5555 | **0.33** | 2669 | 0.30 | **1343** |\n| AIME | 0.33 | 9949 | **0.37** | **5634** | 0.27 | 6060 |\n\nThe results show that `ShorterBetter` successfully generalizes to a different model architecture. Here we choose hyperparameter alpha=5 since it achieves a better balance between accuracy and length reduction.\n\nRegarding the \"aha-moment\" outputs, we view this as a characteristic of certain reasoning trace cases generated during inference, rather than a fundamental property of the DeepSeek distilled model series that would uniquely bias our method's effectiveness. The results on the Llama model support the conclusion that our method is broadly applicable.\n\n---\n\n## **On Quality of Shortened Outputs**\n\nWe thank the reviewer for highlighting the importance of human-involved analysis regarding the quality and interpretability of shortened reasoning traces. To directly address this, we conducted a detailed manual analysis of `ShorterBetter-7B`’s reasoning traces for all 46 problems from the AMC dataset. Our analysis yields two primary insights:\n\n1. **Coarse-level human agreement:** As detailed in our response to Reviewer 3dRz, inter-annotator agreement between humans and our LLM judge is high (sentence-level category accuracy: 94.23%). This supports the reliability of our reasoning structure analysis framework.\n\n2. **Preservation of coherence and readability:** Through careful inspection, we confirmed that the coherence of reasoning traces remains largely intact. Specifically, consider the first problem in the AMC dataset:\n\n> *$\\frac{m}{n}$ is the irreducible fraction value of $3+\\frac{1}{3+\\frac{1}{3+\\frac{1}{3}}}$. What is the value of $m+n$?*\n\nBoth baseline and `ShorterBetter-7B` models answered correctly, but ShorterBetter reduced reasoning length from 1320 to 654 tokens. Human inspection revealed some key insights:\n\n- **Removal of non-substantive statements:** Unnecessary statements (e.g., \"I think that's solid\" or \"each step seems to check out\") were effectively eliminated.\n- **Reduction in redundant verification:** The baseline explicitly and verbatim repeated calculations. ShorterBetter instead employed more concise verification methods, utilizing assumptions inherent in the problem.\n- **More concise pivotal reasoning:** ShorterBetter replaced overly verbalized calculations (e.g., \"So now, adding $\\frac{9}{3}$ + $\\frac{1}{3}$ gives me $\\frac{10}{3}$\") with succinct mathematical expressions, preserving clarity and coherence.\n\nOverall, we observe that `ShorterBetter`’s RL fine-tuning shrinks reasoning trace length without compromising readability and interpretability. In fact, its reductions are very reasonable and human understandable.\n\nWe acknowledge that our current evaluation primarily addresses rigorously structured domains (e.g., mathematics & programming), where criteria such as \"politeness\" or \"helpfulness\" are less pertinent. Extending our framework to open-ended settings may introduce new challenges—such as potential reward hacking—where reducing trace length might negatively impact such human-interaction qualities. We plan to investigate these issues in future work."}]

Author rebuttal:

Meta-review: The paper proposes controlling response lengths in CoT reasoning for avoiding overthinking. This is done via RL with length regularization.
Reviewers note clear presentation, comprehensive experiments, empirical strength, good ablations. On the other hand, more ablations further evaluation also on different models and many hyper parameters. Also concurrent similar work is mentioned.
Given the good experimental results and reviewer consensus, the paper merits acceptance at NeurIPS.
Decision: Accept (poster)