Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
Authors: Bin Hong
Jiayu Liu
Kai Zhang
Jianwen Sun
Mengdi Zhang
Zhenya Huang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Large Reasoning Models
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper analyzes existing preference optimization method and proposes a computationally efficient method to mitigate the problem of overly lengthy outputs for Large Reasoning Models.

Abstract: Recent advances in Large Reasoning Models (LRMs) have demonstrated strong performance on complex tasks through long Chain-of-Thought (CoT) reasoning. However, their lengthy outputs increase computational costs and may lead to overthinking, raising challenges in balancing reasoning effectiveness and efficiency. Current solutions often compromise reasoning quality or require extensive resources. In this paper, we investigate how to reduce the generation length of LRMs with limited tuning. We analyze generation path distributions and filter generated trajectories through difficulty estimation. Subsequently, we analyze the convergence characteristics of various preference optimization objectives under a unified Bradley-Terry loss based framework. Based on the analysis, we propose Length Controlled Preference Optimization (LCPO) that directly balances the implicit reward related to NLL loss. LCPO can effectively learn length preference with limited data and training. Extensive experiments demonstrate that our method significantly reduces the average output length of LRMs by over 50\% across multiple benchmarks while maintaining the reasoning performance. Our work highlights the potential for computationally efficient approaches in guiding LRMs toward efficient reasoning.

Published as a conference paper at ICLR 2026
PRUNING LONG CHAIN-OF-THOUGHT OF LARGE REASONING MODELS VIA SMALL-SCALE PREFER-
ENCE OPTIMIZATION
Bin Hong', Jiayu Liu', Kai Zhang', Jianwen Sun', Mengdi Zhang Zhenya Huang1.2.1
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China
2 Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
3 Artificial Intelligence in Education, Central China Normal University
Meituan NeoShell
hb20022maii il, uste. restc.edu.... en mdzhangmde gmae com huangzhy@ustc.edu
ABSTRACT
Recent advances in Large Reasoning Models (LRMs) have demonstrated strong performance on complex tasks through long Chain-of-Thought (CoT) reasoning. However, their lengthy outputs increase computational costs and may lead to over- thinking, raising challenges balancing reasoning effectiveness and efficiency. Current solutions often compromise reasoning quality or require extensive re- sources. In this paper, we investigate how to reduce the generation length LRMs with limited tuning. We analyze generation path distributions and filter generated trajectories through difficulty estimation. Subsequently, we analyze the convergence characteristics of various preference optimization objectives under a unified Bradley- loss based framework. Based on the analysis, we pro- pose Length Controlled Preference Optimization (LCPO) that directly balances the implicit reward related to NLL loss. LCPO can effectively learn length pref- erence with limited data and training. Extensive experiments demonstrate that our method significantly reduces the average output length of LRMss by over 50% across multiple benchmarks while maintaining the reasoning performance. Our work highlights the potential for computationally efficient approaches in guiding LRMs toward efficient reasoning.
INTRODUCTION
Building upon Large Language Models (LLMs), numerous powerful models have emerged. sig- nificantly enhancing AI capabilities across diverse domains (Jiang et al. 2025; Guo et al. 2024] Wang et al. 2024). Recently reasoning-oriented LLMs, Large Reasoning Models (LRMs) such as Deepseek- RI (Guo et al. 2025) and QwQ-32B (Team 2025 Yang et al. 2024b) have achieved remarkable performance
complex reasoning tasks. These models learn perform long Chain- of-Thought (CoT) (Wii et al. 2022b) thinking through online reinforcement learning with verifiable reward (RLVR) (e.g., PPO (Ouyang al. 2022) and GRPO (Shao al. 2024) with conventional accuracy reward), which explicitly decomposes the given problem and proceeds to solve each sub- problem in a step-by-step manner. Furthermore, they actively recall and verify previously generated steps throughout the reasoning process (Cheng et al., 2025 Xue et al., 2024). This significantly in- creases the output length while empowering LLMs to effectively address cessoonnnn-intensii tasks.
The tendency of LRMs to generate excessively long responses introduces additional chal- lenges (Feng et al., 2025 Qu et al., 2025) Liu et al., 2025b] Xu et al.) 2025c] Liao et al.). As shown in Figure [ for an easy question, an LRM can consume 5465 tokens to solve it. On one hand, this tendency significantly increases computational and memory costs when using LRMss for various downstream tasks, which limits the applicability of continual learning for further enhancing the capabilities of LRMs. On the other hand, overly long outputs may lead to overthinking, where LRMs consume more tokens than standard models and may produce redundant even incorrect
CCoresponding author

---

Published as a conference paper at ICLR 2026
Problem: l'nnd the value log;3 log,4 log45 logs6 log.7-log,8.
ground truth:
DeepSeek-BR Distill-Qwennt
LCPO-7B
Problem Understanding: 188 tokens
Problem Understanding 64 tokens
Athigh,Arrihh,, so have this problem here: need... That the expression is..
<think=Okay, so need to find... with changing bases and arguments..
Hmm, okay, So., Let me think.
Deduction: 841 tokens
Deduction: tokens
recall the change of base formula...
remember something about changing the base.. So the value of the entire product is
Let'sdenote each term follows:loge hr3 in2..
Looking this... Well, 2 cubed 80 log.8
Self-Verification times 3993 tokens
Self- Verification time: 120 tokens
Wait, me verify that step because seems straightforward..
Wait, make sure didn't iedn'tttnn make mistake... The value </think>
Allernaively, in case had any doubls..
I'm confident that the value of the given product...
Final Answer: <</think>
Answer: 443 tokens
Answer: 457 tokens
To find the vflue
To find the value Thus, the value is
Thus, the value of the given expression is
188 + 841 3993 + 443 5465 tokens!
64+487+1204487-111 tokens!
tokeens 4337 (9.37%))
An example math problem from MATH-500. Left: the lengthy output of LRM. Right: the concise output of LRM after trained with LCPO.
Figure I:
outputs on relatively simple queries (Sui et al. 2025). These challenges have prompted a renewed focus on a critical question for LRMs: How to enable efficient reasoning with shorter output length and maintained reasoning performance?
Exploration of this question remains limited. Inference-time pruning methods (Xu et al. 2025b
Muennighoff et al. 2025) append length-related tokens to the prompt generation process to en- courage earlier termination. While these approaches are low-cost, they are unstable in reducing length, and often compromise the model's reasoning capabilities. Subsequent works (Aggarwal & Welleck 2025) Hou et al., 2025 Arora & Zanette 2025) adopt large-scale multi-objective paradigm. They introduce a enngh-regularrieer reward alongside the conventional accuracy reward and perform RL on a large amount of data, aiming to reduce generation length and recover reason- ing capacity. Online RL approaches involve complex training systems and higher resource demand compared to offline fine-tuning Yu et al. 2025 Chen et al.), which contradicts the idea of efficient reasoning. Additionally, when incorporating budget-foreing mechanisms where token limit is set in the prompt (Muennighoff et al., 22025] Aggarwal & Welleck, (025). online RL-based methods are inherently constrained by manually predefined token budgets, limiting their adaptability dynamic task requirements and can again compromise reasoning capabilities.
To address these limitations, we investigate how reduce the generation length of LRMs with lim- ited tuning. As RLVR biases the output distribution toward rewarded trajectories (Yue et al. 2025 explore biasing the trajectory distribution explicitly within the LRMs generation space Yue al. 2025) in an offline manner. In this paper, we ask the following research questions: RQI: Is there still shorter yet effective reasoming paths within the generation space of reasoning models? RQ2: How can we leverage limited training and data to adjust reasoning model's generation distribution?
To answer these questions, we empirically analyze the trajectory distribution of generation space of LRMs. We use the pass rate as proxy for the perceived difficulty of problem and leverage this metric to reduce data volume while reserving concise generation patterns. Subsequently, we employ preference optimization distribution toward the shortest effective paths. To find an ideal algorithm for effective small-scale training. theoretically analyze the convergence conditions of different objective functions used in various preference optimization methods under a Bradley-Terry loss framework. As discussed in Appendix []] we find the implicit reward associated with Negative Log Likelihood (NLL) loss can
efficient self-distillation training method to push the output
2

---

Published as a conference paper at ICLR 2026
Table Data needs for rollout and training of different efficient reasoning methods.
Method
Type
Data
Train Samples
CoD (uu et al.
2025b)
inference-time
L1 Aggarral & Welleck
2025
RL & budget-foreing
645k
645k
TrEff (Arora & Zanctte
2025)
RL
24.8k
24.8k
DAST (henn 2025)
preference optimization
150k
20.6k
Ours
preference optimization
22k
0.8k
hinder length preference alignment. Based on this analysis, we propose Length Controlled Prefer- ence Optimization (LCPO), which incorporates an explicit term of identical form to counterbalance the implicit NLL-related reward. This enables precise length alignment with small-scale training data, yielding an efficient, low-cost, and computationally lightweight reasoning pipeline.
We conducted extensive experiments using DeepSeek- R-i--ittlll-Oe Swenil---ee-1...5 and DeepSeek-RR Seaill-Qwen-7B.. following previous works (Aggarwal & Welleck, 2025; Arora & Zanette 2025)
on many widely used benchmarks. Our method reduces the average generation length by over 50% across all benchmarks while maintaining the original model's reasoning performance. As illustrated in Figure [] our approach can prune unnecessary parts of the reasoning trajectory, reducing length by 79.37%, and guides the model to more concise reasoning. Furthermore, as shown in Table our method requires only 0.8k training samples and 50 training steps, largely reducing computational cost compared to prior approaches. We believe our findings and method provide valuable insights for future research efficient reasoning in LLMs.
In summary, our main contributions are as follows:
We investigate the generation space of LRMs and introduce simple yet effective data filter- ing strategy that preserves a more concise generation mode. Under unified Bradley-Terry loss framework, we further analyze the convergence characteristics of different preference optimiza- tion methods, providing insight of offline alignment objectives.
We propose Length Controlled Preference Optimization (LCPO), an objective specifically de- signed to effectively learn length-related preferences from limited training data, offering enhanced efficiency and generalization ability compared to existing techniques.
We conduct extensive experiments on several popular reasoning benchmarks. The results demon- strate that our method significantly reduces average output length by over 50% across different benchmarks while maintaining reasoning performance. We also perform further experiments to gain deeper insights into the characteristics of our method.
2 RELATED WORKS
Reinforcement Learning on LLM. Recent works typically employ reinforcement learning with verifiable reward (RLVR) to build models with enhanced reasoning capabilities (Guo et al. 2025 Jacch et al.) 2024 OpenAl 2025) through online RL algorithms such Proximal Policy Opti- et al. 2022) (PPO) and Group Relative Policy Optimiza- tion (Shao et al. 2024) (GRPO). However, this paradigm involves alternating generation with pa- rameter updates, which increases training complexity. It typically demands more computational resources and is significantly slower compared offline tuning methods. Direct Preference Opti- mization (Rafailov et al., 2023) (DPO) is an offline alternative to online RL algorithms, introducing implicit reward via pairwise preference data through the Bradley-Teri -eery model. Subsequent advance- ments including SimPO (Meng et al.) 2024), SimPER (Xiao et al. (025) and ORPO (Hong et al.) 2024
mization (Schulman et al. 2017
further eliminate the need for the reference model in the DPO objective.
Test-Time Scaling and Efficient Reasoning. enhance reasoning abilities (Chen et al. 2025 Xu et al. 2025a), known as Test- Test-Time Scaling (Wei
LRMs often generate excessively lengthy outputs to
et al.
2022a Wang et al.] 2h22g Zhang et al. 2025b
Liu et al. 2025b Feng et al. 2025). To reduce
3

---

Published as a conference paper at ICLR 2026
1.0
800
16000
700
connect futter filter)
connect fbefore filseri
14000
600
whomg
208
ABAcuuac
12000
Frequenyy 3
500
Accuracy
10000
400
7R1 ength
-.0.7
150 Length
5000
6.000
4000
05
10
12
14
2000
0
4000
6000
8000
10000
12000
Rank Index of Trajectory Groups
Responese Length
(a) Performance of 7B and 1.5B models on LIMR with varying average output lengths.
(b) Distribution of output length from 7B models on LIMR before and after filtering
Figure 2: (a) As the number of output tokens increases, model performance tends to deteriorate. The x-axis represents the sorted rank of the samples. (b) After filtering, the distribution change is minor.
the computational overhead and redundancy introduced by this, several studies (Muennighoff al. 2025 Zeng et al., 2025 Xu et al., 2025b] Arora & 7anette 2025) incorporate length-related tokens or impose explicit constraints prompt or generation process, which often compromise models' performance. Besides, explicit length penalties naturally leveraged in the reward computation of RLVR for shortening reasoning trajectories (Team et al. 2025b; Aggarwal & Welleck 2025). This approach typically requires large-scale training with high resource demands in online manner and can lead to performance drop when incorporating with budget-forcing (Muennighoff et al.) 2025).
Recent Hybrid-CoT researches provide another perspective by reframing efficient reasoning as implicit text classification problem. LRMs are trained to dynamically choose between thinking mode and non-thinking mode. Nevertheless, they typically rely on large-scale online RLVR (Zhang) iet al.] [2225a] Tu et al.) (2225). There are also some works explore offline tuning paradigms. For instance, DAST (Shen al., (025) designed difficulty
fffficulty-based based ranking function for preference and leveraged SimPO for training. Ada-R1 (Luo et al.) (2025)) employed model merging followed by DPO on bi-level preference data to explicitly distinguish between two thinking modes.
METHOD
3.1 OVERVIEW
We refine the training procedure based on three key ingredients of RL: 1) Data: We harvest outputs from the generation space of the model using only one seamless rollout process and a small-scale prompt data. 2) Reward: Reward is implicitly determined by the ranking of data. We adopt an effective strategy to select data and rank purely by length to maximize the preference signal. 3) Algorithm: We investigate the objectives of existing preference optimization methods, and introduce Length Control Preference Optimization (LCPPO) for more efficient length pruning.
3.2 DATA AND REWARD: EMPIRICAL STUDY ON GENERATION SPACE
In this section, we aim to answer RQ1: Is there still shorter yet effective reasoning paths within the generation space of LRMs? We conduct experiments to analyze the trajectory distribution of the generation space of LRMs. Throughout this process, we introduce our design on data and reward.
Analysis Settings and Notations. In this section, we conduct analysis LIMR (Li et al. 2025) with ---ieeek------tillll------------ objectively difficult, but relatively easy for the model. For each question qi in the dataset D. generate 16 outputs and sort them by length (e.g., the number of tokens in the sequence). This procedure samples a set of reasoning trajectories from the output space of model M:
and
-iiit-RR--isiil------e--- 1.5B. Questions in LIMR are
{(a to {
(I)
where of is the r-th shortest output of gr 8; denotes the average accuracy of question y over {of 16
4

---

Published as a conference paper at ICLR 2026
First, evaluate how LRMs' performance across the whole dataset varies with the r-th shortest reasoning trajectory set |2 As shown in Figure 2a outputs with longer average lengths exhibit noticeable performance decline: the 7B model shows degradation for rankings above 10, while the 1.5B model declines earlier from ranking 7. Conversely, for rankings 0-6 with shorter outputs, performance remains stable despite length variations. This suggests that reasoning models with extended deliberation tendencies can still perform efficient, shorter reasoning without significant accuracy loss, providing a positive answer to RQI. This highlights the potential of leveraging model- generated data for behavior refinement.
Although model performance shows a clear decline with longer outputs, we argue that this is a cor- relation rather than a causal relationship. For reasoning oriented models, their tendency to reflect and self-verify often leads to longer responses on questions they answer incorrectly, as they typ- ically perform more extensive trial-and-error reasoning (Sui et al. 2025: Liu et al. (025c). This suggests that output length is influenced by the model's perception question diffiully,, indicating that responses to more difficult questions may inherently contain more complex and lengthier gen- cration mode. While the model can achieve over 90% accuracy with approximately 2200 tokens, requires average length of 7000 tokens to complete reasoning the same tasks. This indicates a misalignment between the model's 's perceived difficulty and its actual capabilities. Even on tasks well within its capabilities, the model still wastes excessive tokens reasoning due its inflated sense of challenge. To this end, we adopt a coarse and heuristic approach to approximate question from the model's perspective and differentiate the conciseness of generated responses. Specifically, we categorize each question along with its outputs o based on the corresponding score sy
-orsonitg-orietted
eess if 1, medium difficult 8; = 0.
la:)l(g;:)
0 8 < 1,
To encourage the model to maintain exploratory behaviors when encountering difficult problems while correcting its perception to prevent excessive responses on simple tasks, we train on questions the model has already mastered. Thus, for data, we leverage the easy split of the generated data. And for reward (e.g. ranking of preference data), we select the shortest response as the chosen one and the longest as the rejected one. Details about the datasets refer to Table 7
3.3 ALGORITHM: EMPIRICAL STUDY OF PREFERENCE OPTIMIZATION METHODS
In this section, we aim to answer RQ2: How can we leverage limited training and data to adjust reasoning model's generation distribution? First, we provide brief preliminaries of preference opti- mization, which is further detailed in Appendix C Then we begin with preliminary experiment to investigate the potential of existing preference optimization methods in effectively capturing length preferences. After that, leverage Bradley-Teme model (Bradley & Terry 1952) to gain theoret- ical insight of the convergence characteristics of these methods. Finally, based on our findings and analysis, we introduce Length Controlled Preference Optimization (LCPO), effective preference optimization method for shortening output length of reasoning models with small scale training.
Bradley-Terry Model. Let B4 E IR represent the ability value of team i. Let the outcome of a game between teams and be determined by the difference ability values 3, - B The Bradley-Terry model (Bradley & Terry 1952) defines the log-odds that team beats team j as
log
Piá Pij
B, B3 Pij 1 = =(3 Bz),
(3)
DPO and Preference Optimization Let x be the input sampled from the context space X. We define LLM 0 as our policy <)(y)=)), where the probability of generating a sequence y To(y)ar) (y)))) Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al. 2022) uses unpaired data. It adds KI. divergence (Kullback & Leibler 1951) penalty to the reward to restrict the policy distribution from shifting excessively, The objective is to maximize accumulated rewards:
LRLIF Rg go
[r(xr, v) I (To((())))))) (er [y))]
(4)
5

---

Published as a conference paper at ICLR 2026
Table 2: Results of different preference optimization methods. We report aceuracy (Acc) and erage number of tokens in the generation (Len). Bold indicates the best trade-off. A f % denotes length reduction ratio (higher is better).
MATH-500
GSM8K
Method
Acc
Len (A f %)
Acc
Len (A f %)
Original (Guo et al. 2025
92.20
4223
91.81
1677
SFT Ouyang el al. 2022
91.00
3844 (9.97%)
89.41
805 (52.00%)
DPO Ratarly
2023
92.00
2823 (33.15%)
92.84
1059 (36.85%)
SimPO Meng et al. 2024
91.80
3750 (11.20%)
92.87
1687 (-5.96%)
ORPO Hong al. 2024)
91.20
3137 (25.72%)
91.43
988 (41.09%)
SimPER (Xiao et al
2025
91.80
3832 (9.26%)
92.19
1629 (2.86%)
Ours
91.40
2033 (51.86%)
92.95
796 (52.53%)
Direct Preference Optimization (DPO) (Rafailov et al. 2023) derives an implicit reward from RLHF:
(D u) = 3 log To(y
+ 8 log Z(x),
(5)
Teee
where ((x) Ey (z)() exp( r(x, (r., y)) is the partition function that is hard to estimate. DPO
leverages the BT model to eliminate (x)), resulting the following objective:
IDPO = -BggDD.rrsttnn [log o log To (ymlx)) Tref (te (x)
To (y (ar)
33 log
)]
(6)
Tred (yr (x)
3.3.1 DIFFERENCES AMONG PREFERENCE OPTIMIZATIONS
Experimental Setup We train DeepSeek Ba-eeek-RI-Dsttiie Qwen-7B on the easy split of LIMR and evaluate different preference optimization methods on MATH-500 (Lightman et al. 2023) and
GSM8K (Cobbe al. 202T). Training configurations are provided in Appendix B
Table 2 summarizes the results of our experiment. For the MATH-500 dataset, among existing methods, DPO (Rafailov et al. (203) achieves superior performance with proper tuning on hyper- parameters, reducing average output length by 33.15%. SFT (Ouyang al. 2022) shows limited ef- ficacy, yielding only a 9.97% length reduction. Notably, the recently proposed SimPER (Xiao et al.) 2025) underperforms with merely 9.26% reduction. ORPO (Hong al. (024) demonstrates more significant improvement (25.72%). substantially outperforming SFT. On the GSMSK benchmark, SFT and ORPO exhibit accuracy degradation but achieve more substantial length reductions (52.00% and 41 .00% respectively).
3.3.2 INSIGHT OF CONVERGENCE CHARACTERISTICS ON PREFERENCES
Several approaches show clustered patterns. We further investigate where these convergence charac- teristics come from and how they influence preference alignment. We analyze the objective functions of multiple methods in a general setting, applying the insights to scenarios with length preferences.
We reformulate the objective functions of different methods into log-sigmoid function form:
log o (R( 91, (x))
The transformed objectives can then be interpreted as specialized BT models, where the sigmoid's parameters R(ye: 3r (x) explicitly encode the reward difference hetween competing objective func- tions. During preference optimization, the model converges when the sigmoid output approaches 1, indicating near-deterministic preference satisfaction. Thus, we assess convergence by verifying the condition where a Ry : y1 (z) Detailed derivations are provided in Appendix D
We theoretically illustrate in Equation 21 that the less satisfactory outcomes SFT stem from the NLL loss. For ORPO, shown in Inequation 26 the probability of the chosen response rises, the convergence of the objective becomes increasingly dominated by the NLL loss. In Equation we we find that although SimPER can be helpful in non-reasoning tasks relatively easy reasoning tasks

---

Published as a conference paper at ICLR 2026
(e.g. GSM8K (Cobbe al. 2021). shown in (Xiao al. 2025) ), it is less effective in preference
alignment for length control. DPO (Inequation 32) and SimPO (Inequation 33) have more relaxed
convergence conditions but are highly dependent hyperparameters, which can lead to unstable performance with limited training data.
3.3.3 AN OBJECTIVE BETTER FOR CONVERGENCE ON PREFERENCE DATA
Based on the analysis, we can conclude that two ingredients are needed in the context of length control: 1) Bradley-Terry loss with relaxed reward margin and 2) well balanced negative rewards for the NILL loss. So first our objective follows the conventional BT loss form:
log a (Bro (yw (x) - Bro (<) +).
(8)
We set to zero to let our model converge earlier for ingredient 1). Note that the reward of the NLL part can be shown explicitly in an objective of Bradley- Terry form:
1 37
po
LNEI.
log (ogl) log a (log 1 po (7((yxr) [r)
(9)
where
pe p
po(u)æ) exp(
log (z(2)))) re(y)a>) log( 1 po (z)
(10)
|y1
We directly balance this reward with a negative counterpart for ingredient 2). Then our proposed objective is follow:
LECOO = log a (log( po po(Uwlax) log(
Po (yu (z))
(11)
Pe (Vwllx))
Pe (yr (r)
Our objective can converge rapidly without requiring any hyperparameter tuning.
EXPERIMENTS
4.1 MAIN RESULT
and DeepSeek-RI-Disee Qwen-1.5B (Guo al. 2025 base models. We evaluate our method on the following math reasoning benchmarks: MATH-500 (Lightman et al., 2023), GSM8K (Cobbe et al. 2021) Minerva-Math (Lewkowyzz al. 2022), AIME24 (Online) 2025), AMC23 (of Americal (025) and
Experimental Setup We choose -
OlympiadBench (He et al.) 2024). We use accuracy as the metric for evaluation. Considering the
limited size of AIME24 and AMC23, we sample 16 outputs for each question and compute pass @ score (e.g., accuracy averaged over 16 runs). Information about the benchmarks and baselines can be found in Appendix A Detailed experiment settings can be found in Appendix B
LCPO effectively reduces length while main- taining reasoning performance. As shown in Table 3 the experimental results demonstrate that our method achieves consistent effectiveness across diverse benchmarks. Our approach signif- icantly reduces generation length by at least 46% across all datasets. On the challenging Minerva- Math, our method prunes 64% of the output, while bringing slight performance gain. With our method, reasoning performance remains sta- ble across most benchmarks, with only slight drops on a few. Overall, our method achieves more than 50% average length reduction over all benchmarks and yields the best overall trade-off, while the computation demands are lower.
7B-Original
5.0
TOOOOO
LSB-Origintl
155-LCPO
Demity
2.0
1.0
00
10000
20000
3003
40000
50000
6000.
700.0
Tength
Figure 3: The distribution of output length MATH-500 shifts after preference optimization.

---

Published as a conference paper at ICLR 2026
Table 3: (Averaged) accuracy (Acc) and averaged number of tokens in generation (Len) of our method on different benchmarks. Bold denotes the best trade-off results. A denotes change on Acc. A% denotes the change ratio of Len. Total denotes the total change on Acc. Arg denotes the average change ratio of Len.
MATH-500 Aco(A) Leat %)
GSM8K
Minerva-Math Aco(
ATME24
AMC23
OlympiadRench Acct Len/
Method
Aco(A) Len(&%)
Acc(
Aoc(
Total Avg
Lent (%)
Leat
Len(
DeepSockRR- DisSssckRR--istill---ooo------
Original
83.00
86.28
26.84
29.17
70.62
44,66
5665
2457
7211
16355
10162
11715
81.80 (-1.20)
80.89 (-5.39)
25.74 (-1.10)
26.67 (-2.50)
69.84 (-0.78)
43.03 (-1.63)
-12.6
CoD
4788 (-15.48%)
1483 (-39.44%)
6231 (-13.59%)
15468 (-5.44%)
9052 (-10.92%%
11014 (-5.98%)
-15.14%
(-1.20)
87.95 (+1.67)
(-1.47)
21.25 (-7.92)
(-3.43)
42.88 (-1.78)
-14.13
L1-Exact
3335 (-41.1339)
2986 (+21.53%)
3673 (-49.0618)
3665 (-77.59%%
3406 (-66.48)
3657 (-68.78%)
46.92%
X2.20
88.32
27.57 (+0.73)
22.29 (-6.88)
70.62 (0.00)
43.03 (-L63)
-6.54
L1-Max
3374 (-40.444%)
3141 (4%)
3418 (-52.601)
3523 (-78.46%%
3250 (-68.02%)
3452 (-70.53%%
-4704%
82,80 (-0,20)
83.24 (-3.64)
25.74 (-1.10)
28.54 (-0.63)
69.84 (-0.78)
43.62 (-1.04)
-7.39
TrEfft
2813 2-5033550.34455
612 (-75.09%)
2950 (-59.1018)
10719 (-34.4618)
5166 (-49.1698)
7652 (-34.68%)
-50.47%
83,20 (+0.20)
85.82 (-0.46)
27.21 (+0.37)
29.58 (+0.41)
70.47 (-0.15)
44.81 (+0.15)
+0.52
Ours
(-57.69%)
(-61.50%)
2596 (-64.00%%
(-46.13%)
4026 (-60.38%
(-57.99%)
-57.31%
SosSsck-RI-Disiil-OHee
Original
92.20
91.81
36.76
51.46
87.97
56,82
4223
1677
5926
13411
6966
8789
90.06 (-2.14)
(-3.49)
(-0.36)
48.54 (-2.92)
87.34 (-0.66)))
54 (-2.22)
-11.76
CoD
2778 (-34.22%)
416 (175.19%)
2733 (-53.88%)
12029 (-10.30%%
4880 (-29.95%)
7200 (-18.08%)
36.94%
L1-Exact
(-2.40)
92.34 (+0.53)
(-0.73)
(-30.42)
69.53 (-18
53.86 (-2.96)
-5.42
3555 (-15.82(%)
3320 (+97.97%)
3341 (-33.62%)
3442 ( (.74.35%)
3337 (-52. 10%)
1757.777777%
24.28%
88.80 (-3.40)
92.42 (+0.61)
36.40 (-0.36)
25.00 (-26.46)
75,94 (-12.03)
(-2.522)
-44.16
L1-Max
2016 (-52.266%)
1640 (:2.21%)
1908 (7. 80%)
3560 (-73.45%%
3197 (-54.11%)
2510 ( (-65.14%))
-2.50%
90,20 (-2.00)
89.39 (-2.42)
37.13 (+0.37)
48.13 (-3.33)
87.03 (-0.94)
$3.56 (-3.26)
-11.58
TrEIf
(-42.8666)
(-78.71%)
2917 (-50788))
(:23.91
4420 (-36.55%)
(-20.70%%
42.25%
91.20 (-1.00)
91.21 (-0.60)
36.03 (-0.73)
50.83 (-0.63)
87.66 (-0,31)
55.34 (-1-48)
-4.75
DAST
3563 (-15.63/5)
1092 (-34.88%)
8119 (+37.01%)
15430 (+15.05%))
6422 (-7.81%)
10339 (+17.64%)
+11.88%
91.40 (-0.80)
92.95
38.97 (+2.21)
48.75 (-2.71)
(-1.09)
-1.99
Ours
2033 (-51.86%)
796 (-52.53%)
2079 (-64.92 %%
6892 (-48.61 %%)
3108 (-55.38%
4222 (-51.96%)
-54.21% %
Distribution is shifted by preference optimization. Figure 3 shows the distribution of output length on MATH-500 hefore training and after training. It is clear that the peak of the distribution, which indicates the mean, is shifted left expected. Meanwhile, after training, the variance is also decreased after training, resulting in a model with greater consistency on length preference and better alignment with short effective trajectories in the generation space.
4.2 ANALYSIS
We conduct additional experiments on the MATH-500 and GSM8K benchmarks to analyze key components of our pipeline. Based on the framework in 3.1, we investigate how the key ingredients (e.g., data with implicit reward and preference optimization algorithms) affect the effectiveness of our method on length reduction.
Data with implicit reward. Table focuses on the data (with implicit reward), presenting the results of training on the split of different difficulty labels and the full dataset ("w/o filter"). As the length the chosen responses in the training set increases, the average output length of models trained with our method also scales up, which aligns with our assumption that responses to harder problems contain more complex reasoning modes, and confirms the effectiveness of our data filter- ing strategy. Although all the chosen responses of the difficult split are not correct, the reasoning performance of the model is still slightly improved. This indicates that the LCPO objective focuses more capturing preference and is robust noise in the correctness label of training data.
8

---

Published as a conference paper at ICLR 2026
Table 4: Results of training on splits of different difficulty labels. Bold indicates the best trade-off A 1 % denotes length reduction ratio (higher is better).
MATH-500
GSM8K
Avg. chosen Len
Method
Acc
Len (A %)
Acc
Len (A 1 %)
Original Model
92.20
4223
91.81
1677
Ours w/ easy
91,40
2033 (51.86%) %)
92.95
796 (52.53%)
2232
w/ medium
91.80
2468 (41.56%)
92.65
1068 (36.31%)
3637
difficult
92.00
3130 (25 88%)
92.49
1364 (18. (.66))
3681
w/o filter 91.80
2954 (30.05%)
92.34
1180 (29.64%)
3270
Table 5: Results of our method on the out-of-distribution benchmarks. Bold denotes the best trade- off results. denotes change on Acc. A% denotes the change ratio of Len. Total denotes the total change on Acc. Avg denotes the average change ratio of Len.
MMLU
GPQA-D
Winogrande
Method
Acc(A)
Len(A%)
Acc(A)
Len(A%)
A))
Len(A%)
Total Avg.
DeepSeek
Qwen
Original
46.43
2549
34.85
10312
50.04
1516
Ours
46.78 (+0.35)
760 (-70.18%)
39.39 (+4.54)
4340 (-57.91%)
50.83 (+0.79)
309 (-79.62%)
+5.68 -69.24%
DeepSeok- RR-Distill-Queee
Dis--iiill---e--77 Qwen-BB
Original
65.27 64.16 (-1.11)
1858
48.99
9237
61.17
1042
Ours
835 (-55.06%)
52.53 (+3.54)
4301 (-53.44%) 62.12
375 (-64.01%
+3.38 & -57.50%
Algorithm of Preference Optimization. Table 2 also shows the performance of our objective comparing with various preference optimization methods. With limited training, LCPO achieves the best performance in reducing response length on MATH-500 while maintaining reasoning perfor- mance. Compared to SimPO, which is used in several previous works (Chen et al. [Shen al. 2025). ) LCPO performs better at the early training stage. Moreover, LCPO's objective function requires no hyperparameter tuning, making easier use.
4.3 DISCUSSION
Generalizability to OOD scenario. To the effectiveness of our method to generalize in out-of-distribution scenarios, we conduct evaluations on
Oelees Lang
7000
100
6000
MMLU (Hendrycks et al., 2021b a).
GPQA-Diamond (Rein et al.) 2024) and
Length
5000
096
WinoGrande (Sakaguchi et al., (020)) which are distinct from our training data in question form and subjects.
4000
Accuracy
0.2
3000
These benchmarks likely fall outside reasoning models' training distribu- tion (Aggarwal & Welleck 2025). As shown in Table models trained math datasets demonstrate remarkable generalization ability. On MMLU, the output length of the 1.5B and 7B models is reduced by over 70% and 55%, respectively, while the reasoning performance is maintained, On GPQA-Diamond, while the reduction for the models exceeds 53%, the accuracy is improved by 3.54% and 4.54%, respectively. On average, we achieved slight improvement performance while significantly reducing the generation length.
2000
088
1000
Difficulty Level
Figure 4: Changes of accuracy and generation length of DeepSeek- RI RI--Ditilll-wee--1 across different diffi- culty levels on MATH-500.
LCPO adaptively reduces generation length across different difficulty levels. As shown in Figure 4 our method reduces over half of the generation length across different difficulty levels.

---

Published as a conference paper at ICLR 2026
At the same time, accuracy of different levels is preserved. Intuitively, easier problem need less tokens to process, which should result in shorter generation on level 1 problems. However, Figure shows longer average length on level 1 problems than level 2 and level 3. This indicates potential overthinking phenomenon of LRMs, where easier queries consume more tokens and can lead to
performance drop (Liu al.] 2025b Feng al. 2025 Chen al., [Sui al. 2025). While
method reduces the generation length on level 1, accuracy on level improves. After training, the
average generation length is positively correlated with difficulty level, demonstrating consistency of reasoning effort with reasoning complexity.
5 CONCLUSION
In this paper, investigated how to reduce the output length of large reasoning models via small- scale preference optimization. We proposed a pipeline for achieving trade-offs between reasoning performance and output length using limited data. We distilled data from models' generation space and filtered data with question-level difficulty estimation. We analyzed different preference op- timization methods under unified Bradly-Terry loss framework and proposed Length Controlled Preference Optimization (LCPO). Extensive experiments demonstrated that our method reduced out- put length substantially while maintaining reasoning performance. Our work highlights the potential of efficient methods for tuning model behaviors.
ACKNOWLEDGMENTS
This research was partially supported by the National Science and Technology Major Project (No.2022Z00117003) the National Natural Science Foundation China (Grants Anhui Provincial Natural Science Foundation (No. No.2238885QF229, the Fundamental Research Funds for the Central Universities (No. (oo.K221500 10038), the Young Elite Scientists Sponsorship Program by CAST (No. 2024QNRC001), and the Meituan Joint Project.
REPRODUCIBILIT STATEMENT
To facilitate reproduction and verification of results, provide the following details: (1) Im- plementation Details: Comprehensive implementation details, including hyperparameters, prompts, and algorithmic specifics, can be found Appendix Theoretical Analysis: The theoretical foundations and mathematical derivations supporting Section 3.3.2 are presented in Appendix C
and Appendix
(3) Source Code: https://github..cm https//github..oo//leepy WithootCofffe/SSmall Scale.
REFERENCES
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503. 04697, 2025.
Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025. URL https://arxiv. org/ abs / 2502 04463
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng. Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.0956) 2025.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of long reasoning models. In Forty-second International Conference Machine Learning.
Cheng Cheng, Zhenya Huang, Guanhao Zhao, Yuxiang Guo, Xin Lin, Jinze Wu, Xin Li, and Shijin Wang. From objectives to questions: A planning based framework for educational mathematical question generation. In Proceedings of the 63rd Annual Meeting of the Association for Computa- fional Linguistics (Volume 1: Long Papers), pp. 12836-1 12856, 2025.
10

---

Published as a conference paper at ICLR 2026
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. In Forty-second International Conference on Machine Learning.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456. 2025.
Wei Du, Shubham Toshniwal, Branislav Kisacanin, Sadegh Mahdavi, Ivan Moshkov, George Arm- strong, Stephen Ge, Edgar Minasyan, Feng Chen, and Igor Gitman. Nemotron-math: Efficient long-context distillation of mathematical reasoning from multi-mode supervision. arXiv preprint arXiv:2512.15489 2025.
Abhimanyu Dubey, Abhinav Jaubri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Sicheng Feng, Gongfan Fang. Xinyin Ma, and Xinchao Wang. Efficient reasoning models: A survey. 2025.
arXiv preprint
Center for Al Safety Phan Long agibenchmark safe. ai I Gatti Alice Li Nathaniel Khoja Adam Kim Ryan Ren Richard Hausenloy Jason Zhang Oliver Mazeika Mantas Hendrycks Dan dan @ safe. ai 1. A benchmark of expert-level academic questions to assess ai capabilities. Nature, 649(8099): 1111-119444 2026.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yifan Wu, YK Li, et al. Deepseek-coder: when the large language model meets programming- the rise of code intelligence. arXiv preprint arXiv:2401. /4196, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma. Xiao Bi, et al. Deepseek-rl incentivizes reasoning in 1lms through reinforce- ment learning. Nature, 645(808) 1):333663333 638, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 3828 3850, 2024.
Dan Hendryeks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021c.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without refer- ence model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11170-11 11189, 89, 2024.
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of- though of 1lms via reinforcement learning. arXiv preprint arXiv:2504.01240 02296, 2025.
11

---

Published as a conference paper at ICLR 2026
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167100 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirleenth International Conference on Learning Reprexentations.
Fred Jelinek, Robert Mercer, Lalit R Bahl, and James K Baker. Perplexity a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): S63-S63, 1977.
Hao Jiang, Qi Liu, Rui Li, Shengyu Ye, and Shijin Wang. Cursorcore: Assist programming through aligning anything. Forty-second International Conference on Machine Learning.
Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucina- tions via attention lens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25004-25014, 2025.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint 111.:91412.6980, 2014.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statistics, 22(1):79-86, 1951.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo el. Solving quantitative reasoning problems with language models. Advances neural information processing systems, 35:3843-3857 2022.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. A
objective function for neural conversation models. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pp. 110-119, 2016.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less more for rl scaling. arXiv preprint arXiv:2502. 1886, 2025.
Baohao Liao. Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. howard-guided speculative decoding for efficient IIm reasoning. Forty-second International Conference Machine Learning.
Hunter Lightman. Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Jiayu Liu, Zhenya Huang, Wei Dai, Cheng Cheng, Jinze Wu, Jing Sha. Song Li. Qi Liu, Shijin Wang, and Enhong Chen. Cogmath: Assessing llms' authentic mathematical ability from a human cognitive perspective. In Forty-second International Conference on Machine Learning.
Jiayu Liu, Zhenya Huang, Chaokun Wang, Xunpeng Huang, ChengXiang Zhai, and Enhong Chen. What makes in-context learning effective for mathematical reasoning. In Forty-secome Interna- tional Conference on Machine Learning, 2025a.
Siyang Liu, Sahand Sabour, Yinhe Zheng, Pei Ke, Xiaoyan Zhu, and Minlie Huang. Rethinking and refining the distinct metric. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vllue 2: Short Papers), pp. 762-770, 2022.
12

---

Published as a conference paper at ICLR 2026
Yue Liu, Jiaying Wu, Yufei He, Hongcheng Gao, Hongyu Chen, Baolong Bi, Jiaheng Zhang, Zhiqi Huang, and Bryan Hooi. Efficient inference for large reasoning models: A survey, arXiv preprint arXiv:2503.23077, 2025b.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective, 2025c. URL https:
/
20783
Haotian Luo, Haiying He, Yibo Wang, Jinluan Yang, Rui Liu, Naiqiang Tan, Xiaochun Cao, Dacheng Tao, and Li Shen. Ada-rl: Hybrid-cot via bi-level adaptive reasoning optimization. arXiv preprint 25Xiv: 1655, 2025a.
Michael Luo, Sijun Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol -preview with a 1.5b model by scaling rl. 025b. Notion Blog.
Zhiyuan Ma, Jiayu Liu, Xianzhen Luo, Zhenya Huang, Qingfu Zhu, and Wanxiang Che. Advancing tool-augmented large language models via meta-verification and reflection learning. In Proceed- ings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 2078-2089, 2025.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198-12425, 2024.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zeutlemoyer, Percy Liang, Emmanuel Candes, and Tatsunori Hashimoto. sl: Simple test-time scaling. In Workshop on Reasoning and Planning for Large Language Models, 2025.
Mathematical Association of America, American mathematics competitions. https://maa
htwp:studdettttpporrrmamaaa/mai 2025, Accessed: March 2, 2026.
AoPS Online. Art of problem solving. htttas//artofproat wtttt/ofprrbllmollilii..... ng. com/wiki/
index.php/Bi Persc_rooolessse and Solutions] 2025. Accessed: March 2, 2026.
OpenAl. Introducing openai o3 and 04-mini. https:/ //tp://pppaaiaaea i com commindex/ ndex 2025. Accessed: March 2, 2026.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang. Junxian He, al. survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. arXiv preprint arXiv:2503.21614, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 01153728-5377444 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level progle-prooff q&a benchmark. In First Conference on Language Modeling, 2024. URLLtttps://pperriiii net./ feot/forum? d= 67584b98
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8732-8740, 2020.
13

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2xXi::2402.03300 2024.
Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, and Shiguo Lian. Dast: addiptie slow-thinking for large reasoning models. arXiv preprint ar arXiv:2503.04477 2025.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503. 16419, 2025.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint 250. 2025a.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501. 2025b.
Qwen Team. -wh: Embracing the power of reinforcement learning, March 2025. URL withug..thhhh....m http://mww.............oo///////
Songjun Tu, Jiahao Lin, Qichao Zhang. Xiangyu Tian, Linjing Li, Xiangyuan Lan, and Dongbin
Zhao. Learning when to think: Shaping adaptive reasoning r1-style models via multi-stage rl. arXiv preprint arXiv:2505. 2025.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: General- izing direct preference optimization with diverse divergence constraints. In The Twelfth Interna- fional Conference on Learning Representations.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2 191, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824-248311 Curran Associates, Inc., 2022a. URL ht https::/ppocceed...eees/ www..ps.e.ppp...... cc httpp/aarrrrrieeepaapr///// Papee-Paaaa-----eeeeeeeeee pdd
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 2483777-248877 2022b.
Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. Simper: minimalist approach to preference alignment without hyperparameters. In The Thirteenth International Conference Learning Representations, 2025.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang. Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint 2XXv:2501.00666, 2025a.
Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less. arXiv preprint ar0iv:2502. 18600, 2025b.
14

---

Published as a conference paper at ICLR 2026
Yuhui Xu, Hanze Dong, Lei Wang, Doyen Sahoo, Jumnan Li, and Caiming Xiong. Scalable chain of thoughts via elastic reasoning. arXiv preprint arXiv:2505.05313 2025c.
Shangzi Xue, Zhenya Huang, Jiayu Liu, Xin Lin, Yuting Ning, Binbin Jin, Xin Li, and Qi Liu. Decompose, analyze and rethink: Solving intricate problems with human-like reasoning cycle. Advances in Neural Information Processing Systems, 37:357-385. 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu. Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024a.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024b.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source IIm reinforcement learning system scale. arXiv preprint 25Xiv:5503.1447000 2025.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.01 2412.0192..11111 08800198111111 2024.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song. and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity Ilms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Yunhua Zhou, and Xipeng Qiu. Revisiting the test-time scaling of ol-kike models: Do they truly possess test-time scaling capabilities? arXiv preprint arXiv:2502. /2215, 2025.
Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. Adaptthink: Reasoning models can learn when to think. arXiv preprint -xXiv:2505.13417 2025a.
Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? a survey test-time scaling in large language models. arXiv preprint arXiv:2503.24235, 2025b.
Yuze Zhao, Tianyun Ji, Wenjun Feng, Zhenya Huang, Qi Liu, Zhiding Liu, Yixiao Ma, Kai Zhang, and Enhong Chen. Unveiling the magic of code reasoning through hypothesis decomposition and amendment, In The Thirteenth International Conference on Learning Representations.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tume fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational I Linguis- tics. URL http:://rxxxa http://awwwvv....aaaa/./ Sxrg/abs/2A00 220../1303312222222
A BRIEF INTRODUCTION OF DATASETS AND BASELINES
Math Datasets We use several math datasets covering in domain and out of domain data for evaluation. The detailed information is follows:
MATH-500 (Lightman al. 2023) is high-quality math problem solving dataset containing 500 problems extracted from the MATH (Hendrycks et al. 2021c) test set. It is widely used for evaluation of LLM reasoning (Guo et al. 2025 Yang al. 2024b Team 2025).
15

---

Published as a conference paper at ICLR 2026
GSM8K (Cobbe et al. 2021) is famous dataset containing 1,319 primary school level math problems. It is alse widely used for LLM evaluation (Ouyang et al. 2022 OpenAl 2023 Liu
ct al. 2025a).
Minerva-Math (Lewkowyez et al. 2022) is specialized collection designed for training and eval- uating AI models on challenging mathematical reasoning tasks. It includes 272 problems ranging from algebra and calculus to advanced proofs. It is adopted by (Luo et al. 2025b) for evaluation
on LRMs trained via online RI..
AIME24 (Online 2025) is collection of mathematical problems from the American Invitational
Mathematics Examination (AIME) (Online) 2025) held in 2024. It contains 30 extremely chal- lenging problems from the real world and is used for evaluating powerful LRMs.
AIME25 (Online 2025) is a another collection of 30 mathematical problems AIME (Online 2025
held in 2025.
AMC23 of America 2025 is collection of challenging mathematical problems designed for the American Mathematics Competitions (AMC). is also a widely used dataset.
OlympiadBench He al. (2024) is an Olympiad-level bilingual multimodal scientific benchmark, featuring 8,476 problems from Olympic-level mathematics and physics competitions, including
the Chinese college entrance exam. We use the math split in English for evaluation following
previous works (Luo et al. 2025b Aggarwal & Welleck 2025).
HLE (for AI Safety Phan Long agibenchmark @ safe. ai I Gatti Alice Li Nathanicl Khoja Adam Kim Ryan Ren Richard Hausenloy Jason Zhang Oliver I Mazcika Mantas
Hendrycks Dan dan safe. ai 1 2026) is an extremely challenging multi-modal benchmark at the frontier of human deeeledge, globally by subject-matter experts. Following [Du ct al. (2025), we use test data that does not require multimodal capabilities in the "Math" split,
which known HLE-Math.
General Datasets We adopt the widely used dataset MMLU (Hendrycks et al. 2021b), GPQA- Diamond (Rein al. 2024) and WinoGrande (Sakaguchi al. 2020 for evaluation.
MMLU a massive multitask test consisting of 14k multiple-choice questions from various branches of knowledge, spanning subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. This covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability.
GPQA is a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions in GPQA
high-quality and extremely difficult, reaching human PhD level. GPQA-Diamond is a subset of GPQA containing 198 problems. is the cleanest and most reliable part of the GPQA dataset and is widely used test set for reasoning.
WinoGrande is a dataset of fill-in-a-bland commonsense reasoning problems, widely used to eval- uate models' commonsense reasoning and general reasoning capabilities (Dubey al. 2024
Team et al. 2025a b).
Coding Datasets
We use the continuously updated benchmark, LiveCodeBench (Jain et al.)
LiveCodeBench provides holistic and contamination--fee evaluation of coding capabilities of LLMss. Particularly, LivcCodeBench continuously collects new problems over time from contests across competition platforms. We use the latest split v6 for evaluation.
Training Datasets
use LIMR (Li et al. 2025) dataset for training. LIMR is a subset of MATH (Hendrycks et al., 2021c) training set containing 1,389 math problems extracted by assessing consistency of accuracy reward scores during online RL. In our practice, we perform rollout on the whole dataset and use only 400 instances from it for training 50 steps.
Baselines We evaluate baseline methods listed in Table I These baselines span across various types, covering inference-time, RL, RL-with- RL- budget-forcing and preference optimization.
All the baselines are evaluated using the open-source models released by their respective authors. Detailed information about the baselines is provided below:
16

---

Published as a conference paper at ICLR 2026
Table 6: Hyperparameters for various preference optimization methods. It is worth noting that the preference weight parameter is not required for LCPO.
Method
learning rate
preference weight
others
SFT
le-5
DPO
5e-6
0.1
SimPO
5e-7
2
7 0.5
SIMPER
5e-7
ORPO
5e-6
0.2
LCPO (Ours)
5c-6
0.3
CoD (Xu al. 2025b), Chain-of- Draft prompting, training-free frree inference- method per- formed by adding an special instruction asking the model to keep draft of at most 5 words for each thinking step.
LI (Aggarwal & Welleck 2025)
online RL based method incorporating with budget- The model is trained on massive amount data from the DeepScaleR dataset (Luo et (0256). LI provides versions of model. LI-Exact is trained under an accurate length constrained reward penalty. LI-Max is trained under a ceiling limit of length constraint. We use the open- source weights from the official HuggingFace repository
TrEff (Arora & Zanette 2025) is another powerful online RL based method. During training, TrEff adopts an additional normalized length reward similar to the advantage estimation of DeepSeek's We use the open-source model released in the official repository"
GRPO (Shao et al.
2024
2025 is preference optimization based method. It uses pass rate to estimate difficulty of the questions and then calculate token length budget. The length penalty based on the budget is applied when ranking preference data to achieve adaptive slow thinking. The orig- inal paper uses DeepSeek-FP RR-DDistll--Weet 2025) as the base models. Their official repository only contains the 7B model, which we adopt for evaluation.
DAST (Shen et al.
and DeepSeek- RI-Distill Qwen-32B (Guo al.]
B IMPLEMENTAION DETAILS
Common Settings Experiments conducted 4x A800-80G PCIe. We train the model for 3 epochs and save checkpoints for every 50 steps, using the checkpoint at the 50th step for all methods, which corresponds only 0.4k sampled pairs. Methods Table [ also use the same common settings. For all experiments, we set batch size per GPU to 2, resulung in an equivalent batch size of 8. Context length limited to 2.3k. The learning rate scheduler type cosine, and the warmup ratio is 0.1. The optimizer is Adam W (Kingma 2014) from torch (Paszke et al. 2017). The dtype is bfloat16.
For rollout and evaluation, we set temperature to 0.6. The max tokens limit is 32,768 in all experi- ments following Guo et al. (2025). For datasets with limited size (e.g., AIME24, AMC23), we report averaged accuracy over 16 samples (e.g. pass@) I), Additionally, report the average number of tokens as the average generation length which denoted as Len.
Hyperparameters The specific settings for the methods in Section 3.3 are listed in Table 6
Infrastructures Our training framework is built upon LLMMA-Factory (Zhnng et al. 2024) while vLLM (Kwon et al.] 2023) employed for rollout, The evaluation pipeline is based Deep- ScaleR (Lo et al. (0225)). Furthermore, we integrate official evaluation scripts from the repositories of various datasets.
Prompts We use the recommended prompt setting from DeepSeek official repo (Guo et al. 2025
shown in Figure 5
wttp://mnaaaacc.ccaaaaaaaaaa co/daman 209arora/mmddels 1
17

---

Published as a conference paper at ICLR 2026
Evaluation Prompt for Math
(Math Problem}
Please reason step by step, and put your final answer within [boxed[].
Evaluation Prompt for General
[General Problem}
Please reason step by step and put your final answer (eg, B, C, D) within [boxedf), such as "| \bbxxe(A)""
Figure 5: Evaluation prompt.
Table 7: Statistics of all the datasets generated from LIMR. Token Proportion (%) indicates the share of tokens contributed by each category to the overall total, which also reflects the allocation of GPU inference budget.
Model
Split
Number of Questions
Token Proportion (%)
DeepSeek-RR -i--ssiilll Qwen-
casy
988
56.3
DeepSeek-RI-Dingg ----iisill--Qwen-71 Qwen--B
medium
330
38.3
DeepSeek-R1 -I--Dstill-Qwent
difficult all
52
5.4
SraSSec--R--Disill--O- DeepSeck- Dist--istl---ee Qwen- 1.5B
1389
100
casy
488
19.4
Generated Datasets In this paper, we employ a heuristic approach involving self-distillation (roll- out), filtering, and ranking of the data. Through this process, we obtain three splits: easy, medium, and difficult. presen/ the statistics of the dataset generated via this pipeline. For the medium split, we select the shortest correct response the chosen one and the longest wrong response as the rejected one. Furthermore, we filter out questions where the longest wrong response is shorter than the shortest correct answer. For the difficult split, we treat the shortest response as the chosen one and the longest the rejected one.
C INTRODUCION FOR PREFERENCE OPTIMIZATION
In this section, outline the foundational concepts and notations used throughout study of preference optimization, including the formulation of pairwise preference data, reward modeling, and the learning objectives employed in various algorithms such DPO, SimPO, and others.
Cradley-Terry Model. Let 8 E IR represent the ability value of team i. Let the outcome of game between teams i and j be determined by the difference in their aibility values - B, The Bradley- Terry model (Bradley & Terry 1952) defines the log-odds corresponding to the probability Pay that team beats team j as
Pij a B, 3z,
log
(12)
Pij
where is intercept term. Solving for Più yields
Pú [+e- aaa + 3; -8)),
(13)
where o(-) is the sigmoid function. In the LLMs setting, given an input x within a finite space of contexts X, define policy Tg()) For pair of actions V,), obtain partial preference relation y; ž By between them through human annotations or similar methods, indicating y: is preferred yo according to the ground truth. Tuning the policy for preference dataset
18

---

Published as a conference paper at ICLR 2026
D D= 1
yo 1= aims maximize the expected log-probability of yo yi (0) We take
(3) PI
(
the rewards r(x(1), You as their ability values. Then applying the BT model gives
(1)) and r(z(1) Vi (a)) of (7) and !/i (8) given by rule function or reward model
L(r D) = (e (r(z, yw) - r(z, yt]-
(14)
RLHF and DPO. Reinforcement Learning from Human Feedback (RLHF) (Ouyagg et al. 2022) uses unpaired data generated by the policy To(y) It adds KL divergence penalty to the ward to restrict the policy distribution from shifting excessively. The objective is to maximize the accumulated rewards:
L(Ø) Ea- 3 k ([a()))
(15)
Based on the RLHF formation, Direct Preference Optimization (DPO) (Rafailov et al. 2023) derives an implicit reward:
(D y) = Blog To(y
+ 3 log ((r),
(16)
Ter (ylax)
where 7(x) = Ey (yl (r(r,y) y)) is the partition function that hard to estimate. DPO
leverage BT loss to remove (xx) from the objective, resulting in the folllowing objective:
LDPO -Bg-D llog o log (y B log
To (z)
)
(17)
Teef (Nh (x)
Tref (3n (x)
SimPO, SimPER and ORPO. Simple Preference Optimization (SimPO) (Meng al. 2024) in-
troduces a reward margin and uses the average log probability of a sequence the implicit reward to eliminate the need for the reference model DPO objective:
B
B
LsiPOO = -B [log o log Te (Y (x)
log no (u]l)) - ->]-
(18)
3/
1yn
Simple alignment with Perplexity optimization (SimPER) (Xiao et al. linek et al. 1977) and changes the form of BT loss into simpler one:
2025 utilizes perplexity (Je-
1
LSMPPER (Eg-D.yrr (exp(
log na (ywlz)) + exp(
log [x)((=))
(19)
Odds Ratio Preference Optimization (ORPO) (Hong et al.
2024) incorporates an odds ratio penalty
(formulated as BT loss) into the conventional negative log- likeeiill (NLL) loss:
(e (x)
pe (v (z)
LORPO EELDD D. llog pe (ywr X log olog
log
}}, (20)
po (ymml (x)
Po py(
where po (b(xx) = exp( log ()())=)))
We will leverage the definition of pe (y((z) for further derivation.
D ANALYSIS OF CONVERGENCE CHARACTERISTICS OF PREFERENCE OPTIMIZATION METHODS
We aim find ideal method preference optimization that can achieve fast convergence good point in the context of length control. We begin by reformulating the objectives into the form of BT loss. And then we analyze the convergence conditions of different optimization objectives based on practical assumptions.
Background Assumptions From Practice For sequences with length |y/ > 1000, the typical per- token prg-probabilities lie in the range log ((u) E [-5, 0). Let m be the point where the sigmoid function saturates. Let (m) > 0.99, then we have m > 5. We make an assumption about the training dynamics: after updating, a convergence of the model on preference satisfies pa pa( x) > Po z) by some margin, where yo and Yi denote the preferred and less less-preferred responses, respectively. Note that this could lead over-fitting if it holds for every sample. However, this assumption is reasonable when considering the average case over the input distribution X' (c.g. is satisfied for set of imputs with high probability mass).
19

---

Published as a conference paper at ICLR 2026
Table 8: Objective functions of preference optimization methods (expectation omitted).
Name
Objective
SFT (Oyyagg et al.
2022)
log xo (D (x)
DPO Rafailov et al.
2023
log
mo (0(w)l2) ( log
(x(2)12) Blog (xc((())
SimPO
Meng al.
2024)
log o
log To (Uw (r)
8
log No mo(()l -7)
Tvil
log po (U (a:)
Alog
(log
pe po(ullz)
loog Pa( (x)
1 1-po(y
I-polu((z)
ORPO (Hong et al.
2024)
where pa (y)(z)
log To (1(()))) exp [
SimPER Xiao et al.
2025)
exp
|+>|
log To (vrolz))
+ exp
log To (()((2)))
SFT
With some algebra, we can derive the BT form of SFT loss as:
(o(wl)))) po yue (z)
LSFT log o (log
(21)
where the penalty for the rejected responses is absent, or po(()(>) is default to 0.5. Convergence achieved when:
Ps (Yer (r)
C = (o
(22)
1 Po (B (r)
+ em
For m = 5(a se 0.993), requires po po (x)) > 0.993. For sequences longer than 1000, cumulative
probability requires:
II p(v pz, yon " 0.993.
(23)
This is fundamentally incompatible with typical token probabilities (p R 0.05 0.5). Thus SFT cannot effectively enforce preference aligmment, which aligns with intuition. In another words, the convergence of SFT depends completely on the input æ. In the context of length control, fitting the model to limited generation modes leads to potential catastrophic forgetting.
ORPO For ORPO, it is complex due to the additional NLL loss.
+ (1 + (o 1//
LORFO - log o [log(
e- (1 + Ipo (ym (r)1/2 ).
(24)
where = = =logg
(=) log I-Po(uilz)
Ps PM [x)
Pe( ax)
Then we have:
po Po(un (x) po (yww (ar)
2po (po(wll))1 1/)/////////////
+ m
(25)
(o( + pa (Brw (x))
Continue the derivation:
log(exp(=2) - (1 log(1 [1 + (1 exp(-=)))o -
(26)
1+ (1 (1+eep((=) (00(=2))ppo (x) > exp(m - -2) - eexp(m) + expl
(ppp(-2))pp(uee (27)
exp(m - -2) 22-11
2pe 2poll)l))) expm - 21) + exp(m))
exp(m) (1
(28)
1 exp(m 2)
2po (x)
(1 20(2 m))
(29)
+ exp(m)
Note that is inside the sigmoid function of the BT loss from the original objective. Then controls whether the right side is positive. If - m > 0 (e.g, (p (x) > po (31[2) - some margin), the right side is negative, which satisfies the inequality. Then the convergence behavior is determined
20

---

Published as a conference paper at ICLR 2026
by the SFT loss component. In this case, apply the analysis of SFT. If 0 (e.g, Po (yw < po()) + some. some_marae margin), then we have
sf
(e((ll))) po(u) [z) margin,
(30)
exp(m)
where sf is positive scaling factor smaller than 1. Based the assumption that Po po(in (x) by some margin is salisfied for a sel of inpuls with high probability mass. We consider the latter case under-fit. In conclusion, ORPO fits better and faster to the preference data than SFT. However, due to the NLL component in its objective, it fits worse than pure log odds ratio loss.
SimPER SimPER cannot be written as log sigmoid formation. Because the value of log sigmoid function always falls in (-o0, 0). while the inverse of the SimPER objective is always positive under the assumption that po (31 (z) < ())))))))) by some margin is satisfied for set of inputs with high probability mass. This implies that the convergence of SimPER unconstrained by preference. From the formulation of SimPER's 's objective function, can be observed that separately optimizes two objectives of the following form:
po(y]z) = exp ( log mo(ulz) > 0,
(31)
which cannot be written as a log sigmoid formation either. Thus, the convergence of SimPER depends both responses.
DPO and SimPO These two methods have similar objective. We can directly extract the differ- ence of the rewards from their objectives. For DPO we have:
log (10 log Na (n([(r) >> log
Trat (Nw [x)
m
(32)
Teetl (y) (r) 33
For SimPO we have:
1
log (o(()>)) m
log To
(33)
lvul
B
Intuitively, the normalized advantage objective in SimPO potentially provides superior convergence properties for long-form generation tasks, while DPO remains valuable for maintaining response diversity. The convergence characteristics depend on the hyperparameters settings, which decide the right sides of the equation. Note that the left sides can be seen as the reward differences of DPO and SIMPO. Thus, the right sides indicate reward margin floor.
Conclusion From the performance and analysis of SFT, ORPO and SimPER, we can conclude that Bradley-Terry loss provide predictable convergence characteristics from the objective, while NLL loss potentially amplifies the data's impact on convergence. Under the form of BT loss, the tunable component of the objective is the reward function. Thus, a negative reward needed to mitigate the effect of NLL. Besides, from the anlysis of DPO and SimPO, we can conclude that the reward margin is important in length control, which is reflected directly in the effect controlled by the hyperparameters.
E EXTENDED RELATED WORKS: HYBRID CHAIN OF THOUGHT
Chain-of-T Thann-o-TThough (Wei et al. 2022b) is intended to guide models to employ effective reasoning pat- terns in solving complex problems through prompt (Xue et al. 2024 Zhao et al.). To achieve perfor- mance breakthroughs, foundation models undergo post-training that enables them to autonomously apply complex reasoning patterns inspired by CoT during generation Yang et al. (2024a): Guo et al. 2025
Yang al. (2024b); Ma et al. (2025).
Thought or Adaptive Thinking aims to enable LLMs to only apply complex rea- soning patterns challenging prompts. LLMs with adaptive thinking ability dynamically choose between two reasoning modes based on problem difficulty: (1) highly complex thinking with think- ing tags and non-empty thinking content, (2) non-thinking without thinking tags with empty
Hybrid Chain-of- Chain-of-Thought
21

---

Published as a conference paper at ICLR 2026
18000
16000
0.7
14000
Accuracy
7B Accuracy
12000
Avg. Length
0.6
Accuracy
7B Length
10000
1.5B Length
8000
0.5
6000
0,4
4000
2000
0
2
8
10
12
14
Rank Index of Trajectoryy Groups
Figure 6: A replicated experiment of Section 3 on Eurus-2 dataset 2024).
CCui et al.
2025 Yuan et al,
Table 9: (Averaged) accuracy (Acc) and averaged number of tokens in generation (Len) of our method on OOD benchmarks. Bold denotes the best trade-off results. A denotes change on Acc. A% denotes the change ratio of Len. Total denotes the total change Acc. Avg denotes the average change ratio of Len.
MMLU
GPQA-D
WinoGrande
Method
Acc(A)
Leni A%)
AcctA)
Len(A%)
Acc(A)
Lea(
Total Avg.
Original
46.43
2549
34.85
10312
50.04
1516
Col
46.72 (+0.29)
1592 (-37.54%)
33.33 (-1.52)
7328 (-28.948)
46.88 (:3.16)
1572 (43.69%)
.4.99
20.93%
I.I-Fxact
48.64 (+2.22++211)
5373 (+52.56%)
31.31 (33.54)
3253 (468.458)
51.14
4819 (+217.88%)
-0.23 & 33%
LI-Max
49.29 (+2.86)
2219 (-12.95%)
31.82 (-3.03)
2839 (-72.47%)
50.59 (-0.55)
2409 (+58.91
+0.38
-8.84%
TrEff
44.80
(-52
(-32.95%)
-44.46%
Ours
46.78 (+1.35)
760 (-70.18%)
39.39 (+4.54)
4340 (-57.91%)
50.83 (+0.79)
309 (-79.62%)
+5.68 & -69.24%
AxspSeck-II-isttll-Neaaes
Original
65.27
1858
48,99
9237
61.17
1042
CoD
(-1.55)
822 (-33.76%)
(7.98(11.01)
5834 (-36.84%)
(-0.995
505 (-51.545%
-1.61
-48.05
LI-Exact
2834 (452.53%)
(-2.53)
(-64.98%)
62.90
(+64.67%)
-2.30
L1-Max
64.02 (-1.25)
983 (-47.09%)
46.97 (-2.02)
2039 (-77.93%)
62.19 (-1.02)
847 (-18.71%)
-2.25 -47.91%
THill
63.17 (-2.10)
1077 (-42.03%)
44.95 (-4.04)
6470 (-29.968)
60.22 (-0.95)
472 (-54.70%)
-7.09 -42.4398
DAST
65.62 (+0.35)
2468 (+32.83%)
47.98 (-1.0))
10819 (+17.13%)
59.35 (.1.82)
1630 (+56.43%)
2.48 +35.466%
Ours
64.16 (-1.11)
835 (-55.06%)
$2.53 (+3.54)
4301 (-53.44%)
62.12 (+0.95)
375 (-64.01 %)
+3.38
-57.50%
thinking content. Many concurrent works explore this paradigm for reducing the average genera- tion length of LRMs. For instance, Ada-R1 (Luo et al. 2025a) used model merging and bi-level preference optimization explicitly teach the model to distinguish between the types of data. AdapíThink (Zhang et al. 2025a) carefully designed a reward function so that when facing data of different difficulties, the model's behaviors adaptively converge state mixing both modes as training progresses. AutoTThink (Tu et al., (205)) adopted multi-stage RL to enable the model to master this output pattern. Although there are offline methods like Ada-R1, many other works such as AdapiThink and Auto still rely on costly large-scale online RL.
F COMPLETE OOD RESULTS
Our method maintains strong generalization out-of-distribution scenarios and achieves the best trade-off between reasoning capability and generation length, shown in Table
22

---

Published as a conference paper at ICLR 2026
vs
Generation Length A
1002
1550
00.5
ORPO Acc
ORPO 1 cn
LCPO Acc
Generation Length
3000
LCPO Len
2500
900
873
850
900
A
Figure Comparison between ORPO and LCPO on MATH-500.
Accuracy
Generation Length vs
ORPO Acc
8000
ORPOLen
LCPO Acc
Generation Length
LCPO Len
7000
65
2000
30
5000
4001
35
03
Figure 8: Comparison between ORPO and LCPO OlympiadBench.
G ADDITIONAL RESULTS
G.1 REPLICATED EXPERIMENTS OF SECTION
3.2
To verify whether the phenomena observed in Section 3.2 persists on more challenging datasets, we reran the experiment with identical settings on more challenging dataset, the AMC-AIME split from Eurus-2 (Cui et al. 2025 Yuan al. 2024)
The results in Figure 6 show that when confronted with harder problems, the model's performance and length still exhibit similar patterns and short effective paths still exist.
G.2 SENSITIVITY ANALYSIS OF LCPO
To investigate how the weight of the BT loss (denoted as following ORPO) affects LCPO, compared the performance variations between LCPO and ORPO under different BT loss weighting configurations. We tested weights ranging from 10.1, 0.3, 0.5, 0.7, 2, 2, 3, 10, 100], using the 7B
model on MATH-500 and OlympiadBench. Results are shown in Figure 7 and
Sensitivity Analysis LCPO is robust to X, making it a hyperparameter-free preference optimiza- tion method. can match the peak performance of ORPO without any hyperparameter tuning or calculation on additional loss term.
As shown the results, ORPO sensitive A. When the optimization of ORPO becomes domi- nated by SFT, which is not always beneficial for reasoning (Chu et al.), notable manifestation is that as the impact of the BT loss diminishes, the model overfits to the chosen responses. This could lead to collapse in diversity and potentially causes subsequent decline in accuracy (as shown in the figure, ORPO's 's accuracy gradually falls below that of LCPO as increases).
In contrast, LCPO mitigates the effect balancing the implicit NLL term. As the BT weight increases, the performance of ORPO continues to improve. When becomes extremely large (>e 10), it gradually approaches ICPO, which is predictable: the two loss terms in ORPO act together
23

---

Published as a conference paper at ICLR 2026
Table 10: Performance of our method on LiveCodeBench (v6), AIME25 and HLE-Math.
LiveCodeBench (v6)
AIME25
HLE-Math
Method
(A)
(A t
Acc (A)
(A 1 %)
Aoc (A)
Len (A
Original-1 1.5B
18.29 18.86 (+0.57)
16340
22.71
15985
3.07 3.48 (+0.41)
14940
Ours- 1,5B
7871 (-51.83%)
22.23 (-0.48)
7194 (-56.00%)
6217 (-58.39%)
Original-7B
12080
37.50
14622
13956
Ours- 7B
33.71 (+0.57)
8018 (-33.63 %+
36.87 (-0.63)
8187 (-44.01 %)
4.51 (+0.21)
5873 (-57.92%)
Table 11: Cost of 7B model estimated in A100
Method
Type
GPU hours
AdaptThink
RL
1992
AutoThink
RL
~576
TrEff
RL
~50
Ours
preference optimization
100.4
through summation. When the weight of one term increases, its influence and gradually surpasses that the other and becomes dominant. This aligns with our analysis in Appendix as preference learning progresses, the probability of the chosen response gradually becomes higher than that of the rejected response by a certain margin. At this point, the BT loss no longer affects convergence, and the effect of the loss function is almost entirely dominated by SFT. Moreover, this indicates that LCPO directly achieves the theoretically optimal position by removing SFT, which reinforces our key insight: the dominant driver of performance this paradigm is the BT loss part. This also aligns with the principle of Occam's 's Razor.
G.3 ADDITIONAL EVALUATION
Coding is another important part of reasoning (Jiang et al.). We conducted additional evaluation LiveCodeBench v6 split (the latest split) to assess the coding performance of our method. As shown in Table 10 our method also performs well challenging code generation tasks. Acc improves slightly by 0.57, while Len is reduced by 33.63% and 51.83% for the 7B and the 1.5B models, respectively.
We also conducted additional evaluation on the new challenging math benchmarks AIME25 and HLE-Math. Following Du al. (2025), we use an LLM (Guo al. 2025) with their evaluation prompt to verify the generated answer for the HLE-Math evaluation. As shown in Table IO, our method also performs well. Notably, on HLE-Math Ace improves slightly by 0.41 and 0.21, while Len is reduced by 58.39% and 57.92% for the 1.5B and the 7B models, respectively.
G.4 PIPELINE COST
The rollout process is the primary computational cost of our method. Here, we estimate the overall training cost (rollout and training) of the 7B model in A A100 in Table I1
The rollout is a one-time, offline process. Once the preference dataset is generated, it can be reused for multiple training runs and easily shared (e.g., via platforms like HuggingFace).
The overall cost of our method is low. Compare to offline methods (in Table D need less data from rollout. Compare to online RL methods, which necessitate a continuous and interleaved rollout and policy updating throughout the entire training process, the overall computational budget (rollout + training) of our method remains highly competitive.
Due to conversion discrepancies in GPU hours, the specific computational cost may fluctuate, but they provide reference terms of magnitude.
24

---

Published as a conference paper at ICLR 2026
Table 12: Results on generation diversity of our method.
Model
Distinct-1 f
Distinet-2
Distinct-3
EAD
Original-7B
0,0222
0.1132
0.2320
0,0256
Ours-BB
0.0292
0.1260
0.2375
0.0312
Original-I.5B
0,0199
0.1036
0.2179
0,0240
Ours-1.5B
0.0285
0.1316
0.2553
0.0313
Original-7B (on level 5)
0,0046
0.0252
0.0546
0,0058
Ours-7B (on level 5)
0.0062
0.0303
0.0613
0.0070
Original- 1..nal-1..B 1.5B (on level Ours-1.5B (on level
0.0040
0.0222 0.0322
0.0498
0,0055
0.0063
0.0668
0.0074
Table 13: Generation diversity comparisons.
Model
Distinct-1
Distinct-2
Distinet-3
EAD f
Original-7B
0.0222
0.1132
0.2320
0,0256
ORPO-7B
0.0275
0.1220
0.2326
0.0297
Ours-7B
0.0278
0,1253
0.2400
0.0300
Original-7B on level
0.0046
0.0252
0.0546
0.0058
ORPO-7B on level
0.0059
0.0294
0.0597
0,0067
Ours-7B on level 5
0.0060
0.0301
0.0614
0.0068
G.5 GENERATION DIVERSITY
We conducted experiments verify the impact of our method on generation diversity. To quantita- tively measure the diversity of generations, we adopted the following metrics:
Distinct-n (Li et al. 2016 Wang et al.): a metric used to calculate the diversity of a group of sentences,
AppectationAAjjusted----ssi (EAD) (Liu et al. 2022): improved Distinct metric that removes the biases of the original Distinct score (Distinct tends to assign lower scores to longer sequences),
and test our method on MATH-500, which was chosen for its diverse and annotated difficulty levels. We generated 16 responses for each problem using the report results on both the full dataset and the most difficult problems (Level 5, the highest difficulty level) to quantitatively measure the diversity of generations on average and on difficult problems. As shown in Table 12 the model trained with our method achieves higher score under all four metries. This indicates that our method has improved generation diversity while simultaneously reducing overall response length.
generation settings as Table We
We set À to 10 (where ORPO approaches LCPO length reduction) and conduct a similar experi- ment. As shown in Table 13 the model trained with our method still achieves higher scores under all four metrics.
G.6 HYBRID CoT
We further compared our method with AdaptThink and AutoThink, which have publicly available open-source weights. Results shown in TT (cost) and Tabte[T= and generation length). Both methods serve as very strong baselines. Yet, our approach consistently demonstrates comparable performance with much lower cost.
H LIMITATIONS AND FUTURE WORK
We discuss several limitations of our work in this section. 1) Following previous works (Aggarwal & Welleck 2025 Shen et al. 2025), our training set focuses math tasks. The diversity of concise
25

---

Published as a conference paper at ICLR 2026
Table 14: (Averaged) accuracy (Acc) and averaged number of tokens in generation (Len) of our method on different benchmarks. Bold denotes the best trade-off results. A denotes change on Acc. A% denotes the change ratio of Len. Total denotes the total change on Acc. Arg denotes the average change ratio of Len.
MATH-500
Minerva-Math
AIME24
AMC23
OlympiadRench Aect Len(
Method
Acc(.) Lent (%)
Aco( Len(A%%)
Acc(.))
Acc(
Acc(
Total Avg
Len(A%)
Len(&%))
Leaf
83.00
86.28
26.84
29.17
70.62
44,66
Original
16355
10162
11715
AdaptThink
80.20 (-2.80)
82.64 (:3.64)
25.00 (-1 84)
27.08 (-2.09)
68.28 (-2.34)
41.84 (-2.82)
-15.83
1715 (-69.73%)
447 (-81.81%)
1910 (-73,519)
8494 (-48.07%)
3188 (-68.63%)
4309 (-63.22%)
-67.50%
81.00 (-2.00)
78.98 (-6.30)
27.57 (+0.73)
(-0.21)
67.34 (-3.28)
44.36 (-0.30)
-1..66
AutoThink
1820 (-67.87%)
601 (-75.54%)
2397 (-66.76%)
7338 (-55.13%)
3487(-65 69%)
4049 (-65,44%)
66.07%
83.20 (+0.20)
85.82
27.21 (+0.37)
70.47 (-0.15)
44.81 (+0.15)
+0.52
LCPO
2397 (-57.69%%)
946 (-61.509))
2596 (-64.009%)
8810 (-46.139)
4026 (-60.38%)
4921 (-57.9918)
-57.31%
DeepSed--R--Diiil---Mee
92.20
91.81
36.76
51.46
87.97
56.82
Original
4223
1677
3926
13411
6966
8789
90.60 (-1 60)
91.28 (-0.53)
36.40 (-0.36
53.54 (+2.08)
86.56 (-1,4))
55.93 (-0.89)
-2.71
AdapTThink
(-53.23%)
(-78.89%)
2787 (-52.97%)
11002 (-17.96%)
4701 (-32.52%)
6894 (-24.97%)
-43.22%
AutoThink
89 (-2 80)
86.81 (-5.00)
36.03 (-0.73)
50.42 (-1.04)
83.91 (-4.06)
56.97 (-0.15) 15)
-13.78
(-48.21%)
(-51.76%)
2794 (-52.85%)
8215 (-38.74%)
4005 (-42.51%)
(194)119))
-46.44%
(-0.80)
92.95 (+1.14)
38.97 (+2.21)
(-2:71)
86.88 (-1
56.08 (-0.74)
LCPO
2033 (-51.86%)
796 (-52.53%)
2079 (.64.92%)
6892 (-48.61%)
3108 (.55.38%)
4222 (:51.96%)
-54.21%
reasoning behaviors depends the native potential of LRMs. While experiments demonstrate the generalization ability to OOD) scenarios, we believe there can be better results by carefully design mixed training set of different reasoning modes. 2) Due limited computational resources, we only conduct experiments on 1.5B and 7B models. Nevertheless, these extensive experiments demonstrate the effectiveness of our method across different model size.
I THE USE OF LARGE LANGUAGE MODELS (LLMs)
We use LLMs purely for writing assistance. Specifically, we employ LLMs check and correct potential grammatical errors, adjust wording, and streamline the text to avoid exceeding the page limit. The LLMs we use are all freely accessible to the public: DeepSeek-Vi. v3.cekk-.... and ChatGPT
"hups://chatgnt.coml
26


Reviews:
[{"review_id": "hXBbOC11Lw", "reviewer": "Reviewer_HaQu", "summary": "This paper addresses the “overthinking” problem in large reasoning models (LRMs) such as DeepSeek-R1, which often produce excessively long reasoning chains. The authors propose Length Controlled Preference Optimization (LCPO) — a small-scale, offline tuning method under a unified Bradley–Terry framework — that explicitly counterbalances the implicit NLL reward to align model preferences toward concise reasoning trajectories. Empirical results on multiple math reasoning benchmarks (MATH-500, GSM8K, Minerva-Math, AIME24, AMC23, OlympiadBench) show over 50% reduction in output length with minimal accuracy drop and good out-of-domain generalization to MMLU and GPQA-D.", "questions": "see above weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The issue of long, inefficient chain-of-thought generation is increasingly important for practical deployment of reasoning models.\n\n2. The use of a Bradley–Terry loss analysis to interpret and modify preference optimization objectives (e.g., DPO, ORPO, SimPER) is clear and insightful.\n\n3. LCPO achieves competitive results with only ~0.8K training samples and 50 steps — a strong practical contribution.\n\n4. Strong empirical validation: Evaluations across multiple datasets (both in-domain and OOD) support the claimed efficiency and generalizability.\n\n5. Clear ablation design: The authors systematically analyze the impact of data difficulty filtering and algorithmic components.", "weaknesses": "1. Does LCPO risk collapsing reasoning diversity — e.g., over-shortening reasoning even for difficult problems? It would be interesting to measure the diversity of generations.\n\n2. OOD evidence is limited to MMLU and GPQA-D, without baseline comparisons, making the generalization beyond math unclear.\n \n3. The paper emphasizes using only a small number of preference samples, but the actual workflow involves a significant upfront computational cost: it requires multiple rollouts across the dataset. In other words, while the training phase uses limited data, generating this data itself demands substantial computational resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "42QQ9tl8lA", "reviewer": "Reviewer_4RRh", "summary": "The paper first analyzes the generation path distributions of large reasoning models and filters their generated trajectories based on difficulty estimation to identify concise yet effective reasoning patterns.\nThen, it proposes Length Controlled Preference Optimization (LCPO) — a preference optimization method that directly balances implicit NLL-related rewards, effectively shortening reasoning chains by over 50% while maintaining accuracy.", "questions": "See the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper conducts a rigorous theoretical analysis of the characteristics of the Bradley–Terry loss and NLL loss, and proposes the LCPO method, which facilitates convergence in length control while reducing dependence on hyperparameters.\n2. The method achieves strong results on both the 1.5B and 7B models across multiple benchmarks, reducing output length by over 50% while maintaining accuracy, and also demonstrates excellent performance on out-of-distribution (OOD) benchmarks (GPQA, MMLU).", "weaknesses": "1. In Section 3, the experiments on LRMs are conducted using a relatively “simple” dataset (L196), making the conclusions less representative. This is because, in LRMs, reflective behavior on more difficult problems should lead to both an increase in tokens and an improvement in accuracy. Moreover, the final training data only includes the “easy” subset, which may cause the model to overfit or “hack” on simple problems while losing its ability to allocate more tokens to harder ones.\n2. There has been some research on difficulty-aware approaches, such as [1] AdaptThink and [2] Ada-R1, which reduces the novelty of this paper. Meanwhile, in RL-based methods, selecting the shortest response as the chosen one and the longest response as the rejected one is also a very common practice, as seen in works like [3] Kimi 1.5 and [4] Learning When to Think.\n3.  Lacks evaluation on the AIME25 dataset, and AIME24 may have been exposed to the models used in this work.\n4. For Table 2, I notice that the other methods use the different training time, could you provide the results in 50th step for other methods?\n\nBTW, the anonymous code link provided in the paper is no longer accessible.\n\n[1] AdaptThink: Reasoning Models Can Learn When to Think\n\n[2] Ada-R1: Hybrid-CoT via Bi-Level Adaptive Reasoning Optimization\n\n[3] Kimi k1.5: Scaling Reinforcement Learning with LLMs\n\n[4] Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OVsvSWn9NR", "reviewer": "Reviewer_og6S", "summary": "This paper proposes LCPO, a Length Controlled Preference Optimization, to implement efficient reasoning. Through extensive experimental analysis, the paper demonstrates the length difference in the model's own output distribution and the feasibility of using this difference for preference alignment. Furthermore, the paper experimentally analyzes the limitations of the method based on NLL loss. Finally, the authors propose using the log NLL loss as a reward for preference alignment, which achieves good results in length preference alignment.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper empirically demonstrates that short and correct reasoning trajectories exist within the model's generation distribution, which can be leveraged to model length preference.\n2.The proposed LCPO method is highly effective and successfully alleviates the \"overthinking\" problem in large reasoning models.\n3.The paper provides a good insight by analyzing the limitations of the NLL loss from an empirical perspective， and, based on this, proposes using a reward function derived from the NLL loss to model preferences.", "weaknesses": "1. Generalizability of In-Distribution Preference Alignment: While effective, the method of filtering and training on model-generated trajectories is highly dependent on the base model's intrinsic ability to produce short reasoning paths. This significantly limits the method's generalizability across different models and its potential for further compressing reasoning length.\n\n2. Stability of LCPO: Although the authors claim the method is effective in small-scale data scenarios ()()()(), the lack of a reference model in LCPO raises concerns about distribution drift. It is difficult to guarantee that the model will not suffer from significant distribution shift, potentially leading to performance collapse, when trained on a larger volume of data. In this light, the reliance on small-scale data might be a necessary limitation rather than an advantage. Did the authors attempt to train the model on more data?\n\n3. Limited Baselines: The paper primarily discusses methods for in-distribution length preference alignment (like DPO, SimPO, etc.) . It lacks a comprehensive comparison and discussion against other families of methods, such as broader RL-based approaches or prompt-based techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZCTnc7AYn0", "reviewer": "Reviewer_jTBW", "summary": "This paper presents LCPO, a lightweight method for pruning overly long CoT outputs from large reasoning models. The authors mine short yet effective paths via self-distillation and difficulty filtering, then bias the policy toward them with a preference objective. On six math benchmarks, LCPO cuts average output length by >50% while preserving or slightly improving accuracy. Extensive results on OOD tests (MMLU, GPQA-D) show that the approach is general.", "questions": "1. Algorithmic novelty:\n  \n   Have you run an ablation where the BT-loss weight in ORPO is increased? If so, please report the length/accuracy curve and indicate whether LCPO still outperforms this baseline.  \n\n2. Data efficiency:\n  \n    What fraction of the total GPU budget is spent on rolling out medi um/difficult questions that are finally discarded? Could a two-stage or cheaper proxy be used to predict “easy” questions and reduce upfront compute?  \n\n3. OOG Geeralization:\n\n    Have you tried training LCPO on short–long pairs distilled from another model (e.g., GPT-4o) instead of pure self-distillation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper reframes length reduction as “mine the model’s own short paths, then align”, and LCPO’s single-parameter BT loss is reasonable.  \n2. Extensive baselines (SFT, DPO variants) and OOD tests (MMLU, GPQA-D) are included.  \n3. With 0.8 k pairs and 50 steps, LCPO method can reduce average output length by over 50%.", "weaknesses": "1. Algorithmic novelty is limited. \n\n    LCPO amounts to dropping the NLL term of ORPO and keeping the BT loss; a controlled ablation that simply scales up the BT weight in ORPO is missing, so the reader cannot tell whether the observed gains require a new objective or just a different hyper-parameter balance.\n\n2. Data efficiency. \n    \n    The entire dataset must be rolled out 16× to obtain per-question accuracy and retain only the “easy” split. Moreover, all training pairs are self-generated, and the paper does not explore whether distilled high-quality pairs from another model (i.e., OOD preference data) could improve compression or robustness.\n\n3. Hyper-parameter sensitivity \n\n    The manuscript reports only the best value of the key temperature λ, without any sensitivity analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)