How Far Are We from Optimal Reasoning Efficiency?
Authors: Jiaxuan Gao
Shu Yan
Qixin Tan
lu Yang
Shusheng Xu
Wei Fu
Zhiyu Mei
Kaifeng Lyu
Yi Wu
Venue: NeurIPS 2025 poster ()
Keywords: Effiicent Reasoning; Large Reasoning Models; Reinforcement Learning for Reasoning
Primary area: deep_learning
TL;DR: This paper introduces a nove metric (REG) for evaluating  the reasoning efficiency of LRMs and a reinforcement learning method (REO-RL) that significantly reduces reasoning redundancy  while maintaining accuracy.

Abstract: Large Reasoning Models (LRMs) demonstrate remarkable problem-solving capabilities through extended Chain-of-Thought (CoT) reasoning but often produce excessively verbose and redundant reasoning traces. This inefficiency incurs high inference costs and limits practical deployment. While existing fine-tuning methods aim to improve reasoning efficiency, assessing their efficiency gains remains challenging due to inconsistent evaluations. In this work, we introduce the ***reasoning efficiency frontiers***, empirical upper bounds derived from fine-tuning a base LRM (DeepSeek-R1-Distill-Qwen-1.5B/7B) across diverse approaches and training configurations. Based on these frontiers, we propose the ***Reasoning Efficiency Gap (REG)***, a unified metric quantifying deviations of any fine-tuned LRMs from these frontiers. Systematic evaluation on challenging mathematical benchmarks, AMC23, AIME24, and AIME25, reveals significant gaps in current methods: they either sacrifice accuracy for short length or use excessive tokens to achieve sub-optimal accuracies despite high overall accuracy. To reduce the efficiency gap, we propose ***REO-RL***, a Reinforcement Learning algorithm that optimizes reasoning efficiency by targeting a sparse set of token budgets. Leveraging numerical integration over strategically selected budgets, REO-RL approximates the full efficiency objective with low error using a small set of token budgets. Experiments show that, compared to vanilla RL with outcome reward, REO-RL reduces the reasoning efficiency gap by 74.5\% and 64.2\%  in the 1.5B and 7B settings. The 7B LRM fine-tuned with REO-RL achieves reasoning conciseness surpassing frontier LRMs like Qwen3 and Claude Sonnet 3.7. Ablation studies confirm the efficacy of our token budget strategy and highlight REO-RL’s flexibility across design choices. This work establishes a systematic framework for evaluating and optimizing reasoning efficiency in LRMs. We will release the related code, data, and models to support future research on efficient reasoning in LRMs.

How Far Are We from Optimal Reasoning Efficiency?
Jiaxuan Gao!2 Shu Yan
Qixin Tan'
Lu Yang'
Shusheng Xu² Wei Fu
Zhiyu Mei
Kaifeng Lyu'
Yi Wu'
IIS, Tsinghua University Ant Group
Nanjing University
{samjia2000,
jxwuyi)@gmail.cm
Abstract
Large Reasoning Models (LRMs) demonstrate remarkable problem-solving capa- bilities through extended Chain-of-Thought (CoT) reasoning but often produce excessively verbose and redundant reasoning traces. This inefficiency incurs high inference costs and limits practical deployment. While existing fine-tuning methods aim to improve reasoning efficiency, assessing their efficiency gains remains chal- lenging due to inconsistent evaluations. In this work, we introduce the reasoning efficiency frontiers, empirical upper bounds derived from fine-tuning a base L.RM (DeepSeek-R) 1 Diwstill-Qeen-..Sie figurations. Based on these frontiers, we propose the Reasoning Efficiency Gap (REG), unified metric quantifying deviations of any fine-tuned LRMs from these frontiers. Systematic evaluation on challenging mathematical benchmarks, AMC23, AIME24, and AIME25, reveals significant gaps in current methods: they either sacrifice accuracy for short length or use excessive tokens to achieve sub-optimal accuracies despite high overall accuracy. To reduce the efficiency gap, we propose REO-RL, a Reinforcement Learning algorithm that optimizes reasoning efficiency by targeting sparse of token budgets Leveraging numerical integration over strategically selected budgets, REO-RI. approximates the full efficiency objective with low error using small set of token budgets. Experiments show that, compared to vanilla RL with outcome reward, REO-RL reduces the reasoning efficiency gap by 74.5% and 64.2% in the 1.5B and 7B settings. The 7B LRM fine-tuned with REO-RL achieves reasoning conciseness surpassing frontier LRMs like Qwen3 and Claude Sonnet 3.7. Ablation studies confirm the efficacy of our token budget strategy and highlight REO-RL RL's flexibility across design choices. This work estab- lishes a systematic framework for evaluating and optimizing reasoning efficiency in LRMs. We will release the related code, data, and models to support future research on efficient reasoning in LRMss
across diverse approaches and training con-
Introduction
Large Reasoning Models (LRMs) have recently emerged as a powerful class of models capable of solving complex tasks that require advanced reasoning. Frontier LRMs such as OpenAl ol [OpenAl] and DeepSeek R1 [Guo et al. 2025 have obtained superior performance across wide range of tasks, including mathematical reasoning and competitive programming. major factor behind this success is their ability to perform deep, multi-step reasoning through extended Chain-of- ofhouugh (CoT) processes. These reasoning traces often include sophisticated operations such as reflection, verification, and exploration, within a single inference pass.
However, the powerful capability of long CoT reasoning comes cost. LRMs frequently generate overly verbose and redundant reasoning traces, a phenomenon referred to as the overthinking prob- lem Yang et al. 2025b Sui al. 20255] Recent studies [Chen al. 2024b Sui et al., 2025 have
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

shown that even simple questions like "2 = can result in outputs spanning up to 900 tokens. This redundancy brings a significant cost in inference time and limits practical deployment. Several fine-tuning approaches have been proposed to improve reasoning efficiency, mostly focusing on reducing response length [Team et al.] 2025| [ Luo al. 2025a Aggarwal and Welleck] Arora
Arora and Zanette [025 Yeo et al. [025 [Shen et al. 2025h Qu et al. 2025 Yang et al. [025a She et al. 2025| Hou et al. 2025]. However, comparing these methods remains difficult due to inconsistent evaluation setups, including varying models, benchmarks, and mixed performance metrics. It is still
unclear how close current approaches are to the optimal trade-off between length and accuracy.
In this work, we investigate a critical question: How far are current approaches from reaching the optimal reasoning efficiency? To answer this question, we conduct a comprehensive empirical study using two LRMs, SeepSeck-Distilll Seriill--ee----- I.5B and benchmarks, AMC23, AIME24, and AIME25. We introduce the concept of reasoning efficiency frontiers, derived from fine-tuning the base LRMs with 3 types of algorithms and diverse training configurations. These reasoning efficiency frontiers represent the best reward achievable by the current approaches at each token budget, offering practical lower bound optimal efficiency. By comparing current methods to these frontiers, we uncover a substantial gap. Existing methods often fall short in one of two ways, either they aggressively shorten responses at the expense of accuracy, or methods that reach high overall accuracy would consume significantly more tokens than necessary to reach moderate accuracy levels. To quantify this gap, we propose the Reasoning Efficiency Gap (REG), a unified metric that captures both accuracy and response length by measuring the area between the length-accuracy curve of an LRM and the frontier. REG offers practical insights into how much room still remains for improvement.
DasSSeek-Distill-wweaaea S0wen-7B, on three challenging
We further ask: How can an LRM be fine-tueed to minimize this efficiency gap? A natural approach is to optimize the rewards across all possible token budgets at RL training time. However, this approach leads to costly training process since rewards across all token budgets should be evaluated. To overcome the inefficiency of this dense reward approach, we introduce Reasoning Efficiency Opti- mization with Reinforcement Learning (REO-RL), novel RL algorithm that improves reasoning efficiency by targeting a small set of token budgets. The key insight of REO-RL is that the total rewards across all token budgets can be well-approximated using numerical integration over a small set of representative token budgets. We experiment with both an oracle-based greedy selection based on the estimated reasoning efficiency frontiers and a heuristic strategy using exponentially spaced token budgets. Remarkably, with as few as 5 token budgets, both token selection strategies could achieve approximation error of less than 1%, ensuring that the optimization direction of REO-RL could align with the total rewards across all token budgets.
Finally, through systematically evaluating the reasoning efficiency gap for existing methods and REO-RL, we find that REO-RL consistently outperforms baseline methods in terms of reasoning efficiency across model scales. Notably, compared the vanilla RL baseline, REO-RL reduces the efficiency gap by 74.5% and 64.2% for the 1.5B and 7B models, respectively. By conducting a controlled comparison with frontier LRMs including Qwen3 and Claude Sonnet 3.7. we show that the 7B LRM fine-tuned with REO-RL. exhibits more concise reasoning patterns than frontier LRMss Our ablation study on the design choices for REO-RL reveals the success of the exponential token budget selection strategy and effective approximation with a small amount of token budgets. We also show that REO-RL flexible and various design choices, including setting the coefficients uniformly as 1 and using question-specific token budget, both could lead to competitive reasoning efficiency improvements.
2 Related Works
Efficient Reasoning. Prior studies have shown that LRMs often suffer from redundant reasoning. Even for very simple questions, Frontier LRMs often generate lengthy responses spanning thousands of tokens [Chen et al. 2025 Sui et al.
2025]. This redundancy in the reasoning process brings
significant overheads in the inference costs. Several works are then proposed to make LRM reasoning
more concise. Team et al. [2025] Luo et al. (2025a], Aggarwal and Welleck [2025] A.oor, and Zanette 2025), Yeo et al. 2025) Shen et al. 2025b), Qu et al. 20255] Yang et al. 2025a She et al. 22025].
Hou et al. 2025 investigate RL training with length reward designs, mostly focusing on reducing
the reasoning lengths. A line of works apply SFT to fine-tune LRMs on datasets with laria
reasoning traces to elicit concise reasoning (Yu et al. 2024
Wang et al. 2023 Han al. 2024 or

---

2025 Liu et al. 2024, Yu et al. 2025c]. Some works also investigate enhancing the reasoning efficiency through test-time techniques. including reward model guided decoding (Sun et al. 2024 Liao et al. 2025 and uncertainty-basee dynamic reasoning [Fu al. 2024 2. and confidence-based approaches (Taubenfeld et al. 2025 Huang et al. 2025|. In this work, we study the optimal reasoning efficiency for an LRM and focus on training- based approaches for enhancing LRM reasoning efficiency. Our method aims at enhancing the accuracy of the LRM under diverse token budgets without explicitly incentivizing shorter responses.
adjustable length control (Kang et al. 2024 Xia et al., 2025
Ma et al.
RL for LRM Reasoning. Reinforcement Learning is the central technique for eliciting and enhanc- ing the reasoning capability of LRMs. Frontier LRMs, including OpenAI o1 [OpenAl] and DeepSeek RI (Guo et al. 2025 have shown that applying "zero RL" on a base LLM could effectively clicit the ability to utilize long CoTs for complex reasoning. series of works have emerged with the focus on improving the training efficiency of RL for LRMs from the perspectives of data (Luo et al. 2025c
Lab 2025 He et al. 2025 Li et al. 2025
Wang et al. [2025]. algorithms [Guo et al. 2025 He
al.
Yue al. [025] J. and training framework (Sheng et al. Luo et al., 2025b RL Lab 2025 A number of works successfully apply zero RL training on a
2025
Luo et al. [2025c] Yu et al. 2025b
2024
wide range of reasoning-heas heavy domains, including multi-modality (Shen al. 2025a Zhang et al..
2. medical (Yu et al. 2025a Chen et al.) 2024a], and financial [Liu et al. [2225] J. Recent works
also explore efficiency enhancement by encouraging concise reasoning with RL [Team et al.) 2025
Luo et al. 2025a Aggarwal and Welleck [2025 |Arora and Zanette| [0025] Yeo et al. [2025] Shen et al.
2025b Qu et al. 2025 Yang et al. 22025a] [She et al. 20255 Hou et al.) 2025 In this work, we focus on enhancing the reasoning efficiency with RL.
3 Preliminary
LRM Reasoning. In this work, we focus on the task of mathematical reasoning. Given a question r, the goal of an LRM policy is to generate a response y that contains step-by-step reasoning to derive the correct answer. We assume access to a verifier Rxx, y) that evaluates the correctness of a solution In practice, such a verifier is implemented by matching the ground-truth answer and the model-generated answer. The LRM is policy Io parameterized with and generates sequence of reasoning tokens in an auto regressive manner. Given a question distribution D, the objective of the LRM is to maximize the probability of producing correct responses,
y given the question
J(D, f) FarDD [R(r, y)]
(1)
where 0 is in a parameter space θ and the response length [y] is limited to the maximum length Lmax- In practice, 0 is usually obtained through applying fine-tuning approaches such as RL on base LRM. Therefore we assume the existence of base LRM Ousse and ② to be the of all LRMs that could be obtained by fine-tuning Ohuse with any algorithm.
4
Understanding the Limits of Efficient Reasoning
4.1 Defining Optimality in Token-Boundee Reasoning
Evaluating Reasoning under Token Budgets. To assess optimal reasoning efficiency, we must evaluate the performance of an LRM Ne under a fixed token budget Simply truncating a model's output after L tokens, however, may lead to incomplete responses. To address this, we define fallback mechanism following prior works [Muennighoff et al. 20255] Fu et al. 2024). If reasoning trace y To(- (xr) exceeds L tokens, the model is prompted to produce final answer a directly from the truncated trace V.L.
To(-I>; M.L. [The Final Answer is)) Extract Answer
if | >
L
a = Ansserr(To.2.
otherwise
where V.r. denotes the first L. tokens of the reasoning trace and AxtractAAssweric y) extracts the final answer from a complete trace. This approach allows consistent evaluation across different budgets, though it introduces minor additional token cost in the truncation case.
3

---

Fine-tuning w. Diverse Strategies
(b) Reasoning Efficiency Frontier
80
Reusoning Efficiency Gap
70
RL W. Token Budget
LRR
&ccuracy
50
Length Reward
40
LRM
30
Base LRV Bgaase
Reasering Efficiency froner
Preference Learning
LRM
Base
4K
BK
12K
16K
Token Budget
Figure 1: Procedure of estimating the reasoning efficiency frontiers. (a) Starting from a base LRM Tre.. we apply diverse fine-tuning strategies to obtain a large amount of LRMs. (b) We then compute the best achievable accuracy across varying token budgets to obtain the reasoning efficiency frontiers (Eq.
Length-Constrainee Reward and Optimality. We define the length-constranee constrained reward for model Ty over question distribution D as the expected reward obtained when the model is restricted to token budget I,
JDD L.) = EgwD E (-x (ar, Answer(To, x, v.I)))]
The length-construined optimal reward then captures the best possible reward achievable by any model parameter space under the same budget,
Joptimal (D, O L)
max J(D,0,
#5
4.2 Empirical Estimation Reasoning Frontiers
We aim to characterize the optimal reasoning efficiency, denoted by Joptimal (D, O, L). that reflects the best achievable reward at any token budget L across all possible model parameters O. However, computing this optimal frontier exactly is infeasible in practice, it requires exhaustively exploring all algorithms and training configurations. Instead, we construct an empirical reasoning efficiency frontier by fine-tuning a diverse set of models using existing approaches. Let é = (01,..., 0n C θ denote the collection of parameters from m fine-tuned models. Based these, we define the empirical reasoning efficiency frontier as,
Definition 4.1: Reasoning Efficiency Frontier
Given a parameter space O, a set of model parameters Ô {0,, 0m)) and question distribution D, we define the reasoning efficiency frontier as a set of points (Joptimal (D, ô, L) E [max]} where
j Joptimal (D, 0E10,... )m)
J(D,,0,L) VL [1, Imma]]
Note that Joptimal (D, θ L) serves as lower bound of the optimal frontier since Ô is subset of θ Joptinal (D, Ô L) Jeptimal (D,0,
Diverse Fine-tuning Approaches. To obtain close approximation to the optimal frontier in Eq. we fine-tune models using wide range of training strategies,
Online RL with Token Budgets: We conduct online RL training with different token budgets ranging from 512 to 32k. Fine-ture tuning an LRM with a token budget effectively enforces the LRM reason with limited tokens [uu et al. 2025 Hou al. 2025
Online RL with Length Rewards. We test various reward designs that promote concise yet accurate reasoning, including Seangth-harmonzzing rewards [Luo normalized rewards [Arora and Zanette 2025].
2025a and length-group
Preference Learning. We apply SimPO Meng et al. 2024 with preference datasets constructed via methods such as TOPS YYng et al.
2025b and DAST [Shen et al. 2025b). For example,
we contrast short correct responses with
longer ones to promote conciseness Muunkhbat et
2025

---

80
70
Accuracy (%)
more
(%)
60
40
50
1.3x more
30
40
Reasoning Efficieccy Frontier
30
Reasoning Efficiency Frontiar
LRM
20
LRM
10
Vanilla RL
10
Vanilla AL
Budget
Takan Budget
0
0
2K
4K
8K
10K
12K
16K
Token Budget
Token Budget
(a)
(b) --aas------isssii--eee.e-
Figure 2: Reasoning Efficiency Frontiers for DeepSeek-R1-Distee - 1.5B and DeepSeek-RI -11 Distill-Qwen-7B DeepSeek-Ri SeesSee-RR--issill-Nveeess DDistillQQeen-I.53 and 1.3x more tokens than the corresponding frontiers to achieve the same accuracies.
and DeepSeek- BerSSekk-R---- RI-DDssill-QQenn spend 3.4x
Experiment Setup. We conduct our study using two base LRMs,, DeepSeek-R RI Distill-Qween--e. and DeepSeek-RI-Diin E2--Disill-Qwen--7 [Guo et al. [025]. For each LRM, we start from RL-fine-tuned versions and further fine-tune them using the strategies above. We evaluate three challenging mathematical reasoning benchmarks: AMC 2023, AIME 2024, and AIME 2025. More training details can be found in Sec. 6.I Appendix. algorithms and 15 training configurations, resulting in 180+ and 210+ models for 1.5B and 7B scales, respectively.
C We conducted large-scale RL experiments spanning
Reasoning Efficiency Frontiers. Based on the fine-tuned models, we are able to construct the reasoning efficiency frontiers for both base LRMs. Notably, we find that DeepSeek DeepSeek-RI-Dissee -i--ttlll-- Qwen-1.5B and DeepSeek-R1 R1-istill -weill--wee-BBB require approximately 3.4x and 1.3x more tokens, respectively, than their corresponding empirical frontiers achieve the same level accuracy.
Measuring the Gap to Optimality. Given the estimated reasoning efficiency frontiers Joptial D, θ L). can the metric of Reasoning Efficiency Gap (REG) to quantify the distance of any LRM from reaching the optimal reasoning efficiency.
Definition 4.2: Reasoning Efficiency Gap (REG)
Given any LRM Te and the estimated reasoning efficiency frontier {J (D, (D, [1, Lmax]> }, we define Reasoning Efficiency Gap as,
Lana
(f D,, Q)
Σ
j Joptinal
L=1
Methodology
5.1 Boosting Reasoning Efficiency by Optimizing Congth-Constrained Rewards
Optimizing Congtt-Conttr Rewards. To minimize the reasoning efficiency gap. straight- forward idea is to optimize the length-constrained rewards under all token budgets to enhance the reasoning efficiency of Te leading the efficiency objective,
Lexs
Statiiency (0, = Σ J(D,O,L)
where Lasz is the maximum generation length. However, directly optimizing Eq. blis computationally impractical. Evaluating JDD, 0, L) for each budget E [1, [max] requires separate inference runs to evaluate truncated responses discussed in Sec. 4.1 As a result, each training example would require up Taax additional LRM generations, significantly increasing both compute time and memory usage, particularly due to the expanded KV cache usage.

---

5.2 Reasoning Efficiency Optimization with Reinforcement Learning
REO-RL. We introduce Reasoning Efficiency Optimization with Reinforcement Learning (REO- RL), efficient training algorithm that optimizes an approximation for the objective in Eq. 6 In REO-RL, instead of optimizing the length-constrained reward in all token budgets, we approximate the objective with small set of selected token budgets L..... nn to ensure high training efficiency. Specifically, following the Trapezoidal rule in numerical integration, we could approximate the objective in Eq. 6 with,
N J(D,O,L)
L1+1 L--1 J(D,0, L J(D,0,0)
Lmax
LN
J(D, 0, Lmmas)
2
i=1
(7)
f(D,0, -...... LnN]
(8)
where assume Lu and LN+1 Lmax and fD, 0, {L,. objective with token budgets Li. Lv. As the number of selected token budgets increases, (D,00, (L, Lx) }) would become closer and closer to the objective in Eq.
Ln}) denotes the approximated
The approximated objective S(D,0, (I,,..., [NN) could be equivalently represented as an RI. objective with dense rewards,
N+1
Capprox (0, D) )- E E Eywas(-la))
Σ
Answer((e y.L.))
where C;
L441-hi11 for S and CN+1 Lon are the coefficient for the i-th token
budget following Eq.
Directly computing the derivative of in Eq. would lead to the following equation with terms,
N+1
V Vo Lapproo (0, D) = EAND
Vo VoLyyxo(11a)
Σ
G, R(x, Answer(Tu, B.r..))
(10)
Rewards of Truncated Responses
N+
+ B
E Bywwryr (-1x)
Σ G
VoR(x, Answer(r,, 2,
(11)
Probubiiiyy of Outputting Correct Answer Directly Given Truncated Responses
where 0' = sg(0) is equivalent to 0 but does not propagate gradients. Note that the first term can be computed through standard reinforcement learning algorithm. On the other hand, the second term focuses on direct answer generation from truncated responses. In practice, we neglect the answer generation since this term focuses on a much smaller amount of tokens compared with the first term and we also find optimizing answer generation not bringing empirical benefits.
REO-RL.
To this end, we propose REO-RL that focuses on improving the rewards responses truncated under different token budgets. Suppose use REINFORCE algorithm, the objective of REO-RL is derived by
V TaCREO-RI(O,)
N+1
VotolyL, aagl V.r. .)
Σ Rg 25,
In Eq. 8 different sets of token budgets LI,... Ln would cause different approximation error Ideally, the approximation error induced by the selected token budgets should be low to ensure that REO-RL aligns with the original objective in Eq. 4

---

80
40%
Accuracy (%)
Approx. Error
30%
Exp
40
20%
20-
Ressoning Efficieccy Fronder
10%
Selccld Takan Busigels
32K 0%
10K
10
Token Budget
(a)
(b)
Figure 3: Selection of Token Budgets in REO-RL. (a) Token budgets selected roughly follow an exponential pattern. (b) Both the oracle greedy approach and the exponential approach achieve lower approximation errors with few token budgets.
REO-RL (Oracle). Note that the original objective in Eq. 4 bounded by the theoretical reasoning efficiency frontier in Sec. S.2. i.e. Eim J(D, 0, L.) 21 Joptimal (D, θ L.). A natural idea is to determine the optimal token budget selection scheme based on the estimated reasoning efficiency frontiers. Specifically, for any set of token budgets L2, LN. EL=1 Japtimal (D, θ L) could
be approximated in similar way Eq.
N Σ
Ti+1 1--1 Jeptinul (D.0, Tn Joptimal (D, (12)
Joptinal (D,0, LNl)
mmax
LN Japtinal (D, O Immax)
(13)
2
where Soptimal (D, θ (I,, [,)) denotes the approximated value for ELer Joptimal (D, Ô I.) given token budgets T.,,..,, I.
We adopt greedy approach that iteratively selects the token budget with the lowest approximation error. Note that this is an oracle approach since the reasoning efficiency frontiers should be known in advance. This greedy selection approach leads to oracle algorithm, REO-RL (Oracle). In REO-RL (Oracle), the token budgets L.. LN are selected according to,
Lon Σ
I arg min E'
Suptimal (D, (D,6, (T,,... I.-1, I'})
Japtiaal (D, U T)
L=1
The oracle greedy approach could produce set of token budgets that achieves low approximation error in Fig. 3(a) As illustrated in Fig. 3(b) the approximation error gradually degrades with more token budgets. Notably, the approximation error could be lower than 1% with N 2 5.
REO-RL (Exp). However, in cases when the reasoning efficiency frontiers are unknown, it is infeasible to apply REO-RL (Oracle), We observe that token budgets selected by the greedy selection approach roughly follow an exponentially spaced pattern as shown in Fig. 3(a] Therefore, we propose adopt exponentially spaced scheme for token budget selection, leading to the algorithm, REO-RL (Exp), that selects a set of exponentially spaced token budgets,
L = Tomin (Tmaxx/ Tmmin)
where Lmin/ Lmax are the minimum/maximum token budgets.
Experiments
6.1 Experimental Setup
Models, Datasets & Metrics. We use Di-ecc----iittil---eee 1.5B and DeepsSek-R11-Dssilll
Qwen-7B the base LRMs. For training, we adopt mixture of training data consisting of 135k problems sourced from DeepScaleR Luo et al. [025c] and AReaL RL Lab 2025] For evaluation, we use three challenging mathematical benchmarks: AMC 2023, AIME 2024, and AIME 2025. We report the average accuracy of 32 responses generated with temperature T = 0.6 and top_p= 0.95 with maximum length T.max = 32K. The main results are averaged over three benchmarks, When evaluating REG, we smaller length Lmax 16K to focus on the area under lower token budgets.

---

REO-RL & Baselines. For REO-RL, we consider both REO-RL (Exp), REO-RL (Oracle), and variant of REO-RL (Exp) that uniformly sets all coefficients c; For baselines, we consider,
Online RL: We consider online RL. with length- based rewards, including length-harmoniin rewards [Luo al.
2025a], and length group normalized rewards [Arora and Zanette [025]. We also compare with Meta Reinforcement Fine-tuning (MFT) [Qu et al. 2025]. which minimizes regret for individual steps to enhance reasoning efficiency. Additionally, we also adopt online
RL with hard token budgets as a baseline, which is also adopted in recent works [Hou et al.
2025
Xu et al. 2025).
Supervised Line-Tunin:: For each problem in the training data, we generate multiple responses and perform SFT on the shortest correct one. Two strategies are used for data generation. The first strategy is direct generation with the problems as inputs [Munkhbat et al. [025). 'he second strategy follows TOPS prompt the LRM with different levels of reasoning efforts Yang et 2025b].
Preference Learning:
We apply SimPO Meng et al. 2024 on various preference datasets. In
Sim.POshortest, we use the shortest correct response the longest response as the preference pairs. We also follow TOPS Yang et al.] 20256) and DAST [Shen et al. 2025b to construct
preference datasets.
have also tried RL-based length control methods such as [Aggarwal and Welleck 2025 Xu et 2025] but find these methods only achieve successful length control under low token budgets and yield similar performance as RL w. Token Budgets. More details about the baselines and our other investigations could be found in Appendix.
Training Details. For REO-RL and all baselines, we use the same training data and implement all methods based the AReaL framework. Instead of directly fine-tuning the base LRMs, fine tune the corresponding RL-trained versions. Specifically, we use
Haal--Boba-L--..3 [RL Lab 2025
and SkyWork-OO -Oa 1-Math-7BB (He et al. 2025 the starting points for further fine-tuning in 1.5B and 7B experiments, respectively. For REO-RL (Exp) and REO-RL (Oracle), we use N = token budgets since N already obtains sufficiently accurate approximations illustrated in Fig. 3(b) For more training details, please refer to Appendix.
6.2
Main Results
60
Accuracy (%)
Accuracy (%)
(Exp) Coaf=1 jours) REC RL (Exp) lours)
REC-RL (Exp) (ours)
40
Base LAM
Basa LAM
Vanilla
Vanila
MRT
Budget=2K
Budiget=2K
Len Group horm.
RL w. Lon Lon-Hammnianng Rew
RL Len Harmonizinn Rew.
SimPO
SimPCruer
Efficiency
Eificieccy
4K
8K
12K
16K
4K
8K Token Budget
12K
16K
Token Budget
(a) 1.5B
(b) 7B
Figure 4: Performance comparison of REO-RL (Exp) with representative baseline methods REO-RL (Exp) notably im-
DeepSeck-R1-Distee -iii------istilll-ee------- 1.0--eeenBBB and proves the reasoning efficiency of LRMs. Although existing approaches can approach the efficiency frontier when sufficient token budgets are available, there still exist large performance gaps under tight token constraints. Results are averaged over AMC 2023, AIME 2024 and AIME 2025.
Fig. presents the accuracy of LRMs fine-tuned using REO-RL (Exp) and range of strong baseline methods across varying token budgets F Full quantitative results detailed in Tab. 1 We draw several key observations from the experiments,
There exist fundamental gaps between existing approaches and the frontiers (Fig, 4. As shown in Fig. several existing approaches can approach or even reach the reasoning efficiency frontier
For clearer visualization, we include only set of representative approaches and plot their performance under limited token budgets.

---

DeepSeckRR--B Disicc--RR--isii---weeeee
SeepSeck-R--iaull---
Accuracy (56)
Length
REG
Accurncy (%)1
Length |
REG 4
Base LRM
41.4
14430.5
2036.7
62.0
11160.4
1782.1
Vanilla RL.
51.6
11635.5
1257.0
71.0
12609.9
1691.6
REO-RL (Exp) (ours)
13.1
7467 188.8%
551.8.66.1
68,7,18
6725.7
816.7.5..
REO-RL (Exp) Codf=l (ours)
44.3
9042.7 39
320.8 68
67.6pa.4
161.13
.64.25
RED-RL (Oracle) (ours)
53.9 53,9
71.9.7
606.9 J01.799
69.2
7660.8
7660.8.90.20
771.1_64.08
RL Token Budget=IK
12.2
1293,9
2823.6,
39.8,a1.a
1123.6,0.118
3627.1g114.00
Budget-2K
18.7
1979.0
2077.5 305.3%%
49.5.21.
2256.5 -111-9
RL w. Token Budget-4K 4K
46.1
3441.0 p7
1045.3 116.85
57.6
2978.4
1343.3 _03.3
RL Lon Group Norm,
52.6 52.6
p0 7%
891.2,pp US
69.8
8961
1057
(107.2_07.00
RL w. Len Harmonizing Rew.
52.7
7075.4 150.29
622.8 8u0
70.4
7956.8 195,988
970.8 42.
MRT
53,41.
9031.8,22.4
770.2 736
70.1,0.
8252.2
976.4,41
SPTa
51.9
11544.5yaast
11.6.0
71.6,0.
11711.3,7a%
1542.4.s.55
SFT
51.7
26666.216.995
1090.6
70.4,a.
11549. 11549.1_s.15
1519
SimPOuusT
36.8,
10882.2.10.88
3318.3re4.45
66. 14.9
6489.7 244.05
987.8
SinPPOsveaa
46.6gs.0
6274.6146.
1250.7
20.39%
69.9
8144.5195.es
1060.3
SimPOmrs
30.8
2185.8
3137.0 pa4k.ss
58.3_19.7
4379.5
1453.3
Table Accuracy, Generation Length, and Reasoning Effciiecyy Gap (REG) for all methods. The relative changes compared to the vanilla RL baseline are also reported. REO-RI could significantly reduce the gap from the LRM to the reasoning efficiency frontier with minor or even no accuracy drop at the same time.
given sufficient token budgets. However, they could require significantly more tokens than the efficiency frontier to achieve moderate-level accuracies. For example, on On the other hand, some methods, such as RL with Token Budget, perform well under tight token constraints but exhibit substantially lower overall accuracy. Our benchnarking results suggest that, optimizing the base LRM to precisely match reasoning efficiency frontiers remains an open problem.
REO-RL consistently outperforms baselines in reasoning efficiency (Tab. 1. Compared to baseline methods, the three variants of REO-RL are much closer to the reasoning efficiency frontier, exhibiting much lower Reasoning Efficiency Gap (REG). REO-RL maintains high accuracy while generating shorter outputs. REO-RL (Exp) Coef=1 1 performs the best in terms of reasoning efficiency, reducing the REG by 74.5% and 64.2 for the 1.5B and 7B models, respectively.
Vanilla RL does not yield consistent improvements acruss budgets (Fig, 4). Although Vanilla RL improves overall accuracy relative to the base LRM, it fails to provide stable gains across different token budgets. In the 7B experiments, its accuracy is lower than the base LRM when the token budget below 8K.
DeepSeek R1 Dististill--ae Distiill-wenn Qwen-1.50 1.50
DeepSeek R1 Distill Owen 70
80%
70
50
Accuracy (%)
Accuracy (%)
45-
60
55
50%
40
50
0%
*
40
30 2000
4000
6000
8000
10000
12000
14000
2000
4000
6000
8000
10000
12000
Higher R1000%
Base
REO-RL (ours)
w/ Length Rewards
SimPO
Vanilla RL
RL w/ Token Budget
SFT
Figure 5: REG effectively captures the trade-off between accuracy and response length. Achiev- ing low REG require both competitive accuracy and short response length. By minimizing the efficiency gap, REO-RL outperforms baselines in terms of reasoning efficiency.
REG effectively captures the trade-off between accuracy and response length (Fig. 5). Achieving low REG requires balancing high accuracy with concise output. For example, REO-RL (Exp) maintains strong accuracy while generating compact responses, resulting in low REG. contrast, methods that optimize only one aspect, such as minimizing length at the cost of accuracy, fail to achieve low REG. RL with Token Budgets, despite producing shorter responses, suffers from significant accuracy loss.

---

6.3 Comparison with Frontier LRMs
To further evaluate the reasoning efficiency LRMs trained with REO-RL (Exp) in comparison to advanced LRMs, we conduct controlled analysis focused on the correct response length. Since frontier LRMs and those fine-tuned with REO-RL (Exp) differ significantly in overall accuracy. we construct balanced subset of 71 questions from the test set, where all models achieve an accuracy exceeding 50%. For each model, we compute the average length of correct reasoning traces for each problem. The final length metric for each model is then obtained by averaging the correct response lengths across all 71 questions. As shown Tab. 2 the 7B model trained with REO-RL (Exp)
exhibits more concise reasoning patterns than frontier LRMs, with much shorter response length.
(Thinking) DeepSeek
Qwen3-4B
Qwen3-BB
Qwen3-32B
Vanilla
(Exp)- (ours)
Length Accuracy
17478.87
5156,39
8631.61
8898.57
7755,99
7732.11
4524,02
90.0%
98.2%
95.8%
94.9%
97.2%
93.5%
93.5%
Table Comparing the response length with frontier LRMs.
6.4 Ablation Study of REO-RL
We conduct an ablation study on the design choices of REO-RL using the -i-iseee-R--sssil--------- model to better understand its flexibility and perfor- mance characteristics. By varying key components, we demonstrate that REÔ-RL can maintain compet- itive performance across different configurations.
Method
Accuracy (9)
Length
REG
Vanilla RL
71.0
12009.9 9
1691.6
LRM
62.0
111604
1782.
Eep
68 u
6725 7 2.1.5
816-7
Oeacle
69.2,3.
766088
771.
11.1ysa
Exp Coef-l
67.6.3.4
6160.3 us..es
608 9
Exp-N ---10
69.0,pa
7979 1.3.75
923.2
Lincar
69.1...
7424
536.1
Token budget selection strategy. We evaluate REO- RL (Oracle) that adopts the oracle greedy strategy. This variant achieves slightly better performance than REO-RL (Exp), reflected in a lower REG. We also investigate linearly spaced token budgets, which leads to less optimal efficiency and higher REG.
Questlon Specttic Oracle
69.5
7312.3
687.7
Table 3: Ablation Study of REO-RL on DeepSeek-RI-Disee -inSSeetRR--Distill--ee-----
Coefficient in REO-RL objective. Setting all coefficients uniformly to 1 leads the model to align more closely with the efficiency frontier and shorter response lengths. However, this approach comes at the cost of reduced overall accuracy.
Number of selected token budgets N. We increase the number of selected token budgets to N 10 to explore whether a finer granularity improves performance. In practice, we observe that using more token budgets results in slower convergence and higher rate of training instability. Consequently, this configuration produces weaker results overall.
Question-Specific Oracle Budgets. Finally, we investigate question-specific oracle strategy that assigns two token budgets to each problem, the minimum reasoning length derived through the frontier estimation experiments (Sec. 4.2) and the full budget Lmax. This oracle approach proves to be competitive, further reducing the efficiency gap of REO-RL.
Conclusion
In this work, we investigate efficient reasoning for LRMs. We introduce reasoning efficiency frontiers, which characterize the empirically optimal trade-off between response length and accuracy for LRMs. To quantify the reasoning efficiency of a fine-tuned LRM, introduce the Reasoning Efficiency Gap (REG), a unified metric that captures both accuracy and length. We benchmark existing methods and reveal a substantial gap between current fine-tuning approaches and the frontiers. Our proposed method, REO-RL, consistently obtains better reasoning efficiency than strong baselines across model scales. Despite these gains, achieving full match with the reasoning efficiency frontiers remains an open problem.
10

---

References
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.0m 04697, 2025.
Daman Arora and Andrea Zanette. Training language models to reason efficiently. arXiv preprint arXiv:2502.04443. 2025.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang. Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-ol, towards medical complex reasoning with Ilms, 2024a. URL
https://arxiv org/ abs/ 2412. 18925
Xingyu Chen, Jiahao Xu, Tian I. Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang. al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. arXiv preprint arXiv:2412.21185 187, 2024b.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei Hc, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like Ilms, 2025. URL https //arxiv. cwexxiv.rrg/ae abs/ 2412. 21187
Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. Efficiently serving IIm reasoning programs with certaindex. arXiv preprint 2tl::2422.2999. 2024.
Yichao Fu, Junda Chen, Yonghao Zhuang, Zheyu Fu, Ion Stoica, and Hao Zhang. Reasoning without self-doubt: More efficient chain-of-thought through certainty probing. In ICLR 2025 Workshop on Foundation Models in the Wild, 2025. /RLLattss//opperriviie www.iewwwweeforrur Id=wpK4IMJfaX
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint 22X1 2025.
Tingxu Han, Chunrong Fang, Shiyu Zhao, Shiqing Ma, Zhenyu Chen, and Zhenting Wang. Token- budget-aware I1m reasoning. arXiv preprint arXi::2412..8544 2024.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang. Peng Cheng. Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng. Tianwen Wei. Cheng Cheng. Yang Liu, and Yahui Zhou. Skywork open reasoner series. httpe://aaprrcccoou--byrrrggr notion.. site/ www.ssiittstttoee Skywork-Opent 2025. Notion Blog.
Series- oateeeeeeeobbbee2222000034444443
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning, arXiv preprint arXiv:2504.01296 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zerro An open source approach to scaling up reinforcement learning on the base model, 2025. URL httpss://axxi...rrtttt wtwpx/wwww.rr//b///....3 2550g 24290
Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. Efficient test-time scaling via self-calibration. arXiv preprint arXiv:2503.0000: 2025.
Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter chain-of-thouge without compromising efffectiveness. arXiv preprint arXiv:2412.11664, 2024.
Xuefeng Li. Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling, 2025. URL https://arxiv org/ aww./b//00 11886
Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. Reward-guided speculative decoding for efficient Ulm reasoning. arXiv preprint arX1v:2501.19334, 2025.

---

Tengxiao Liu, Qipeng Guo, Xiangkun Hu, Cheng Jiayang, Yue Zhang, Xipeng Qiu, and Zheng Zhang. Can language models learn skip steps? arXiv preprint arXiv:2411.01855, 2024.
Zhaowei Liu, Xin Guo, Fangqi Lou, Lingfeng Zeng. Jinyi Niu, Zixuan Wang, Jiajie Xu, Weige Cai, Ziwei Yang, Xueqian Zhao, Chao Li, Sheng Xu, Dezhi Chen, Yun Chen, Zuo Bai, and Liwen Zhang. Fin-rl: A large language model for financial reasoning through reinforcement learning, 2025. URL https:: haapp:///arxiv org/ abs/ 16252
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: Length-harmonizing fine-tuning for ol-like reasoning pruning. arXiv preprint 2rXiv:2501. 2025a.
Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaox- iang Shi, Rachel Xin, Colin Cai. Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at 03-mini level. https://reett-rrdiobbb7s http:://noayyyaaaloooomtttttttte Saller---Fully------Sore---e- mini-Level- lef8 1902c 4680033eee3555534445522222 2025b. Notion Blog.
Michael Luo. Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o -p-eeiiww with 1.5b model by scaling rl. Docaling-RL-19660 ELL-196819020 444-1680000022440000000000000020 2025c. Ol-Preview-with-a Baattha---SB-Modeeee
Notion Blog.
Xinyin Ma, Guangnian Wan, Rumpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length- compressible Shain-of-thoughle tuning. arXiv preprint arXiv:2502.0900 2025.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. Advances in Neural Information Processing Systems,
2024.
Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL https:://ttssee http///axvvv... www.aatwww.aa.aaaaaae.eaa abs/ 2501 19393
Tergel Munkhbat, Namgyu Ho, Seohyun Kim, Yongjin Yang. Yujin Kim, and Se- Young Yun. Self- training elicits concise reasoning in large language models. arXiv preprint arXiv:5502.20122. 2025.
OpenAI. Learning reason with llms. uutttpps/oppnai.cc//eeeee///mete cow/index//eaminggg
to-reason- co-rrason-with-llnss- with-llms/ Accessed: 15 March 2025.
Yuxiao Qu, Matthew YR Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine- tuning. arXiv preprint arXiv:2003.07572, 2025.
Ant Research RL Lab. 2025.
Areal: Ant reasoning rl.
Jianshu She, Zhuohao Li, Zhemin Huang. Qi Li, Peiran Xu, Haonan Li, and Qirong Ho. Hawk- eyesefficient reasoning with model collaboration, 2025. URL httth:ttp///arree http:://xxxvv.rrr htrgg/bb/2504 00424.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vim-rl: A stable and generalizable rt-style large vasion-language model. arXiv preprint XiXx:2504.076155 2025a.
Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, and Shiguo Lian. Dast: Diffieulty-adaptive slow-thinking for large reasoning models. arXiv preprint A.Xv:2503300477 2025b.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rlhf framework. arXiv preprint arXiv: 19256, 2024.
12

---

Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Hanjie Chen, Xia Hu, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503. 16419, 2025.
Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang. Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. arXiv preprint 2iXi2:240.220290, 2024.
Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves confsscoy in llms. arXiv preprint arXiv:2502.06233, 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint 1.XXi::2501.12999 2025.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. cenficoons improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
Yiping Wang. Qing Yang. Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training example, 2025. URL // org/
Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable thouuht compression in 1lms. arXiv preprint arXhv:2502.1206) 2025.
Yuhui Xu, Hanze Dong. Lei Wang. Doyen Sahoo, Jumnan Li, and Caiming Xiong. Scalable chain of thoughts via elastic reasoning, 2025. URLttttps: //ae:iiv//xiv wwwpsiii.rrg//oooooo.
org/ arg/abs/25055. 05315
Junjie Yang. Ke Lin, and Xing Yu. Think when you need: Self-adaptive Mfain--ff-thuueee learning,
2025a. URLbtsss://arxix org/ awwaab///.... 03234
Wenkai Yang. Shuming Ma, Yankai Lin, and Furu Wei. Towards thinking-optimal scaling of test-time
compute for IIm reasoning, 2025b.
hbsp//bb//2202/ 2202. 18080
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long
chain-of-thought reasoning llms. arXiv preprint MiXi:22502203373 2025.
Hongzhou Yu, Tianhao Cheng, Ying Cheng, and Rui Feng. Finemedlm-ol: Enhancing the medical
reasoning ability of I1m from supervised fine-tuning to test-time training, 2025a. URL https
7/arxiv. org/ abs/ 0000/551.000 09213
Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1. arXiv preprint arXiv:2407.06022 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi
Wang, Hongli Yu, Weinan Dai, Yuxuan Song. Xiangpeng Wei, Hao Zhou, Jinginng Liu, Wei- WingYing Ma, Ya-Qi fa-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open- source I1m reinforcement learning system scale, 2025b. URL https://: //arsiv////ivv///// wtw.aawww..ga..aaeeeaaaeaee htrg//bbs25033 14476]
Zhaojian Yu, Yinghao Wu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. Zl: Efficient test-time scaling with code, 2025c. URL wttph:ttt///et //arxivv org/ htw.////04.000 00810
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang. TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu, and Lin Yan. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025. URL httpl:://////// //arxiv...vvv/a/ www.i.irrggaaa.aeaaaaeaaa.aaolla abs 2004.2504..00 05118
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vI: Learning to reason with multimodal large language models via siep group relative policy optimization. arXiv preprint arXiv:2503. 12937, 2025.
13

---

A Limitations
Limitations. Our study is currently only taken place on 1.3B and 7B LRMs. Extending our research larger and more powerful models could potentially uncover more efficient reasoning capabilities. Additionally, our evaluations and training are primarily based on mathematical reasoning tasks. Broadening our scope to include more diverse tasks, such competitive programming and multimodal reasoning, may enhance the generalizability and impact of our approach.
B Reproducibility
We provide our code in the http://annyyomoo.ooooeeeeeeee/// for the details reasoning efficiency frontiers and reasoning efficiency gap, and Sec. C for the implementation detials.
Please refer Sec.
C Implementation Details
Training Data.
For training data, integrate data from DeepScaleR Luo et al.
[205cc] and
AReaL
RL Lab
2025]. For 7B we directly use the training data of AReal R RL 7B RL Lab
2025
For 1.3B, adopt the mixture of training data from DeepScaleR Luo al. 2025c and
AReaL
RL Lab
[025 and remove duplicated problems.
We implement the training algorithm with the AReaL framework [RL Lab 2025 which supports SGLang [?] for rollout generation. Below we detail implementation details of each training algorithm.
Online RL Training. We use PPO as the default online RL algorithm. Following standard practices in RL training for LLM reasoning YYu et al. 2025b Hu et al. 2025]. we do not utilize value model and KL regularization. The default training setting and hyperparameters for PPO training are listed in Tab. 4
For REO-RI, and baseline methods, we do not carry out RL. training directly from the base LRM since it would lead to prolonged training and slower convergence. Instead, we perform further fine-tuning on the RL trained versions for both 1.5B and 7B settings. Specifically, we adopt AReaL-Boba- 1.5B RL Lab 2025 and Skywork-OR 1011-Mtth-77 [He et al. 2025) as the starting points for further fine-tuning. Following the cluster config in Tab. each experiment could finish within 48 hours.
Supervised Fine- Munn-Tunng. The default training configurations and hyperparameters for SFT are
listed in Tab. 5
Preference Learning. We implement SimPO [Meng et al. 2024 in the AReaL framework [RL Lab
2025 The default training configurations and hyperparameters for SimPO are listed in Tab.
D
Baselines
RL with Token Budgets. In our online RL training with token budget constraints, we control the maximum generation length during each training phase. Rather than fine-tuning the LRM directly on a fixed token budget, we adopt a progressive length-shrinking strategy. We begin training with 16K token budget. Once RL training at this level converges, we reduce the budget to 8K and continue training. This process repeated, halving the token budget each time, until we reach the minimum budget of 512 tokens. This staged approach enables the model to gradually adapt to shorter generation lengths while maintaining reasoning performance.
RL with Length Rewards. the "RL with Length Group Normalized Rewards" baseline Arora and Zanette 2025). for each question and the corresponding of sampled responses .'...... the reward of response y is computed as,
r(x, v²) = I[y² is correct) (1 ((v())
14

---

Table 4: Default training configurations and hyperparameters for PPO.
Training Configuration
Batch size (number of prompts)
512
Rollouts per prompt
16
Random seed
Cluster Config
8 H800 (for 1.5B) 16 8 H800 (for 7B)
PPO Parameters
PPO Minibatches
Clipping
0.2
Advantage normalization
True
Discount factor y
1.0
GAE A
1.0
Epochs
2.0
Optimizer Parameters
Optimizer
Adam
Learning rate
2.0 x 10-5
Weight decay
0.05
0.9
32
0.95
Adam
10-5
Gradient norm clipping
1.0
Learning rate scheduler
constant
Warmup steps proportion
0.001
Generation Parameters
Temperature
1.0
Top-p
1.0
Top-k
-1
Max prompt length
1024
Min generation length
Max generation length
24376 (for 1.5B) 32768 (for 7B)
where the function f normalizes ly"| according the lengths of correct responses and applies sigmoid function. Specifically,
[ MEAN(x)
A(y51) =
(TD(x)
where
MEAN(x)) =
E (-s). correclly]]]
STD(2) Var y a.). y. correct [byl]
In the "RL with Length-Harmone Harmonizing Rewards" baseline Luo al. 2025a for each question æ and the corresponding set of sampled responses y " the reward of response yf is computed as,
T (x) 1 + (IVy is correct) - (x))(x))
r(x, v²)
lyl
where Lre/ (z) is the average response length of the reference model when taking input and Are (x:) is the average accuracy of the reference model. In our implementations, set the models that serve as the starting point of RL training as the reference models.
different from the original paper that only implements single- step optimization with offline collected response prefixes, we implement the online RI. training version with dense rewards for MRT. For each question and the corresponding set of sampled
In the MRT baseline QQu al. 2025
15

---

Table 5: Default training configurations and hyperparameters for SFT.
Training Configuration
Batch size (number of prompt-answer pairs)
512
Cluster Config
16 8 H800
SFT Parameters
Epochs
10
Save Frequency Steps
100 True 32768
use_bfl6
Max Seq Length
Optimizer Parameters
Optimizer
Adam
Learning rate
1 x 10-5
Weight decay
0.05
31
0.9
32
0.95
Adam
10-5
Gradient norm clipping
1.0
Learning rate scheduler
constant
Warmup steps proportion
0.03
Table 6: Default training configurations and hyperparameters for SimPO.
Training Configuration
Batch size (number of preference pairs)
128
Cluster Config
16 x 8 H800
SimPO Parameters
Epochs
2
Save Frequency Steps
10
use ose_bf16
True 32768
Max Seq Length
SimPO Coefficient 3
1/2
SimPO Coefficient y
1.2/1.4
Optimizer Parameters
Optimizer
Adam
Learning rate
10-5 5 (for 1.5B) x 10-6 (or 7B)
Weight decay
0.05
31
0,9
82
0.95
Adam
1 x 10-5
Gradient norm clipping
1.0
Learning rate scheduler
constant
Warmup steps proportion
0.03
responses partition each response into several steps y (vi.-.. v.). In each training step, the model updated by computing the policy gradient for the following objective,
E Eg.y=(, a (12))
R(r,, Answer(mer [V v/T)
Answer(T + R(x, v)]l
where a is the weight for the overall accuracy and is set 0.2 in our experiments.
16

---

Supervised Fine- Tuning In SFT Shortest, we generate 16 outputs for each question in the training dataset. Then we select the correct response with the shortest length for each question to construct the SFT dataset. In TOTS we follow (Yang et al. 2025b) to prompt the LRM to generate responses with three different types of reasoning efforts. We strictly follow the prompts used in (Yang et al.. 2025b]. For each type of reasoning effort, we generate 16 responses. To construct the SFT dataset, the shortest correct response among all 48 responses are gathered.
Preference Learning. We adopt three strategies for constructing the preference datasets. In SimPOShoness, we adopt the responses generated for SFT and select the shortest correct response and the longest response as the preference pair for each question. In SimPOrops, we use the same preference construction strategy as SimPO Shertest but on the responses generated through TOPS, which contain reasoning traces with different reasoning efforts. Finally, SimPODAST. we again utilize the responses generated for SFT SFTsetttest but adopt different preference construction strategy. In the preference dataset of SimPOpAST: each pair falls into one the two cases: either contains two correct responses where the positive sample is much shorter than the negative sample, or contains two incorrect responses where the positive sample is much longer than the negative sample.
Other Methods We Have Tried. We have also experimented with Z1 Yu et al. 2025c that con- structs code-based reasoning traces, and L1 [Aggarwal and Welleck 2025 that fine-tunes the LRMs follow token budget instruction with RL. We find ZI having poor performance on mathematical reasoning tasks, demonstrating significantly lower accuracy to that of the base LRMs as illustrated in Tab. 7 For L.1, we find L1 mainly works under tight token budgets, i.e. less than 4K. When we extend the training approach of L1 to a larger context length, i.e. 24K for 1.5B models, we find it hard to make the LRM learning to follow strict token budget instructions through RL. Consequently, the resulted models, LI-Exact-24K- instruction, as shown Tab, g
and L1-Max 24K-1x--44-1551 could not follow the token budget
AIME24 Aecuracy (%)
MATH500 Accuracy (%)
GPQA
Method
Length
Length
Accuracy (%)
Length
Base LRM
31.5
16747.6
83.6
5633.1
44.6
10325.2
Z1
10.0
15106.0
63.6
4904.1
61.6
9004.1
Table 7: Result 21 on DeepSeek-RIl RI-Distill Qwen-
Yu et al.
2025c
AMC23 Accuracy (5)
AIME24 Accuracy (%)
AIME25
Instructed Token Budget
Length
Length
Accuracy (%)
Length
2048
63.9
20267.7
38.6
18067.4
27.2
19657.7
4096
63.5
20142.6
38.4
17961.3
26.9
19711.1
8192
62.9
20191.6
38.4
17868.7
26.8
19673.5
Table 8: Result of 1akacct244-11.555
E REO-RL
E.1
Implementing REO-RL
In the generation phase of REO-RL, there are two rounds of LRM generation. In the first round, multiple responses are generated for each question in the training batch. In the sec- ond round, to compute the length-constrained rewards for each of the responses and across all selected token budgets, we choose all truncated responses y.r., and apply a prompt enforce the LRM to gen- erate the final answer given incomplete reasoning traces, i.e. a = To( y.r. (The Final Answer is). We follow the prompt employed by [Fu et al. 2024 and Fu et al. 2025
Generation Phase.
17

---

Prompt for Forcing LRM Produce Answer
Oh, suddenly got the answer to the whole problem. **Final Answer** [[boxed
Dense Reward RL. REO-RL obtains dense rewards through forcing the LRM to generate answer under various token budgets. The objective of REO-RL is as follows,
N+1
REO-RL: LI = EEND
Bywna(--a)
E
Σ
cR(2, (.o. x,
where c; 1111-1111 for S and CN+1 Lun are the coefficient for the i-th token budget.
To perform policy update, compute the return for each section between consecutive token budgets I, and Li+1. For 1 s i s N, we compute,
N+1
Return; Σ
R,,, Answer(zo, V:L,))
Since we disable value model in the training process, the computed returns are then used directly as advantages for PPO loss computation. Specifically Return;(r, y) would be used to update the tokens yr
F
Reasoning Efficiency Frontiers & Reasoning Efficiency Gap
Reasoning Efficiency Frontiers. for points the estimated reasoning efficiency frontiers for DeepSsek-RI--Dssill---eeee 1.5 and DeepSeek-R1- -ineeekk----iiiee 11-Distill-ween11 respectively.
The two boxes below record the detailed lengths and accuracies
Reasoning Efficiency Frontier for DeepSeek-RR--Disiil---eeeee
[(0, 0.0722), (64, 0,0616), (128, 0.0626), (192, 0.0928), (256, 0. 0.1121), (320, 0. 02116,), (384,
0.1562), (448, 0.1884), (512, 0.2126). (576, 0.2219), (640, 0.2296), (704, 0.237), (768, 0.2486), (832, 0.2646), (896, 0.2809). (960, 0.2892), (1024, 0.3034), (2048, 0.4035), (3072, 0.4413), (4096, 0.471). (5120, 0.4851), (6144, 0.4941), (7168, 0.5088). (8192, 0.5195), (9216,
0.5273), (10240, 0.5339), (11264, 0.541), (12288, 0.5431), (13312, 0.5469), (14336, 0.5491),
(15360, 0.5506), (16384, 0.5517), (17408, 0.552), (18432, 0.5526), (19456, 0.5516), (20480,
0.5517), (21504, 0.5517), (22528, 0.5523), (23552, 0.5523), (24576, 0.5523), (25600, 0.5527),
(26624, 0.5527), (27648, 0.5527), (28672, 0.5527), (29696, 0.5527), (30720, 0,5527), (31744,
0.5527), (32768, 0.5527)]
Reasoning Efficiency Frontier for DeepSeek-RI -iaissek-RR--isstil---eeeee
[00, 0.0872). (64, 0.0892), (128, 0.0977), (192, 0. (256, 0.1512), (320. 0.1959), (384,
0.2522), (448, 0,2766), (512, 0.2896), (576, 0.2939), (640, 0.294), (704, 0.2944), (768, 0.2991), (832, 0.3155), (896, 0.3265), (960, 0.3418), (1024, 0.3889), (2048, 0.4885), (3072,
0.5345), (4096, 0.5742), (5120, 0.5926), (6144, 0.6128), (7168, 0.6293), (8192, 0.6486),
(9216, 0.662), (10240, 0.6742), (11264, 0.6872), (12288, 0.6936), (13312, 0.6977), (14336,
0.7023), (15360, 0.7039), (16384, 0.7066), (17408, 0.7123), (18432, 0.7147), (19456, 0.7167),
(20480, 0.7173), (21504, 0.7177), (22528, 0.7201), (23552, 0.7208), (24576, 0.7212), (25600,
0.7219), (26624, 0.7219), (27648, 0.7219), (28672, 0.7222), (29696, 0.7222), (30720, 0.7222),
(31744, 0.7225), (32768, 0.7221))
18

---

Evaluating Reasoning Efficiency Gap. To practically evaluate REG, instead of strictly fol-
lowing Eq. 5 obtain approximations of Eim Jopúnal and Σ -1=1 JDD 0, L) through numer- ical integration on set of token budgets respectively, in a similar way to Eq. g We select IN {L,,
f64i|0 i 16) U {1024i|1 S 16). Note that set Imma 16K instead of Lmax 32K to focus on the efficiency gap under lower token budgets.
G Additional Results
G.1 More Results Frontier LRMs
To enable a more comprehensive comparison as discussed in Sec 6.3 we also evaluated Claude Sonnet 3.7 under two different configurations. The detailed results are presented in Table Row shows the performance of Claude Sonnet 3.7 without the use of thinking mode, while Row reflects its performance under the low reasoning effort mode (with maximum token limit of 16,384 and 30% of the tokens allocated as budget tokens). Under both configurations, Claude Sonnet 3.7 produces significantly shorter responses, but this comes the cost of noticeable drop in accuracy.
Model
Length
Accuracy
Claude Sonnet 3.7 (Thinking)
17478.87
90.0%
Claude Sonnet 3.7 (Without Thinking)
906.01
54.15%
Claude Sonnet 3.7 (Low Reasoning Effect)
3491.14
67.62%
DeepSeek R1 Qwen3-4B
5156.39
98.2%
8631.61
95.8%
Qwen3-8B
8898.57
94.9%
Qwen3-32B
7755.99
97.2%
Vanilla RL - 7B
7732.11
93.5%
REO-RL (Exp) 7B (ours)
4524.02
93.5%
Table 9: Comparing the response length and accuracy with frontier LRMs.
G.2 Results on all Benchmarks
Generalization to Other Tasks
To demonstrate the generalizability of REG and REO-RL, we conduct cross-domain evaluations on coding tasks using models trained solely math data. Additionally, we apply the REG framework and REO-RL to coding by fine-tuning Qwen3-4B on DeepCoder data Luo et al. 2025b
1. Cross-Domain Evaluation (Math - Coding)
We evaluate models trained on math on two coding benchmarks, CodeContests and LiveCodeBench. construct cross-domain efficiency frontiers by truncating responses under different token budgets and allowing extra 512-token budget for python code generation. We then compute REG based on the cross-domain efficiency frontiers. 8 responses are generated for evaluation for each method.
Even when trained only on math, REO-RL achieves strong generalization to coding, significantly improving efficiency (lower REG) while maintaining accuracy.
Applying the REG REO-RL Coding (Qwen3-4B)
For frontier construction, we run RL with different token budgets to estimate the efficiency frontier. Regarding REO-RL, we run REO-RL (Exp) using exponentially spacing token budgets. To obtain length-con conetrinnntt rewards, we use the model to generate python codes from truncated responses with an additional generation budget 512 tokens. REO-RL (Exp) reduces REG and response length by 51. 14% and 32.49% on coding while maintaining the overall accuracy.
19

---

Method
Accuracy (%) f
Length +
REG
Base LRM
29.2
16757.1
2136.8
Vanilla RL
42.7
12829.1
1073.3
REO-RL (Exp) Coef=1
46.9
10624.2417.2%
197.29
REO-RL (Exp)
42.9
8780.3 131.6%
146.51
REO-RL (Oracle)
42.5
8443.8
134...8 134.2%
784.0126.9%
RL w. Token Budget=
17.3
1487. - 488.4%
3321. 1 1209.4%
RL w. Token Budget=2K
22.4 120.3
2423.9 181.1%
2610.3 143.2%%
RL w. Token Budget=4K
31.4 31.4 111.4
3991.0468.99
1490.4 138.9%
RL w. Len Group Norm. Rew.
44.4
10486.0 418.3%
689.8 133.7%
RL w. Len-Harmonizing Rew.
41.5 41.541.2 11.2
9119.2 128.9%
767.2
128.5%
MRT
44.0
10365.8 119.2%
610.1 J43.2%
SFT Shortest SFTTOPS
43.2
12898. 10.5%
1024.0
J4.6%
41.8,0.9
10823.9 115.6%
1057.7 11.4%
SimPO DAST
22.9 119.8
14029.4
2583.4 140.7%
SimPO Shortest
35.8 |6.9
7422. 142.1%
1170.0
SimPOToPS
15.6127.1
2647.4 479.4%
3639.77239.1%
Table 10: Results of 1.5B Models on AIME 2024
Method
Accuracy (%) 1
Length +
REG
Base LRM
23.5
16577.6
906.7
Vanilla RL
28.3
14109.4
749.2
REO-RL (Exp) Coef=1
32.9
10298.4.27.00
250.5 466.6%
REO-RL (Exp)
31.9
8630.5 138.8%
292.8 (90.8%
REO-RL (Oracle)
34.7
8288. 7 141.3%
232.2,69.0%
RL w. Token Budget=lK
13.5
1282.8 190.9%
1403.
187.3%
RL w. Token Budget=2K
17.5 110.8
1966.0 1966.0186.1% 186.1%
1105.9
147.6%
RL w. Token Budget=4K
26.1
3770.8 173.3%
474.5
474.5L36.7%% -36.7%
RL w. Len Group Norm. Rew.
30.7
10599.6 124.9%
553.5
126.1%
RL w. MRT
Rew.
31.673.2
8113.9 142.5%
316.0 157.8%
31.8+3.4
10496.2|25.66
458. 138.9%
SFT Shortest SFTTOPS
29.9
14221.8 870.8%
629.7 11..0%
30.7
11720.5116.99
549. 126.7%
SimPOpAST
19.4
11161.7 1266..72009%
1008.3p34.60
SimPO Shortest
26.5 11.9
7380.9 147.7%
672. 110.3%
SimPOToPe
18.3 18.30
2412.2 182.9%
1056.8 141.1%
Table 11: Results of 1.5B Models
AIME 2025
20

---

Method
Accuracy (%) f
Length +
REG
Base LRM
71.6
9956.8
1709.0
Vanilla RL
83.8
7968.0
1424.0
REO-RL (Exp) Coef=1
84.5
1990.8137.499
648.5 154.5%
REO-RL (Exp)
83.2
6205.6122.19
562.8 460.8%
REO-RL (Oracle)
84.4
4626.6 161.99
821.2
RL w. Token Budget=
65.6118.2
1111.3 186.1%
1354.244.9%
RL w. Token Budget=2K
73.1 110.7
1547. 180.6%
827.8
RL W. Token Budget=4K
80.9
2561.2
2561.2167.9% 167.9%
493.8
RL w. Len Group Norm. Rew.
82.7
5911.5 1911.5125.8% 125.8%
1097.2 122.99
RL w. Hermnninnn Rew.
85.1
3993.2 149.9%
648.5 154.5%
MRT
84.6,0.8
6233.6 6233.6121.8%
1051.8
SFT Shortest
82.5
7513.7 15.7%
1370.4 43.8%
SFT
82.7
6454. 1 119.0%
1283.5 49.9%
SimPOpAST
68.1 115.7
5955.5 425.3%
1514.4
SimPO Shortest
77.7 J6.2 4020.8 1449.%
1038.2,27.1%
SimPOToPP TOPS
58.5 18..3
1497.8 1497.8.81.29%
2179.7 153.1%
Table 12: Results of 1.5B Models on AMC 2023
Method
Accuracy (%) f
Length 4
REG
Base LRM
55.3
13062.1
2471.8
Vanilla RL
66.2
14264.7
1770.8
REO-RL (Exp)
64.0 J2.3
7671.5
849.3 152.0%
REO-RL (Oracle)
63.9
9348.6134.6%
1015.6142.69
RL w. Token Budget=1K
26.7139.6
1242.4 191.3%
6119.5 1549.69
RL w. Token Budget=2K
36.5
1929.6
1999.6686659%
4372.7 146.9%
RL W. Token Budget=4K
48.3
3480.9 175.6%
2458.7
1338..738.89% +38.8%
RL w. Len Group Norm. Rew.
64.2
10608. 125.6%
1353.4 4 123.6%
RL w. Len-Harmonizing Rew.
65.5 jo.7
9167.7 135.7%
1169.7 133.9%
MRT
66.1
9210.6 435.4%
1051.8 140.6%
SFT Shortest
67.7
13197.2 17.5%
1512. J14.6%
SFT SFTTOPS TOPS
66.0 Jo.2
13204. 17.44.1.7.4%
1609. 19.
SimPO DAST
60.1 j6.1 60.5
7500.5 J47.4%
1348.4
SimPO Shortest
9348. 134.5%
1138.9 -35.7%
Table 13: Results 7B Models on AIME 2024
21

---

Method
Accuracy (%) f
Length +
REG
Base LRM
39.7
14241.9
2055.6
Vanilla RL
52.9
16305.3
1502.3
REO-RL (Exp)
48.8
8361.1148.77
767.2 148.9%
REO-RL (Oracle)
49.0
9189.6443.69
628.2 158.22%
RL w. Token Budget=1K
19.7
1168.0 192.8%
4789.6,218.8
RL w. Token Budget=2K
27.6125.3
1831.6 188.8%
$354.8,123.55
RL w. Token Budget=4K
35.4
3345.2
2119.4441.19%
RL w. Len Group Norm. Rew.
50.8
11486.8 8 129.6%
851.4 451.4143.3%
RL w. Len-Harmnnizing Rew.
51.1 11.8
10394.6 136.2%
881.5 141.3%
MRT
50.5
10559.3 185.2%
711.4152.6%
SFT Shortest
53.610.7
1,2008 46.7%
1309.2.12.9%
SFTTOPS
52.0
15170.8
17.0% 17.0%
1349.6 110.2%
DAST
46.2 50.8 j2.1
7916.9 151.4%
1013.8 2013.8332.5% 132.5%
SimPO Shortest
10292.3 -36.9%
862.4 862.4.42.6%
Table 14: Results of 7B Models on AIME 2025
Method
Accuracy (%) t
Length |
REG
Base LRM
90.9
6177.3
1652.7
Vanilla RI
93.9
7259.8
1841.5
REO-RL (Exp)
93,4
4144.6
(144.6142.999 142.9%
1067. (42.1%
REO-RL (Oracle)
94.7
4444.
138.8%
960.9
RL w. Token Budget=1K
73.0
960.4 86.8%
2098. 119.91
RL w. Token Budget=2K
84.3
1427.5
180.9%
906.7
906.715u.856
150.8%
RL W. Token Budget=4K
89.1
2109.2 770.9%
708.5
RL w. Len Group Norm. Rew.
94.5
4788.9 134.0%
1181.3_as
_35.9%
w.
Len-Harmoniiing Rew.
94.5
4307.9 110.7%
1027.8
L44.2%
MRT
93.8
4986.6
131.3%
1242.6 53255%
SFT Shortest SFT TOPS
93.610.3
6728.2
1755.5
93.3
6272.5113.66
1653.4
_10.2%
SimPO
92.0 93.4
405 J44.2%
1050.1
L43.0%
SimPO Shortest
4793.
134.0%
1276.0 330%%
Table 15: Results of 7B Models on A.C. 2023
22

---

Method
Accuracy (%) f
Length +
REG
Vanilla RL
66.2
14264.7
1770.8
Base LRM
55.3
13062.1
2471.8
REO-RL (Exp) Coef=1
62.6.,3.6
7218.5
129..5149.499 149.4%
156.5.667.3%
REO-RL (Exp) - N=10
64.0
9352.7 134.4%
1088.9 138.5%
REO-RL (Linear)
63.5
8982.0
0tt.00033.000 137.0%
1097.8
1097.8438.09
438.0%
REO-RL Sntestion-Speciiie Oracle)
63.9
8407.8 141.1%
831. 831.4 L53.0%
REO-RL (Oracle)
63.9
9348.6134.59)
1015.6142.6%
REO-RL (Exp)
64.0
7671.5.44.229
199.0%
Table 16: Results of Ablation Study on -i-iskk--------iiiile--------- on AIME 2024
Method
Accuracy (%) 1
Length +
REG
Vanilla RL
52.9
16305.3
1502.3
Base LRM
39.7
14241.9
2055.6
REO-RL (Exp) Coef=1
47.2
7525.9 453.8%
611.1 (99.1%%9.%%.
REO-RL (Exp) - -=10
48.9
9804.9 439.9%
927.0
138.3%
REO-RL (Linear)
49.9
9001.6
100116.44.899 444.8%
715.1 452.4%
REO-RL (Question-Specifee Oracle)
51.4
9298.5 143.0%
401.2 773.3%
REO-RL (Oracle)
49.0
9189.6 1189.6443.6% 443.6%
628.2
J58.2%%
REO-RL (Exp)
48.8
8361. 448.7%
767.2
167.2448.99
Table 17: Results of Ablation Study DeepSiek-RI--Dittill-----ae on AIME 2025
Method
Accuracy (%) f
Length +
REG
Vanilla RL
93.9
7259.8
1841.5
Base LRM
90.9
6177.3
1652.7
REO-RL (Exp) Coef=1
93.1
3736.4
REO-RL (Exp) - N=10
94.1+0.2
4779.6634.2
1027.3 144.2%
REO-RL (Linear)
93.90.0
4290.9
140.9%
1093.9 440.6%
REO-RL Prrestion-Specffii Oracle)
93.4
4230.6 6 141.7%
970.6 L47.3%
REO-RL (Oracle)
94.7
4444.1
960.9 960.9L47.88%
138.8%
REO-RL (Exp)
93.4
4144.6142.99%
1067.1142.19%
Table 18: Results of Ablation Study on
on AMC 2023
Table 19: RinnsSclkRR--isisill--eeeeee//e Results (Trained on Math, Eval on Coding)
Method
Acc.
Len.
REG
Vanilla RL
13.55
15593.15
881.45
REO-RL (Oracle)
14.76
13064.89
556.57
REO-RL (Exp)
14.83
14047.32
510.56
RI w. Len Group Norm. Rew.
13.2
14328
996.29
RL w. Hen-Harmnn Rew.
14.34
12670.71
646.61
RL w. Token Budget=4K
13.35
6144.13
876.90
23

---

Table 20: DeepSeck-RI-Disilll-aaaaaa Results (Trained Math, Eval Coding)
Method
Acc.
Len.
REG
Vanilla RL
33.2
14722.48
849.27
REO-RL (Oracle)
32.29
10246.18
747.14
REO-RL (Exp)
31.98
9143.80
789.20
RL Len Group Norm. Rew.
32.48
13183.96
839.00
RL w. Lan-Haamnnizing Rew.
31.49
13240.02
1070.17
RL Token Budget=4K
29.40
4968.69
1353.91
Table 21: Qwen3-4B Results on Coding
Method
Acc.
Length
REG
Base LRM (Qwen3-4B)
41.14
16850.23
514.86
REO-RL (Exp)
41.03
11375.06
251.55
RL w. Token Budget=8K
28.78
7059.58
2572.08
RL w. Token Budget=16K
32.58
10443.31
1961.00
24

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We believe our introduction and the abstract sections both clearly state the contributions made in this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
25

---

Justification:
Please refer to Sec. 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide full guideline for reproducing the results of our experiment results in Appendix. C Appendix. and Appendix.
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide our training code through anonymous link in Appendix. Our training data are gathered from open-sourced projects, which we have already properly cited. We also provide the estimated reasoning efficiency frontiers, that are necessary for computing the introduced metric REG. in Appendix. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //ipps.cc/pubicc/uiieee
CooeeSumiiss ssooiiycyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
27

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Please refer to Sec. 6.1 and Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We did not include error bars for end-to-end experiments because this paper includes large volume of expensive experiments. We present results within single trial across different settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Appendix,
Guidelines:
28

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: N/A
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper investigates training algorithms, which has limited social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]

---

Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: ensure existing assets are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, thcors thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the code for implementing REO-RL and the exact numerical value of our estimated efficiency frontiers. We will also release necessary models in the future.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
30

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This work does not use LLMs as any important, original, or non-standard components
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy httpp://reerrpppcccooooofeeerrre for what should or should not be described.
31


Reviews:
[{"review_id": "a2EZtAPOQW", "reviewer": "Reviewer_s9q1", "summary": "This paper addresses the problem of reasoning efficiency in Large Reasoning Models (LRMs), which often generate excessively verbose reasoning traces despite their strong problem-solving capabilities. The authors introduce the concept of reasoning efficiency frontiers - empirical upper bounds on the optimal trade-off between accuracy and response length - and propose the Reasoning Efficiency Gap (REG) metric to quantify how far current methods are from optimal efficiency. The main contribution is REO-RL (Reasoning Efficiency Optimization with Reinforcement Learning), an algorithm that optimizes reasoning efficiency by targeting a sparse set of token budgets using numerical integration approximation. Experiments on mathematical reasoning benchmarks (AMC23, AIME24, AIME25) with DeepSeek-R1-Distill-Qwen models (1.5B and 7B) show that REO-RL reduces the efficiency gap by 74.5% and 64.2% respectively compared to vanilla RL.", "questions": "Please see the weaknesses. Addressing these issues will increase the clarity of the paper and help the readers better understand the main contribution of the paper.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Novel Framework and Metrics: The introduction of reasoning efficiency frontiers and the REG metric provides a principled way to evaluate and compare reasoning efficiency methods. \n- Solid Experimental Design: The paper conducts comprehensive experiments with diverse fine-tuning approaches. The evaluation covers multiple baselines and includes proper ablation studies.\n- Practical Algorithm: REO-RL cleverly approximates the computationally expensive full efficiency objective using numerical integration over strategically selected token budgets. The exponential spacing strategy achieves <1% approximation error with only 5 budgets, making the approach practical.\n- Strong Empirical Results: The model finetuned with REO-RL demonstrates substantial REG reductions and competitive performance against frontier LRMs like Qwen3 and Claude Sonnet 3.7 in terms of reasoning conciseness.\n\nWeaknesses\n- Confusion in math notations: In section 4.1, when defining $a = Answer(π_\\theta, x, y)$, the $L$ term is not shown as the input.  however, in equation (2), the $L$ term is shown as the input for $Answer()$. This may cause confusion while reading. Also, in definition 4.1 (equation (4)), as far as I am concerned, the reasoning efficiency frontier should be \"a set\" of best achievable reward $J$. In this case, maybe a notation like {$\\hat{J}_{optimal}(D, \\hat{\\theta}, L) \\mid L \\in [1, Lmax]$} will help the reader better understand the technical details.\n- Confusion in Table 1: Table 1 contains the main result of the paper. However, a lot of information (diverse baselines with different budgets, token length, accuracy across baselines, etc.) is contained in a single table. This could cause the readers unable to fully capture the main takeaways from the table. Maybe dividing the table into several sub-tables may be helpful (one table one main-takeaway).", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable review to our work. These valuable feedback could help us better improve the clarity of our work.\n\n**[W1] Confusion in Math Notations**\n\nWe appreciate the reviewer’s helpful suggestions regarding notation clarity. We have accordingly revised the mathematical notations to ensure consistency and readability. Specifically, following the suggestion of the reviwer, \n- we use $a=\\text{Answer}(\\pi_\\theta, x, y_{:L})$ to denote the answer for a truncated response $y_{:L}$, ensuring consistency with Eq.(2).\n- we use {$(L, \\hat{J}_{optimal}(D, \\hat{\\theta}, L)))$ | $L\\in[1, L{max}]$} to denote the reasoning efficiency frontier. \n\n\n\n**[W2] Confusion in Table 1**\n\nWe agree with the reviewer that splitting Table 1 into more focused visualizations will enhance clarity. Specifically, in the revised version, we have included:\n- A figure comparing REG, showing that REO-RL consistently outperforms baselines in reasoning efficiency.\n- A line plot of accuracy vs. token budget, showing that Vanilla RL does not consistently improve accuracy across token bugets.\n- A visualization for accuracy, response length, and REG, illustrating how REG effectively balances trade-off between accuracy and response length.\n\nWe believe these changes would greatly improve the clarity of our work. We deeply appreciate the reviewer’s time and valuable suggestions."}, {"review_id": "O0KybkrDLu", "reviewer": "Reviewer_VFri", "summary": "This paper tackles the problem of reasoning efficiency in Large Reasoning Models (LRMs). The authors observe that LRMs like DeepSeek-R1 often produce unnecessarily long reasoning traces - sometimes hundreds of tokens for simple problems. To address this, they introduce \"reasoning efficiency frontiers\" - empirical upper bounds showing the best accuracy achievable at each token budget. They then propose REO-RL, a reinforcement learning method that optimizes models to work well across multiple token budgets simultaneously. The key insight is using numerical integration to approximate performance across all budgets while only training on a small set (5 budgets).", "questions": "Have you tested on other domains like coding or general reasoning? Math problems might have specific properties that make this approach work well. Maybe presenting results such as puzzle or GPQA would be nice,w", "limitations": "n/a", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1.The paper identifies a real issue, the LRMs waste tokens. The \"reasoning efficiency frontier\" concept provides a concrete way to measure how far models are from optimal.\n2.They test multiple fine-tuning approaches (RL with token budgets, length rewards, preference learning) to construct the frontiers. This gives a comprehensive view of current methods.\n3.Only tested on math problems (AMC/AIME). It's unclear if this generalizes to other reasoning tasks like coding or general QA.\n4. The \"optimal\" frontiers depend entirely on which methods they tried. If someone finds a better fine-tuning approach tomorrow, all the REG measurements become outdated.\n5. The paper introduces many metrics (REG, length-constrained rewards, etc.) but it's not always clear which matter most for real applications.\n\nMajor concern is that the paper is not very well-written, not clear", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback, which has helped us improve the clarity and scope of our work. Below, we address each concern in detail.\n\n## Generalization to Other Tasks\n\n> \"Only tested on math problems (AMC/AIME). It's unclear if this generalizes to other reasoning tasks like coding or general QA\"\n\nTo demonstrate the **generalizability of REG and REO-RL**, we conduct **cross-domain evaluations on coding tasks** using models trained solely on math data. Additionally, we apply the **REG framework and REO-RL** to coding by fine-tuning **Qwen3-4B** on DeepCoder data[1].\n\n**1. Cross-Domain Evaluation (Math -> Coding)**\n\nWe evaluate models trained on math (as in the paper) on two coding benchmarks, CodeContests and LiveCodeBench. We construct **cross-domain efficiency frontiers** by truncating responses under different token budgets and allowing an extra 512-token budget for python code generation. We then compute REG based on the corss-domain efficiency frontiers. 8 responses are generated for evaluation for each method.\n\nEven when trained only on math, **REO-RL achieves strong generalization to coding**, significantly improving efficiency (lower REG) while maintaining accuracy.\n\nDeepSeek-R1-Distill-Qwen-1.5B Results:\n\n| Method | Acc. | Len.$\\downarrow$ | REG $\\downarrow$ | \n|----|-|-|-|\n| Vanilla RL | 13.55 | 15593.15 | 881.45 |\n| **REO-RL (Oracle)** |14.76 | 13064.89| **556.57** |\n| **REO-RL (Exp)** | 14.83 | 14047.32 | **510.56** |\n| RL w. Len Group Norm. Rew.  | 13.2 |14328\t|996.29|\n| RL w. Len-Harmonizing Rew.|14.34|12670.71|646.61|\n|RL w. Token Budet=4K|13.35|6144.13|876.90|\n\nDeepSeek-R1-Distill-Qwen-7B Results:\n| Method | Acc. | Len.$\\downarrow$ | REG$\\downarrow$ | \n|----|-|-|-|\n|Vanilla RL|33.2|14722.48|849.27|\n|**REO-RL (Oracle)**|32.29|10246.18|**747.14**|\n|**REO-RL (Exp)**|31.98|9143.80|**789.20**|\n|RL w. Len Group Norm. Rew.|32.48|13183.96|839.00|\n|RL w. Len-Harmonizing Rew.|31.49|13240.02|1070.17|\n|RL w. Token Budet=4K|29.40|4968.69|1353.91|\n\n\n**2. Applying the REG & REO-RL to Coding (Qwen3-4B)**\n\nFor frontier construction, due to time constraints, we run **RL with different token budgets** to estimate the efficiency frontier. \n\nRegarding REO-RL, we run REO-RL (Exp) using *$N=5$ exponentially spacing token budgets*. To obtain length-constrained rewards, we use the model to generate python codes from truncated responses with an addtional *generation budget of 512 tokens*. **REO-RL (Exp) reduces REG and response length by 51.14% and 32.49% on coding** while maintaining the overall accuracy.\n\n\n*Qwen3-4B Results on Coding:*\n| |Acc.| Length$\\downarrow$|REG $\\downarrow$|\n|-|-|-|-|\n|Base LRM (Qwen3-4B)|41.14|16850.23|514.86|\n|REO-RL (Exp)|41.03|11375.06|**251.55**|\n|RL w. Token Budget=8K|28.78|7059.58|2572.08|\n|RL w. Token Budget=16K|32.58|10443.31|1961.00|\n\n\n[1] DeepCoder: A Fully Open-Source 14B Coder at O3-mini Level. https://huggingface.co/agentica-org/DeepCoder-14B-Preview\n\n## Potential Outdated Issue of REG\n\n> \"The 'optimal' frontiers depend entirely on which methods they tried. If someone finds a better fine-tuning approach tomorrow, all the REG measurements become outdated. \"\n\nWe appreciate the reviewer’s insightful observation about the REG metric. Indeed, as with many performance metric in deep learning, REG’s frontiers may require updates as methods improve. We highlight two key points here:\n- Our large-scale benchmarking reveals a **fundamental gap that no existing method can fully match the estimated frontier**. This suggests that surpassing our current frontier would represent a meaningful advance in the field.\n- We present a comprehensive framework for estimating the efficiency frontiers. **New models or new methods can be seamlessly incorporated into the framework to update the frontiers and REG measurement**, keeping REG up-to-date.\n\n## Clarity on Metrics (REG vs. Length-Constrained Rewards)\n\n>  \"The paper introduces many metrics (REG, length-constrained rewards, etc.) but it's not always clear which matter most for real applications.\"\n\nWe first would like to clarify the distinction:\n- **REG (Reasoning Efficiency Gap)** proposed as **a metric for evaluating reasoning efficiency**. It measures how close a model is to the optimal accuracy-length trade-off.\n- Length-constrained rewards are not metrics but used as rewards in REO-RL training, and for frontier estimation. \n\nImportantly, our results also suggest that **REG captures the trade-off between accuracy and generation length.** As shown in Table 1, methods that achieve low accuracy or long generation length tend to have high REG. In practice, REG can be used as an effective metric for measuring reasoning efficiency for any fine-tuned model.\n\n## Improving Clarity\n\nWe thank the reviewer for pointing out the clarity issue. We have taken the following steps to enhance the presentation:\n\n- **Training Efficiency Analysis**: We have added a detailed comparison with other methods, including a theoretical analysis of REO-RL’s computational cost, to explicitly demonstrate its training efficiency. (See our response to Reviewer 7ETN for details.)\n- **Frontier Construction Details**: We now include comprehensive statistics on the specific methods and models used for constructing the frontiers. (See our response to Reviewer 7ETN for details.)\n- **Improved Table Visualization**: As suggested by Reviewer s9q1, we have split Table 1 into sub-figures and sub-tables to improve readability and highlight key takeaways.\n\nWe thank the reviewer again for the thoughtful comments. Please let us know if additional clarifications or revisions would be helpful."}, {"review_id": "Rwgdd7Yz7g", "reviewer": "Reviewer_7ETN", "summary": "This paper tackles the problem of verbose reasoning in Large Reasoning Models (LRMs). It first introduces Reasoning Efficiency Frontiers to define the empirical upper bound of the accuracy-length trade-off for any token budget. It then proposes the Reasoning Efficiency Gap (REG), a single metric that quantifies a model's deviation from this frontier. The core contribution is REO-RL, a novel reinforcement learning algorithm designed to minimize this gap. By leveraging sparse token budget sampling and numerical integration, REO-RL efficiently approximates the full-budget optimization problem, leading to substantial reductions in computational cost.", "questions": "The paper states on lines 136-137 that you \"construct an empirical reasoning efficiency frontier by fine-tuning a diverse set of models using existing approaches.\" Could you please provide a more detailed explanation of how this frontier, which is used to estimate J_optimal, is constructed?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The proposed REG metric effectively unifies model accuracy and efficiency into a single, coherent evaluation standard. The REO-RL algorithm, through sparse budget sampling, offers a computationally efficient optimization method.\n2. The work demonstrates a high degree of experimental rigor, supported by a detailed appendix and the open-sourcing of code and data to ensure reproducibility.\n3. The paper is clearly written with a smooth logical flow, and its core arguments are well-supported by intuitive figures and diagrams.\n\nWeakness\n1. The method's effectiveness is only demonstrated on mathematical reasoning tasks (AMC/AIME). Its applicability to other domains, such as programming, remains unverified, and the paper should either include cross-domain experiments or discuss the transfer potential of the REG framework.\n2. Experiments are limited to relatively small models (1.5B/7B), and the paper does not assess the scalability of REO-RL on larger models (>10B). An analysis of its performance on larger-scale LRMs is necessary to confirm its broader utility.  My primary concern with this paper lies in the choice of models used for experimentation. The study relies on 1.5B and 7B models, which have limited practical value in real-world applications and are not representative of the state-of-the-art Large Reasoning Models currently in use. It is well-understood that the reasoning processes and emergent capabilities of models at this smaller scale can differ substantially from those in the 30B to 230B parameter range.\n3. The paper does not quantify the additional computational overhead introduced by REO-RL, which requires generating multiple truncated responses for reward calculation. A comparative analysis of training efficiency is needed to provide a complete picture of the method's cost.", "was_revised": true, "final_justification": "The author's supplementary experiments have addressed some key shortcomings of the paper to some extent, and I will raise my rating accordingly. Considering the maturity and completeness of the paper, the author needs to make significant adjustments to the experimental section in the final submission version.", "rebuttal": "We sincerely appreciate the reviewer’s time and thoughtful feedback, which has helped us strengthen our work. Below, we address the key concerns raised:\n\n1. **Cross-Domain Generalization (Coding Tasks)**\n    - We evaluate the math-trained models on coding tasks to show the generalizability of REO-RL.\n    - We further include additional results **applying the REG framework and REO-RL to Qwen3-4B on coding** to further demonstrate the adaptability across domains.\n2. **Scalability to Larger Models**\n    - To address the lack of evaluation on advanced large reasoning models, we **apply REO-RL to Qwen3-32B**, showing consistent improvements in efficiency-performance trade-offs\n3. **Training Efficiency of REO-RL**\n    - We provide an **empirical comparison** showing that *training efficiency of REO-RL is competitive with baselines.*\n    - We provide a **theoretical analysis** showing why *REO-RL is training-efficient.*\n4. **Efficiency Frontier Details**\n    - We included dedicated description including the specific training algorithms and training configuraitons, and construction method for the frontiers.\n\nWe hope this clarifies our methodology and adequately addresses the reviewer's concerns. We’re grateful for the opportunity to improve our work and welcome further discussion.\n\n## [W1] Cross-Domain Generalization (Coding Tasks)\n\n\nTo demonstrate the **generalizability of REG and REO-RL**, we conduct **cross-domain evaluations on coding tasks** using models trained solely on math data. Additionally, we apply the **REG framework and REO-RL to coding** by fine-tuning **Qwen3-4B** on DeepCoder data[1].\n\n**1. Cross-Domain Evaluation (Math -> Coding)**\n\nWe evaluate models trained on math (as in the paper) on two coding benchmarks, CodeContests and LiveCodeBench. We construct **cross-domain efficiency frontiers** by truncating responses under different token budgets and allowing an extra 512-token budget for code generation. We then compute REG based on the cross-domain efficiency frontiers. 8 responses are generated for evaluation.\n\nThough only trained on math, **REO-RL achieves strong generalization to coding**, significantly improving efficiency (lower REG) while maintaining accuracy.\n\nDeepSeek-R1-Distill-Qwen-1.5B:\n\n|Method|Acc.|Len.$\\downarrow$|REG$\\downarrow$| \n|----|-|-|-|\n|Vanilla RL|13.55|15593.15|881.45|\n|**REO-RL (Oracle)**|14.76|13064.89|**556.57**|\n|**REO-RL (Exp)**|14.83| 14047.32 |**510.56**|\n|RL w. Len Group Norm. Rew.|13.2 |14328|996.29|\n|RL w. Len-Harmonizing Rew.|14.34|12670.71|646.61|\n|RL w. Token Budet=4K|13.35|6144.13|876.90|\n\nDeepSeek-R1-Distill-Qwen-7B:\n|Method|Acc.|Len.$\\downarrow$|REG$\\downarrow$| \n|----|-|-|-|\n|Vanilla RL|33.2|14722.48|849.27|\n|**REO-RL (Oracle)**|32.29|10246.18|**747.14**|\n|**REO-RL (Exp)**|31.98|9143.80|**789.20**|\n|RL w. Len Group Norm. Rew.|32.48|13183.96|839.00|\n|RL w. Len-Harmonizing Rew.|31.49|13240.02|1070.17|\n|RL w. Token Budet=4K|29.40|4968.69|1353.91|\n\n\n**2. Applying the REG & REO-RL to Coding (Qwen3-4B)**\n\nFor frontier construction, due to time constraints, we run **RL with different token budgets** to estimate the efficiency frontier. \n\nRegarding REO-RL, we run REO-RL (Exp) using *$N=5$ exponentially spacing token budgets*. To obtain length-constrained rewards, we use the model to generate codes from truncated responses with an addtional *generation budget of 512 tokens*. **REO-RL (Exp) reduces REG and response length by 51.14% and 32.49% on coding** while maintaining the overall accuracy.\n\n*Qwen3-4B Results on Coding:*\n|Method|Acc.|Len.$\\downarrow$|REG$\\downarrow$|\n|-|-|-|-|\n|Base LRM (Qwen3-4B)|41.14|16850.23|514.86|\n|REO-RL (Exp)|41.03|11375.06|**251.55**|\n|RL w. Token Budget=8K|28.78|7059.58|2572.08|\n|RL w. Token Budget=16K|32.58|10443.31|1961.00|\n\n[1] DeepCoder: A Fully Open-Source 14B Coder at O3-mini Level. https://huggingface.co/agentica-org/DeepCoder-14B-Preview\n\n## [W2] Scalability to Larger Models\n\nWe apply REO-RL (Exp) to an advanced LRM, Qwen3-32B using N=5 token budgets for training. Each model is evaluated on AIME24/AIME25/AMC23 with 32 responses. The results show that **REO-RL can reduce response length of Qwen3-32B by 10.03% while maintaining the accuracy**,indicating that *REO-RL can effectively scale to larger models*.\n\n||Acc.|Length|\n|-|-|-|\n|Qwen3-32B|82.83|11961.23| \n|REO-RL (Exp)|83.47|10757.33| \n\n## [W3] Computational Cost Analysis of REO-RL\n\nWe thank the reviewer for raising concerns about training efficiency. Here we provide training efficiency comparison with baselines, and a theoretical analysis for the cost of REO-RL. \n\nTo start with, REO-RL generates a response and then computes rewards by,\n1. generate a response $y\\sim\\pi_\\theta(\\cdot|x)$\n2. For each truncated response $y_{:L_i}$, an answer $a_i$ is generated, i.e. $a_i\\sim \\pi_\\theta(\\cdot|x, y_{:L_i},\\text{[The Final Answer is]})$, with a budget of 10 tokens. Reward is then computed by matching the answer with the ground-truth answer.\n\n### 1. Training Efficiency Comparison\n\n Below, we provide empirical evidence showing that REO-RL is computationally efficient:\n\n(a) **Low Overhead from Answer Generation** \nREO-RL uses *a small number of token budgets $N=5$* and *short answers*, i.e. $|a_i|\\le 10$. The per-question latency breakdown shows that **answer generation adds only 6.4% overhead relative to response generation**:\n\n| Generating a response y | Generating Answers a[1]..a[N] |\n|---|---|\n|  110.63 $\\pm$63.80 s| **7.53$\\pm$5.02s**|\n\n(b) **Faster Training Than Baselines**\n\nREO-RL trains faster than Vanilla RL and other RL methods with length rewards because *REO-RL encourages shorter responses.*\n\n| Method                          | Avg. Time per Training Step (s) | Avg. Length (Test) | Avg. Acc. (Test) |\n|---------------------------------|------------------------|-------------|-----------|\n| Vanilla RL                      | 1147.39                | 12609.9     | 71.0      |\n| **REO-RL (EXP)**                 | **877.36**                 | 6725.7      | 68.7      |\n| **REO-RL (Oracle)**                 | **873.26**                 | 7660.8      | 69.2      |\n| RL w. Length Group Norm. Reward | 938.58                 | 8961.3      | 69.8      |\n| RL w. Length-Harmonizing Reward | 903.93                 | 7956.8      | 70.4      |\n| RL w. Token Budget=4K           | 400.13                 | 2978.4      | 57.6      |\n\nSpecially, RL w. Token Budget excessively shrinks length at the cost of severe accuracy drop,also having fast training speed. \n\n### 2. Theoretical Cost Analysis\n\nFollowing [1] and [2], we analyze the computation cost for a single input $x$. Suppose the LRM is a transformer with $P$ parameters, KV size $D$ and GQA ratio $r$. Let $L_{in}$ be the length of the input, $L_{out}$ be the length of generated response $y$ and  $L_{ans}$ be the maximum lengths of answers generated from truncated responses. The computation in REO-RL consists of three parts:\n\n- Prefill for Input $x$: $2PL_{in}+2DL_{in}^2$\n- Decoding Response $y$: $2PL_{out}+r(2L_{in}+L_{out})L_{out}D$\n- Decoding Answers $a_1,\\cdots,a_N$: $2PNL_{ans}+r(2L_{in}+2L_{out}+L_{ans})NL_{ans}D$\n\n**Why REO-RL is Efficient**:\n\n1. **Negligible Answer Generation Cost.** The input length $L_{in}$ is small ($\\le 500$) and the response length $L_{out}$ could span from 5K to 32K. In REO-RL, we use $N=5$ and $L_{ans}\\le 10$. Therefore the computation cost is dominated by decoding $y$ since $L_{out}\\gg NL_{answer}$.\n\n2. **KV Cache Reuse:** The KV cache of $x$ and $y$ could be reused during answer generation. For more details on KV cache, please refer to [SGLang Blog](https://lmsys.org/blog/2024-01-17-sglang/).\n\n[1] Kinetics: Rethinking Test-Time Scaling Laws. https://arxiv.org/pdf/2506.05333\n\n[2] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. https://arxiv.org/abs/2407.21787\n\n\n## [Q1] Further Details on Empirical Reasoning Efficiency Frontiers\n\nTo ensure representative coverage of the best methods across varying token budgets, we run RL training with **3 classes of algorithms**, covering **8 different distinct algorithms** and **15 unique training configurations** in total. Each training configuration required substantial computational effort to ensure convergence, with individual run at least 2K and 4K GPU hours for each run in the 1.5B and 7B scales, respectively. For each run, a set of checkpoints are sampled for evaluation. In total, **186 and 218 models in 1.5B and 7B settings**, respectively, are evaluated for constructing the efficiency frontiers.\n\nFor each model $\\pi_{\\theta_i}$, we first generate $32$ responses under a 32K token budget and then truncate each response under token budgets $\\{32,64,\\cdots,512,1K,2K,\\cdots,32K\\}$. We then generate answers from the truncated responses to compute the length-constrained rewards at different token budgets. Finally, the efficiency frontier for each benchmark is derived by selecting the **maximum accuracy across all models** for every token budget. \n\nThe table below summarizes the algorithm classes, configurations, and model counts:\n\n| Algorithm Class         | Specific Algorithms and Training Configurations                                                               | # of Models (1.5B) | # of Models (7B) |\n|-------------------------|-------------------------------------------------------------------------------------|--------------------|------------------|\n| RL w. Token Budget      | Token Budget $\\in \\{512, 1K, 2K, 4K, 8K, 16K, 32K\\}$                                    | 52                 | 118              |\n| RL w. Length Reward     | MRT; RL w. Length Group Norm. Reward ($\\alpha\\in\\{0.1, 0.2\\}$); RL w. Length-Harmonizing Reward ($\\lambda \\in \\{1, 2\\}$) | 83                 | 70               |\n| Preference Learning     | SimPO(Shortest, TOPS, DAST)        | 51                 | 30               |\n| Others | Base LRM, Vanilla RL | - | - |\n\n\nFurther details about the algorithms and frontiers can be found in **Line 141-151** and **Appendix C**."}, {"review_id": "wMIeF2ykCi", "reviewer": "Reviewer_uvtF", "summary": "This paper addresses the overthinking behavior in reasoning models which often produce verbose outputs. The authors introduce Reasoning Efficiency Frontiers which is an empirical upper bounds on the accuracy-length trade-off, and define Reasoning Efficiency Gap (REG) to measure deviation from these frontiers. To optimize REG, they propose REO-RL, a reinforcement learning algorithm that approximates a dense multi-budget reward objective using a small set of token budgets selected via either oracle or exponential strategies. Experiments on three math reasoning benchmarks with 1.5B and 7B models show REO-RL improves reasoning efficiency over baselines.", "questions": "- Could authors include an algorithm pseudo code for REO-RL?\n- Could authors provide more details on how the reasoning efficiency frontier is constructed? Does the author train RL with three different objectives, evaluate on each token budget, and select the best one over the three methods?\n- Since exp with coef = 1 outperforms oracle method, I wonder what would be the actual use for such an empirical frontier.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The empirical results are strong, with comprehensive comparisons across baselines, model sizes, and ablations.\n- The paper is well-structured.\n- REG metric offers a useful lens for benchmarking reasoning efficiency.\n- The idea of optimizing a sparse approximation to a dense budgeted reward objective is novel in this context.\n\nWeaknesses:\n- The entire framework relies on empirical frontiers derived from a limited set of fine-tuned models. There is no theoretical guarantee that these frontiers are close to optimal, which weakens the central premise. \n- Prompting the model to generate the final answer directly from a truncated trace to be used for intermediate rewards is not justified, since the objective in Equation 2 is on $y$ sampled from $\\pi$ rather than the truncated response. (Maybe the authors are actual training on the truncate trace for each truncation under different token budget. It would be helpful to include an algorithm pseudo code.)\n- The complexity of the algorithm is higher than the baselines as it needs to regenerate for N times after each generation ends.\n- The details on how the reasoning efficiency frontier is constructed are missing.\n- Empirically the oracle method is close to exp with coef = 1. Exp with coef = 1 is essentially just optimizes the mean rewards for different token budgets which defeats the purpose of optimizing with an oracle frontier. The frontier does not seem to be useful except for being a metric.", "was_revised": true, "final_justification": "The frontier estimation and training efficiency issues have been resolved. The estimated frontier helps to establish an exponential budget selection.\n\nThe issue remain unresolved is the supplied pseudo-code appears to be misaligned with the stated objective. According to Equation (6), we aim to train a model that performs optimally at every token budget (which optimally would be the red dashed line in Figure 2). Yet both the pseudo-code and Equation (9) train only on the full response y, but not on any truncated responses y_trunc. As a result, the model may encounter out-of-distribution issues when generating under smaller budgets, since it has never been trained to predict directly from y_trunc.\n\nI hypothesize that the underlying objective of the algorithm is to encourage the model to generate important reasoning steps as early as possible. For example, the model generates two responses, one where it first interprets the question for 1k tokens and solves it for 1k tokens, and the other generation interprets and solves together in 2k tokens. Then, the second generation would have a higher chance that, if you truncate at 1k token budget, it will directly produce the right answer. Therefore, the model would learn to generate important reasoning steps as early as possible, which might be the reason that REO-RL is able to outperform other token efficiency algorithms. The current draft feels that there is a disconnection between the empirical frontier and the proposed algorithm, as the algorithm does not directly optimize to improve towards the frontier. Therefore, I have a final rating of 3.", "rebuttal": "We sincerely appreciate the reviewer's time and feedback. Below, we address the key concerns point-by-point:\n\n1. **Large-Scale Efficiency Frontier Estimation**\n    - To rigorously estimate optimal reasoning efficiency, we conducted **large-scale RL experiments** spanning **8 algorithms and 15 training configurations**, resulting in **180+ and 210+ models for 1.5B and 7B scales**, respectively. This far exceeds the limited fine-tuned models mentioned by the reviewer. This large-scale evaluation ensures robust, statistically meaningful efficiency frontiers.\n2. **Training Efficiency of REO-RL**\n    - We provide an **empirical comparison** showing that *training efficiency of REO-RL is competitive with baselines.*\n    - We provide a **theoretical analysis** explaning why *REO-RL is training-efficient.*\n3. **Are the the efficiency frontiers useful in our design of REO-RL?**\n    - It is **through the efficiency frontiers that we found using the exponential budget selection is principled in REO-RL.** (Sec 5.2). This further motivates us to propose REO-RL(Exp) ad REO-RL(Exp)-Coef=1 as practical implementations.\n    - We show via an ablation study that **exponential budget selection is critical for reasoning efficiency**. Please see our response to [W3,Q3] below.\n\nTo summarize, our work introduces:\n- **Efficiency frontiers** as the first large-scale estimation of optimal reasoning efficiency.\n- **REG** as a practical metric for quantifying gaps to optimality.\n- **REO-RL(Exp)** as a *training-efficient* and *practical* algorithm, empirically and theoretically justified.\n\nWe hope this clarifies our methodology and adequaetly addresses the reviewer's concern. We appreciate the reviewer's consideration and are happy to address any further questions.\n\n## [W1,W4,Q2] Large-Scale Efficiency Frontier Estimation\n\nTo ensure representative coverage of the best methods across various token budgets, we run RL training with **3 classes of algorithms**, covering **8 distinct algorithms** and **15 unique training configurations** in total. Each  individual run consumes at least 2K and 4K GPU hours for each run in the 1.5B and 7B scales, respectively. For each run, a set of checkpoints are sampled for evaluation. In total, **186 and 218 models in 1.5B and 7B settings**, respectively, are evaluated for constructing the efficiency frontiers. **Our large-scale experiments provide statistically meaningful frontiers that empirically approximate the optimal reasoning efficiency.**\n\nFor each model $\\pi_{\\theta_i}$, we measure the accuracy across benchmarks under token budgets $\\{32,64,\\cdots,512,1K,2K,\\cdots,32K\\}$. Finally, the efficiency frontier for each benchmark is derived by selecting the **maximum accuracy across all models** for each token budget.\n\nThe table below summarizes the algorithm classes, configurations, and model counts:\n\n| Algorithm Class         | Specific Algorithms & Training Configurations                                                               | # of Models (1.5B) | # of Models (7B) |\n|-------------------------|-------------------------------------------------------------------------------------|--------------------|------------------|\n| RL w. Token Budget      | Token Budget $\\in \\{512, 1K, 2K, 4K, 8K, 16K, 32K\\}$                                    | 52                 | 118              |\n| RL w. Length Reward     | MRT; RL w. Len. Group Norm. Rew. ($\\alpha\\in\\{0.1, 0.2\\}$); RL w. Len.-Harmonizing Rew. ($\\lambda \\in \\{1, 2\\}$) | 83                 | 70               |\n| Preference Learning     | SimPO(Shortest, TOPS, DAST)        | 51                 | 30               |\n| Others | Base LRM, Vanilla RL | - | - |\n\n\nFurther details about the algorithms and frontiers can be found in **Line 141-151** and **Appendix C**.\n\n## [W3] Training Efficiency of REO-RL\n\nWe thank the reviewer for raising concerns about training efficiency. Actually, REO-RL is training-efficient. Here we provide training efficiency comparison with baselines, and a theoretical analysis for the cost of REO-RL. \n\nTo start with, REO-RL generates a response and then computes rewards by,\n1. generate a response $y\\sim\\pi_\\theta(\\cdot|x)$\n2. For each truncated response $y_{:L_i}$, an answer $a_i$ is generated, i.e. $a_i\\sim \\pi_\\theta(\\cdot|x, y_{:L_i},\\text{[The Final Answer is]})$, with a budget of 10 tokens. Reward is then computed by matching the answer with the ground-truth answer.\n\n### 1. Training Efficiency Comparison\n\n Below, we provide empirical evidence showing that REO-RL is computationally efficient:\n\n(a) **Low Overhead from Answer Generation** \nREO-RL uses *a small number of token budgets $N=5$* and the *answers are short*, i.e. $|a_i|\\le 10$. The per-question latency breakdown shows that **answer generation adds only 6.4% overhead relative to full response generation**:\n\n| Generating a response y | Generating Answers a[1]..a[N] |\n|---|---|\n|  110.63 $\\pm$ 63.80 s| **7.53 $\\pm$ 5.02s** |\n\n(b) **Faster Training Than Baselines**\n\nREO-RL trains faster than Vanilla RL and other RL methods with length rewards. This is because *overhead from answer generation is low and REO-RL  incentivizes shorter response length*. \n\n| Method                          | Avg. Time per Training Step (s) | Avg. Length (Test) | Avg. Acc. (Test) |\n|---------------------------------|------------------------|-------------|-----------|\n| Vanilla RL                      | 1147.39                | 12609.9     | 71.0      |\n| **REO-RL (EXP)**                 | **877.36**                 | 6725.7      | 68.7      |\n| **REO-RL (Oracle)**                 | **873.26**                 | 7660.8      | 69.2      |\n| RL w. Length Group Norm. Reward | 938.58                 | 8961.3      | 69.8      |\n| RL w. Length-Harmonizing Reward | 903.93                 | 7956.8      | 70.4      |\n| RL w. Token Budget=4K           | 400.13                 | 2978.4      | 57.6      |\n\nSpecially, RL w. Token Budget excessively reduces length at the cost of severe accuracy drop,also having fast training speed. \n\n### 2. Theoretical Cost Analysis\n\nFollowing [1] and [2], we analyze the computation cost for a single input. Let $L_{in}$ be the length of the input, $L_{out}$ be the length of generated response $y$ and  $L_{ans}$ be the maximum lengths of answers generated from truncated responses. For a transformer with $P$ parameters, KV size $D$ and GQA ratio $r$, the computation in REO-RL consists of three parts:\n\n- Prefill for Input $x$: $2PL_{in}+2DL_{in}^2$\n- Decoding Response $y$: $2PL_{out}+r(2L_{in}+L_{out})L_{out}D$\n- Decoding Answers $a_1,\\cdots,a_N$: $2PNL_{ans}+r(2L_{in}+2L_{out}+L_{ans})NL_{ans}D$\n\n**Why REO-RL is Efficient**:\n\n1. **Negligible Answer Generation Cost.** The input length $L_{in}$ is usually small ($\\le 500$) and the response length $L_{out}$ is large, spanning from 5K to 32K. In REO-RL, by default we use $N=5$ and $L_{ans}\\le 10$.  Since $L_{out}\\gg NL_{answer}$, the computation cost is dominated by decoding $y$.\n2. **KV Cache Reuse:** The KV cache of $x$ and $y$ could be reused during answer generation. For more details on KV cache, please refer to [SGLang Blog](https://lmsys.org/blog/2024-01-17-sglang/).\n\n[1] Kinetics: Rethinking Test-Time Scaling Laws. https://arxiv.org/pdf/2506.05333\n\n[2] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. https://arxiv.org/abs/2407.21787\n\n## [W5,Q3] Utility of Frontiers in REO-RL & Novelty of Exponential Budget Selection\n\nWe appreciate the reviewer's thoughtful comments questioning the utility of the frontiers.  Below, we clarify practical utility and the novelty of the frontiers:\n\n**1. In REO-RL, the frontiers reveal that exponential budget selection is principled for efficient reasoning.**\n\nAs shown in Fig. 3 (Sec 5.2), oracle budget selection naturally follows an exponential spacing pattern. This further motivates exponential spacing as a principled and practical way to select token budgets. Thus, REO-RL (Exp) is not an arbitrary heuristic, but derived from the inherent structure of the frontiers.\n\n**2. Exponential budget selection is critical for reasoning efficiency**.\n\nWe make a comparison with a linear variant of REO-RL (with Coef=1). Clearly using linear budget selection underperforms the exponential budget selection:\n\n|  | Accuracy | Length $\\downarrow$ | REG $\\downarrow$ | \n|-|-|-|-|\n| REO-RL(Exp) | 68.7 | 6725.7 | **816.7** |\n| REO-RL(Exp) - Coef=1 | 67.6 | 6160.3 | **605.9** |\n| REO-RL(Linear) - Coef=1 | 69.1 | 7424.8 | 886.1 |\n\n\n## [W2, Q1] Pseudo-Code of REO-RL\n\n```\nInput: \n    - LLM π\n    - Dataset D consisting of input prompts and ground-truth answers\n    - Token budgets L = [L[1], ..., L[N]]\n    - Training steps T\n\nOutput: Fine-tuned LLM π*\n\ndef Generate(π, x, ans, L):\n    y ~ π(· | x)  ▷ Generate full response\n    a = []  ▷ Stores answers for truncated responses\n    \n    ▷ Generate truncated answers in parallel for budgets L[i] < |y|\n    for i = 1 to N do\n        if L[i] < |y| then\n            y_trunc = x + y[:L[i]] + \"</think> The Final Answer is: \"\n            a[i] ~ π(· | y_trunc)  ▷ Answer from truncated response\n        else:\n            a[i] = ExtractAnswer(y) ▷ Answer from the full response\n        \n    ▷ Compute reward (Eq. (2) and Eq. (9) in paper)\n    reward = compute_reward(y, a, ans, L)  \n    \n    return reward, y\n\nfor t = 1 to T do\n    ▷ Sample a batch of prompts and answers\n    B ← Sample a training batch from D  \n    rewards, responses = [], []\n    \n    ▷ Parallel rollout for each (x, ans) in B\n    for (x, ans) in B do\n        reward, response <- Generate(π, x, ans, L)\n        \n    ▷ PPO update step\n    π ← Update with PPO\nend for\nreturn π\n```\nNote that the responses and rewards are generated in parallel to ensure high training efficiency, known as async training in modern LLM training frameworks such as [AReaL](https://github.com/inclusionAI/AReaL) and [veRL](https://github.com/volcengine/verl/pull/2231)."}]

Author rebuttal:

Meta-review: Summary of the paper: This paper claims that current LRMs are far from optimal in the accuracy-vs-length trade-off and introduces the concept of “reasoning efficiency frontiers” to capture the empirically best accuracy achievable at every token budget. To quantify how far a model is from this frontier, the authors define the Reasoning Efficiency Gap (REG). They then propose REO-RL, a reinforcement-learning algorithm that minimizes REG by optimizing performance across a sparse set of token budgets whose rewards are integrated via numerical approximation. Experiments on AMC23, AIME24, and AIME25 with 1.5 B and 7 B DeepSeek-R1-Distill-Qwen models show that REO-RL reduces the efficiency gap by 74.5 % and 64.2 %, respectively, compared to vanilla RL, achieving competitive accuracy while being markedly more concise.

Strengths of the paper:
1. Useful framing: the “reasoning efficiency frontier” and REG metric give the community a concrete yardstick for evaluating conciseness vs. accuracy.
2. Practical algorithmic contribution: REO-RL’s use of sparse token budgets plus numerical integration keeps computational cost low (<1 % error with only 5 budgets).
3. Strong empirical results: large REG reductions across two model sizes, extensive baselines, ablations, and open-source code/data.

Weaknesses of the paper: I would like to express my heartfelt thanks to Reviewer uvtF—the most engaged and responsive reviewer. I am deeply grateful for the thorough and insightful review, as well as for the continuous discussions during the rebuttal period that helped us identify the shortcomings and refine the paper. I fully agree with the final comment: the current narrative feels disjointed and should be restructured. As for the other three reviewers, it seems to me that the authors have addressed most of the concerns. Please add the additional experiments and analyses, reorganize the paper, and correct the technical details to the camera-ready version of the paper.

Reasons for the decision:  This paper aims to rigorously quantify reasoning efficiency, provide a practical optimization method that yields measured gains, and is backed by solid experiments and open resources. In my opinion, while the frontier is empirical rather than theoretical and a structural concern was noted by Reviewer uvtF, these limitations are outweighed by the contributions; I therefore slightly lean toward acceptance. However, I would also comment to SAC on the concern that the Reviewer uvtF still has after the rebuttal.
Decision: Accept (poster)