Title: How Far Are We from Optimal Reasoning Efficiency?
Venue: NeurIPS 2025
Accepted: yes
Keywords: Effiicent Reasoning; Large Reasoning Models; Reinforcement Learning for Reasoning
Abstract: Large Reasoning Models (LRMs) demonstrate remarkable problem-solving capabilities through extended Chain-of-Thought (CoT) reasoning but often produce excessively verbose and redundant reasoning traces. This inefficiency incurs high inference costs and limits practical deployment. While existing fine-tuning methods aim to improve reasoning efficiency, assessing their efficiency gains remains challenging due to inconsistent evaluations. In this work, we introduce the ***reasoning efficiency frontiers***, empirical upper bounds derived from fine-tuning a base LRM (DeepSeek-R1-Distill-Qwen-1.5B/7B) across diverse approaches and training configurations. Based on these frontiers, we propose the ***Reasoning Efficiency Gap (REG)***, a unified metric quantifying deviations of any fine-tuned LRMs from these frontiers. Systematic evaluation on challenging mathematical benchmarks, AMC23, AIME24, and AIME25, reveals significant gaps in current methods: they either sacrifice accuracy for short length or use excessive tokens to achieve sub-optimal accuracies despite high overall accuracy. To reduce the efficiency gap, we propose ***REO-RL***, a Reinforcement Learning algorithm that optimizes reasoning efficiency by targeting a sparse set of token budgets. Leveraging numerical integration over strategically selected budgets, REO-RL approximates the full efficiency objective with low error using a small set of token budgets. Experiments show that, compared to vanilla RL with outcome reward, REO-RL reduces the reasoning efficiency gap by 74.5\% and 64.2\%  in the 1.5B and 7B settings. The 7B LRM fine-tuned with REO-RL achieves reasoning conciseness surpassing frontier LRMs like Qwen3 and Claude Sonnet 3.7. Ablation studies confirm the efficacy of our token budget strategy and highlight REO-RL’s flexibility across design choices. This work establishes a systematic framework for evaluating and optimizing reasoning efficiency in LRMs. We will release the related code, data, and models to support future research on efficient reasoning in LRMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the overthinking behavior in reasoning models which often produce verbose outputs. The authors introduce Reasoning Efficiency Frontiers which is an empirical upper bounds on the accuracy-length trade-off, and define Reasoning Efficiency Gap (REG) to measure deviation from these frontiers. To optimize REG, they propose REO-RL, a reinforcement learning algorithm that approximates a dense multi-budget reward objective using a small set of token budgets selected via either oracle or exponential strategies. Experiments on three math reasoning benchmarks with 1.5B and 7B models show REO-RL improves reasoning efficiency over baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The empirical results are strong, with comprehensive comparisons across baselines, model sizes, and ablations.\n- The paper is well-structured.\n- REG metric offers a useful lens for benchmarking reasoning efficiency.\n- The idea of optimizing a sparse approximation to a dense budgeted reward objective is novel in this context.\n\nWeaknesses:\n- The entire framework relies on empirical frontiers derived from a limited set of fine-tuned models. There is no theoretical guarantee that these frontiers are close to optimal, which weakens the central premise. \n- Prompting the model to generate the final answer directly from a truncated trace to be used for intermediate rewards is not justified, since the objective in Equation 2 is on $y$ sampled from $\\pi$ rather than the truncated response. (Maybe the authors are actual training on the truncate trace for each truncation under different token budget. It would be helpful to include an algorithm pseudo code.)\n- The complexity of the algorithm is higher than the baselines as it needs to regenerate for N times after each generation ends.\n- The details on how the reasoning efficiency frontier is constructed are missing.\n- Empirically the oracle method is close to exp with coef = 1. Exp with coef = 1 is essentially just optimizes the mean rewards for different token budgets which defeats the purpose of optimizing with an oracle frontier. The frontier does not seem to be useful except for being a metric.\n\nQUESTIONS:\n- Could authors include an algorithm pseudo code for REO-RL?\n- Could authors provide more details on how the reasoning efficiency frontier is constructed? Does the author train RL with three different objectives, evaluate on each token budget, and select the best one over the three methods?\n- Since exp with coef = 1 outperforms oracle method, I wonder what would be the actual use for such an empirical frontier.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of verbose reasoning in Large Reasoning Models (LRMs). It first introduces Reasoning Efficiency Frontiers to define the empirical upper bound of the accuracy-length trade-off for any token budget. It then proposes the Reasoning Efficiency Gap (REG), a single metric that quantifies a model's deviation from this frontier. The core contribution is REO-RL, a novel reinforcement learning algorithm designed to minimize this gap. By leveraging sparse token budget sampling and numerical integration, REO-RL efficiently approximates the full-budget optimization problem, leading to substantial reductions in computational cost.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The proposed REG metric effectively unifies model accuracy and efficiency into a single, coherent evaluation standard. The REO-RL algorithm, through sparse budget sampling, offers a computationally efficient optimization method.\n2. The work demonstrates a high degree of experimental rigor, supported by a detailed appendix and the open-sourcing of code and data to ensure reproducibility.\n3. The paper is clearly written with a smooth logical flow, and its core arguments are well-supported by intuitive figures and diagrams.\n\nWeakness\n1. The method's effectiveness is only demonstrated on mathematical reasoning tasks (AMC/AIME). Its applicability to other domains, such as programming, remains unverified, and the paper should either include cross-domain experiments or discuss the transfer potential of the REG framework.\n2. Experiments are limited to relatively small models (1.5B/7B), and the paper does not assess the scalability of REO-RL on larger models (>10B). An analysis of its performance on larger-scale LRMs is necessary to confirm its broader utility.  My primary concern with this paper lies in the choice of models used for experimentation. The study relies on 1.5B and 7B models, which have limited practical value in real-world applications and are not representative of the state-of-the-art Large Reasoning Models currently in use. It is well-understood that the reasoning processes and emergent capabilities of models at this smaller scale can differ substantially from those in the 30B to 230B parameter range.\n3. The paper does not quantify the additional computational overhead introduced by REO-RL, which requires generating multiple truncated responses for reward calculation. A comparative analysis of training efficiency is needed to provide a complete picture of the method's cost.\n\nQUESTIONS:\nThe paper states on lines 136-137 that you \"construct an empirical reasoning efficiency frontier by fine-tuning a diverse set of models using existing approaches.\" Could you please provide a more detailed explanation of how this frontier, which is used to estimate J_optimal, is constructed?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of reasoning efficiency in Large Reasoning Models (LRMs). The authors observe that LRMs like DeepSeek-R1 often produce unnecessarily long reasoning traces - sometimes hundreds of tokens for simple problems. To address this, they introduce \"reasoning efficiency frontiers\" - empirical upper bounds showing the best accuracy achievable at each token budget. They then propose REO-RL, a reinforcement learning method that optimizes models to work well across multiple token budgets simultaneously. The key insight is using numerical integration to approximate performance across all budgets while only training on a small set (5 budgets).\n\nSTRENGTHS AND WEAKNESSES:\n1.The paper identifies a real issue, the LRMs waste tokens. The \"reasoning efficiency frontier\" concept provides a concrete way to measure how far models are from optimal.\n2.They test multiple fine-tuning approaches (RL with token budgets, length rewards, preference learning) to construct the frontiers. This gives a comprehensive view of current methods.\n3.Only tested on math problems (AMC/AIME). It's unclear if this generalizes to other reasoning tasks like coding or general QA.\n4. The \"optimal\" frontiers depend entirely on which methods they tried. If someone finds a better fine-tuning approach tomorrow, all the REG measurements become outdated.\n5. The paper introduces many metrics (REG, length-constrained rewards, etc.) but it's not always clear which matter most for real applications.\n\nMajor concern is that the paper is not very well-written, not clear\n\nQUESTIONS:\nHave you tested on other domains like coding or general reasoning? Math problems might have specific properties that make this approach work well. Maybe presenting results such as puzzle or GPQA would be nice,w\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of reasoning efficiency in Large Reasoning Models (LRMs), which often generate excessively verbose reasoning traces despite their strong problem-solving capabilities. The authors introduce the concept of reasoning efficiency frontiers - empirical upper bounds on the optimal trade-off between accuracy and response length - and propose the Reasoning Efficiency Gap (REG) metric to quantify how far current methods are from optimal efficiency. The main contribution is REO-RL (Reasoning Efficiency Optimization with Reinforcement Learning), an algorithm that optimizes reasoning efficiency by targeting a sparse set of token budgets using numerical integration approximation. Experiments on mathematical reasoning benchmarks (AMC23, AIME24, AIME25) with DeepSeek-R1-Distill-Qwen models (1.5B and 7B) show that REO-RL reduces the efficiency gap by 74.5% and 64.2% respectively compared to vanilla RL.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Novel Framework and Metrics: The introduction of reasoning efficiency frontiers and the REG metric provides a principled way to evaluate and compare reasoning efficiency methods. \n- Solid Experimental Design: The paper conducts comprehensive experiments with diverse fine-tuning approaches. The evaluation covers multiple baselines and includes proper ablation studies.\n- Practical Algorithm: REO-RL cleverly approximates the computationally expensive full efficiency objective using numerical integration over strategically selected token budgets. The exponential spacing strategy achieves <1% approximation error with only 5 budgets, making the approach practical.\n- Strong Empirical Results: The model finetuned with REO-RL demonstrates substantial REG reductions and competitive performance against frontier LRMs like Qwen3 and Claude Sonnet 3.7 in terms of reasoning conciseness.\n\nWeaknesses\n- Confusion in math notations: In section 4.1, when defining $a = Answer(π_\\theta, x, y)$, the $L$ term is not shown as the input.  however, in equation (2), the $L$ term is shown as the input for $Answer()$. This may cause confusion while reading. Also, in definition 4.1 (equation (4)), as far as I am concerned, the reasoning efficiency frontier should be \"a set\" of best achievable reward $J$. In this case, maybe a notation like {$\\hat{J}_{optimal}(D, \\hat{\\theta}, L) \\mid L \\in [1, Lmax]$} will help the reader better understand the technical details.\n- Confusion in Table 1: Table 1 contains the main result of the paper. However, a lot of information (diverse baselines with different budgets, token length, accuracy across baselines, etc.) is contained in a single table. This could cause the readers unable to fully capture the main takeaways from the table. Maybe dividing the table into several sub-tables may be helpful (one table one main-takeaway).\n\nQUESTIONS:\nPlease see the weaknesses. Addressing these issues will increase the clarity of the paper and help the readers better understand the main contribution of the paper.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}