Title: ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reasoning

Efficient Inference Methods

Reinforcement Learning
Abstract: Recent models such as OpenAI o1 and DeepSeek-R1 have demonstrated strong performance on reasoning-intensive tasks by generating extended Chain-of-Thought (CoT) traces. While longer reasoning helps with thorough exploration of solution paths for complex problems, it also often leads to inefficient and redundant outputs—a phenomenon commonly described as $\textit{overthinking}$.  In this paper, we propose $\texttt{ShorterBetter}$, a simple yet effective reinforcement learning method that enables reasoning models to learn their own optimal CoT lengths without manual supervision. We define the $\textit{Sample Optimal Length}$ (SOL) as the length of the shortest correct response among multiple generations, which serves as a dynamic reward signal to guide the model toward efficient reasoning. Applied to DeepSeek-Distill-Qwen-1.5B/7B as base models, $\texttt{ShorterBetter}$ achieves 50\%-80\% reduction in output lengths in both in-domain and out-of-domain reasoning tasks while maintaining accuracy. Our reasoning trace analysis shows that $\texttt{ShorterBetter}$ refines the structure of the reasoning traces by reducing unnecessary repetition, excessive self-verification, and over-exploration of alternatives.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study proposes a new method to reduce the length of the reasoning chain without sacrificing the model's performance. Specifically, in the proposed method, the shortest and correct reasoning chain is first determined via sampling, and then based on such statistics, the model reward is refined, and the model parameter is updated via Group Relative Policy Optimization. The proposed method shows more reduction in the output while maintaining the accuracy across broad benchmark sets.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- The proposed approach achieved substantial output length reduction without compromising the accuracy\n- Broad evaluation tasks, including out-of-domain ones, are adopted, and the advantage of the proposed method is consistently observed.\n- Analysis of the reasoning structure is appreciated, which is informative about what kind of change is really made for the output\n\n\nWeakness\n- It is unclear which part of the proposed method truly contributed to the improvement over baseline methods, such as Training Inefficient (see Questions)\n- The proposed method is only applied to DeepSeek-R1-Distill-Qwen-1.5B/7B models (see Questions)\n- Further evaluation (or clarification) on the quality of the shortened reasoning chain (beyond correctness of the answer) will be encouraged (see Questions)\n\nQUESTIONS:\n[Comparison with baselines]   \nThe proposed method and baseline methods differ in several dimensions, such as training data,  hyperparameters (Appendix A.3), and formulation details, such as GRPO. Thus, it is not clear what was truly the key to achieving improvement over baselines. The use of out-of-domain seems a weak argument to skip such concerns (ll. 483–485).\n\nParticularly, if the main claim of the paper is the use of the shortest correct response (as stated below and abstract), one should conduct clearer ablations about the definition of l^{SOL} (l.131).\n>(Luo et al., 2025a), for example, compares generated lengths against those from a reference model. More in line with our proposal of sampling multiple responses, Kimi 1.5 (Team, 2025) and Training Efficient (Arora and Zanette, 2025) utilize the shortest and average lengths, respectively, among all sampled responses as reward baselines. In contrast, ShorterBetter innovatively anchors its reward function to the length of the shortest correct response\n\nSpecifically, one can examine the method variants without the correctness condition I(y_j=y_i*)=1 or just using the absolute length of the output in defining l^{SOL}.\nSection 5.3 partially handles the concern around this point, but the analysis focuses more on orthogonal concerns about alpha and the case when there is no correct response. \n\n[Experiments with other models]  \nI’m concerned about the possibility that the success of proposed methods is biased toward Deepseek-based models. These models yield somewhat distinctive outputs with, e.g., aha-moment, and I’m interested in whether the conclusions hold even with other models.  \n\n[Quality of shortened outputs]  \nAlthough coarse analysis on the reasoning structure has been done in Section 5.2, I'm more curious about the quality (human interpretability) of the shortened reasoning chains. If the shortened chain is no longer coherent, not helpful for human users (e.g., due to substantial omission in explanation), or amplifies inappropriate properties (e.g., hurting politeness), these will be limitations in a real human-LLM interaction. Such human-involved analysis will also be informative for the readers.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces \"ShorterBetter,\" a reinforcement learning method designed to mitigate the \"overthinking\" phenomenon in Large Reasoning Models. The core contribution is a novel, self-supervised reward mechanism based on the concept of \"Sample Optimal Length\" (SOL). For a given problem, the model generates a small batch of responses, and the SOL is defined as the token length of the shortest correct response within that batch. If no response is correct, the average length of the incorrect responses is used as a neutral baseline. This SOL-based reward is then used to fine-tune the LRM using the GRPO algorithm, guiding the model to produce more concise yet accurate reasoning traces without requiring manual length specifications. The authors demonstrate empirically that applying ShorterBetter to 1.5B and 7B parameter models results in a 50-80% reduction in output length on both in-domain and out-of-domain benchmarks, while largely maintaining task accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThis paper presents a compelling and timely solution to a significant problem in modern LRMs. Its primary strengths lie in its originality, significance, and strong empirical results. The proposed method is novel in its use of SOL as a dynamic, data-driven reward signal, offering an elegant alternative to methods requiring explicit length budgets. The problem of computational inefficiency from \"overthinking\" is highly significant for the practical application of LRMs, and this work provides a promising path toward more efficient and deployable models. The quality of the empirical work is high; the reported length reductions are substantial and consistently demonstrated across multiple models and diverse benchmarks, supported by ablation studies that validate key design choices. The paper is also written with excellent clarity, making the methodology and results easy to follow.\n\nWeaknesses\n\nThe robustness of the core method is questionable due to its reliance on a small sample size. The entire reward signal hinges on the SOL, which is derived from a small group of n=8 rollouts per prompt during training. This small sample size can lead to a high-variance, unstable reward signal. The calculated SOL in any given step might be an artifact of a stochastic sampling outcome rather than a true reflection of the model's optimal reasoning capability for a given problem. If a truly concise and correct reasoning path is not among the 8 samples, the model will be guided toward a suboptimal target. The paper lacks a crucial sensitivity analysis exploring how the choice of n impacts training stability and final model performance. This omission undermines the quality and robustness of the proposed method, as its effectiveness may be highly dependent on this unexamined hyperparameter.\n\nA key claim of the paper is that ShorterBetter refines the structure of reasoning traces, not just their length. However, the evidence for this claim is based on an analysis performed by another LLM (Gemini 2.5 Flash Preview) acting as an automated judge. The authors use this LLM-as-a-judge to categorize sentences into functional roles like \"Pivotal Reasoning\" or \"Exploring Alternatives\" and then report shifts in their token-level proportions. The critical flaw in this evaluation is the lack of any validation for the judge's accuracy or objectivity. The paper does not report inter-annotator agreement between the LLM judge and human experts, nor does it provide any metrics to quantify the reliability of these automated categorizations. Without such validation, the conclusions about structural improvements are built on an unverified and potentially biased foundation, weakening the quality and significance of this portion of the paper's contribution.\n\nQUESTIONS:\nrefer to weakness\n\nLIMITATIONS:\nrefer to weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nReasoning models tend to generate redundant and inefficient chain-of-thought (CoT) traces for complex tasks, a phenomenon commonly known as overthinking. The paper proposes a simple yet effective reinforcement learning method to enable models to learn their own optimal CoT lengths without manual supervision. The length of the shortest correct response among multiple generations is used as a dynamic reward signal to guide model training. The proposed method has been shown to be able to reduce unnecessary repetitions, excessive self-verification, and over-exploration of alternatives, achieving significant reduction in output length while maintaining reasoning accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n-\tThe proposed method is effective in reducing the length of reasoning traces while being able to maintain model accuracy. The obtained models perform well on both in-domain and out-of-domain benchmarks with reduced reasoning lengths.\n-\tThe proposed reasoning trace analysis frameworks, including analyzing the length of extra reasoning after reaching correct answers and fine-grained analysis of the reasoning trace structure, are useful for this line of research.\n-\tThe proposed sample optimal length serves as a dynamic and effective reward for learning efficient reasoning models. \n-\tThe paper is well-written and easy to follow.\n\n**Weaknesses**\n\n-\tThe proposed method, ShorterBetter, is motivated from the optimal reasoning length hypothesis. To approximate the optimal length, the method adopts a sample-based approach to select the shortest length of correct responses from $n$ generated rollouts. It is possible that the selected response is still long. It is also possible that the selected response is too short, so the model takes some reasoning shortcuts, resulting in reduced performance on out-of-distribution benchmarks.  It is unclear how ShorterBetter can address these issues.\n-\tThere are three important hyperparameters in  ShorterBetter, i.e., $\\alpha$, $\\beta$, and $n$. There lacks a discussion on how these parameters are chosen and what are the tradeoffs for setting these parameters.\n\nQUESTIONS:\n-\tCan ShorterBetter address the issues when the sampled response is still long or encourages taking reasoning shortcuts that adversely affect out-of-distribution generalization? \n-\tAre there any tradeoffs in determining the hyperparameters of the proposed method?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nShorterBetter introduces a reward function that explicitly accounts for Optimal Reasoning Length (OL). In addition to answer accuracy, the function assigns a penalty to any generation whose length diverges from the shortest correct solution—or, when no correct solution appears in the roll-outs, from the batch-average length. They assess ShorterBetter on both in-distribution and out-of-distribution mathematics benchmarks and perform ablation studies to isolate and validate the contribution of the length-based penalty.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths:**\n- The presentation is clear, and the method is easy to understand.\n- The experimental study seems comprehensive, with 4 ID and 4 OOD tasks. \n- The authors conduct a comprehensive ablation study with varying values of the hyperparameters in the reward function. \n\n### **Weakness:**\n- **Novelty:** ShorterBetter implements an augmented version of GRPO with length regularization on top of the task metrics. Previous works like L1 (Aggarwal et al., 2025) and concurrent works such as ConciseR (Song et al., 2025) have similar notions, where L1 also leverages GRPO to fine-tune LLMs. I saw the authors compare with L1:\n```\nExisting approaches targeting efficient LLM reasoning primarily focus on explicitly controlling reasoning length, for instance by imposing a length budget based on user specifications (L1; Aggarwal and Welleck, 2025) … In contrast to these explicit control strategies, our work is based on the hypothesis that reasoning models implicitly possess an optimal reasoning length (OL).\n```\nIf I understand correctly, ShorterBetter can also be viewed as a self-supervised version of L1 where the optimal length is defined by the model itself. In this context, the novelty seems insufficient.\n\n- **Robustness:** There are some interesting yet not fully explained conclusions. Specifically, Figure 4 shows that naively adopting the shortest response as the target length leads to training collapse. The resulting accuracies and output lengths are all zeros, even for “Partial.” Can the authors provide more explanation? If the training dynamics are sensitive to the length penalty, how do the authors, in practice, choose the hyperparameters? \n\n### **Minor:**\n- I don’t think the section “Output Length after First Appearance of Correct Answer” is necessary. It seems to convey the same information as comparing the total answer length. Also, why “after” and not “before”? Redundancy can also occur before the model reaches the correct answer, no?\n\nQUESTIONS:\nPlease see weakness. \n\nAlso, what do you mean by \"may inform the design of more fine-grained and behavior-aware reasoning optimization strategies\" (in section 6)?\n\nLIMITATIONS:\nYes the authors mentioned future improvements such as 1) applying to larger-scale reasoning model, 2) tasks with non-binary correctness score, 3) more fine-grained reasoning optimization strategies.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}