Title: Thinkless: LLM Learns When to Think
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Reasoning Models

Hybrid Reasoning

Reinforcement Learning

Efficienct Deep Learning
Abstract: Reasoning Language Models, capable of extended chain-of-thought reasoning, have demonstrated remarkable performance on tasks requiring complex logical inference. However, applying elaborate reasoning for all queries often results in substantial computational inefficiencies, particularly when many problems admit straightforward solutions. This motivates an open question: Can LLMs learn when to think? To answer this, we propose Thinkless, a learnable framework that empowers an LLM to adaptively select between short-form and long-form reasoning, based on both task complexity and the model's ability. Thinkless is trained under a reinforcement learning paradigm and employs two control tokens, \<short\> for concise responses and \<think\> for detailed reasoning. At the core of our method is a Decoupled Group Relative Policy Optimization (DeGRPO) algorithm, which decomposes the learning objective of hybrid reasoning into two components: (1) a control token loss that governs the selection of the reasoning mode, and (2) a response loss that improves the accuracy of the generated answers. This decoupled formulation enables fine-grained control over the contributions of each objective, stabilizing training and effectively preventing the collapse observed in vanilla GRPO. Empirically, on several benchmarks such as Minerva Algebra, MATH-500, and GSM8K, Thinkless is able to reduce the usage of long-chain thinking by 50% - 90%, significantly improving the efficiency of Reasoning Language Models. The code is available at \url{https://github.com/VainF/Thinkless}


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework designed to teach LLMs to decide when to use short think versus long think. The model learns to select between a <think> token for long reasoning and a <short> token for concise responses. The work also proposes a reinforcement learning algorithm, Decoupled GRPO (DeGRPO), which solves the critical issue of \"mode collapse\" by separately optimizing the policy for mode selection and answer accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The algorithm is well-motivated. The authors show that GRPO fails due to mode collapse and correctly identify the token imbalance as the cause. The proposed DeGRPO is a direct solution. It decouples the loss for the control token from the response tokens, which makes the algorithm a targeted fix for a clearly documented empirical problem.\n2. The experiments are well-designed. Author provides experiments covering many areas. \n3. This paper provides many insightful findings. \n\nWeaknesses\n1.  Performance Degradation: The main weakness is that the method's efficiency gains come at the cost of performance. On AIME 2024, the final Thinkless model achieves a Pass@1 of 0.2733, which is lower than the 0.2800 of its own base model (DeepSeek-R1-1.5B). While token usage is reduced, this negative trade-off is concerning, especially as other recent methods for reasoning truncation have demonstrated the ability to simultaneously reduce tokens and improve accuracy. For instance, concurrent work such as Laser-D and Laser-DE achieve a 6.1% improvement on AIME2024 while reducing token usage by 63%. While the methods differ, this comparison suggests the proposed approach may be suboptimal.\n2. Limited Scope of Evaluation: The experiments are limited to math reasoning. This leaves the method's generalizability to other domains like commonsense reasoning or code generation.\n3. Dependence on Distillation Quality: The performance of the RL stage appears dependent on the quality of the initial \"warm-up\" model from the distillation phase. The paper does not adequately explore this sensitivity. \n4. The analysis of key hyperparameters is not thorough enough. For example, when examining the control token update weight, the paper only compares 0.5 and 0.001. This is a large gap, and a more progressive sweep is needed to understand the trend and sensitivity of this critical parameter.\n\nQUESTIONS:\n1. When observing the collapse in Vanilla GRPO, what is the final Pass@1 accuracy of the collapsed policy?\n2. Reward shaping: have you tried finegrained reward shaping to control the response length?\n3. Could you do log-scale sweep of alpha values (e.g., 0.1, 0.01, 0.001, 0.0001)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Thinkless, a reinforcement learning framework that enables Large Language Models to adaptively choose between detailed chain-of-thought reasoning and direct short-form responses based on task complexity. The authors propose a two-stage training approach: first, knowledge distillation using control tokens (<think> for long reasoning and <short> for concise answers) to establish dual response capabilities; second, a novel Decoupled Group Relative Policy Optimization (DeGRPO) algorithm that separates the learning objectives for mode selection and response accuracy, addressing the gradient imbalance issues in vanilla GRPO that lead to mode collapse. Experiments on mathematical datasets (AIME, MATH-500, Minerva Algebra, GSM8K) demonstrate that Thinkless reduces the use of long-form reasoning by 50-90% while maintaining comparable accuracy (within 1% degradation), achieving significant computational savings by intelligently allocating reasoning resources—using extensive thinking only when necessary for complex problems while providing quick responses for simpler queries.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1、 The authors address an interesting problem: how to enable models to autonomously think based on question difficulty, thereby answering more efficiently. This is a highly interesting problem, and the authors propose an effective solution that can contribute to the community.   \n\n2、The authors design control tokens to manage two different thinking modes and discover that directly using the original GRPO algorithm leads to mode collapse, where the model only learns the short mode rather than adapting. Based on this, the authors propose Decoupled GRPO and improve both parts of the loss function, achieving excellent experimental results.\n\n## Weaknesses and Questions\n1、I am curious about what would happen if we directly proceed with reinforcement learning training without the first-stage warm-up? A related question: after the first-stage warm-up, would long responses still appear under the short mode in the second-stage reinforcement learning training?   \n\n2、Similar to the previous question, in the reward function, when the model uses short mode and gets the correct answer, it receives a reward of 1. Could this design lead the model to prefer outputting <short> while still using long responses to answer questions? I would like to see analytical experiments on the changes in average response lengths for both short and long modes.   \n\n3、In the second plot of Figure 3-b, it seems that the accuracy of think mode shows a significant decline after 200 steps. Why does this happen?\n\nQUESTIONS:\nAs I stated in Weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-stage approach to teach large language models (LLMs) when to engage in longer reasoning. In the first stage, supervised fine-tuning on distilled examples introduces two new control tokens and teaches the model their intended semantics. In the second stage, reinforcement learning trains the model to use these tokens effectively for efficient problem solving. The paper also introduces corrections to the reinforcement learning procedure to prevent mode collapse.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper presents a clear and straightforward idea that brings measurable benefits.\n- The proposed reinforcement learning formulation, with its modifications to mitigate mode collapse, is sound and well-motivated.\n- I appreciate that the experimental design is structured around specific findings, with empirical results provided to support each of them.\n\nWeaknesses:\n\n- The novelty of the work appears limited. There is substantial prior research on controlling when language models should “think” more, such as “Think Before You Speak: Training Language Models with Pause Tokens” and “Training Language Models to Reason Efficiently”. In addition, there are several recent and concurrent works in this space that are not cited as related work.\nWhile the paper cites related baselines ([1, 13, 18, 24]), it does not actually compare to them in the experiments—only [26] is included as a baseline. This makes it hard to judge the empirical significance of the proposed approach relative to existing methods.\n- Finding 1 claims that the model can distinguish simple from complex queries, but the evidence for this is indirect. Table 1 only shows pass@1 and reasoning trace lengths. A stronger evaluation would involve explicitly classifying queries into simple and complex (e.g., by baseline success rate) and demonstrating that Thinkless activates the <think> mode significantly more often on harder instances, similar to Fig. 5 but stratified by problem complexity.\n\nQUESTIONS:\n- How does the proposed method compare quantitatively to other relevant baselines? Why are there no reported results for some of the methods cited in the related work section?\n- In Figure 3b, there appears to be some degradation in performance on examples using the <think> token as training progresses. What might explain this decline? Should we expect the accuracy on <think> samples (the default mode) to remain relatively stable throughout training, while accuracy on <short> samples to improve over time?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a learnable framework to enable LLMs' capability of learning when to think. It designs two types of thinking: short-form and long-form reasoning, and lets LLMs learn to choose the suitable reasoning type via the decoupled group relative policy optimization (DeGRPO) algorithm. The results on multiple Math benchmarks show that it can reduce 50%-90% thinking tokens.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* This paper focuses on the important problem in current long-chain-of-thought reasoning: the model may waste its thinking time on easy problems. It then proposes two thinking modes to make the reasoning more effective while keeping it accurate. \n* To mitigate the imbalance between the single-mode token and the response, it modifies the original GRPO to decouple the mode loss and the generation loss. It introduces a hyperparameter to balance the optimization. Experimental results show that this hyperparameter can effectively avoid mode collapse caused by the length difference between the thinking mode and the response.  \n* It provides a comprehensive investigation on the training dynamics for thinking mode learning. It shows that LLMs will first learn to use long-chain outputs to improve accuracy, and then learn to generate more high-quality short answers to achieve a higher reward as defined in Section 3.2. The decoupling hyperparameter also has a high influence on the training stability. \n\nWeaknesses:\n\n* The idea is simple and straightforward: introducing the thinking mode tag to encourage the controlled generation under different models. The proposed DeGRPO is also an incremental improvement over the existing GRPO by adding a hyperparameter in special tokens (mode). \n* The definition and boundary between the think mode and the short mode are not very clear.  Is there a specific length threshold between two modes? If these two modes are mainly distinguished by different distillation sources (general-purpose LLMs Qwen2.5-Math-1.5B-Instruct v.s. reasoning model DeepSeek-R1-Distill-Qwen-1.5B) , it may overfit on learning which model is more suitable to solve the problem, instead of learning the better thinking mode.\n* More descriptions should be added for notations in Table 1. For example, the bold numbers are not the highest ones in the column. The baseline to calculate the token percentage is not mentioned.\n\nQUESTIONS:\n1. Why is it important to have the paired thinking response and the short response for the same problem in the warm-up? The data imbalance issue can also be avoided by sampling the same number of thinking and short responses from different sets of problems.  There could be some difficult problems that only have correct thinking responses. In this case, these hard problems will be removed from distillation. \n\n2. What are the average lengths of the responses in think mode and short mode? How well is the model trained to follow the mode token to generate the corresponding response? For example, are the responses under the short mode always shorter than the thinking mode?\n\n3. Do you think that if the RL is fully trained, the final model will finally learn to always generate shorter responses because of a higher reward? Because once the model learn to correctly solve the problem in the think mode, it can still be optimized to a shorter version.\n\nLIMITATIONS:\nThey discuss the potential limitations in their paper, and there is no noticeable negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}