Title: Think Only When You Need with Large Hybrid-Reasoning Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Hybrid Reasoning

Large Reasoning Models

Reinforcement Learning
Abstract: Recent Large Reasoning Models (LRMs) have shown substantially improved reasoning capabilities over traditional Large Language Models (LLMs) by incorporating extended thinking processes prior to producing final responses. However, excessively lengthy thinking introduces substantial overhead in terms of token consumption and latency, which is unnecessary for simple queries. In this work, we introduce Large Hybrid-Reasoning Models (LHRMs), the first kind of model capable of adaptively determining whether to perform reasoning based on the contextual information of user queries. To achieve this, we propose a two-stage training pipeline comprising Hybrid Fine-Tuning (HFT) as a cold start, followed by online reinforcement learning with the proposed Hybrid Group Policy Optimization (HGPO) to implicitly learn to select the appropriate reasoning mode. Furthermore, we introduce a metric called Hybrid Accuracy to quantitatively assess the model’s capability for hybrid reasoning. Extensive experimental results show that LHRMs can adaptively perform hybrid reasoning on queries of varying difficulty and type. It outperforms existing LRMs and LLMs in reasoning and general capabilities while significantly improving efficiency. Together, our work advocates for a reconsideration of the appropriate use of extended reasoning processes and provides a solid starting point for building hybrid reasoning systems.


Peer reviews:
comments: SUMMARY:
In this paper authors introduce a new method to make reasoning models perform better on conversational benchmarks. The method consists of two stages - Hybrid Finetuning and Hybrid Group Policy Optimization. The resulting models support two generation modes - thinking and no thinking which makes them both competitive on reasoning tasks and usable in conversational scenarios.

STRENGTHS AND WEAKNESSES:
### Strengths

- NHTMs outperform baselines on both reasoning and chat benchmarks.
- Good baselines - authors were comparing HGPO with both HFT and HFT+DPO/RFT.
- Only fully-accessible data and open-source models were used in this study.
- Two model scales - 1.5B and 7B.

### Weaknesses

- No comparison with DeepScaleR-1.5B-Preview[1] although the DeepScaleR data was used.
- No evaluations on GPQA-Diamond[2], HumanEval, HumanEval++.
- No information on training data decontamination. E.g. AIME dataset[3] might contain samples from AIME24 benchmark.
- Grammar errors ("to lean a effective NHTMs").

### References

1. https://huggingface.co/agentica-org/DeepScaleR-1.5B-Preview
2. https://openreview.net/forum?id=Ti67584b98
3. https://huggingface.co/datasets/di-zhang-fdu/AIME_1983_2024

QUESTIONS:
- How evaluations were done? Were all models evaluated with the same hyperparameters?
- Can authors provide evaluations on GPQA-Diamond, HumanEval, HumanEval++ for fair comparison with other LRMs?

LIMITATIONS:
- Only one base model was used - Qwen2.5-math-base.
- No comparison with the same compute (both training and test-time).
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a complete pipeline to build a native hybrid thinking LLM which can decide whether conduct long CoT thinking according to the difficulty of the problem by themselves. The entire pipeline follows a standard SFT-then-RL pattern. The SFT stage integrates both reasoning-intensive (Thinking) and direct-answer (No-Thinking) data to establish a robust initialization for following RL training. In RL training phase, this paper proposes a novel HGPO and delicate reward functions to enable the LLM to internally choose the thinking mode when dealing problems with different difficulties. Extensive experiments demonstrate that NHTMs not only possess strong reasoning capabilities but also can largely save computation resources while dealing simple problems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The overall writing of this paper is excellent, and the logic is easy to follow.\n2. The proposed training pipeline is well-designed, and the final experimental results are solid, comprehensive, and convincing.\n\nWeaknesses:\nThis paper does not have any significant weaknesses. Here are some of my suggestions:\n1. In Figure 5, the y-axis label for \"Thinking Ratio\" appears to be missing.\n2. A key contribution of the proposed NHTMs is their ability to autonomously decide whether to think, thereby saving computational budget. It would be more intuitive to include the average number of output tokens for both the baselines and NHTMs across all benchmarks.\n3. In my understanding, Hybrid Accuracy is a model-specific metric, as $m_\\text{gt}$ is determined by the evaluated LLM. If multiple NHTMs are evaluated, the proposed Hybrid Accuracy metric may not fairly compare them.\n\nQUESTIONS:\n1. In Equation (10), I am particularly interested in the term “GRPO for inter-group advantage $A_{\\text{inter}}$.” However, I do not fully understand how this term effectively contributes to the decision of whether to think or not. I would appreciate it if the authors could provide a more detailed explanation. It may also be helpful for other readers if the authors include additional clarification or discussion of this term in the paper.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Native Hybrid Thinking Models, which adaptively decide whether to engage in extended thinking based on user query context, addressing the overthinking issue in Large Reasoning Models. It proposes a two-stage training pipeline: Hybrid Fine-Tuning (HFT) for cold start and Hybrid Group Policy Optimization for learning to select appropriate thinking modes, along with a Hybrid Accuracy metric to assess hybrid thinking capability. Experiments show Native Hybrid Thinking Models outperform LLMs in reasoning and general capabilities while improving efficiency by reducing unnecessary thinking for simple queries.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. Making reasoning models more efficient is an important research problem.\n2. The paper proposes a reasonable approach (including the complete pipeline of SFT and RL) with certain novelty.\n3. The paper is well-written and easy to follow.\n\n**Weakness**\n1. Direct comparison with DeepSeek-R1-Distill-Qwen-7B/1.5B & NHTMs-7B/1.5B is insufficient to demonstrate the method's effectiveness. Based on Stage I, training a baseline model using the same RL data (76K queries from Deepscaler and Tülu3) with standard GRPO to achieve high performance would better illustrate whether implementing adaptive thinking compromises the model's capability ceiling. Simply comparing against DeepSeek-R1-Distill-Qwen-7B/1.5B is inadequate, as you have expended additional computational resources that might enable the model to reach a significantly higher ceiling, making adaptive CoT no longer cost-effective. This point requires experimental validation.\n2. As research on adaptive CoT, reporting the thinking token length statistics for each dataset and comparing with CoT compression works [1][2][3] would strengthen the contribution of this work.\n3. Llama-3.1-Tulu-3-8B-RM is used both for RL training and for computing Hybrid Accuracy in Section 2.4. Could this lead to unfair Hybrid Accuracy evaluation?\n\n[1] Luo H, Shen L, He H, et al. O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning[J]. arXiv preprint arXiv:2501.12570, 2025.\n\n[2] Ma X, Wan G, Yu R, et al. CoT-Valve: Length-Compressible Chain-of-Thought Tuning[J]. arXiv preprint arXiv:2502.09601, 2025.\n\n[3] Shen Y, Zhang J, Huang J, et al. Dast: Difficulty-adaptive slow-thinking for large reasoning models[J]. arXiv preprint arXiv:2503.04472, 2025.\n\nQUESTIONS:\n1. It is surprising that adaptive thinking shows significantly greater improvement on AIME 2024 than on Math500. Intuitively, simple problems' thinking process should be easily compressible, while difficult problems' thinking process should not be compressed or should even benefit from length scaling to improve performance. Could you compare the thinking ratio and thinking token length changes between MATH500/AIME 2024 to explain this phenomenon?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new post-training pipeline to enable \"hybrid\" thinking in language models. Unlike how existing \"thinking\" models such as DeepSeek-R1 always employ a long thinking phase, i.e., `<thinking>...</thinking> Final response`, hybrid thinking models adaptive choose whether to employ thinking, or directly respond to the question, i.e., `<no_thinking>Final response</no_thinking>`.\n\nThe method employs a two-stage pipeline, similar to that of DeepSeek-R1. The first cold-start SFT phase includes samples that (1) employ thinking for reasoning tasks, and those that (2) provide direct responses to non-reasoning tasks. In the second RL phase, the model is prompted to generate multiple responses in thinking and non-thinking modes, respectively. Positive rewards are given to (1) the mode that achieved the best outcome on average, and (2) individual responses that achieved high reward for the given task.\n\nThe method enables higher quality responses across various tasks, and potentially more efficient inference.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Hybrid thinking models trained with the proposed method (1) outperform existing thinking models of the same size and (2) outperform baseline hybrid thinking models proposed by the authors, using standard RL algorithms, under equivalent settings as the proposed method.\n    - *(W1) However, there is no result on a non-hybrid thinking model under equivalent settings.*\n- Analysis shows that the trained model choses between thinking and non-thinking modes adaptively according to the task. Strong evidence of adaptive thinking is shown within the training domain (Figure 3(c), 4) and some evidence is shown for out-of-domain settings (Figure 5).\n- Extensive ablations on the choice of specific advantage estimators (GRPO, RLOO, Reinforce++) and the margin hyperparameter are conducted. These confirm robustness to the choice of advantage estimator, and confirm that the margin hyperparameter work as expected.\n- Each stage of the training methodology, as well as the proposed metric on hybrid thinking, are simple and straightforward.\n- The methodology and experimental settings are well-explained in detail.\n\n### Weaknesses\n\n1. The authors do not compare the proposed hybrid thinking model with an equivalent non-hybrid thinking model. The baseline model in Table 1, DeepSeek-R1-Distill-Qwen, may use a different training protocol from that of the proposed hybrid thinking models.\n    - This makes it unclear whether hybrid thinking degrades performance in reasoning-intense tasks.\n\nQUESTIONS:\n### Questions\n\n- In Table 2 in Appendix C, is WildChat-1M used as the Thinking dataset, and the rest used for Non-Thinking?\n- Is the hybrid-thinking model more efficient compared to the thinking model, i.e., by using less output tokens?\n\n### Suggestions\n\n- Some parts of the method explanation could be simplified by using plain words instead of, or supplementing, the mathematical equations. E.g., equation (5) could simply be explained as generating N responses, with `<think>` appended to the prompt for one half, and `<no_think>` to the other.\n- (Future work) the method could be improved by enabling the model to reason before making the decision to `<think>` or `<no_think>`. The current training pipeline forces to make this decision in the first decoding step.\n\n### Criteria\n\nI am willing to increase my score if W1 is adequately addressed.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}