Think Only When You Need with Large Hybrid-Reasoning Models
Authors: Lingjie Jiang
Xun Wu
Shaohan Huang
Qingxiu Dong
Zewen Chi
Li Dong
Xingxing Zhang
Tengchao Lv
Lei Cui
Furu Wei
Venue: NeurIPS 2025 poster ()
Keywords: Hybrid Reasoning
Large Reasoning Models
Reinforcement Learning
Primary area: deep_learning

Abstract: Recent Large Reasoning Models (LRMs) have shown substantially improved reasoning capabilities over traditional Large Language Models (LLMs) by incorporating extended thinking processes prior to producing final responses. However, excessively lengthy thinking introduces substantial overhead in terms of token consumption and latency, which is unnecessary for simple queries. In this work, we introduce Large Hybrid-Reasoning Models (LHRMs), the first kind of model capable of adaptively determining whether to perform reasoning based on the contextual information of user queries. To achieve this, we propose a two-stage training pipeline comprising Hybrid Fine-Tuning (HFT) as a cold start, followed by online reinforcement learning with the proposed Hybrid Group Policy Optimization (HGPO) to implicitly learn to select the appropriate reasoning mode. Furthermore, we introduce a metric called Hybrid Accuracy to quantitatively assess the model’s capability for hybrid reasoning. Extensive experimental results show that LHRMs can adaptively perform hybrid reasoning on queries of varying difficulty and type. It outperforms existing LRMs and LLMs in reasoning and general capabilities while significantly improving efficiency. Together, our work advocates for a reconsideration of the appropriate use of extended reasoning processes and provides a solid starting point for building hybrid reasoning systems.

Think Only When You Need with Large Hybrid-Reasoning Models
Lingjie Jiang" ti
Xun Wu Shaohan Huang
Qingxiu Dongi:
Zewen Chit
Li Dong
Xingxing Zhang Tengchao Lvt Lei Cui Furu Weit
Microsoft Research $ Peking University
singjiejiangletu.pku.cc
{xunnu, shachanh, (mweiooiicooooot..cm
https//aka.ms/GGeeraalll
Abstract
Recent Large Reasoning Models (LRMs) have shown substantially improved reasoning capabilities over traditional Large Language Models (LLMs) by incorpo- rating extended thinking processes prior to producing final responses. However, excessively lengthy thinking introduces substantial overhead in terms of token consumption and latency, particularly unnecessary for simple queries. work, we introduce Large -oriid-Re Models (LHRMs), the first kind of model capable of adaptively determining whether to perform thinking based on the contextual information of user queries. To achieve this, propose two-stage training pipeline comprising Hybrid TineTTnnng (HFT) as a cold start, followed by online reinforcement learning with the proposed Hybrid Group Policy Optimization (HGPO) to implicitly learn to select the appropriate thinking mode. Furthermore, we introduce a metric called Hybrid Accuracy to quantitatively assess the model's capability for hybrid thinking. Extensive experimental results show that LHRMs can adaptively perform hybrid thinking on queries of varying difficulty and type. It outperforms existing LRMs and LLMs in reasoning and general capabilities while significantly improving efficiency. Together, our work advocates for reconsider- ation of the appropriate use of extended thinking processes, and provides a solid starting point for building hybrid thinking systems.
this
Introduction
Recent remarkable advancements in Large Reasoning Models (LRMs), such DeepSeek-RI R1 [15], OpenAl ol/o3 series [38] and others [46, 37], have catalyzed a significant paradigm shift from conventional Large Language Models (L.LMs) toward LRMss. Compared to LLMss LRMs exhibit substantially improved reasoning capabilities and generalization across tasks such as programming, mathematics, and commonsense reasoning [49, 15,48). These improvements are primarily attributed to the generation of extended thinking traces, often marked by special tokens such as <think>, prior to producing the final responses. Despite the effectiveness of LRMs, existing work primarily focuses on converting LLMs into LRMs to enhance performance across various domains, typically through reinforcement learning algorithms such as GRPO [15] and REINFORCE+ [17], or by fine-tuning on distilled thinking traces [36].
However, broader implications regarding the potential challenges faced by LRMs in real-world
applications remain underexplored. A prominent issue is the problem of overthinking [? ? J. where excessive computational resources are expended on simple queries with negligible performance gains, even for trivial inputs such single word (e.g. "Hello"). These findings highlight several pressing limitations current LRMs and raise critical question:
Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Question 1: a. e: real numbers
laz + 1or < x 1. Find largest possible
+ [6] [e]. Let's think step by step and output the final answer within boxed[). Ground Truth: 17
Sween.5-7B-Insttruc To lind largest possible value of [a] + [6] + |e] given that Jax ba +
all
we
anulyze the constraints imposed by this inequality specific points.
All conditions are satisfied. Now, calculate |a| + |6] [e]: [a]
|6]
2| + [1] 2 2 +
5 Thus, the largest
possible value of [a] + [b] + lol 5 DeepSeek- RRl-Disill--wennt
<think>> Okay, have this prohlem where need the largest possiibee value of + lel given that [az? ba + ol < I for all x between O and Hmm, let me think about how to approach this </tthink> The maximum value of lal + lbl is achioved when the quadratic function touches at x=0 and x=1, and -1 at x=1/2.
For 0 S a S 1,0 < (x =)² < 5, -1 1 & (a) & 1. Therefore, the largest possible value of [a] + [6] + |e|
the largest possible value lal ki given the quadratic function Jax 2 bor + is bounded by in absolute value for all between and I. Hmm, let's start by understanding the problem. </think>>
LHRMMs-B: <think>> Okay, so need
Given the problem fioding the largest possible value [o] + [6] + |e] under the constraint that jax + be + cl S for all 0 S x S 1. by considering the Chehyshev polynomials and their propeties.
polyuomial indeed satisties given constraint and considering other potential polynemials. we couclude the maximum value of [a] + MM + |e| is achieved by the polynomial Sa* 8ar + 1. The largest possible value of |a] + [6] + Je] is 17
After veritying
Question Can you help please?
Oween2.577B-Instren Of course! I'd he happy to help you. Could you please provide some more information about what you need assistance with? /
DeepSeek-HE RI- AL--Disill-QWee Qwen-BB:
<thinks Okay, the user just said "Can you help me please!" need to respond in friendly and helpful way.
should acknowledge their request, show willingness to asiist, and ask them to specily what they need help with. Keeping it open ended will
cacourage them to provide mere details.
Himn, mayhe slart with positive afliraation. Something like, "Yes, can help! What du you need assistance with today?" ?" That sounds good.
polite and inviting them share more about their request.
should make sure response is clear and concise. No need for complicated words. Just a struighfforward offer to help. Let me pur that
together. </think>>
of course! I'm here help. What do you need assistance with today? x
LHRMs-7B: <ao_ bo_think> Of course" What do you need help with wiith?//no_tbaiiii think> /
DeepSeek-R1 Dieeeee---Disiil--eveeee -waeill---e--BB... and our LHRMs-7B on reasoning related task (top) and a daily QA task (bottom). While LLMs respond concisely to simple queries, they struggle with complex reasoning. LRMs handle reasoning well via explicit thinking steps but often overuse them on simple tasks, resulting in slower responses and degraded user experience. In contrast, LHRMs adaptively determines when to engage in thinking, preserving strong reasoning ability while enabling faster, more natural interactions in everyday scenarios. Additional examples are provided in Appendix G.
Figure 1: Example responses from Dwwe2.5--B-Insttren
How to build a hybrid thinking system that can achieve an optimal balance between system 2 reasoning and system ahility?
To fill these gaps and make LRMs more efficient, intelligent, and better aligned with human usage requirements, we draw inspiration from human cognitive processes, where complex problems often require deliberate thinking while simple ones can be handled intuitively.
Building on this insight, we introduce Large Hybrid- Reasoning Models (LHRMs), a novel class of models that, to the best of our knowledge, are the first to explicitly address the limitations of existing LRMs by dynamically deciding whether to invoke extended thinking based on the semantic and contextual characteristics of user queries. As shown in Figure 1, this adaptive thinking mechanism allows LHRMs to balance computational efficiency with task complexity, better aligning with real- world deployment scenarios and enhancing user experience without compromising performance. To achieve this, we propose a two-stage training pipeline. The first stage, Hybrid Fine- Tine-TTnnig (HFT), provides an effective cold start by enabling the model to learn to support two distinct thinking modes on the same query without mode collapse. This is followed by Hybrid Group Policy Optimization, a variant of online RL designed to teach the model when to engage in extended thinking while simultaneously generating more helpful and harmless responses. Furthermore, we introduce metric termed Hybrid Accuracy, which correlates strongly with human expert judgment, to quantitatively evaluate the model's capability for hybrid thinking.
Extensive experimental results and human studies conducted on Qwen-2.5 series models ranging from 1.5B to 7B parameters across multiple domains (including mathematics, programming, and general tasks) demonstrate that our LHRMs effectively performs hybrid thinking by adapting to queries of varying difficulty and types. Moreover, LHRMs outperforms existing LRMs and LLMs
2

---

in both reasoning and general capabilities, while significantly improving efficiency. In general, our main contribution is follows:
We introduce Large determine whether to perform thinking based the contextual information of user queries, mimicking human reasoning patterns and mitigating the over-thinking problem.
Models, the first kind of model which can adaptively
2. We develop two-stage training pipeline comprising Hybrid Fine- Tuninn cold start and novel RL method, Hybrid Group Policy Optimization, which enables the model to learn when to engage in thinking while simultaneously generating more helpful and harmless responses.
3. Extensive experiments across diverse model scales and domains demonstrate that LHRMs accu- rately performs hybrid thinking and significantly improves both efficiency and general capabilities compared to existing LRMs and LLMs.
2 Large Hybrid- Models
2.1 Problem Formulation
Specifically, let q be an input query. The LHRMs has access to two thinking modes: Thinking and No-Thinking, denoted as .M = J, respectively. Each mode defines a conditional distribution P(a q. m) over answers A.A P(a q, m), m M.
(I)
For each query q. the model aims to select the optimal reasoning mode m (q) from the candidate set M, such that a task-specific utility function (g a) is maximized in expectation:
m (a) arg max (awP(al).m) ((qaaa)
MEM
The overall objective is to learn a policy + M select mode E M that maximizes the expected utility U(q, A) over the query distributions θ = ((D.,
N
E Ba-P(aly anD, [u( (c)
mIxX 37
ED, De
We summarize two key challenges and our corresponding solutions as follows:
C1. How to learn the policy based on the U4 We first propose two-stage training phase to learn an effective LHRMs by: Hybrid Fine-Tuning as cold start ($ 2.2). followed by online reinforcement learning with Hybrid Group Policy Optimization ($ 2.3) to guide LHRMs to more effectively select between two reasoming modes to generate more appropriate and high-quality responses, respectively.
C2. How to evaluate the hybrid thinking capability of a policy т. We propose evaluation metric named hybrid accuracy, which designed to assess the hybrid thinking ability of models in $ 2.4.
2.2 Stage I: Hybrid Fine-Tunine
We begin with a wybrid-formatted supervised fine-tuning stage named Hybrid Fine-Tuning (HFT) that integrates both reasoning intenssiee (Thinking) and direct-answer (No-Thinking) data. This approach mitigates the instability often observed in cold-start scenarios [15], and establishes a robust initialization for next stage reinforcement learning.
Data Construction. The hybrid fine- tuning dataset consists of both answer examples. The think-style subset includes high-quality math, code, and science questions sourced from existing datasets [35, 12, 39, 45, 50], with answers generated by Deepseek-RR [15] and verified for correctness. Each instance is annotated with <think> and </think> tags to mark intermediate reasoning steps. For the non-think-style subset, we collect simple queries from WildChat- 1M [58] using FastText-based classifier [21] to exclude complex reasoning tasks. The remaining factual and conversational examples are wrapped with mo-think>> and </no_think> tags indicate that direct answers sufficient.
intensive and direct-
After deduplication and the removal of overlaps with evaluation benchmarks, we obtain a final set of 1.7M hybrid-fore formattddd training examples. This curated collection provides high-quality,
3

---

query q
XL Update
Reference Model
Algorithm Hybrid Group Policy Optimization
Input model trained at Stage Tour's reward models
Pollcy Model
Update
AE
Austra olaast
Rg: queries P; byperparameters e, B, μ
1: policy model To + Tourr
No Thinkin
Thinking
Autra
2: for iteration = I,..., do
respanses of Pi.... 9/yz
r Fl you
3: reference model Wna t n0
4: for step I.... M
Auster
5:
Sample a batch Ps, from P
rasponses of 004/3
rewards
6: Update the old policy model Tew
Yt lig you 8//2
7:
Sample N outputs Ô -eau (- q, m) for
q E Pa and m E .M
8:
Compute rewards r(o,") for o, E by running Rg
Reward
Reward
Model
Assignment
9:
Assign each r(o,") Timer (o.") and
Figure 2: Demonstration of Hybrid Group Pol- icy Optimization. HGPO proceeds by: (1) sam- pling multiple responses for each query 9 using both reasoning modes; (2) scoring the responses
Tintra ( using Eq.
Compute A for the 1-th token of of
through Eq. 10
11:
for iteration = ...., do Update the by maximizing the objec- tive JHUPO in Eq.
12:
with the reward model and assigning these re- wards based on Eq. 9; and (3) computing the advantage and policy loss, followed by updat-
13:
end for end for
14:
ing the policy model. AE denotes advantage
15:
end for
estimator and reward assignment denotes Eq. 9. Output T3
format-consistent foundation for cold-start training, enabling the model to effectively handle both reasoning-heavy and direct-response tasks.
Optimize Objective. HFT trains the model to predict the next token based on prior context. For the constructed dataset DHET = {(x², y ) the objective
lvl
Lurr(0) (0) = - ((.
Σ
log To Ir vht-
(4)
2.3 Stage II: Hybrid Group Policy Optimization
After Stage I, LHRMs acquires an initial ability to support two distinct reasoning modes in M on the same query q without collapsing. However, our ultimate goal is to enable the LHRMs to adaptively select the most appropriate reasoning mode m (q) (Eq. 2) for each query to improve its applicability in real-world scenarios. To this end, we propose an effective RL algorithm named Hybrid Group Policy Optimization (HGPO) to explicitly teach the model how to adaptively select between reasoning modes, i.e., to learn the policy æ Eq. 3, while simultaneously improving the model's 's foundational ability (e.g., helpfulness and harmlessness).
HGPO is illustrated in Figure and Algorithm 1. Following prior RL approaches [1, 17, 41], HGPO discards the critic model and instead estimates the values using multiple samples for each prompt to reduce the computational cost training. Specifically, for each question q in the task prompt set P. HGPO samples two groups of outputs O from the old policy Te using Thinking and No- No-Thinking mode, respectively.
Sampling Strategy, Given a query & E P, we sample N candidate responses from the old policy Thur under two distinct reasoning modes M [-, Specifically,
(o N/2 (. qq, +. (of 1 Tourt ( 1. K)
(5)
We define the complete output candidate set as:
O
N/2 (ot } | u (
N/2
(6)

---

Reward Scoring and Assignment. A reward function R is applied to each candidate output, yielding two reward sets:
R = (r(o)) N/2 R =
Since the reward scores assigned by the reward model may vary significantly across different domains, we apply rule-based assignment scheme to normalize the existing reward scores. Specifically, we compute two types of binary reward: inter-group rewards rinter and intra-group rewards Tintra to jointly capture both the relative quality across different reasoning modes and the answer quality within each individual reasoning mode. Given the average reward for each mode computed as
N/2
N/2
Σ r(o5); 1=1
R
2
r(d)):
(8)
N
N
we have Tinter and Tintra assigned as:
1, m arg max {R. R
+ 5}
1, if = arg max
Vinter (o,
m'
Fintra (0
(E(1....N//3
(9)
0, otherwise
otherwise
Here margin 8 is hyperparameter that governs the trade-off between two reasoning modes.
Advantage Estimation. After getting the final two kinds of rewards for O, we can adopt advantage estimators like REINFORCE+ [17], GRPO [41], and RLOO [I] to estimate the advantages. Here we use GRPO as the default advantage estimator, i.c.,
Tintra (o:) mean /'intr (o₂))
Tinter (o;) mean (Tinter (0,))
A
+ +1 E Φ
(10)
std((innma ('intra (0,))
std (Tinter (0z))
GRPO for intra-group advantage Aum
GRPO for inter-group advantage Aus
where A the final estimated per-token advantage for each response in O, + (<think>, <no_thol moo_thnnk>] } and a is the hyperparameter for controlling the weight of inter-group advantages. We also explore the impact of using different estimators in 3.3.
Optimize Objective. Following [1, 17, 41]. HGPO optimizes the policy model my by maximizing the following objective:
JuCIOO (8)
(p-P,, (or')% rum (oll), mEM]
N loil
(0, 14, (ct)
(o s..ct)
(11)
min
Kourr (0: 1
Af, clip
Towr (0t 19: m.
e.
e,1+e) A1)
DKa. (STll(TTa))
where
Tref (0," (q)
log Trer(o
DKL. (Trpl [Te)))
1.
(12)
To (o (q)
To (o Tm (q)
Here ThuT denotes the model we get from Stage in $ 2.2 and e and 3 are hyper-parameters.
2.4
Evaluating Hybrid Thinking Capability
To more comprehensively evaluate the performance of LHRMs, beyond conventional downstream task metrics, propose new metric called Hybrid Accuracy (HAre), which aims measure the LHRMs' ability to correctly select the appropriate reasoning pattern.
Given the task prompt set P {pp J LHRMs is first applied to sample N responses under two distinct reasoning modes 1- and K. reward model is then employed as a scorer to evaluate and assign scores to both sets of generated responses. The mode with the higher average score is regarded as the ground-trutt -ooun preferred reasoning mode for each prompt, denoted as Mg- In cases where the average scores of the two modes equal the difference between them falls below predefined margin, the mode yielding the shorter response selected as mgt Subsequently, we allow LHRMs to autonomously select a reasoning mode mp for each prompt. The proportion of prompts for which LHRMss selected mode matches the ground-truth mode is reported as
K
Hose
1
Σ
Equal (Mg. mp)
s.t. Mgs mp (+.F).
(13)
K
For queries with definitive answers, we use rule-based rewards [41, 15] for better reward estimation; otherwise, trained parametric reward model is applied.
5

---

3 Experimental Results
3.1 Experimental Setup
Compared Baselines. To validate the effectiveness of our proposed method LHRMs, we conduct a comprehensive comparison against state-of-the-ar LLMs and LRMs derived from the same base models. Specifically, we build our LHRMs on both 1.5B and 7B parameter versions of Qwen-2.5- math-base [52], and compare our method with several Qwen- 2.5-dervved variants, including:
LLMs: we compare our model with Qwen-2.5 Math series [52] and Instruct series [5 I], which show great capabilities in coding, mathematies and general tasks.
LRMs: Here we use DeepSeek-RI-D Ri-iistilll series [10], which are distilled using the reasoning
data generated by PrppSeek-R1100) and attain strong reasoning ability.
Hybrid: Due the absence of existing models capable of hybrid thinking, we compare our final model against the baseline we obtained in our Stage ($ 2.2) (denoted as HFT). Additionally, we construct two further baselines by applying Direct Preference Optimization (DPO) [40] and Rejection Sampling Fine-Tuning Tuning (RFT) [55] to the checkpoint Tewe from Stage I, using the same training data as in Stage II ($ 2.3). These baselines are referred as HFT-DPO and HFT-RFT, respectively. Implementation details of DPO and RFT can be found in Appendix A.
Evaluation Settings. We primarily evaluate model performance based on the following aspects:
Reasoning Capabilities. We evaluate models on a comprehensive set of widely-used reasoning benchmarks, including math related like MATH500 [26], AIME24 [2]. AMC23, Olympaid
Bench [16], and code related like LiveCodeBench [20]. MBPP [4] and MBPP+ [30].
General Capabilities. We assess the models' general capabilities through open-ended generation tasks using LLMS as judges. Specifically, we adopt AlpacaEval 2.0 [11) and Arena-Hard [25] to assess instruction-following ability and alignment with human preferences.
Hybrid Thinking Capabilities. We compute Hybrid Accuracy (HAce) ($ 2.4) on MATH500 to evaluate the model's ' ability to correctly select the appropriate reasoning mode.
More details about the evaluation settings can be found in Appendix D.
Training Settings. For stage I, we finally obtained 1.7 M SFT data and details about construction pipeline, sources and statisties can be found in & 2.2 and Appendix C. All models are trained for 3 epochs with the Adam optimizer, employing a 10% linear warmup followed by cosine learning rate decay schedule. The maximum learning rate is set to 1c-4, with a batch size of 128 and a maximum sequence length of 32k tokens. Training the 7B model in the SFT phase takes approximately 2.5 days on 4 nodes of NVIDIA H100 stations.
For stage II, we construct the training dataset by randomly sampling 76K queries from Deepscaler [32] and Tülu3 [23] (details can be found in Appendix C). We use
3 as the parametric reward model in Eq. 7. We use VeRL [43] to conduct experiments. By default, we use constant x 10-6 learning rate together with AdamW optimizer for policy model, and use a batch size of 256 and micro batchsize of 8. The rollout stage collects 256 prompts and samples 4 responses for each prompt. We set a = 1.0 and margin = 0.2 for RL training. We set KL coefficient to 0.001 and 0.5 in Eq. 11 in all experiments. The RL phase takes 2 days on NVIDIA 4xH100 Stations.
3.2 Main Results
Overall Performance.
The overall results of different models on the aforementioned benchmarks are presented Table 1. We observe that L.RMSs consistently outperform all comparable baselines across both the 1.5B and 7B model scales, achieving average improvements of 9.2% and 7.1 % compared to HFT-DPO, and 18.3% % and 11.5% compared to HFT-RFT, at the 1.5B and 7B scales, respectively. Specifically, it surpasses the strongest competing baseline, HFT-DPO, by 10.3% % and 13.6% on the AIME24 benchmark at 1.5B and 7B scales, respectively. On the 7B scale, LHRMs further outperforms HFT-DPO by 50.2% 2% on Alpaca and 93.4% Arena-Hard. Notably, AIME24 and Arena-Hare represent the most challenging benchmarks in the math and general domains, respectively. These results demonstrate the strong reasoning and general capabilities of our LHRMs.
http::/hwuggigggacccccolllll//m/ 1-Tulu-3-8B-B

---

Table Performance comparison across different tasks. "_" indicates that the model does not support hybrid thinking and is therefore left blank. The last column (Avg.) reports the average performance across all evaluation tasks, excluding the Hace metric. Bold numbers indicate the best performance. Type refers to the model's reasoning mode, where Hybrid denotes models that adaptively select between Thinking and No- No-Thinking modes for each query.
MATH
Code
General
Methods
Type
MATH500
AIME24
AMC21
Olympiad
LiveCode
MBPP
MBPP
Alpaca
Arena
Nare Avg.
size model
Qwen2.5 Swen2.5-Math-11 M.55
L.LMs
42.4
3.3
22.5
16.7
0.4
16.1
14.3
0.1
1.8
13.1
Owen2.5-1.5B-Ime
LLMs
51.0
52.8
38.7
60.1
51.9
8.8
30.0
Qwen2.5-Maie Math-l.5B 1.5t-Instruet Instruct
LLMs
72.0
6.7
60.0
38.1
3.7
26.7
23.8
2.8
4.7
26.5
DeepSeck-RI-DD Mil--istill--eee-
LRMs
83.9
28.9
62.9
43.3
16.8
54.2
46.3
5.6
2.7
38.3
HFT-1,5B
Hybrid
87.8
32.7
48,9
15.7
47.4
13.1
41,4
42.5
HFT-RFT-I.5B
Hybrid
82.2
22.0
67.5
44.1
14.2
49.7
42.6
13.6
8.5
48.1
38.3
HFT--PPO-1155
Hyhrid
86.8
32.6
75.0
48.7
17.2
50.5
42.6
13.3
6.9
45.8
41.5
LHRMs- 1.5B
Hybrid
87.8
35.3
75.0
50.4
17.2
61.1
54.0
16.9
10.4
54.4
45.3
7B size model
Qwen2.5 M-tt---
LLMs
57.0
13.3
22.5
21.8
6.0
31.5
27.3
2.0
7.0
20.9
Breen.5-7B-Insttenn
LLMs
77.0
52.8
29.1
14.6
79.9
67.5
36.2
44.0
Doven.5-Mutt-7B--nnni
LLMs
82.4
10.0
62.5
41.6
2.6
40.2
34.7
3.8
10.0
32.0
DeepSeek-RI
LRMs
92.8
55.5
91.5
58.1
37.6
74.3
64.3
19.1
17.9
56.8
HFT178
Hyhrid
93.6
56.7
95.0
58.5
34.7
70.6
59.8
23.7
14.0
34.2
56.4
HFT-RFT-7B
Hybrid
87.8
82.5
55.0
68.8
28.1
56.6
HFT-DPO-7B
Hybrid
93.8
58.7
92.5
60.6
38.8
80.1
68.3
23.3
13.0
37.1
58.9
LHRMs-7B
Hybrid 93.8
66.7
95.0
61.2
38.8
81.5
69.6
35.0
26.0
71.9
63.1
Furthermore, find that LHRMs achieves the best hybrid thinking performance, measured by Hace, significantly outperforming all baselines. For example, it exceeds HFT-DPO by 93.8% and RFT by 44.7%. This further demonstrates that our HGPO effectively enables the model learn correct hybrid thinking behaviors, providing a promising and practical pathway for building hybrid thinking systems. Next, we provide detailed investigation into each of the training stages.
Effect of HFT at Stage I. By comparing HFT with the Qwen2.5 and Deepseek- -ireeckk-----isili series, we observe that HFT significantly enhances both reasoning and general capabilities, while demonstrating robust cold-start performance by maintaining stability during hybrid reasoning without failure or collapse. These results validate the effectiveness of the proposed HFT 2.2) framework.
Effeet of HGPO at Stage II. By comparing LHRMs with HFT, we find that HGPO further substan- tially improves the model's reasoning and general capabilities, while enabling more effective selection of thinking modes (over 31.4% and 110.2% % Hase improvements at 1.5B and 7B size, respectively).
These findings demonstrate the effectiveness of HGPO.
Furthermore, when comparing LHRMs with HFT-DPO and HFT-RFT, we observe that LHRMs achieves superior downstream performance on both reasoning and general tasks, along with higher accuracy in reasoning mode selection. This highlights the effectiveness of HGPO.
Notably, we also observe that LHRMs exhibits stronger cross-domain generalization capabilities. Although RL training is conducted solely on math and general- data, LHRMs-1.55 achieves substantial improvements on code-related tasks, with gains of 11.1 % on MBPP and 13.9% on MBPP+. In contrast, both HFT-DPO-1.55 1.5BB and HFT-RFT-1.5B show performance drops (7.8% % and 9.3%, respectively) on the same tasks. This indicates that LHRMs is able to generalize learned hybrid thinking patterns across domains property not observed in DPO or RFT.
3.3 Analysis
Effect of Different Advantage Estimators. To investigate the impact of different advantage estimators HGPO training, we replace GRPO in Eq. 10 with RLOO [1] and REINFORCE+ [17]. Implementation details can be found in Appendix B. As shown Figure (a), all estimators yield competitive performance, indicating that HGPO is robust to the choice of advantage estimator.
Effect of Margin & in Eq. 9. investigate how varying the margin influences the model's bias toward the two reasoning modes. As shown in Figure (b), different values of result in distinct
7

---

Think Ratio WildChat
0.6
Reward Score
3.2
Reinforce++
RLCO
GRPD
0.4
marain 0.0
margin
margin 0.2
marain - 0.3
margin
0.2
0.0
50
100
150
200
250
300
350
400
25
50
75
100
125
150
Training Steps
Training Steps
Ablation study advantage estimators.
(b) Ablation study on margin
Figure 3: Ablation study on the effects of advantage estimators and margin 8. Eq. 9.
hybrid reasoning behaviors. Specifically, a larger 8 encourages the model favor the No-Thinking mode. This suggests that 5 can serve as a control knob for tailoring hybrid reasoning behavior to specific application needs. For instance, higher 8 may be preferred when real-time responsiveness is prioritized, whereas lower d is more suitable when reasoning quality is the primary concern.
Thinking Ratio Study within Domain. Figure 4 illus- trates the distribution of thinking ratio across varying diffi- culty levels in the MATH500 benchmark for both HFT-7B and LHRMs-7B. We observe that HFT-7B maintains a consistently high thinking ratio across all difficulty levels.
Ralio of Thinssing 187
100
80
60
In contrast, after applying the Stage II HGPO, LHRMs-7B exhibits decreasing thinking ratio the difficulty level decreases while getting higher overall performance (as shown in Table 1). This trend suggests that HGPO can effectively enable the model to adaptively perform hybrid thinking based the input query. Specifically, the model learns to respond to simpler queries using a Nh-Think strategy, thereby reducing reasoning cost and inference latency without sacrificing accuracy. For more challenging queries, the model consistently engages in full thinking, achieving higher reasoning accuracy where it matters most.
20
Stagel
Difficulty @ Figure Analysis of thinking ratio of
LHRMs within single domain.
Thinking Ratio Study across Domain. We investigate how HGPO affects model performance on unseen domains encountered during the RL phase. Specifically, duct RL using data from the math and general domains, and then evaluate the resulting models the code domain. As shown in Figure 5, we observe that the model attained from HFT exhibits a relatively low think ratio in the code domain. However, as RL progresses despite being con- ducted on unrelated domains such as math the think ratio in the code domain gradually increases, accompanied by improved performance. This suggests that our HGPO capable of transferring hybrid thinking patterns learned in specific domain (e.g., math) to other domains (e.g., code), demonstrating strong generalization and transferability.
100
Think Ratio
HFT
HGPO
Accurasy
70
61.1
40
1.58
Figure 5: Analysis of thinking ratio of LHRMs across different domains.
Thinking Ratio Study across Model Size. We analyze how model size influences hybrid thinking behavior during the RL. phase. In Figure 6, we present the think ratio of the
Think Rato MATHSB0
1.00
0.75
1.5B and 7B models on the MATH500 benchmark. We
observe that, as RL training progresses, the think ratio of the 7B model gradually decreases, while that of the 1.5B model increases. This indicates that hybrid thinking behav- ior during RL is strongly correlated with model capacity and knowledge storage. Specifically, larger models with stronger knowledge reserves tend to shift towards using No-Thinking strategies on solvable problems to reduce reasoning overhead and improve response efficiency. In contrast, smaller models are more inclined to invoke Thinking more frequently, as a way to compensate for their limited capabilities through enhanced reasoning.
0.25
NHTMs-1.58 1.58
0.00
Training Steps
Figure 6: Ablation study model size.

---

Question: If
9g 26 oh 5, what the value of
DeepSeck-RR. -I-issi-- --1-Disill-Queenn Sueilll--e---BB <thinks Okay, let's lackle this problem. So, we have this operation delined as a star equals 9a plus minus ab plus And need to find the value of star 1. Hmm, let me make understard the question... (2941 characters) </think>
To solve 5 + 1 using the detined operation a b 9u + 26 ab 5:
Substitute
and
expression:
9(5) + 2(1) (5)(1) +
2. Calculate each term: 9 x 5 45 2 = 2-5 x =
3. Substitute these values back into the expression: 15 + 5
4. Perform the arithmetic operations step-by-step:
45
47
5 42 42
Thus,
47
LHRMM-7B: <to__titne think> To solve the problem, we noed to substitute = =5 and into the given function a b 9a
Step-by-step solution:
1. Start with the given function: a 9a + 26
ab +
Substitute
function;
9(5) (21))++(11)))))) 5(1) +
3. Perform the multiplication operations: 5 = 45
5
4. Simplity the expression by performing the addition and subtraction operations:
+ 5
Therefore, the final answer is: 47 /
Figure 7: Example outputs from Deepseek-R1-Disiil---eeeee and our LHRMs-7B model. For easy questions, LHRMs-7B selects the No-Thinking mode.
Case Analysis. We present representative model responses in Figure and We observe that for relatively simple queries such as everyday questions and basic math questions, LHRMs tends to autonomously select the No-Thinking mode, achieving fast and accurate responses. contrast, for more complex problems that require deeper reasoning, LHRMs adaptively switches to the Thinking mode to produce more precise solutions. More example responses can be found in Appendix G.
Additional Analysis. To further verify the effectiveness of our method, we conduct additional analyses in Appendix E, including the impact of hybrid training reasoning ability (E.1), an efficiency analysis (E.2), an ablation comparing HGPO with standard GRPO (E.3), and an annotation consistency validation for Hybrid Accuracy (E.4).
4 Related Works
Test-Time Scaling (TTS). Test-time scaling [12, 15, 37, 19] has been validated as promising approach enhance model performance beyond scaling model size [44]. There are two primary approaches to implementing TTS [37]: Parallel and Sequential. The former is achieved by sampling multiple solutions and select the one by a selector (e.g., an outcome reward model), like Best-of- N [5, 19, 24), and Monte-Carlo Tree Search (MCTS) [28, 56, 61, 8). The latter aims to achieve TTS by enabling the model to generate longer outputs, i.c., ooain--o-TThuuhtss (CoT) within a single sampling pass by prompting, finetuning reinforcement Icarning. Beyond the field of NLP, TTS has also been shown to effectively improve the test-time performance of trained models in other domains, like image generation, video generations [27] and multi modliittddllty learning [13, 18, 6).
Large Reasoning Models. Recent advances in Large Reasoning Models (LRMs), such as DeepSeek- R1 [10], OpenAl ol/o3 series [38], and others [46, 37, 3, 14], have led to a growing focus Large Reasoning Models (LRMs). Compared to general LLMs, LRMs extend the capabilities LLMs by generating long chains of reasoning steps with reflection before outputting the final answers. LRMs are typically developed by applying reinforcement learning such as GRPO [15], REINFORCE++ [17], and related approaches [7, 53], distilled from stronger models [45, 35, 12, 5715
5 Conclusion
In this work, we focus on building a large language model that effectively balances reasoning capabilities and general-purpose performance. To this end, propose novel evaluation metric, Hace- designed to consistently assess a model's 's ability to perform hybrid thinking across diverse tasks. We further introduce two-stage training pipeline consisting HFT and HGPO. Experimental results demonstrate that this pipeline significantly improves performance on both crasoning-centii and general downstream tasks. Moreover, it enhances the model's 's hybrid thinking capabilities, leading to a better user experience by reducing unnecessary reasoning on simple queries commonly observed in L LRMs, and mitigating the insufficient reasoning capabilities in conventional LLMs.

---

References
[1] A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. In L. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 2224-11227, Association for Computational Linguistics, 2024.
[2] AI-MO. Aime 2024, 2024.
[3] Anthropic. Claude 3.7 sonnet and claude code. http::/www..atttnriicccco//ev/ea sonnet, 2025.
[4] Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang. C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 2021.
[5] B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. Ré, and A. Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024.
[6] L. Chen, L. Li, H. Zhao, Y. Song, and Vinci. RI-v: Reinforcing super generalization ability in vision-language models with less than S3. https: hetp/////ttbbbb..m/eeeeep-p- Agent/R1-V, 2025. Accessed: 2025-02-02.
[7] D. Cheng. S. Huang, X. Zhu, Dai, W. X. Zhao, Zhang, and Wei. Reasoning with exploration: An entropy perspective on reinforcement learning for llms, 2025.
[8] S. Choi, T. Fang. Z. Wang, and Y. Song. Kets: Knowledgee-consrrinned tree search decoding with token-level hallucination detection, 2023.
[9] G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with hugh-quality feedback, 2023.
[10] DeepSeck-AL. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement
learning, 2025.
[11] Y. Dubois, B. Galambosi, P. Liang. and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475 2024.
[12] H. Face. Open rl: A fully open reproduction of deepseek-rl, January 2025.
[13] K. Feng. K. Gong, B. Li, Z. Guo, Y. Wang. T. Peng, B. Wang, and X. Yue. Video-rl: Reinforeing video reasoning in milms. arXiv preprint arXiv:2503.21776, 2025.
[14] Google. Gemini 2.5 flash. 2025.
[15] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[16] C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad- -eeel bilingual multimodal scientific problems, 2024.
[17] J. Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501 003662, 2025.
[18] W. Huang. B. Jia, Z. Zhai, S. Cao, Z. Ye. F. Zhao, Y. Hu, and S. Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint #xXiv:2503.06749 2025.
[19] R. Irvine, D. Boubert, V. Raina, A. Liusie, Z. Zhu, V. Mudupalli, A. Korshuk, Z. Liu, F. Cremer, V. Assassi, C.-C. Beauchamp, X. Lu, T. Rialan, and W. Beauchamp. Rewarding chatbots for real-world engagement with millions of users, 2023.
10

---

[20] N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar- Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07977 077744, 2024.
[21] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov. Fasttext.zip: Com- pressing text classification models. arXiv preprint arXiv: 1612.0661, 2016.
[22] A. Köpf. Y. Kilcher, D. Von Rütte, S. Anagnostidis, Z. Tam, K. Stevens, A. Barhoum, D. Nguyen, O. Stanley, R. Nagyfi, et al. Openassistant snnversationsddmmrrddae large language model alignment. Advances in Neural Information Processing Systems, 36:47669- 47681, 2023.
[23] N. Lambert, J. Morrison, V. Pyatkin, S. Huang. H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, Malik, V. Graf, J. D. Hwang, J. Yang. R. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training, 2024.
[24] N. Levi. A simple model of inference scaling laws, 2024.
[25] Li, W.-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica. From crowdsourced data to high-quality benchmarks: Arena-hare and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024.
[26] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike. J. Schulman, I. Sutskever, and K. Cobbe. Let's verify step by step, arXiv preprint arXiv:2305.20050, 2023,
[27] Liu, H. Wang, Cai, K. Zhang. X. Zhan, and Y. Duan. Video-tl: Test-time scaling for video generation. arXiv preprint arXiv:2503. 18942. 2025.
[28] J. Liu, A. Cohen, R. Pasunuru, Y. Choi, H. Hajishirzi, and A. Celikyilmaz. Don't 't throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding, 2024.
[29] J. Liu, S. Xia, Y. Wang, and L.. Zhang. Is your code generated by chatgpt really correct? rigor- ous evaluation of large language models for code generation. Advances Neural Information Processing Systems, 36:21558-22 215722, 2023.
[30] J. Liu, S. Xie. J. Wang. Y. Wei, Y. Ding, and L. Zhang. Evaluating language models for efficient code generation. arXiv preprint arXiv:2408. 00.40650, 2024.
[31] H. Luo, L. Shen, H. He, Y. Wang, S. Liu, W. Li, N. Tan, X. Cao, and D. Tao. Ol-pruner: Length-barmonizing fine-tuning for ol-like reasoning pruning, 2025.
[32] M. Luo, S. Tan, J. Wong, X. Shi, W. Tang, M. Roongta, C. Cai, J. Luo, T. Zhang, E. Li, R. A. Popa, and I. Stoica. Deepscaler: Surpassing ol op-preveew with 1.5b model by scaling rl, 2025. Notion Blog.
[33] X. Ma, G. Wan, R. Yu, G. Fang, and X. Wang. Cot-valve: Length-compressible chain-of-thought tuning, 2025.
[34] S. Mahdavi, M. Li, K. Liu, C. Thrampoulidis, L. Sigal, and R. Liao. Leveraging online olympiad-level math problems for llms training and coaiminntiio-resistaai evaluation. arXiv preprint arXiv:2501. 14275,, 2025.
[35] J. Mattern, S. Jaghouar, M. Basra, J. Straube, M. D. Ferrante, F. Gabriel, J. M. Ong, V. Weisser, and J. Hagemann. Synthetic-l: Two million collaboratively generated reasoning traces from deepseck-rl, 2025.
[36] I. Moshkov, D. Hanley, I. Sorokin, S. Toshniwal. C. Henkel, B. Schifferer, W. Du, and I. Git- man. Aimo-2 winning solution: Building state-of-the-an mathematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.16891 2025.
[37] N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393 2025.
11

---

[38] OpenAl. Openai gpt-4.5 system card. OpenAI Publication, 2025.
[39] G. Penedo, A. Lozhkov, H. Kydlfček, L. B. Allal, E. Beeching, A. P. Lajarín, Q. Gallouédec, co /
N. Habib, L. Tunstall, and L. von Werra. Codeforces.
open-r1/codeforces, 2025.
[40] R. Rafailov, A. Sharma, F. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53744
2023.
[41] Z. Shao, P. Wang. Q. Zhu, R. Xu, J. Song, X. Bi. H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepscekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.055 030000000000..00000000000000000 2024.
[42] Y. Shen, J. Zhang, J. Huang, S. Shi. W. Zhang, J. Yan, N. Wang, K. Wang, Z. Liu, and S. Lian. Dast: Difiiuult--daaptive slow-thinking for large reasoning models, 2025.
[43] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng. H. Lin, and C. Wu. Hybridflow:
A flexible and efficient rihf framework. arXiv preprint arViv: 2409. 24256, 2024.
[44] C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03311 03314, 2024.
[45] O. Team. Open Thoughts. https//open-thoughtte ai, Jan. 2025.
[46] Q. Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024.
[47] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q V. Le, D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
[48] 7.. Xie, L. Chen, W. Mao, J. Xu, I.. Kong, et al. Teaching language models to critique via reinforcement learning. arXiv preprint arXiv:2502.03492 2025.
[49] Xu, Q. Hao, Z. Zong, J. Wang, Y. Zhang, Wang, X. Lan, Gong, T. Ouyang, F. Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arxiv. 10.48550. arXiv preprint arxiv.2501. 1,0666,, 2025.
[50] Z. Xu, Y. Liu, Y. Yin, M. Zhou, and R. Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding, 2025.
[51] A. Yang, B. Yang, B. Zhang. B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang. H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang. K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang. T. Xia, X. Ren, X. Ren, Fan, Y. Su, Y. Zhang, Wan, Y. Liu, Z. Cui, Z. Zhang, and Qiu. Qwen2.5 technical report. 2024.
arXiv preprint arXiv:2412.15115
[52] A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. D. Liu, J. Tu, J. Zhou, J. Lin, K. Lu, M. Xue, R. Lin, T. Liu, X. Ren, and Z. Zhang, Qwen2.5-mal technical report: Toward mathematical expert model via self-improvemennt arXiv preprint arXiv:2409.
2024.
[53] C. Yang, Q. Si, Y. Duan, Z. Zhu, C. Zhu, Q. Li, M. Chen, Z. Lin, and W. Wang. Dynamic early exit in reasoning models, 2025.
[54] Y. Ye, Z. Huang, Y. Xiao, E. Chem, S. Xia, and P. Liu. Limo: Less is more for reasoning, 2025.
[55] Z. Yuan, H. Yuan, C. Li, G. Dong, K. Lu, C. Tan, C. Zhou, and J. Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825. 2023.
[56] S. Zhang, Z. Chen, Y. Shen, M. Ding, J. B. Tenenbaum, and C. Gan. Planning with large language models for code generation, 2023.
[57] H. Zhao, H. Wang, Y. Peng, Zhao, X. Tian, S. Chen, Y. Ji, and X. Li. 1.4 million open-source distilled reasoning dataset to empower large language model training, 2025.
12

---

[58] W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng. Wildchat: Im chatGPT interaction logs in the wild. The Twelfth International Conference on Learning Representations, 2024.
[59] L. Zheng, W.-L. Chiang, Y. Sheng. S. Zhuang, Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
[60] Y. Zheng, R. Zhang. J. Zhang, Y. Ye, Z. Luo, Z. Feng. and Y. Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics.
[61] A. Zhou, K. Yan, M. Shlapentokh- H. Wang, and Y.-X. Wang. Language agent tree search unifies reasoning acting and planning in language models, 2024.
13

---

Contents
1
Introduction
2 Large
Models
2.1 Problem Formulation
2.2 Stage I: Hybrid Fine-Tuning
2.3 Stage II: Hybrid Group Policy Optimization
2.4
Evaluating Hybrid Thinking Capability
3
Experimental Results
3.1 Experimental Setup
3.2
Main Results
3.3
Analysis
4 Related Works
5
Conclusion
A
Implementation Details for DPO and RFT
15
Implementation Details for RLOO and Reinforce++
15
Dataset Statisties
16
Evaluation Settings
16
Additional Experiments
17
E.1
Comparison between Hybrid and Non-Hybrid Models
17
E,2
Efficiency and Compression Analysis
17
E,3 HGPO vs. Standard GRPO under Equivalent Training Resources
18
E,4
Annotation Consistency Analysis for Hybrid Accuracy
19
Future Work
19
G Example Outputs
20
14

---

A Implementation Details for DPO and RFT
In this section, detail the construction pipeline of training data for both DPO and RFT used in $ 3. To construct datasets for DPO and RFT, we adopt the same set of queries used in HGPO for offline sampling. Specifically, for each query q, sample responses using each of the two thinking modes m E (H, resulting in a set of four responses per query as
( = for } of
(14)
Then the reward function R. in Eq. 15 applied to score these responses as:
R² = (r(o
R = (r(d)))
(15)
Given the average reward for each mode computed as
R Σ
r(d)),
R = Σ r(os))
(16)
DPO, For DPO, the training dataset which contains win and lose sample is constructed following:
= (1. Ow, ot)
{(
DrRO
Ow arg max (r)), of arg min (r (r())
(17)
oE (o " Piz
o (o Pe
while
mw arg max R mi arg min Rm mE(+r)
(18)
m /}
After getting DDrO, optimize the model To initialized from Tewe by using training objective:
max E ((..ow.:) logo Blog ( a:) B log T Tour (or 2)
To xx)
(19)
ne
Thur (Ow x:)
RFT. For RFT, the training dataset is constructed as following:
DRFT
(g.0) o)
arg max (R URTURR AEO
(20)
The training objectives for RFT are presented as:
|o]
CRIT (0) = -(((
log To (or 01:-1)
(21)
For implementations, ensure a fair comparison, we maintain the same learning rate, batch size, and total number of training samples as used in Stage II of HGPO.
LLaMA-Factory [60] the codebase for both DPO and RFT. To
B Implementation Details for RLOO and Reinforce++
First all, We compare different advantage estimators including REINFORCE+ [17], GRPO [41], and RLOO [1], toggling the existence of our HGPO. To make different algorithms compatible with the compound of intra- intra-group group rewards and inter-group rewards, we accordingly make adaptions similar to Eq. 10. For Reinforce++, we have
|o.|
A =
Σ
Tintra (0:)
+
1f E Φ) [Fimter (o:)
(22)
REINFORCE++ for inter-group advantage Anna
REINFORCE+ intra-group advantage Anee
'Same with $ 2.3, for queries definitive answers, we use rule-based rewards [41, 15]; otherwise, trained parametric reward model [23] applicd. htnns//gittubboom/yyyggggLLLLLL
15

---

Here is a hyperparameter representing the decay factor, which is set to 0.99 in our experiments. For RLOO, we have
A
Pintra (0s)
Σ
Tintra (0z) J.f
+1fo, E o Timeer (o.)
1
Σ 394
(rimter (oz))
(23)
N 1--1
N
RLOD for intra-group advantage Aben
RLO00 for inter-group advantage Aisoe
From $ 3.3, we show that HGPO is general plug-in for almost any advantage estimator, which largely extends the use cases of HGPO. We implement both RLOO and Reinforce++ on VeRL [43]
C Dataset Statistics
Stage Figure shows the token length distributions for the Thinking and No- No-Thinking datasets we construct in Stgae I. The Thinking data has an average length of 575 tokens, with the 25th and 75th percentiles 362 and 672 tokens, respectively, and maximum length of 9,148 tokens. contrast, the No-Thinking data exhibits significantly higher average length of 4,897 tokens, with the 25th and 75th percentiles at 1,719 and 6,738 tokens, and maximum of 23,997 tokens.
We present the scores and statistics of our constructed dataset for HFT in Table 2. The dataset covers diverse range of domains, primarily including reasoning- intensive tasks such as mathematics and code, as well as veneral-purpose question answering.
-aasoning-intensivee
Stage II. We report the details of the training data used in Stage II in Table 3.
Table Data distribution and source of Stage I.
Category
Source
Data Size
Total
Link
General
WildChat-IM [58]
649,569
674,908
htrps /hoggig codamnettaallcma//iilhhss///////
OSSAT2 (22)
25,339
hitps /huggingacee hat.aaacwwouuuatooooossssssssaaa
SYNTHETIC-1 [35]
343,988
https Mhuggingface www.aa/eweeeiicccnnneeesee//eeee
OpenKI-Mall [12]
93,533
ttpss//ugging www.ggggggggggggoooooooooooooooe
MATH
OpenThought [45]
55,566
631,325
hipps https//hugiingg ing www..oo/aatteeeeeeneennt thwneeeeeetooottoooooooooogogoeo 114k
AoPS [34]
137,497
htpps htt.g/hggggggaoooooooe codddtaneerddiiihaagggaaaa/aaa
AIME
741
https//hugginn /hugging ing www..../daattttte/eeoo..o zhang fdu/AIME 1493 2024
SYNTHETIC-I [35)
107,543
htpps /huggingface www.aa/ewwwooiiooooooeooeoeeeeee
OpenR 1-RRI-CCoe Codeeforces [39]
8,926
htpp://hugghn hwwge//gggggaanoo wwwpe//wwwwaaeeeeeeeeeeeeeeeeeee
Coding
OpenThought [45]
19,447
381,845
htrps httpg//iiii..ttoooottooooooeeeee
KodCode (50)
245,929
htips https///uugllaaaccccoo/ooo
Others
SYNTHETIC-I [35]
6,508
6,508
https
Total
1,694,586
1,694,586
Table 3: Data distribution and source of Stage II. Note that our experiments are conducted on subsets of the two datasets rather than their full versions.
Caurta tatt
12ok
Think
00k
40k
Category
Source
Data Size
Used
10kk
15k
20k
Length Takann
General
Tulu [23]
Figure 8: Token length distributions of Thinking and No- do-Thinking data in Stage
Tulu preference dataset
337,000
50,000
DeepScaler [32]
MATH
DeepScaleR Preview: Dataset
40,300
40,000
D Evaluation Settings
In this section, introduce the details for downstream tasks evaluation in $ 3. During inference. we set the temperature to 0.6, the top-p value to 0.95, and the maximum generation length to 35,000 tokens.
16

---

For MATH500, AIME24, GPQA Diamond. and LiveCodeBench, we adopt the OpenRI [12] evaluation framework. For MBPP and MBPP+, we use the EvalPlus [29, 30] framework with Chain-of-Thought (CoT) [47] prompting. We report pass@) for all reasoning related benchmarks. For AlpacaEval 2.0, adopt evaluation queries from -ecna-Hard-2..0 and use GPT-4o-mini as the baseline. To reduce length-related bias, only the final generated summary is submitted to the judge in think mode.
4PT---Turbo as the baseline model, while for Arena-Hand Hard,
We provide summarization and links of evaluation tools we used experiments Table
Table 4: Summarization of evaluation tools we used in experiments.
Task
Tool Name
Source
MATH500 126). AIME24 (2), Livecodebench (20)
open-rl [12] maih evaluntion hurness
Olympaid Bench, AMC23
bltps://giiubb.. httpuuuuo///uuutaaeeeae Zown//uiiGoo/maala evaluntion
evaltiatievvlltttoon hurness
MBPP14. 30), MBPP (4]
evalplus (29, 30]
https:// https:///tttubbb. http///gtuubbbbgb www//valpluss/vallpaa..a
AlpacaEval [11]
aplaca _evaleellllll
hllps:// bttpe://giluub.co comn/aasa lab/alpoca. eval
ArenaHard v2.0 [25]
arema-huin homa-hard-auto
http::/githuh.coo com//mareen//anenee hard auto
E
Additional Experiments
E.1
Comparison between Hybrid and Non-Hybrid Models
To further validate the robustness of our hybrid training framework, perform controlled compari- son between hybrid-trained and non-hybrid models to assess whether hybrid training compromises reasoning capability.
Specifically, we fine-tuned both the 1.5B and 7B variants of Qwen-2.5- 2.5-Mahhh on approximately I.IM math and code reasoning samples formatted in the Thinking-style. These models were trained using identical optimization settings as our hybrid models, allowing for a direct and fair comparison. The two model configurations are defined as follows:
WureThinn-SFT) models. These models are trained purely on ~..IM Thinking-style math and code data using standard supervised fine-tuning (SFT), without any hybrid formatting or No-Thinking examples.
HFT (Think-Only) models. These models are trained using our Hybrid Fine-Tuning (HFT) pipeline, which jointly incorporates both Thinking and No-Thinking data. During inference, the models are constrained to always operate in the Thinking mode to enable a direct comparison with PureThink-SFT.
As presented in Table 5, both model types achieve nearly identical results across multiple reasoning benchmarks at both the 1 1.5B and 7B scales. This consistency indicates that hybrid training does not impair reasoning performance. Instead, provides the additional flexibility of supporting adaptive reasoning modes without compromising capability.
Table 5: Comparison between Hybrid and Non- Hob-Hy Models on Reasoning Intensive Benchmarks.
Model (Training Type)
MATH500
AIME24
AMC23
Olympiad
LiveCodeBench
MBPP
MBPP+
Avg.
TureThin--FFT1155B
87.6
34.7
72.5
17.2
63.5
53.2
54.1
HFT-1.5B (Think Only)
87.8
32.7
75.0
50.0
16.8
62.2
52.9
53.9
PureThink-SFFE
94.0
56.7
92.5
59.9
38.7
80.7
68.3
70.1
HFT-7B (Think Only)
93.6
56.7
95.0
59.7
38.4
80.3
68.9
70.4
E.2 Efficiency and Compression Analysis
To further analyze the efficiency of the proposed hybrid reasoning approach, we compared the average thinking token length and corresponding accuracies of our models against several leading Chain-of- Thought (CoT) compression methods, including O1-Pruner [31], CoT-Vale Valve [33], and DAST [42].
17

---

As shown in Table 6, our Large compression and accuracy. Compared to full-length thinking baselines (HFT-1. 111-1..BBThink and HFT-7B-Think), L.RMs significantly reduce reasoning lengths across most tasks while maintaining or improving performance.
Models (LHRMs) achieve better balance between
Although some prior CoT compression approaches produce shorter outputs (e.g., Ol-Pruner, DAST), they often exhibit accuracy degradation. In contrast, our adaptive hybrid framework attains com- petitive compression with stronger accuracy, validating its effectiveness for efficient and reliable inference.
Table 6: Efficiency and compression analysis: benchmark accuracy (outside parentheses) and average thinking token length (inside parentheses) across different CoT compression methods and model variants.
Model
MATHE500
AIME34
AMC13
LiveCadeBnnch
MBPP
MBPPPs
AlpacaEval2.
ArenaHarra2.0
1.58 Size Midelss
HPT-1 SR-Tiink
(438
12-3 (13311
25.0(10111)
(1049)))))
(686)1362))
62.2 (5399)
$2.9 (5830)
(1927)
(.3(106)))
44.7 (8314)
HFT-1.58
87.8 (4330)
(13-31)
75.0(10111)
(1049))
15.7 (7464)
54.8 (2913)
47.4 (2888)
13.1
6.0 (4287)
42.5((333))
01 Priner
(2884)
(10655)
(6155
89.7 (0268)
16 (125))
(711)
(713
130 (895)
0.3 (5420)
40. (4887)
04.812428)
(4968)
(6.3)
(1413)
40.1-3909)
7U Stze Models
HET TB Think
(3.5
56.7 (10778)
(G455)
59.7 (3376)
38.0112000)
$0.3 (3251)
68.9 (3251)
(L7231
(7442)
60.7 (6332)
HPI-78
(3.6(3604)
56.7 (10778)
05.00
56.5 (7870)
34.7 (7739))
70.6 (1658)
59.8 (1638)
23.7
(2206)
36.4 (4728)
Ol-Pruncer
03.0.2885)
53.0 (9815)
(4725)
60.8 (5901)
35.1 (8351)
79.4 (22 10)
67.9 (22:00)
24.4 (966)
(202))
57.7 (4196)
DAST
(2.6772977
06. (8835)
(472)
(4725)
58.1 (4905)
32.9 (84155
06.4 (48471
(1074)
21: (855)
(1765))
53.3 33.5(9887)
Mive
(3.4(2964)
(3122)
(2067)
(2957)
(1553)
38.9(5097)
03.8(2616)
(11031)
(4976)
(7540)
(8432)
(1906)
(1086)
(3400
63.1(4758)
These findings highlight the practicality of our adaptive hybrid reasoning framework, which maintains strong accuracy while significantly improving inference efficiency.
E.3 HGPO vs. Standard GRPO under Equivalent Training Resources
To verify that the improvement brought by our Hybrid Guided Policy Optimization (HGPO) is due to additional computational resources, we conducted controlled reinforcement learning (RL) experiments comparing HGPO with standard GRPO under identical setups.
Both baseline models (HFT-1.5B and HFT-7B) were trained using the same RL dataset consisting of 76K queries (sourced from DeepScaleR and Tülu3) and identical hyperparameters. This ensures that any observed differences arise purely from the optimization objective rather than training cost.
As summarizzd in Table 7, both strategies yield similar performance on ceasoning-intensii tasks. general-purpose However, HGPO-traiie trained models consistently outperform GRPO-traine ones general purpose benchmarks, resulting in higher overall averages. We attribute this improvement to HGPO's adaptive selection constraint, which promotes more effective utilization of the "no-think" mode for simple queries and reserves reasoning capacity for complex ones.
In contrast, GRPO lacks this adaptive mechanism and underutilizes the "no-think" mode during rollouts, leading to weaker general-tat task performance. Moreover, HGPO models achieve significantly higher hybrid accuracy (Hice), reflecting stronger ability to select the appropriate reasoning mode per query.
These results confirm that adaptive thinking enhances model effectiveness without requiring extra computational resources, establishing HGPO key contributor the efficiency and capability of hybrid reasoning systems.
Table 7: Performance comparison between HGPO and standard GRPO under equivalent training resources. Hase represents hybrid accuracy.
Mode
MATHS50
AIME24
AMU23
OlympiadBench
LiveCodeBescc
MIPP
MBPP,
AlpaccaEval2..
ArcnHHard2.0
H
Aug.
HFT--.88 LSB
X7.8
32.7
75.0
45.9
15.7
54.5
47.4
13.1
6.9
41.4
42.5
HFT-GRPO-1.58
89.0
34.7
73.1
50.2
17.9
61.4
12.9
14.3
8.1
47.4
44.6
LHRMi-1-55 1.53
X7.8
35.3
75.0
$8.4
17.2
61.1
54.0
16.1
10.4
54.4
45.3
HFT77
93.6
56.7
95.0
38.5
34.7
71.6
59.8
23.7
14.7
34.2
56.4
HFT GRPOOR -7
95.2
61.0
98.4
61.5
39.6
79.6
67.2
25.8
18.7
41.9
60.7
LIERMs 7B
93.8
66.7
95.0
61.2
38.8
$1.5
69.6
35.0
26.7
71.9
63.1
18

---

E,4 Annotation Consistency Analysis for Hybrid Accuracy
To ensure the fairness and reliability of our Hybrid Accuracy (Haee) evaluation, we further computed Hace using both GPT-4o and human annotations in addition to the Tulu-3-8B-Rul reward model. For human evaluations, we randomly sampled 100 queries per model and engaged three annotators, each labeling 800 samples independently.
As shown in Table the relative ranking of models based Hybrid Accuracy remains consistent across all annotation sources (GPT-4o, human, and Tulu-3-8B-RMY, All evaluation methods con- firm that LHRMs-7B and LHRMs-1.55 achieve substantial improvements in selecting the correct reasoning mode.
Furthermore, analyzed the agreement across different annotation sources, summarized in Table 9. Human-to tantan-to-humaaal human agreement averaged around 60%, while human-to-Ra R.M agreement was slightly higher (62.5%). Notably, GPT-4o showed the highest consistency, comparable agreement levels reported in MT-Bench [59] and Ultraeeddback [9]. These results validate the robustness and fairness of using Tulu-3-8B-RM for hybrid accuracy evaluation.
Table 8: Hybrid Accuracy (%) of different mod- els evaluated by Tulu-3-8B-RM. GPT-4o, and human annotators.
Table 9: Agreement rates (%) between annota- tion sources. GPT-4o shows the highest consis- tency, confirming reliable alignment with human judgment.
Model
RM
GPT-40
Human
Avg.
HFT-
41.4
41.3
42.0
41.6
GPT-4o
RM
H-1
H-2
H-3
Avg.
HFT-1.5B
48.1
48.2
46.3
47.5
GPT-4o
78.5
69.8
70.1
69.5
69.8
HFT-DPO-1.5B
45.8
43.4
43.3
44.2
RM
61.0
64.1
62.3
62.5
LHRMs-1.55
54.4
52.8
53.7
53.6
H-1
59.6
61.8
60.7
H1-2
58.6
59.1
HFT-7B
34.2
30.0
39.0
34.4
H-3
60.2
HFT-RFT-7B
49.7
48.6
47.7
48.7
HFT-DPO-7B
37.1
35.4
39.0
37.2
LHRMs-7B
71.9
71.5
72.0
71.8
Future Work
While our current work focuses enabling adaptive hybrid thinking based query context, there remain several promising directions future exploration. One potential enhancement to allow model to first reason about the context and complexity of the input before deciding whether to engage in extended thinking or direct response generation. This preliminary reasoning step could improve the accuracy of mode selection, leading to better balance between performance and efficiency. We plan to explore this approach in future work to further strengthen the adaptability and generality of hybrid thinking framework.
19

---

G
Example Outputs
Question: value 513² 487 Let's step by step output answer boxed(). Ground 26000
Okay, upproach this. remember that there's ' formula for the difference of squares, which is oF (a byta + b). Maybe I can use that bere calculating each square separately, which might consuming. (3837 characters) </thinkk>
squared minus squared.
think about
To solve 513 4872. we the difference squares formula a² 6 (a b)(a b3.
1. Identify a 513 and 487. Calculate a
3. Calculate b:
513 + 487 1000
4. Multiply the results from steps and 3:
1000 26000
Verification direct calculation: 5135 263169- 4877 237169 237169-Subrractin these values: 263169 237169 26000
Both methods confirm the result Therefore,
26000
L LHRMs-7B: <bo_ <bo_think think> To solve 513 4872, we can use the difference of squares foruula, which states that - = (a++ Here, a 513 and = 487.
Step Calculate b:
513 + 487 1000
Step 2: Calculate a b:
513 487 26
Step 3: Use the difference of squares formula:
513
487 (613 487)(313 487)
313² 487 1000 x 26
Step
multiplication:
1000 x 26 26000
Therefore, final
Figure LHRMs-78 uses the No- No-Thinking mode solve easy math problems.
20

---

Question: You are given an arrny of integers nuno of length n and positive integer k. The power of array is defined as: its maximum element
elements are consecutive and sorted ascending order, and otherwise. need to the power where resulta [i] is the of nune (1. (i
all subarrays of nuns of size Return integer results of size
k 1)].
HFFT-::
Kno, thinks To solve this problem, we need to evaluate the power of all subamays of a given size x from the inpur array nuns. The power of subarray defined maximum element if elements are consecutive and sorted ascending order: otherwise,
step by step plan
problem:
I. lterate through each subarray of size We will slide a window of size k through the array and examine each subarray.
2. Check if the subarray is sorted and consists of consecutive integers:
First, check the subarray sorted ascending order.
check if
of the subaray consecutive integers.
3. Calculate the power each subarray: If subarray the criteria, the the maximum element:; otherwise,
Python implements this logic:
class Solution:
resultsArray(solf, nuns: [int], int) List [int]:
(euns))))
result
for 1 in range(n + 1):
subarray muns(i:1 sorted_eubarray sorted(eubarray)
nax max_element subarray[-1]
if subarray sorted_ subarray and max_element subarray result. append (nax, (max_alemant))
elae:
append())
result
//no_think3
LHRMs-7B:
<think> Okay, to solve problem
power all subarrays the maximum element if all clements consecutive and sorted in asceuding order. Otherwise, it's Hmmm, let's break down the problem. So for each subarray of length k. have to check two things: whether the elements are consecutive and whether they are sorted in ascending order (2645 characters) </think>
given array. power
To solve this problem. we need to determine the power of all subarrays of given size from an arry of integers. The power of subarray is do- inlegers. Otherwise, the power
maximum
suburay both
ascending orler consists
Approach
1. Iterate through each possible subarray: For each starting index from 0 to 'a x', extract the subarray of length
Check the subarray is sorted: Verily each element in the suburray is less than the next element. not, the power is Check the subarray consists consecutive integers: Ensure each element exactly power -1
than the previous elemeut. the
power: both conditions are met, power consecutive soquence). Otherwise, the power is -1.
suburray
clase Solution:
def rerullssrrrayselff, nuns: List(int), x: int) List[int]:
result []
u (nun))
range(n aub nuna[i:1+k]
Check the subarray sorted ascending order is_eorted True for j in range(1en(aub) 1): it sub[j] >= sub[]+i]:
is_sorted False break
if not 1e_sorted:
reeult.appead(-1)
continue
Check 1f the subarray consists of consecutive integers
True
for j in range(1en - 1):
if sub[j+1] sub[i] is_consecutive False
break ss_consecutive: result, appand
result, (-1)
return result
Figure 10: LHRMs-7B learned to select the Thinking mode to solve hard code problems.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, our main contribution is building hybrid thinking system, which is clearly and accurately described both the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes].
Justification:
We discuss the limitations of work in Appendix ??.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
We start from the construction of a hybrid thinking system and propose corresponding training algorithms. Our work does not involve theoretical derivation or validation.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided detailed reports on our experimental data (see in Ap- pendix C), the implementation details of the baseline methods (see in Appendix A) and own algorithm (see in $ 3), the evaluation tools (see in Appendix D) we used in experiments at both maintext and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes].
Justification: Our training data is constructed by combining multiple publicly available datasets, with all details provided in Appendix C. The training code will be included the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
cc/
wasssoonooicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
wttp:cccpppppppp...e.eeeeeelepee gww.///g///////eeeeeeeecc httpr/gguudeeeeeeeeeebiieeeeeeee /Co///oodeeeeeeeeeeeeeeeeee orsiionnoocccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided detailed reports on our experimental data (see in Ap- pendix C), the implementation details of the baseline methods (see in Appendix A) and our own algorithm (see in $ 3), the evaluation tools (see in Appendix D) we used in experiments at both maintext and the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: NA
Guidelines:
The answer NA means that the paper does not include experiments.

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: have provided details of computer resources in experiments in 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tturr////./.../.eeeee Spppblic/tthissuiiiii
Answer:
[Yes]
Justification:
We confirm that our paper align well with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work
Guidelines:
25

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not involve such risks, and all data used in our study publicly available, detailed in Appendix C.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: For all assets used in our experiments, we have properly credited and respected the license and use explicitly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/coo/////ooooo conddataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: We will organize our training code according to the official template and include it in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our model training, data collection, and evaluation entirely based automated metrics and models, without any human involvement.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper doesn't 'itst'tiin involve crowd-sourcing with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes].
Justification: Our main contribution building a LLMs system that can perform hybrid thinking.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components. wwl.//////ooeeeeeeeeee
Please refer to our LLM policy (https: http:/pp..../ooofeeeeee///////// for what should or should not be described.
28


Reviews:
[{"review_id": "HnwNj3JuPB", "reviewer": "Reviewer_tCEC", "summary": "The authors propose a new post-training pipeline to enable \"hybrid\" thinking in language models. Unlike how existing \"thinking\" models such as DeepSeek-R1 always employ a long thinking phase, i.e., `<thinking>...</thinking> Final response`, hybrid thinking models adaptive choose whether to employ thinking, or directly respond to the question, i.e., `<no_thinking>Final response</no_thinking>`.\n\nThe method employs a two-stage pipeline, similar to that of DeepSeek-R1. The first cold-start SFT phase includes samples that (1) employ thinking for reasoning tasks, and those that (2) provide direct responses to non-reasoning tasks. In the second RL phase, the model is prompted to generate multiple responses in thinking and non-thinking modes, respectively. Positive rewards are given to (1) the mode that achieved the best outcome on average, and (2) individual responses that achieved high reward for the given task.\n\nThe method enables higher quality responses across various tasks, and potentially more efficient inference.", "questions": "### Questions\n\n- In Table 2 in Appendix C, is WildChat-1M used as the Thinking dataset, and the rest used for Non-Thinking?\n- Is the hybrid-thinking model more efficient compared to the thinking model, i.e., by using less output tokens?\n\n### Suggestions\n\n- Some parts of the method explanation could be simplified by using plain words instead of, or supplementing, the mathematical equations. E.g., equation (5) could simply be explained as generating N responses, with `<think>` appended to the prompt for one half, and `<no_think>` to the other.\n- (Future work) the method could be improved by enabling the model to reason before making the decision to `<think>` or `<no_think>`. The current training pipeline forces to make this decision in the first decoding step.\n\n### Criteria\n\nI am willing to increase my score if W1 is adequately addressed.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- Hybrid thinking models trained with the proposed method (1) outperform existing thinking models of the same size and (2) outperform baseline hybrid thinking models proposed by the authors, using standard RL algorithms, under equivalent settings as the proposed method.\n    - *(W1) However, there is no result on a non-hybrid thinking model under equivalent settings.*\n- Analysis shows that the trained model choses between thinking and non-thinking modes adaptively according to the task. Strong evidence of adaptive thinking is shown within the training domain (Figure 3(c), 4) and some evidence is shown for out-of-domain settings (Figure 5).\n- Extensive ablations on the choice of specific advantage estimators (GRPO, RLOO, Reinforce++) and the margin hyperparameter are conducted. These confirm robustness to the choice of advantage estimator, and confirm that the margin hyperparameter work as expected.\n- Each stage of the training methodology, as well as the proposed metric on hybrid thinking, are simple and straightforward.\n- The methodology and experimental settings are well-explained in detail.\n\n### Weaknesses\n\n1. The authors do not compare the proposed hybrid thinking model with an equivalent non-hybrid thinking model. The baseline model in Table 1, DeepSeek-R1-Distill-Qwen, may use a different training protocol from that of the proposed hybrid thinking models.\n    - This makes it unclear whether hybrid thinking degrades performance in reasoning-intense tasks.", "was_revised": true, "final_justification": "All of my concerns have been addressed", "rebuttal": "We are grateful to the Reviewer for the extensive review. We address your questions point by point below.\n\n> **Q1:** In Table 2 in Appendix C, is WildChat-1M used as the Thinking dataset, and the rest used for Non-Thinking?\n\n**A1**: In Appendix C, Table 2, WildChat-1M is actually used as the Non-Thinking dataset, while the remaining datasets are employed as the Thinking dataset. We apologize for any confusion caused and will clarify this point explicitly in the latest version of our manuscript.\n\n---\n> **Q2:** Is the hybrid-thinking model more efficient compared to the thinking model, i.e., by using less output tokens?\n\n**A2**: We have compared the output lengths between the proposed NHTMs (hybrid-thinking models), the thinking-only models (NHTMs-Thinking), and the DeepSeek-R1-Distill-Qwen model in **Table 1 & 2**. We find that:\n\n* Across most benchmarks, the hybrid-thinking models consistently produce shorter outputs while maintaining equal or better performance, demonstrating their ability to reduce unnecessary reasoning for simpler queries and thus improve inference efficiency.\n* On the more challenging AIME24 benchmark, the hybrid-thinking models generate longer outputs compared to both the thinking-only and DeepSeek models, as they engage in more elaborate reasoning to solve complex problems. This additional reasoning leads to significantly improved performance—NHTMs achieve the best accuracy at both the 1.5B and 7B model sizes, with the 7B variant outperforming DeepSeek-R1-Distill-Qwen by **8** percentage points.\n\nThese results clearly demonstrate that our hybrid-thinking model achieves improved inference efficiency by adaptively reducing unnecessary reasoning steps for simpler queries.\n\n\n***Table 1**. Output Length Comparison Across Hybrid, Thinking-Only, and DeepSeek-R1-Distill-Qwen Models on Various Benchmarks (1.5B size)*\n\n| **Model**                        | MATH500 | AIME24 | AMC23 | Olympiad | LiveCodeBench | MBPP | AlpacaEval2.0 | ArenaHard2.0 | **Avg.** |\n|----------------------------------|---------|--------|-------|-----------|----------------|------|----------------|---------------|-------------|\n| DeepSeek-R1-Distill-Qwen-1.5B   | 4059    | **13366**  | 9556  | 10921     | 13002          | 4146 | 4090           | 8285          | 8428        |\n| NHTMs-1.5B-Thinking              | 4463    | 13491  | 10093 | 10680     | 12969          | 4863 | 1986           | 9286          | 8479        |\n| NHTMs-1.5B                       | **3722**    | 13491  | **9065**  | **9490**      | **9342**           | **3103** | **1250**           | **5289**          | **6844**        |\n\n***Table 2**. Output Length Comparison Across Hybrid, Thinking-Only, and DeepSeek-R1-Distill-Qwen Models on Various Benchmarks (7B size)*\n\n| **Model**                        | MATH500 | AIME24 | AMC23 | Olympiad | LiveCodeBench | MBPP | AlpacaEval2.0 | ArenaHard2.0 | **Avg.** |\n|----------------------------------|---------|--------|-------|-----------|----------------|------|----------------|---------------|-------------|\n| DeepSeek-R1-Distill-Qwen-7B     | 3558    | **9488**   | 6255  | 8635      | 11669          | 2824 | 2209           | 5282          | 6240        |\n| NHTMs-7B-Thinking                | 3547    | 11031  | 6328  | 8736      | 11791          | 3076 | 1628           | 6542          | 6585        |\n| NHTMs-7B                         | **2616**    | 11031  | **4976**  | **7540**      | **8432**           | **1906** | **1086**           | **3416**          | **5125**        |\n\n---\n> **Weaknesses1**: The authors do not compare the proposed hybrid thinking model with an equivalent non-hybrid thinking model. The baseline model in Table 1, DeepSeek-R1-Distill-Qwen, may use a different training protocol from that of the proposed hybrid thinking models.\n\n**A3**: We appreciate the reviewer’s concern and have conducted an additional experiment for a direct comparison. Specifically, we fine-tuned both the 1.5B and 7B Qwen-2.5-Math models on **~1.1M math and code reasoning data in Thinking-style format**, and compared them against our hybrid-trained models under controlled settings. The two model types are defined as follows:\n\n- **PureThink-SFT models:**  \n  These models are trained exclusively on ~1.1M math and code **Thinking-style data** using standard supervised fine-tuning. No hybrid formatting or No-Thinking data is involved.\n\n- **HFT (Think Only) models:**  \n  These models are trained via our Hybrid Fine-Tuning (HFT) pipeline, which includes both Thinking and No-Thinking data. During evaluation, they are constrained to always generate responses in the Thinking mode.\n\nAs shown in **Table 3**, both variants achieve highly similar performance across multiple reasoning benchmarks at both 1.5B and 7B scales. This demonstrates that **hybrid thinking does not degrade performance on reasoning-intensive tasks**, validating the soundness of our hybrid training design.\n\n***Table 3**. Comparison of Hybrid and Non-Hybrid Models on Reasoning-Intensive Tasks*\n\n| **Model (Training Type)**         | MATH500 | AIME24 | AMC23 | Olympiad | LiveCodeBench | MBPP | MBPP+ | **Avg.** |\n|----------------------------------|---------|--------|-------|-----------|----------------|------|--------|----------|\n| PureThink-SFT-1.5B               | 87.6    | 34.7   | 72.5  | 49.9      | 17.2           | 63.5 | 53.2   | 54.1     |\n| HFT-1.5B (Think Only)            | 87.8    | 32.7   | 75.0  | 50.0      | 16.8           | 62.2 | 52.9   | 53.9     |\n| PureThink-SFT-7B                 | 94.0    | 56.7   | 92.5  | 59.9      | 38.7           | 80.7 | 68.3   | 70.1     |\n| HFT-7B (Think Only)              | 93.6    | 56.7   | 95.0  | 59.7      | 38.4           | 80.3 | 68.9   | 70.4     |\n\n---\n\n> **Suggestions1:** Some parts of the method explanation could be simplified to help readers better understand\n\n**A4**: We sincerely thank the reviewer for this valuable suggestion. We fully agree that simplifying parts of the method explanation, such as equation (5), using plain language can significantly improve reader understanding. As suggested, we will revise equation (5) and other parts of the method explanation in the latest version to improve clarity and help readers better understand our method.\n\n---\n> **Suggestions2:** (Future work) Consider allowing the model to reason before deciding on  or <no_think> instead of making the decision at the first decoding step\n\n**A5**: We appreciate the insightful idea! Indeed, enabling the model to first reason about the context and complexity of the query could potentially enhance the accuracy of selecting the appropriate thinking mode, thereby improving both the overall performance and efficiency. We plan to explore this approach in future work to further improve our hybrid thinking model. Meanwhile, we will include a discussion of this potential enhancement in the latest version of the manuscript.\n\n----\nWe sincerely thank the Reviewer tCEC for the thoughtful comments and constructive suggestions :). Your insights have been invaluable in improving the clarity and completeness of our work. In response, we will incorporate the additional comparisons, clarify the dataset usage, simplify the method description, and include a discussion of this direction in the future work section of the latest version of our manuscript.\n\nIf you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you. :)\n\nBest regards,\n\nAll Authors"}, {"review_id": "zLDcFYFc8z", "reviewer": "Reviewer_jtLz", "summary": "This paper presents Native Hybrid Thinking Models, which adaptively decide whether to engage in extended thinking based on user query context, addressing the overthinking issue in Large Reasoning Models. It proposes a two-stage training pipeline: Hybrid Fine-Tuning (HFT) for cold start and Hybrid Group Policy Optimization for learning to select appropriate thinking modes, along with a Hybrid Accuracy metric to assess hybrid thinking capability. Experiments show Native Hybrid Thinking Models outperform LLMs in reasoning and general capabilities while improving efficiency by reducing unnecessary thinking for simple queries.", "questions": "1. It is surprising that adaptive thinking shows significantly greater improvement on AIME 2024 than on Math500. Intuitively, simple problems' thinking process should be easily compressible, while difficult problems' thinking process should not be compressed or should even benefit from length scaling to improve performance. Could you compare the thinking ratio and thinking token length changes between MATH500/AIME 2024 to explain this phenomenon?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n1. Making reasoning models more efficient is an important research problem.\n2. The paper proposes a reasonable approach (including the complete pipeline of SFT and RL) with certain novelty.\n3. The paper is well-written and easy to follow.\n\n**Weakness**\n1. Direct comparison with DeepSeek-R1-Distill-Qwen-7B/1.5B & NHTMs-7B/1.5B is insufficient to demonstrate the method's effectiveness. Based on Stage I, training a baseline model using the same RL data (76K queries from Deepscaler and Tülu3) with standard GRPO to achieve high performance would better illustrate whether implementing adaptive thinking compromises the model's capability ceiling. Simply comparing against DeepSeek-R1-Distill-Qwen-7B/1.5B is inadequate, as you have expended additional computational resources that might enable the model to reach a significantly higher ceiling, making adaptive CoT no longer cost-effective. This point requires experimental validation.\n2. As research on adaptive CoT, reporting the thinking token length statistics for each dataset and comparing with CoT compression works [1][2][3] would strengthen the contribution of this work.\n3. Llama-3.1-Tulu-3-8B-RM is used both for RL training and for computing Hybrid Accuracy in Section 2.4. Could this lead to unfair Hybrid Accuracy evaluation?\n\n[1] Luo H, Shen L, He H, et al. O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning[J]. arXiv preprint arXiv:2501.12570, 2025.\n\n[2] Ma X, Wan G, Yu R, et al. CoT-Valve: Length-Compressible Chain-of-Thought Tuning[J]. arXiv preprint arXiv:2502.09601, 2025.\n\n[3] Shen Y, Zhang J, Huang J, et al. Dast: Difficulty-adaptive slow-thinking for large reasoning models[J]. arXiv preprint arXiv:2503.04472, 2025.", "was_revised": true, "final_justification": "The author has addressed my concerns regarding the baseline setup, the detailed think ratio statistics for different difficulty levels, and the token length statistics. Therefore, my confidence in supporting the acceptance of this paper has increased.", "rebuttal": "We are grateful to the Reviewer for the extensive review. We address your questions point by point below.\n\n> **Q1:** It is surprising that adaptive thinking shows significantly greater improvement on AIME 2024 than on Math500. Could you compare the thinking ratio and thinking token length changes between MATH500/AIME 2024 to explain this phenomenon?\n\n**A1:** We compared the thinking ratio and average reasoning length across MATH500 and AIME24 (see Table 1). The results are consistent with the intuition that simple problems benefit more from reasoning compression, while difficult problems require full reasoning.\n\nOn MATH500, which contains many relatively simple problems, NHTMs significantly reduce both the thinking ratio and the average reasoning length, indicating that the model effectively skips unnecessary reasoning by selecting the No-Thinking mode. This improves efficiency without sacrificing accuracy. In contrast, on the more challenging AIME24, the model applies 100% Thinking with slightly longer reasoning chains, reflecting its consistent use of detailed step-by-step reasoning where it is truly needed.\n\nTogether, these trends explain the greater performance gains on AIME24: the model allocates full reasoning to difficult tasks while leveraging No-Thinking to bypass reasoning for simpler ones.\n\n\n**Table 1: Comparison of thinking ratio (%) and average reasoning length (tokens) between HFT and NHTMs on MATH500 and AIME24.**\n\n|Model|MATH500 (Think Ratio)|MATH500 (Length)|AIME24 (Think Ratio)|AIME24 (Length)|\n|-|-|-|-|-|\n|HFT-1.5B|99.2|4330|100|13431|\n|NHTMs-1.5B|90.2|3722|100|13491|\n|HFT-7B|98.2|3604|100|10778|\n|NHTMs-7B|75.2|2616|100|11031|\n\n---\n> **W1**: Direct comparison with DeepSeek-R1-Distill-Qwen-7B/1.5B & NHTMs-7B/1.5B is insufficient to demonstrate the method's effectiveness....as you have expended additional computational resources that might enable the model to reach a significantly higher ceiling, making adaptive CoT no longer cost-effective. This point requires experimental validation.\n\n**A2:** We thank the reviewer for pointing out the need to isolate the effect of adaptive thinking from computational budget. To address this, we conducted additional experiments where baseline models (HFT-1.5B and HFT-7B) were trained with standard GRPO using the exact same RL dataset (76K queries from DeepScaleR and Tülu3) and training setup as our HGPO method.\n\nAs summarized in Table B, both training strategies yield similar performance on reasoning-intensive tasks. However, HGPO-trained models consistently outperform GRPO-trained counterparts in general tasks, leading to higher overall performance. We attribute this to HGPO’s adaptive selection constraint, which encourages more effective use of the \"no-think\" mode on simpler queries, and preserves reasoning capacity for complex problems. In contrast, GRPO lacks this adaptive mechanism and underutilizes the \"no-think\" mode during rollouts, limiting gains on general tasks.\n\nMoreover, HGPO models achieve significantly higher hybrid accuracy ($H_{\\text{acc}}$), demonstrating stronger ability to select the appropriate reasoning mode per query. This confirms that adaptive thinking not only avoids additional computational cost, but also enhances the model's effectiveness across diverse tasks.\n\nThese results validate the value of HGPO in building a more efficient and capable hybrid reasoning system under equivalent training resources.\n\n**Table B: Performance Comparison of Standard GRPO vs. HGPO**\n\n|Model|MATH-500|AIME 2024|AMC23|Olympiadbench|LiveCodeBench|MBPP|MBPP+|AlpacaEval2.0|ArenaHard2.0|$H_{acc}$|Avg.|\n|-|-|-|-|-|-|-|-|-|-|-|-|\n|HFT-1.5B|87.8|32.7|75.0|48.9|15.7|54.8|47.4|13.1|6.9|41.4|42.5|\n|HFT-GRPO-1.5B|**89.0**|34.7|73.1|50.2|**17.9**|**61.4**|52.9|14.3|8.1|47.4|44.6|\n|NHTMs-1.5B|87.8|**35.3**|**75.0**|**50.4**|17.2|61.1|**54.0**|**16.1**|**10.4**|**54.4**|**45.3**|\n|HFT-7B|93.6|56.7|95.0|58.5|34.7|70.6|59.8|23.7|14.7|34.2|56.4|\n|HFT-GRPO-7B|**95.2**|64.0|94.4|**61.5**|**39.6**|79.6|67.2|25.8|18.7|41.9|60.7|\n|NHTMs-7B|93.8|**66.7**|**95.0**|61.2|38.8|**81.5**|**69.6**|**35.0**|**26.7**|**71.9**|**63.1**|\n\n---\n> **W2**: Reporting the thinking token length statistics for each dataset and comparing with CoT compression works would strengthen the work.\n\n**A2:** We thank the reviewer for the insightful suggestion on analyzing thinking token lengths and comparing with existing CoT compression methods [1–3]. In response, we present detailed results in Table C, including average reasoning lengths and corresponding accuracies:\n\n* Compared to the full-length thinking baselines (HFT-1.5B-Think and HFT-7B-Think), our NHTMs significantly reduce token lengths across most tasks while achieving comparable or better accuracy. This confirms the efficiency and effectiveness of our adaptive reasoning strategy.\n\n* When compared with prior CoT compression methods—O1-Pruner, CoT-Valve, and DAST—our approach achieves a better balance between compression and accuracy. Although some methods (e.g., O1-Pruner, DAST) yield shorter outputs, they often suffer from performance drops. In contrast, NHTMs maintain or improve accuracy with meaningful reductions in reasoning length.\n\nThese results validate the practical advantage of our adaptive CoT framework for efficient and reliable inference. We wil add these results into our newest version.\n\n**Table C. Comparison of Accuracy and Average Thinking Token Length (in Parentheses) Across CoT Compression Methods and Model Variants**\n\n|**Model**|MATH500|AIME24|AMC23|Olympiad|LiveCodeBench|MBPP|MBPP++|AlpacaEval2.0|ArenaHard2.0|**Avg**|\n|-|-|-|-|-|-|-|-|-|-|-|\n|**1.5B Size Models**|||||||||||\n|HFT-1.5B-Think|87.8 (4379)|32.7 (13431)|75.0 (10181)|50.0 (10480)|16.8 (13628)|**62.2** (5090)|52.9 (5090)|**15.9** (1927)|9.3 (10616)|44.7 (8314)|\n|HFT-1.5B|87.8 (4330)|32.7 (13431)|75.0 (10181)|48.9 (10480)|15.7 (**7460**)|54.8 (2803)|47.4 (2803)|13.1 (1101)|6.9 (4287)|42.5 (6320)|\n|O1-Pruner|85.4 (2884)|29.3 (10655)|69.0 (6158)|49.7 (6568)|16.8 (12233)|49.7 (**711**)|41.8 (**711**)|13.0 (**595**)|6.3 (3470)|40.1 (4887)|\n|DAST|84.8 (**2428**)|26.9 (**8429**)|64.0 (**4637**)|46.4 (**4968**)|16.8 (8238)|55.8 (1412)|46.3 (1413)|14.3 (898)|6.0 (**2760**)|40.1 (**3909**)|\n|CoT-Valve|87.0 (3399)|32.0 (11078)|62.5 (8054)|46.1 (7827)|17.1 (9614)|60.1 (4042)|51.9 (4042)|14.1 (1491)|8.1 (6253)|42.1 (6200)|\n|NHTMs-1.5B|**87.8** (3722)|**35.3** (13491)|**75.0** (9065)|**50.4** (9490)|**17.2** (9342)|61.1 (3103)|**54.0** (3103)|16.9 (1250)|**10.4** (5289)|**45.3** (6428)|\n|**7B Size Models**|||||||||||\n|HFT-7B-Think|93.8 (3658)|56.7 (10778)|95.0 (6456)|59.7 (8376)|38.4 (12046)|80.3 (3251)|68.9 (3251)|30.6 (1731)|23.3 (7442)|60.7 (6332)|\n|HFT-7B|93.6 (3604)|56.7 (10778)|95.0 (6253)|58.5 (7870)|34.7 (7739)|70.6 (1658)|59.8 (1658)|23.7 (779)|14.0 (2216)|56.4 (4728)|\n|O1-Pruner|93.0 (2885)|52.0 (9015)|91.0 (4725)|60.8 (**5801**)|35.1 (8351)|79.4 (2210)|67.9 (2210)|24.4 (**566**)|16.0 (2023)|57.7 (4198)|\n|DAST|92.6 (2697)|46.0 (**8835**)|86.0 (**4723**)|58.1 (4965)|32.9 (**6045**)|69.4 (**1074**)|58.9 (**1074**)|22.3 (835)|13.8 (**1765**)|53.3 (**3557**)|\n|CoT-Valve|93.4 (2964)|50.0 (9692)|93.0 (5122)|58.5 (7151)|37.3 (9970)|80.1 (2987)|67.8 (2987)|26.5 (1553)|23.2 (3445)|58.9 (5097)|\n|NHTMs-7B|**93.8** (**2616**)|**66.7** (11031)|**95.0** (4976)|**61.2** (7540)|**38.8** (8432)|**81.5** (1906)|**69.6** (1906)|**35.0** (1086)|**26.0** (3416)|**63.1** (4768)|\n\n---\n> **W3**: Tulu-3-8B-RM is used both for RL training and for computing Hybrid Accuracy. Could this lead to unfair Hybrid Accuracy evaluation?\n\nWe appreciate the reviewer’s thoughtful question. To ensure the fairness and reliability of our Hybrid Accuracy (HAcc) evaluation, we have additionally computed HAcc using both GPT-4o annotations and human annotations. For the human annotations, we sampled 100 queries per model and engaged three annotators, each labeling 800 samples independently.\n\nAs presented in Table D, the results indicate that the relative ordering (partial ordering) of models based on Hybrid Accuracy remains consistent across annotations derived from GPT-4o, human evaluators, and Tulu-3-8B-RM. All annotation methods confirm that NHTMs-7B and NHTMs-1.5B achieve substantial improvements in selecting the correct reasoning mode.\n\nFurthermore, we conducted an agreement analysis across these annotation approaches (see Table E). The average agreement between human annotators was 60%, while the agreement between human annotators and Tulu-3-8B-RM was slightly higher at 62.5%. Notably, GPT-4o demonstrated the highest agreement rates, comparable to previously reported agreement levels in MT-Bench [4] and UltraFeedback [5]. These results collectively validate the reliability and fairness of employing Tulu-3-8B-RM for Hybrid Accuracy evaluation.\n\n**Table D: Hybrid Accuracy (%) of Different Models**\n\n|Model|Tulu-3-8B-RM|GPT-4o|Human|Avg|\n|-|-|-|-|-|\n|1.5B-SFT|41.4|41.3|42.0|41.6|\n|HFT-1.5B|48.1|48.2|46.3|47.5|\n|HFT-DPO-1.5B|45.8|43.4|43.3|44.2|\n|NHTMs-1.5B|54.4|52.8|53.7|53.6|\n|HFT-7B|34.2|30.0|39.0|34.4|\n|HFT-RFT-7B|49.7|48.6|47.7|48.7|\n|HFT-DPO-7B|37.1|35.4|39.0|37.2|\n|NHTMs-7B|71.9|71.5|72.0|71.8|\n\n\n**Table E: Agreement Rates (%) Between Annotators**\n\n||GPT-4o|RM|H-1|H-2|H-3|Average|\n|-|-|-|-|-|-|-|\n|GPT-4o|-|78.5%|69.8%|70.1%|69.5%|69.8%|\n|RM|-|-|61.0%|64.1%|62.3%|62.5%|\n|H-1|-|-|-|59.6%|61.8%|60.7%|\n|H-2|-|-|-|-|58.6%|59.1%|\n|H-3|-|-|-|-|-|60.2%|\n\n[4] Cui, Ganqu, et al. \"Ultrafeedback: Boosting language models with scaled ai feedback.\"\n\n[5] Xu, Jiazheng, et al. \"Imagereward: Learning and evaluating human preferences for text-to-image generation.\"\n\n---\nWe sincerely thank the Reviewer jtLz for the thoughtful comments and constructive suggestions :). If you have any further questions or concerns, please feel free to contact us at any time.\n\nBest regards,\n\nAll Authors"}, {"review_id": "Fw2iUwXBC7", "reviewer": "Reviewer_ofyf", "summary": "This paper proposes a complete pipeline to build a native hybrid thinking LLM which can decide whether conduct long CoT thinking according to the difficulty of the problem by themselves. The entire pipeline follows a standard SFT-then-RL pattern. The SFT stage integrates both reasoning-intensive (Thinking) and direct-answer (No-Thinking) data to establish a robust initialization for following RL training. In RL training phase, this paper proposes a novel HGPO and delicate reward functions to enable the LLM to internally choose the thinking mode when dealing problems with different difficulties. Extensive experiments demonstrate that NHTMs not only possess strong reasoning capabilities but also can largely save computation resources while dealing simple problems.", "questions": "1. In Equation (10), I am particularly interested in the term “GRPO for inter-group advantage $A_{\\text{inter}}$.” However, I do not fully understand how this term effectively contributes to the decision of whether to think or not. I would appreciate it if the authors could provide a more detailed explanation. It may also be helpful for other readers if the authors include additional clarification or discussion of this term in the paper.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The overall writing of this paper is excellent, and the logic is easy to follow.\n2. The proposed training pipeline is well-designed, and the final experimental results are solid, comprehensive, and convincing.\n\nWeaknesses:\nThis paper does not have any significant weaknesses. Here are some of my suggestions:\n1. In Figure 5, the y-axis label for \"Thinking Ratio\" appears to be missing.\n2. A key contribution of the proposed NHTMs is their ability to autonomously decide whether to think, thereby saving computational budget. It would be more intuitive to include the average number of output tokens for both the baselines and NHTMs across all benchmarks.\n3. In my understanding, Hybrid Accuracy is a model-specific metric, as $m_\\text{gt}$ is determined by the evaluated LLM. If multiple NHTMs are evaluated, the proposed Hybrid Accuracy metric may not fairly compare them.", "was_revised": true, "final_justification": "All of my concerns have been well addressed.", "rebuttal": "> **Q1.**: In Equation (10), I am particularly interested in the term “GRPO for inter-group advantage .” However, I do not fully understand how this term effectively contributes to the decision of whether to think or not. I would appreciate it if the authors could provide a more detailed explanation. It may also be helpful for other readers if the authors include additional clarification or discussion of this term in the paper.\n\n**A1.** Thank you for your valuable suggestion. We agree that the “GRPO for inter-group advantage” term in Equation (10) plays a crucial role and deserves a more detailed explanation. To clarify, the process unfolds as follows:\n\n1. **Scoring responses (Eq. 7):** For each query, we sample both *Thinking* and *No-Thinking* responses, and score them using the reward model.\n2. **Computing average rewards (Eq. 8):** We then compute the average reward for both reasoning modes across these samples.\n3. **Selecting the better mode:** By comparing the two average rewards, we identify which reasoning mode—*Thinking* or *No-Thinking*—is more appropriate for the given query.\n4. **Assigning learning signals (Eq. 9):** We then get inter-group reward by assigning a high reward (1) to responses sampled from the better mode and a low reward (0) to those from the less effective mode.\n5. **Learning via GRPO (Eq. 10):** The “GRPO for inter-group advantage” then learns from this inter-group reward contrast, encouraging the model to favor the reasoning mode that is more beneficial for a specific query.\n\nWe will revise the manuscript to enhance the explanation of this term and its role in reasoning-mode selection, as we believe this will indeed help readers better understand our method. Thank you again for your insightful comment.\n\n> **W1:** In Figure 5, the y-axis label for \"Thinking Ratio\" appears to be missing.\n\n**A2**：Thank you for your careful review and for pointing out the issue with Figure 5. We agree with your observation — the \"Thinking Ratio\" and \"Accuracy\" currently share the same y-axis, but the label for \"Thinking Ratio\" was inadvertently omitted. This was an oversight on our part. In the revised version of the paper, we will explicitly add the missing label to ensure clarity and avoid any potential confusion for readers. We appreciate your attention to detail in helping us improve the quality of our paper.\n\n> **W2:** A key contribution of the proposed NHTMs is their ability to autonomously decide whether to think, thereby saving computational budget. It would be more intuitive to include the average number of output tokens for both the baselines and NHTMs across all benchmarks.\n\n**A3:** As suggested, we have provided a detailed comparison of the output lengths across all benchmarks in **Tables 1 and 2**, alongside the corresponding accuracy results. In each cell, the number in parentheses indicates the average number of output tokens. These tables cover the proposed NHTMs (Hybrid-Thinking Models), the full-thinking baselines (HFT-1.5B-Think and HFT-7B-Think), and the DeepSeek-R1-Distill-Qwen models.\n\n* Across most benchmarks, the hybrid-thinking models consistently produce shorter outputs while maintaining equal or better performance, demonstrating their ability to reduce unnecessary reasoning for simpler queries and thus improve inference efficiency.\n* On the more challenging AIME24 benchmark, the hybrid-thinking models generate longer outputs compared to both the thinking-only and DeepSeek models, as they engage in more elaborate reasoning to solve complex problems. This additional reasoning leads to significantly improved performance—NHTMs achieve the best accuracy at both the 1.5B and 7B model sizes, with the 7B variant outperforming DeepSeek-R1-Distill-Qwen by **8** percentage points.\n\nThese results clearly demonstrate that our hybrid-thinking model achieves improved inference efficiency by adaptively reducing unnecessary reasoning steps for simpler queries.\n\n***Table 1*. Accuracy and Output Length Comparison on Various Benchmarks (1.5B size)**\n\n| **Model**                      | MATH500           | AIME24            | AMC23             | Olympiad          | LiveCodeBench     | MBPP             | MBPP++           | AlpacaEval2.0     | ArenaHard2.0       | **Avg.**        |\n|-------------------------------|-------------------|-------------------|-------------------|-------------------|--------------------|------------------|------------------|--------------------|---------------------|------------------|\n| DeepSeek-R1-Distill-Qwen-1.5B | 83.9 (4059)       | 28.9 (13366)      | 62.9 (9556)       | 43.3 (10921)      | 16.8 (13002)       | 54.2 (4146)      | 46.3 (4146)      | 5.6 (4090)         | 2.7 (8285)          | 38.3 (7952)       |\n| HFT-1.5B-Think                | 87.8 (4379)   | 32.7 (13431)      | 75.0 (10181)      | 50.0 (10480)      | 16.8 (13628)       | **62.2** (5090)  | 52.9 (5090)      | 15.9 (1927)        | 9.3 (10616)         | 44.7 (8314)       |\n| HFT-1.5B                     | 87.8 (4330)   | 32.7 (13431)      | 75.0 (10181)      | 48.9 (10480)      | 15.7 (**7460**) | 54.8 (**2803**)  | 47.4 (**2803**)  | 13.1 (**1101**)    | 6.9 (**4287**)      | 42.5 (**6320**)   |\n| NHTMs-1.5B                   | **87.8** (**3722**) | **35.3** (13491)  | **75.0** (**9065**) | **50.4** (**9490**) | **17.2** (9342)    | 61.1 (3103)      | **54.0** (3103)  | **16.9** (1250)     | **10.4** (5289)     | **45.3** (6428)   |\n\n***Table 2*. Accuracy and Output Length Comparison on Various Benchmarks (7B size)**\n\n| **Model**                      | MATH500           | AIME24             | AMC23              | Olympiad           | LiveCodeBench     | MBPP              | MBPP++            | AlpacaEval2.0      | ArenaHard2.0        | **Avg.**        |\n|-------------------------------|-------------------|--------------------|--------------------|--------------------|--------------------|-------------------|-------------------|---------------------|----------------------|------------------|\n| DeepSeek-R1-Distill-Qwen-7B   | 92.8 (3558)       | 55.5 (9488)        | 91.5 (6255)        | 58.1 (8635)        | 37.6 (11669)       | 74.3 (2824)       | 64.3 (2824)       | 19.1 (2209)         | 17.9 (5282)          | 56.8 (5860)       |\n| HFT-7B-Think                  | 93.8 (3658)   | 56.7 (10778)       | 95.0 (6456)    | 59.7 (8376)        | 38.4 (12046)       | 80.3 (3251)       | 68.9 (3251)       | 30.6 (1731)         | 23.3 (7442)          | 60.7 (6332)       |\n| HFT-7B                        | 93.6 (3604)       | 56.7 (10778)       | 95.0 (6253)    | 58.5 (7870)        | 34.7 (**7739**) | 70.6 (**1658**)   | 59.8 (**1658**)   | 23.7 (**779**)       | 14.0 (**2216**)      | 56.4 (**4728**)   |\n| NHTMs-7B                      | **93.8** (**2616**) | **66.7** (11031) | **95.0** (**4976**) | **61.2** (**7540**) | **38.8** (8432) | **81.5** (1906) | **69.6** (1906) | **35.0** (1086) | **26.0** (3416) | **63.1** (4768) |\n\n\n> **W3:** In my understanding, Hybrid Accuracy is a model-specific metric, as $m_{gt}$\n is determined by the evaluated LLM. If multiple NHTMs are evaluated, the proposed Hybrid Accuracy metric may not fairly compare them.\n\n**A4:** Thank you for the insightful comment. We agree with your observation that Hybrid Accuracy is indeed a model-specific metric, as it directly depends on the selection strategy employed by the evaluated hybrid model.\n\nWe would like to clarify that Hybrid Accuracy is not intended as an absolute metric to evaluate overall model performance across different models. Instead, it specifically reflects the effectiveness of a given model in making correct choices between the \"thinking\" and \"no-thinking\" modes within its own decision-making mechanism.\n\nFor example, consider a hypothetical hybrid model that always performs better in the \"thinking\" mode compared to the \"no-thinking\" mode (perhaps due to insufficient training in the \"no-thinking\" mode). If this model always selects \"thinking,\" it will achieve a high Hybrid Accuracy score, but this does not necessarily indicate that the model itself has superior overall capabilities—it merely signifies that it correctly identifies its optimal internal mode.\n\nConversely, if this model mistakenly prefers the \"no-thinking\" mode under similar conditions, the Hybrid Accuracy score will be low, clearly indicating poor internal decision-making. Thus, Hybrid Accuracy primarily assesses the model’s internal decision logic rather than its absolute performance on tasks.\n\nIn the revised version of our paper, we will explicitly clarify this aspect and provide illustrative examples to better convey the scope and intention of the Hybrid Accuracy metric.\n\n---\n\nWe sincerely thank the Reviewer ofyf for the thoughtful comments and constructive suggestions :). Your insights have been invaluable in improving the clarity and completeness of our work. \n\nIf you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you. :)\n\nBest regards,\n\nAll Authors"}, {"review_id": "wJNXCyU3Uu", "reviewer": "Reviewer_hw6N", "summary": "In this paper authors introduce a new method to make reasoning models perform better on conversational benchmarks. The method consists of two stages - Hybrid Finetuning and Hybrid Group Policy Optimization. The resulting models support two generation modes - thinking and no thinking which makes them both competitive on reasoning tasks and usable in conversational scenarios.", "questions": "- How evaluations were done? Were all models evaluated with the same hyperparameters?\n- Can authors provide evaluations on GPQA-Diamond, HumanEval, HumanEval++ for fair comparison with other LRMs?", "limitations": "- Only one base model was used - Qwen2.5-math-base.\n- No comparison with the same compute (both training and test-time).", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- NHTMs outperform baselines on both reasoning and chat benchmarks.\n- Good baselines - authors were comparing HGPO with both HFT and HFT+DPO/RFT.\n- Only fully-accessible data and open-source models were used in this study.\n- Two model scales - 1.5B and 7B.\n\n### Weaknesses\n\n- No comparison with DeepScaleR-1.5B-Preview[1] although the DeepScaleR data was used.\n- No evaluations on GPQA-Diamond[2], HumanEval, HumanEval++.\n- No information on training data decontamination. E.g. AIME dataset[3] might contain samples from AIME24 benchmark.\n- Grammar errors (\"to lean a effective NHTMs\").\n\n### References\n\n1. https://huggingface.co/agentica-org/DeepScaleR-1.5B-Preview\n2. https://openreview.net/forum?id=Ti67584b98\n3. https://huggingface.co/datasets/di-zhang-fdu/AIME_1983_2024", "was_revised": true, "final_justification": "I would like to thank the authors for their clarifications and for running additional experiments. My main concerns regarding additional benchmark evaluations and data decontamination were addressed therefore I will raise my score. I still think that for the fair comparison the HFT and NHFT models should have been compared to other SoTA models taking into account training compute/dataset size (e.g. DeepSeek-R1-Distill-Qwen-7B[1] was trained on 800M samples using SFT while the HFT stage only in the paper involves 1.7M samples.\n\n[1] https://arxiv.org/pdf/2501.12948", "rebuttal": "We are grateful to the Reviewer hw6N for the extensive review. We address your questions point by point below.\n\n---\n> **Q1:** How evaluations were done? Were all models evaluated with the same hyperparameters?\n\n**A1**: Thank you for raising this question. We performed rigorous evaluations on our models across various benchmarks, **ensuring that both NHTMs and baseline models were evaluated under the same experimental setup to guarantee a fair comparison**. As detailed in Appendix D (\"Evaluation Settings\"), we consistently used a temperature of 0.6, a top-p value of 0.95, and a maximum generation length of 35,000 tokens for all models across all tasks. To ensure the stability and reliability of our results, **the reported scores for all reasoning tasks are the averages of five independent runs**.\n\nFor specific benchmark suites, we adhered to widely-accepted evaluation frameworks. MATH500, AIME24, and LiveCodeBench were evaluated using the OpenR1 evaluation framework, while MBPP and MBPP+ were evaluated using the EvalPlus framework. Further details are provided in Appendix D.\n\n---\n> **Q2:** Can authors provide evaluations on GPQA-Diamond, HumanEval, HumanEval++ for fair comparison with other LRMs?\n\n**A2:** Thank you for the valuable suggestion. We have conducted these additional evaluations to ensure a fair and comprehensive comparison at **Table A** and **Table B**. **Our experimental results demonstrate that our NHTMs-1.5B and NHTMs-7B models outperform the DeepSeek-R1-Distill-Qwen counterparts on both HumanEval and HumanEval++ benchmarks, and achieves the highest average performance among all compared models**.\n\nAlthough our NHTMs-7B performs slightly lower on GPQA-Diamond compared to DeepSeek-R1-Distill-Qwen-7B, it still shows significant improvements over the HFT-1.5B baseline. This is particularly noteworthy because our training dataset explicitly excluded Science-related data, which is primarily the domain covered by the GPQA-Diamond benchmark (as illustrated in Appendix C, Table 2 and Table 3). Therefore, the observed performance indicates that our Hybrid Group Policy Optimization (HGPO) method effectively enhances generalization, allowing the model to achieve robust performance even on tasks outside its direct training distribution.\n\nIn summary, the additional evaluations further validate the effectiveness and generalization capability of our proposed model.\n\n***Table A**: Performance Comparison on Comprehensive Benchmarks (1.5B size)*\n\n| Model                          | HumanEval | HumanEval++ | GPQA-Diamond | MATH-500 | AIME 2024 | AMC23 | Olympiad | LiveCode-Bench | MBPP | MBPP+ | Alpaca-Eval2.0 | Arena-Hard v2 | Avg |\n|-------------------------------|-----------|-------------|---------------|----------|------------|-------|----------------|----------------|------|--------|----------------|---------------|---------|\n| DeepSeek-R1-Distill-Qwen-1.5B | 67.7      | 59.8        | **33.8**      | 83.9     | 28.9       | 62.9  | 43.3           | 16.8           | 54.2 | 46.3   | 5.6            | 2.7           | 42.2    |\n| HFT-1.5B                      | 65.9      | 62.2        | 27.3          | **87.8** | 32.7       | **75.0** | 48.9           | 15.7           | 54.8 | 47.4   | 13.1           | 6.9           | 44.8    |\n| NHTMs-1.5B                    | **75.0**  | **70.7**    | 31.3          | **87.8** | **35.3**   | **75.0** | **50.4**       | **17.2**       | **61.1** | **54.0** | **16.1**       | **10.4**      | **48.7** |\n\n***Table B**: Performance Comparison on Comprehensive Benchmarks (7B size)*\n| Model                          | HumanEval | HumanEval++ | GPQA-Diamond | MATH-500 | AIME 2024 | AMC23 | Olympiad | LiveCode-Bench | MBPP | MBPP+ | Alpaca-Eval2.0 | Arena-Hard v2 | Avg|\n|-------------------------------|-----------|-------------|---------------|----------|------------|-------|----------------|----------------|------|--------|----------------|---------------|---------|\n| DeepSeek-R1-Distill-Qwen-7B   | 88.4      | 81.7        | **49.1**      | 92.8     | 55.5       | 91.5  | 58.1           | 37.6           | 74.3 | 64.3   | 19.1           | 17.9          | 60.9    |\n| HFT-7B                        | 84.8      | 79.3        | 31.3          | 93.6     | 56.7       | **95.0** | 58.5           | 34.7           | 70.6 | 59.8   | 23.7           | 14.7          | 58.6    |\n| NHTMs-7B                      | **89.6**  | **82.9**    | 41.4          | **93.8** | **66.7**   | **95.0** | **61.2**       | **38.8**       | **81.5** | **69.6** | **35.0**       | **26.7**      | **65.2** |\n\n---\n> **W1:** No comparison with DeepScaleR-1.5B-Preview[1] although the DeepScaleR data was used.\n\n**A3**: Thank you for your valuable feedback. We have now included the results of DeepScaleR-1.5B-Preview in the comparison table below, alongside NHTMs-1.5B for a fair evaluation across key benchmarks:\n\n| Model                         | MATH500 | AIME24 | AMC23 | Olympiad | MBPP | MBPP+ | Alpaca-Eval2.0 | ArenaHard v2 | Avg  |\n|-------------------------------|---------|--------|-------|----------|--------|-------|----------------|--------------|------|\n| DeepSeek-R1-Distill-Qwen-1.5B | 83.9    | 28.9   | 62.9  | 43.3     | 54.2 | 46.3  | 5.6            | 2.7          | 41.0 |\n| DeepScaleR-1.5B-Preview       | 87.8    | **43.1**   | 73.6  | 50.0     | 56.6 | 46.6  | 6.4            | 6.3          | 46.2 |\n| NHTMs-1.5B                     | **87.8**    | 35.3   | **75.0**  | **50.4**     | **61.1** | **54.0**  | **16.9**           | **10.4**         | **48.8** |\n\nAs shown in the table, **our NHTMs-1.5B model achieves the highest average performance among all compared models, outperforming both DeepScaleR-1.5B-Preview and DeepSeek-R1-Distill-Qwen-1.5B**.\n\nIn particular, NHTMs-1.5B achieves comparable results to DeepScaleR-1.5B-Preview on math-focused tasks such as MATH500, and demonstrates clear advantages on tasks requiring adaptive reasoning, including MBPP and AlpacaEval2.0. Although it underperforms on AIME24, we attribute this to the difference in training budgets—DeepScaleR was trained for 1.7k steps, whereas our model was only trained for fewer than 500 steps in this comparison.\n\nThese results highlight the efficiency and generalization capability of our hybrid-thinking framework, even under limited training. We appreciate your suggestion and will put DeepScaleR-1.5B-Preview as a comparison point in next version of our paper to provide a more comprehensive evaluation.\n\n---\n> **W2:** No information on training data decontamination. E.g. AIME dataset[3] might contain samples from AIME24 benchmark.\n\n**A4:** Thank you for pointing this out. Although we briefly mentioned the removal of overlapping samples in Section 2.2 (\"Data Construction\", line 96), we agree that a more detailed explanation of our decontamination process is necessary.\n\nTo ensure the fairness of our evaluation, **we uniformly applied the Simhash algorithm [1] to calculate the similarity between each training sample and all samples across our evaluation benchmarks. Any instances with a similarity score greater than 0.8 were removed to strictly prevent data contamination**.\n\nSpecifically regarding the AIME dataset, we took extra precautionary measures to avoid any temporal or content overlap with the AIME24 benchmark:\n\n* **Source Filtering**: Our training data only included AIME problems from 2023 and earlier, thereby ensuring there was no temporal overlap with the AIME24 benchmark at the source level.\n\n* **Precise Deduplication**: In addition to source filtering, we applied the aforementioned Simhash deduplication process against the AIME24 test set as an additional safeguard. As reflected in Table 2 of our paper, the initial AIME dataset contained 933 problems. After applying this rigorous decontamination process, 741 problems were retained for training, which further confirms the execution of our decontamination procedure.\n\nThrough these two measures—source control and precise deduplication—we have ensured there is no data leakage from the AIME24 benchmark. We will add a more detailed description of our decontamination process to the appendix in the final version to improve the transparency and reproducibility of our work.\n\n**Reference**\n\n[1] Detecting near-duplicates for web crawling. Manku, Gurmeet Singh and Jain, Arvind and Das Sarma, Anish.\n\n---\n> **W3:** Grammar errors (\"to lean a effective NHTMs\").\n\n**A5:** Thank you for pointing out the grammatical error, and we will correct the grammar in the latest version of the paper.\n\n----\nWe want to express our sincere gratitude for your review. If you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you. :)\n\nBest regards,\n\nAll Authors"}]

Author rebuttal:

Meta-review: This was a consensus accept. Please take into account how the reviewers initially interpreted the baselines when preparing the final version.
Decision: Accept (poster)