Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Time
Authors: Van Yang
Xiang Yue
Vipin Chaudhary
Xiaotian Han
Venue: COLM 2025 (2025)
Keywords: LLM Reasoning; LLM Inference; Speculative Decoding
TL;DR: a training-free framework that enables large reasoning models to guide smaller ones during inference at the reasoning level, distinct from speculative decoding, which operates at the token level

Abstract: Recent advances leverage post-training to enhance model reasoning performance, which typically requires costly training pipelines and still suffers from inefficient, overly lengthy outputs. We introduce **Speculative Thinking**, a training-free framework that enables large reasoning models to guide smaller ones during inference at the reasoning level, distinct from speculative decoding, which operates at the token level. Our approach is based on two observations: (1) reasoning-supportive tokens such as $''wait''$ frequently appear after structural delimiters like $''\n\n''$, serving as signals for reflection or continuation; and (2) larger models exhibit stronger control over reflective behavior, reducing unnecessary backtracking while improving reasoning quality. By strategically delegating reflective steps to a more capable model, our method significantly boosts the reasoning accuracy of reasoning models while shortening their output. With the assistance of the 32B reasoning model, the 1.5B model’s accuracy on $MATH500$ increases from 83.2\% to 89.4\%, marking a substantial improvement of 6.2\%. Simultaneously, the average output length is reduced from $5439$ tokens to $4583$ tokens, representing a 15.7\% decrease. Moreover, when applied to a non-reasoning model (Qwen-2.5-7B-Instruct), our framework boosts its accuracy from 74.0\% to 81.8\% on the same benchmark, achieving a relative improvement of 7.8\%.

Published as a conference paper at COLM 2025
Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Time
Wang Yang', Xiang Yue², Vipin Chaudhary?, Xiaotian Han
"Case Western Reserve University ?Carnegie Mellon University (wxy320, vxc204, xhan] edu xyue28andrew cmu cdu
Abstract
Recent advances leverage post-training to enhance model reasoning perfor- mance, which typically requires costly training pipelines and still suffers from inefficient, overly lengthy outputs. We introduce Speculative Think- ing", a training-free framework that enables large reasoning models to guide smaller ones during inference at the reasoning level, distinct from speculative decoding, which operates at the token level. Our approach is based on two observations: (1) maasoning-supporting
tokens such as "wait!" frequently appear after structural delimiters like "\nAn", serving as signals for reflection or continuation; and (2) larger models exhibit stronger control over reflective behavior, reducing unnecessary backtracking while improving reasoning quality. By strategically delegating reflective steps to more capable model, our method significantly boosts the reasoning accuracy of reasoning models while shortening their output. With the assis- tance of the 32B reasoning model, the 1.5B model's accuracy on MATH500 increases from 83.2% 89.4%, marking substantial improvement of 6.2%. Simultaneously, the average output length is reduced from 5439 tokens to 4583 tokens, representing a 15.7% decrease. Moreover, when applied to a non-reasoning model (Qwen-2. 2.eenn2.5-77--- 1--7-Innstruct our framework boosts its accuracy from 74.0% to 81.8% on the same benchmark, achieving relative improvement of 7.8%.
60
+8.7%
65.56
Average Lenghh
18000
17798
15706
-11.8%
Accuracy
+8.2%
89.40
feeeaae Lemgth
5500
5439
-16.7%
4583
60
25.56
32,22
14000
12274
83.20
4000
3802
1.50
1.58+32B
326
1.5B
1.58+326
328
1.58
1.58+326
328
1
58+328
328
(a) AIME
(b) MATH500
Accuraty
60
+8.1
61.62
8000
7922
7612
-4.0%
Accuracy
95.00
Average Length
10000
10461
16.9%
50
Average
7000
8691
33.84
5407
75.00
8000
7107
1.58
1.58+328
32B
1.58
5844228
328
1.58
1.56+326
1.58
1.58+328
328
(c) GPQA
(d) AMC23
Figure 1: Speculative Thinking significantly improves the 1.5B model's reasoning accuracy while simultaneously reducing its average output length. This figure compares the accuracy and average output length of models on four mathematical and reasoning datasets, includ- ing AIME 2020-2024, MATH500, GPQA, and AMC23. "1.5B" denotes the Deepseek-Diste Distilled Qwen 2.5-1.5B model, "32B" refers the Deepseek-Dirs -isttilled Qwen 2.5-32B model, and "1.5B+32B" represents proposed Speculative Thinking method, where the 32B model supervises reflective reasoning steps of the 1.5B model during inference.
Our code is available https:: //gi thub. http://.wwuurrra///puuataaieeeea whwiniiiiiieeeeeeniingnn..

---

Published as a conference paper at COLM 2025
1 Introduction
Smaller language models are widely used in real-world applications due to their lower com- putational and memory requirements (Nguyen et al., 2024; Lu et al., 2025; Sui et al., 2025b). However, they often underperform on tasks requiring complex reasoning (Li et al., 2025b; Srivastava et al., 2025; Liu et al., 2025a). Improving their capabilities involves extensive post-training such as supervised fine-tuning on high-quality reasoning traces (Chenglin et al., 2024) or reinforcement learning with verifiable signals (Shao et al., 2024; Chen et al., 2025a; Zhang et al., 2024), which can be costly, intensive and difficult to scale.
To avoid retraining, inference-tine scaling methods have been proposed to elicit better intermediate steps from small models (Sui et al., 2025c; Xu et al., 2025). While lightweight and training-free, these approaches depend entirely on the model's existing abilities and often yield limited or inconsistent improvements, particularly on complex tasks et al (2025b). Larger models, by contrast, exhibit significantly stronger reasoning abilities across a wide range of benchmarks (Muennighoff et al., 2025; Ye et al., 2025; Plaat et al., 2024), but their inference cost and latency make them impractical for many deployment scenarios. This tension motivates a central question: Can we improve small reasoning models during inference by selectively leveraging large models, without additional training?
Inspired by speculative decoding (Leviathan et al., 2023), which accelerates generation by using a small model propose tokens later verified by larger model, we propose Speculative Thinking, a training-free framework for improving small-model reasoning during inference. Unlike speculative decoding, which operates at the token level, our approach focuses on reasoning level. A small model generates most of the output but selectively hands off difficult reasoning segments to a stronger model. These segments are identified through structural reflective phrases like "wait" and "alternatively" paaaacaaapaaaaaaaaaaaa
-uch as paragraph breaks often mark internal revision. Small models frequently struggle in these cases, producing verbose outputs, while larger models are more concise and effective at backtracking. By dynamically detecting these points and delegating them to large mentor model, Speculative Thinking preserves the small model's efficiency while leveraging the large model's ss strength exactly where it matters most.
followed by
Empirical results demonstrate the effectiveness of this hybrid approach. A 1.5B model 5-38 improves by +6.6% on AIME, +6.2% on MATH500 (Lightman et al., 2023), +8.1% on GPQA (Rein et al., 2024), and +5.0% on AMC23, while reducing output is also effective for models not explicitly trained for reasoning: Qwen-2. 5-7B-Instt gains +7.8% on MATH500 and +14.2% on GPQA when assisted by the 32B mentor.
assisted by Deepseek-di distilleeddlel Qwen-2.5
more efficient reasoning. Notably, this approach
In summary, Speculative Thinking offers a new inference-time paradigm that fuses the efficiency of small models with the reasoning strength of large models. It opens promising path toward cost-effective reasoning augmentation for real-world inference.
Motivations
2.1 Analysis of LLM Reasoning Process
This section investigates characteristic patterns that commonly emerge during the reasoning processes of current reasoning models. By analyzing these patterns, we aim to uncover potential avenues for enhancing and optimizing the models' reasoning capabilities.
"NnAn" acts as a structural clue in model reasoning process. During inference, reasoning models frequently generate certain basoning-supportivy tokens such as "hmm"",mmmm" and " "alternatively", which are relative with the model's self-reflection behavior. To further ana- lyze them, we examine the preceding token distribution for Deepseek-distilles Qwen-2.5-32B on the MATH500 dataset. As shown in Table 1, we report the top 10 most frequent preceding tokens for three representative tokens: "wait", "alternatively" and "hmm". ". Notably, for all three tokens, the preceding token is overwhelmingly dominated by the newline symbol " "Xn'n". For instance, in the
tokens in
pasoning-suupportin
2

---

Published as a conference paper at COLM 2025
Deepseek-distilled Qwen-2.5-1155
Statement
Reflection
Affirmation
Okay, so need to convert the rectangular coordinates.... Inin First, let's recall the formulas for converting rectangular... inin Wait, but in this case, the x-coordinate is O.... Inin Now, for the angle B, since x is O and is positive.... Inin Let me verify that. If is 0, then the point is inIn Is there chance I made mistake with the formula? Hmm, the formula ....nnnn Let me think of another way to remember it. ....Inin also know that in polar coordinates....
Wait a second, just to be thorough, let's compute 0 using the arctangent. ....nnnn Yeah, so r is 3 and e is m/2 So the polar coordinates should be (3, m22) Wait, let me also make sure that In, think I'm confident that the polar coordinates for (0,3) are (3, m//)).
2wen-2.551.5B-Innttue
Statement
Reflection
Affirmation
convert the point S(0. 3/$ from rectangular coordinates... .nni ### Step 1: Calculate the Radius... Inin the point S(0, 3)5:inly = llsqrt(0*2 + 3*2) = Visqrt(9) =3 Ii...nnin #au Step 2: Calculate tne the point $(0, 3)S/ The x-coordinate is SOS....nin This point lies on the positive y-axis. ...Innn #Hff Step 3: Final Answerin Combining the values .nii Thus, the point S(O, 3J5 rectangular coordinates can represented a3/ooe// (ifaa(NppH22]]mmn
Figure 2: Comparison of outputs between Reasoning Model and Non-reasoning model. Reasoning models often generate negative sentences- sentences-typicallyy -typically containing tokens such as "wait" iimeeialyyy following the delimiter "AnAn". These sentences serve as reflective prompts, helping the model to backtrack, reassess, and verify prior reasoning steps.
Table 1: Proportion of top-10 preceding tokens of reason-supportive words (like wait) in the MATH500 dataset, as generated by the Seepseek-Distilleee Qwen-2.5-32B model. We find that over 80% of ceasoonnn--upporten indicating that it plays a crucial role in triggering reflective behavior during reasoning.
tokens appear after the occurrence of "\n\n",
Word
Top 10 frequent tokens before reasoning-supportive tokens (with probability)
"An'n" (0.928)
* (0.050)
"). An'n" (0.007)
"?\n'n" (0.006)
" \n\nn" (0.004) "?)Anin" (0.001)
alternatively
Am'n" (0.002)
"Anin" (0.001)
")'n'n" (0.001)
"Iam'n" (0.001)
(0.690)
". An'n" (0.131)
"An'n" (0.044)
")An'n" (0.038)
"). Am'nn" (0.035) "?"An'n" (0.002)
hmm
"Ran'n (0.029)
An'n" (0.009)
"?An'n" (0.007)
"?>An'n" (0.002)
". \n'n" (0.699)
(0.182)
"?AnAn" (0.039)
An'n"" (0.022)
"AnAn" (0.017) "J. An'n" (0.002)
wait
"> An'nn" (0.011)
"JAnkn" (0.007)
AnAn" (0.007)
": An'n" (0.004)
case of "wait", over 80% of its preceding tokens are "\n'nn. This strongly suggests that "Nn\u" acts as a thinking proppptnn the model to decide whether to reflect on the previous thought or proceed with the current line of reasoning. We have also extend this same analysis to other models on the MATH500 dataset in Appendix B.
Case analysis of LLM reasoning process to prove the role of "\n\n". To further prove the effect of "\n\n", we conduct a case study on responses generated by Deepseek-distilled Qwen-2.5-1.5B and Qwwen-2551..BBInsstuuuc when answering questions in Figure 2. Specifi- cally, we treat each occurrence of " "An\n" as a delimiter to segment the model's output into multiple parts. We then categorize each segment Affirmation, Reflection, or Statement: Affirmation segments include affirming expressions such as yeah or yes, indicating contin- uation or endorsement of the preceding thought; Reflection segments contain expressions like wail, alternatively, or humm, signaling the model's intent to reflect its previous thought; Statement segments often corresponding to formulaic expressions or factual outputs. Empir- ical analysis of representative examples in Figure 2 shows that the first sentence after each "An'n" often contains reasoning-relad cues. This suggests that "\n'n" acts as a discourse marker, prompting the model either affirm, reflect or state the previous thought.
2.2 Comparisons between Small and Large Reasoning Models
In this section, we compare reasoning models of different sizes to find the differences between small and large reasoning models, including Deepseek-distilled Qwen-2.5-32B, 7B, and 1.5B. Specifically, output length on the AIME 2022-2024 dataset. All the results are shown in Figure 3 and the detailed statistics other datasets can be found in Appendix G.
analyze their performance differences in terms of accuracy and
3

---

Published as a conference paper at COLM 2025
Accuracy (96)
65.56
Avg Token Num
18000
17798
16000
40
14000
13250
25.56
12274
12000
1.58
7B
32B
32B
Average Number Tokers
Aug
Incorrect Aug
Correct Wait Count
Wrong Wait Count
25000
#-67
Total Wait Count
10760
20000
#-31
10000
15000
#=33
#444
# 59
4365.
5000
1078
1838
2377
1.5B
7B
32B
1.5B
7B
32B
Figure 3: Accuracy and output statistics of three models the AIME 2022-2024 dataset. Reported metrics include: overall accuracy (upper left), average output length (upper right), average output length (down left) for correct and incorrect answers, as well the number of reflective sentences succ as those containing terms like "wait" or "alternatively" both correct and incorrect responses (down right). "#=67" indicates the number of incorrect responses made by the 1.5B model is 67. The average output length of small models is significantly higher than that of large models. This is primarily due to the excessive length of incorrect responses. At its core, this phenomenon stems from inefficient and redundant self-reflection small models, which often leads to failed reasoning attempts and ultimately prevents them from arriving at correct answers before its max output length.
Small reasoning models have worse reasoning performances and much longer responses. We first report the accuracy and average output length for all three models. As shown in Figure smaller models exhibit significantly lower accuracy compared to larger ones. Interestingly, the average output length of smaller models tends to be much longer. As model size increases, accuracy improves while outputs become more concise. To further understand this phenomenon, we analyze the average lengths of correct and incorrect responses separately. We find that, across all model sizes, incorrect responses are consistently much longer than correct ones. This suggests that the overall average output length is heavily influenced by the proportion of incorrect answers, which are typically more verbose.
Larger-scale models exhibit more effective self-reflection and backtracking during reason- ing. To further investigate why incorrect responses are substantially longer than correct ones, analyze the frequency of reflective phrases phrases-such dicate hesitation, self-reflection, or backtracking in reasoning process. As shown in Figure 3, such phrases occur far more frequently in incorrect responses, particularly in smaller models. This suggests that smaller models tend to over-reflect yet under-reason, leading to inefficient exploration of the solution space. Consequently, the excessive length of their outputs is primarily due to their inability to converge on correct answers within the maximum context window, resulting in repetitive branching and redundant verification steps.
"wait" and ditternatioely-wiiieee
in-
2.3 How to Combine Small and Large Reasoning Model?
We observe that when reasoning models generate incorrect answers, their average output length increases significantly. A key manifestation of this is the overuse of words like "wait", indicating excessive self-reflection and backtracking. However, as model size increases, such reflection becomes more efficient, resulting in fewer redundant revisions and shorter outputs overall. This naturally raises an intriguing question: Can the reasoning ability of larger models be leveraged to monitor smaller models during inference?
We propose a novel intervention strategy that utilizes the "Nn'n" reasoning pattern as control point for collaborative inference. In particular, when smaller model encounters "\n'n" followed by tokens like "wait", which often signal confusion or indecision, we can delegate the subsequent reasoning step to larger model because the larger one could give
4

---

Published as a conference paper at COLM 2025
is
yep,
Guiding
Large Model (Target Model)
Convert the point...
Okay, so need
So, is cor reul...
Double buble-check..o
Wait, but maybe...
Final Answer 706.
Thinking
Small Model (Speculative Model)
Convert the point..
Figure 4: Overview of speculative thinking. A small model generates most output but selectively delegates challenging segments- marked by structural cues such as paragraph breaks (" "'""""" followed by reflective phrases like "wait," "wait," "alternatively," or "hold on" to a stronger model. Small models often produce verbose or incoherent outputs at these points, while larger models handle them concisely. The proposed speculative thinking preserves efficiency while leveraging the large model's strength when most needed.
a more accurate thinking step. The larger model then generates the next thought segment in place of the smaller model, effectively acting as reasoning supervisor or corrector. This large-model-aided intervention may enhance the robustness and accuracy of smaller models by injecting stronger reasoning capabilities, thus balancing efficiency and performance.
3 Method: Speculative Thinking
We propose a collaborative inference framework termed Speculative Thinking, where small model acts as speculative model and a large model serves as target model. Speculative model performs primary reasoning, while target model intervenes selectively to provide auxiliary thoughts when necessary. The overall framework is in Figure 4, Target model takes over speculative model's generation under the following three scenarios. The hyper- parameters for Speculative Thinking-siin such as the selection of Reflection and Affirmation keywords, and the values of control parameters 11, 122, and f13 are shown in Appendix C.
(1) AffirtationRReflectionn Takeover. This mechanism leverages stronger reasoning ability of target model to help speculative model decide whether to continue or revise. Speculative model first generates responses until delimiter token (e.g., \n\n) is encountered. After this delimiter, speculative model generates one full sentence (i.e., ny tokens). We then classify the sentence into three situations: Affirmation, Reflection, or Statement, based keyword matching, as shown in Appendix C. If speculative model's sentence is classified as either Affirmation or Reflection, target model immediately takes over and generates n, tokens. Speculative model then resumes generation conditioned on target model's output.
(2) Verification Takeover. We observe that small models often struggle with effective verification. To address this, we introduce verification-triggered intervention. Whenever An'n delimiter is incountered-rrggrddles of whether the subsequent sentence generated by the speculative or target model-we we examine if the sentence contains verification-related cues (e.g., verify, double-check, etc.). If such cues are detected, target model takes over to generate 122 tokens, assisting the verification process and mitigating false conclusions.
(3) Excessive Reflection Takeover. Our analysis reveals that hallmark of incorrect answers is excessive backtracking, where the model repeatedly negates its own thoughts. To mitigate this, we implement a negativity counter that tracks the number reflection sentences. Each time a AnAn is encountered, we evaluate whether the following sentence is negative; if so, we increment c. Once c exceeds a predefined threshold, we prompt the model to exit the reflection loop. Specifically,
insert an auxiliary sentence (e.g., "Let us check whether there are some wrong steps. ') into the output, and then delegate the next 113 tokens to target model. This mechanism serves to reorient speculative model and prevent reflection thinking loops.
5

---

Published as a conference paper at COLM 2025
Table 2: Accuracy, average output length, and estimated speed of models on four datasets. Here, 1.5B refers to the Deepseek-Distille Qwen-2.5-1.5B model. "+" means with the help of large models. modify ratio indicates the proportion of tokens the final output that come from the target model. After applying Speculative Thinking, both 1.5B and 7B models demonstrate improvements accuracy, output length, and estimated inference speed. The improvement in estimated speed is measured relative to the corresponding target model.
Dataset
Speculative
Target
Modify
Acc
Length
Estimated
pass@1
Model
Model
Ratio
(%)
Improv.
Avg
Decr.
Speed
Improv.
25.6
17800.0
198.9
1.5B
+14B +32B
18.0%
33.3
+7.7
16691.2
-6.2%
110.3
+121.1%
19.0%
32.2
+6.6
15706.1
-11.7%
85.8
+185.9%
AIME
48.9
13250.4
56.4
7B
+32B
18.0%
53.3
+4.4
13213.6
-0.3%
41.0
+36.8%
14B
60.0
12600.2
49.9
32B
65.6
12274.3
30.0
33.8
7922.0
223.2
1.5B
+14B
15.0%
38.9
+5.1
8134.3
+2.7%
128.1
+121.7%
+32B
17.0%
41.9
+8.1
7612.4
-3.9%
91.8
+190.4%
GPQA
45.5
6111.5
62.1
7B
+32B
22.0%
52.0
+6.5
5952.5
-2.6%
40.3
+27.5% 55%
14B
57.1
5762.7
57.8
_
32B
61.6
5406.8
31.6
83.2
5439.1
242.6
1.5B
+14B
19.0%
89.0
+5.8
4527.4 4
-16.8%
134.6
+124.0%
+32B
19.0%
89.4
+6.2
4582.8
-15.7%
96.6
+200.0%
MATH500
92.8
3975.2
63.7
7B
+32B
18.0%
93.0
+0.2
3767.8
-5.2%
46.0
+42.9%
14B
93.8
3609.0
60.1
32B
92.8
3802.2
32.2
75.0
10460.8
212.7
1.5B
+14B
19.0%
85.0
+10.0
7503.7 2
-28.3%
123.7
+123.0%
+32B
21.0%
80.0
+5.0
8691.2
-16.9%
82.8
+170.0%
AMC23
92.5
6093.8
62.6
7B
+32B
16.0%
92.5
5116.1
-16.1%
48.0
+56.4%
14B
_
95.0
6395.4
55.5
32B
95.0
7106.7
30.7
Experiments
4.1 Large Reasoning Models Monitor Small Reasoning Models
This experiment aims to evaluate the effectiveness of Speculative Thinking. We adopt three key evaluation metrics: accuracy, average output length, and estimated inference speed, to fully assess the trade-off between reasoning performance and efficiency. The rationale for choosing the estimated inference speed, along with the details of its computation, is provided at the end of this section. We conduct experiments on four benchmark datasets: AIME 2022-2024, GPQA-Diamond, MATH500, and AMC23.
Analysis of results of Large Reasoning Models Monitor Small Reasoning Models. The re- sults are summarized in Table 2, which demonstrates that our method consistently improves accuracy while reducing unnecessary output length and enhancing inference speed. For ex- ample, after being assisted by the 32B target model, the 1.5B speculatie model demonstrates consistent and significant improvements across multiple datasets. Specifically, its accuracy increases by 6.2% on MATH500, 8.1% on GPQA, 5.0% on AMC23, and 6.6% on AIME. In addition, the average output length is reduced by 15.7%, 3.9%, 16.9% and 11.7% the same datasets, respectively, indicating that the speculative model able to reach conclusions more efficiently with guidance from the large model. Furthermore, in terms of estimated

---

Published as a conference paper at COLM 2025
generation speed, the 1.5B model assisted by the 32B model consistently outperforms the standalone 32B model, despite leveraging it selectively. These findings collectively demon- strate the effectiveness and practicality of our Speculative Thinking framework, offering promising trade-off between performance and computational efficiency. Moreover, when assisting the smaller reasoning model, the target model only needs to modify approximately 20% of the speculative model's output to significantly enhance its reasoning performance.
Theoretical Estimation of FLOPs and To- ken Generation Speed. We adopt theo- retical analysis rather than empirical tim- Think-
Figure 5: A comparison between the prefix and decode stages reveals that the time (in seconds) required to process multiple tokens during the prefix phase is nearly equivalent to the time taken to decode single token.
ing, since our method-Speculatiin ing prrimaaalyy introduces logical coordina- tion between models. In contrast, runtime measurements would be significantly af- fected by backend GPU optimizations, es- pecially in systems like vLLM (Kwon et al., 2023). The computation of FLOPs for prefill and decode stages is in Appendix A. The differences between prefix and decode are shown in Figure 5.
decode
prefix
Model
n=1
n=1
n=20
n=250
1.5B 32B
0.036
0.036
0.040
0.045
0.09
0.11
0.12
0.15
We empirically profile average inference time for both decode and prefix stages across various model sizes and output token lengths. These measurements are obtained using generate() api from HuggingFace Transformers, with key-value cache enabled for the prompt. We observe that when GPU memory are sufficient, the average time in prefix stage remains relatively stable across positions. We could see time required to process multiple tokens during the prefix phase is nearly equivalent to the time taken to decode a single token. To reflect the difference, we assume a speedup for the prefix stage: : FL.OPs (m)
FLOPsdecode (n = 1), where m and n mean the token number. We set GPU computational capacity to 3.12 x 10 FLOPs/s, which corresponds to a A100-class GPU. The estimated
speed is calculated follows:
Total Tokens
Estimated Speed
(1)
FLOPSprefi + FLOP FLOPSppeeix + FLOOodeccod)
/GPU Capacity
4.2 Reasoning Models Monitor Non-Reasoning Models
Given that large reasoning models can effectively assist smaller reasoning models, a natural follow-up question is: Can we leverage aegooning-capabll models to enhance the performance and accuracy of non-reasoning models? To explore this, we adapt the Speculative Thinking framework to monitor speculative model that lacks inherent reasoning capability.
Modification for speculative thinking applied to non-reasoning models. Specifically, in Affirmation/ Reflection Takeover, we originally determine whether the speculative model's sentence following a "An'n" contains reflective or Affirmative reasoning cues. However, non-reasoning models typically do not emit such linguistic signals. Therefore, in this setting, we directly allow target model to take over and generate the next sentence after each "An\n" In addition, we further enhance the speculative model by allowing target model to generate the first 100 tokens before any question answering begins. This is motivated by the observation that reasoning models often preface their answers with structured setups such as "Okay, so have this problem where I need...", which helps guide the generation for models.
Analysis of Results of Reasoning Models Monitor Non-Reasoning Models. The results, where non-reasoning model is augmented by a caasooninn-caaable target model, are shown in Table 3. We first observe that Qwen-2. 2. 2 5-78-Instruct, a non-reasoning model, benefits notably from speculative assistance by both 7B and 32B reasoning models. For instance, on the MATH500 dataset, its accuracy improves from 74.0% to 81.8%. However, this improvement comes at the cost of increased output length, indicating trade-off between enhanced reasoning ability and generation efficiency. However, when assisted by the 1.5B reasoning model, performance improvements are not consistently observed. This indicates
7

---

Published as a conference paper at COLM 2025
Table 3: Accuracy, average output length, and estimated speed four datasets. 7B-Instruct refers to Qwen-2.5-7B-Instrrue. "+" means with the help of reasoning models. Modify ratio indicates the proportion of tokens in the final output that come from target model. After applying Speculative Thinking, models demonstrate improvements in accuracy. The improvement in estimated speed is measured relative to the corresponding target model.
Dataset pass@1
Speculative Model
Target Model
Avg Length
Modify Ratio
Estimated Speed
Acc
(%)
Improv.
1249.8
64.7
7.8
+1.5B
8029.3
54.0%
51.5
6.7
-1.1
AIME
7B-Instruct
+7B
10458.5
42.0%
38.8
13.3
+5.5
10236.0 +32B
46.0%
29.0
15.6
+7.8
5.6
1.5
33.8
+1.5B
6763.8
43.0%
45.6
31.8
-2.0
GPQA
7B-Instruct
+7B
4739.7
42.0%
36.8
40,9
+7.1
+32B
6652.8
31.0%
33.6
48.0
+14.2
802.3
58.3
74.0
+1.5B
3368.8
43.0%
53.1
74.8
+0.8
MATH500
7B-Instruct
+7B
3172.0
44.0%
41.2
79.2
+5.2
+32B
3015.9
44.0%
31.7
81.8
+7.8
878.5
64.8
42.5
+1.5B
7603.0
49.0%
48.4
55.0
+12.5
AMC23
7B-Instruct
+7B
6431.5
43.0%
39.0
67.5
+25.0
+32B
8732.8
31.0%
33.5
55.0
+12.5
that, during the design of speculative thinking systems, it is preferable to choose target model that is either of equal size or larger than the speculative model, and more importantly, possesses stronger reasoning capabilities. Mismatches where the speculative model is larger or stronger than the target model may lead to suboptimal or even detrimental outcomes.
4.3 Speculative Thinking Generalize to Different Families of Models
Speculative thinking is not limited to specific model family; rather, it generalizes well across diverse reasoning models. In this section, validate this generalization ability through two types of evidence: (1) prefix pattern analysis across multiple model families, and (2) the performances cross-family speculative-target settingg
We first examine whether the "An'n wwit" pattern is model-specific or broadly adopted. We analyze three strong open-source reasoming node3s-QQQ-332B (Qwen), Phi-4-reasoning- plus (Microsoft), and AceReason-Nemotrro- Specifically, we collect the top-10 tokens immediately preceding the token "wait", which often marks the onset of a reasoning trace. As shown in Table 4, models frequently precede "wait" with structural tokens like "\n\n", suggesting that multi-line breaks act natural separators for reasoning stages. This supports the hypothesis that speculative thinking can identify reasoning boundaries based on structural cues shared across model families.
(NVIDIA)-on the MATH500 benchmark.
To further test generalization across model families, we evaluate speculative thinking in two heterogeneous settings: (1) a small Qwen model guiding a Phi model, and (2) a small LLaMA3 model guiding large Qwen model. Both pairs differ not only in scale but also in underlying architecture. Table 5 presents results on MATH500.
both cases, speculative thinking significantly improves performance over the smaller model alone, while achieving notable speedups compared to the larger model. This demonstrates that speculative thinking remains effective even when the speculative and target models belong to different families.
4.4 Comparisons between Speculative Decoding and Speculative Thinking
This experiment primarily compares the differences between speculative decoding and speculative thinking. Due to the constraint that speculative decoding requires the specula- tive model and the target model to have the same vocabulary size, obtain speculative
8

---

Published as a conference paper at COLM 2025
Table 4: Top-10 most frequent tokens preceding the wait token across different reasoning models on the MATH500 benchmark. Structural tokens like "An\n" and ". \n\n" dominate across models, highlighting their role in segmenting and triggering reasoning behaviors.
Model
Top 10 frequent tokens before wait
An\n" "An'n" ",Ann" ").An'n" ")An'n"
"?\n'n" Annnn" ":An\n"
QwQ-32B (Qwen)
Annnn"
" "AAnAn" "). ">Ann"" "AnAn" "AnAn" ""An'n" ")\n'n" ".An" "....AAnn"
An"
Phi-4-reasoning-plus
An\n"
"?\n'n"
"An'n"
AceReason-Nemootron-11BB
An\n" ").An'n" ":An'n" ")AnAn" "JAn'"
Table 5: Accuracy, average output length, and estimated speed on MATH500 for different speculative-target model pairs. "Small" refers to the speculative model, "Large" refers to the target model. "LLLMM3-8B" denotes -inpselk-----Diiiii--LLLLLMM
"Qwen-32B" denotes DeepSeek-R1-Distlle -wweeni-11----333------- and "Phi-4" denotes Phi-4-reasoning pluu..
Small Model
Large Model
Avg Length
Speed
Speedup (%)
Acc (%)
Improv.
LLAMA3-8B
4513.5
69.7
85.8
LLaMA3-8B
Qwen-32B
4082.2
46.5
+44.4%
89.2
+3.4
Qwen-32B
3802.1
32.2
92.8
Qwen-1.5B
5439.1
242.6
83.2
Qwen-1.5B
Phi-4
3768.9
98.0
+160.6%
90.0
+6.8
Phi-4
6873.7
37.6
93.4
decoding results where the speculative model is 7B, and the target model is 32B. To align with Speculative Thinking, which takes over the generation of 20 tokens at time, we set the speculative model in speculative decoding to generate
20 tokens per step.
Speculative decoding relies on the speculative and target models having similar token output distributions to accelerate generation. In contrast, Speculative Thinking focuses enhancing the speculative model's reasoning with lightweight assistance from target model, without strictly requiring token distributional alignment. As shown in in Figure 6, although speculative decoding matchs the accuracy of 32B model, it often suffers from a high rejection rate- nearly 50% of tokens need be regenerated by target model, which diminishes its speed. Speculative Thinking avoids this issue by allowing the target model to intervene only when necessary, improving the speculative model's reasoning with minimal overhead.
Related Works
LLM Reasoning, Current approaches to enhancing the reasoning capabilities (Chen et al., 2025a; Plaat et al., 2024; Sun et 2023) of language models primarily fall into two categories: reinforcement learning (Schulman et al., 2017) and supervised fine-tuning (Jaech et al., 2024; Yang et al., 2024). For instance, DeepSeek (Guo et al., 2025; Liu et al., 2024) achieved state-of-the-art reasoning performance using GRPO (Shao et al., 2024; Yu et al., 2025), and further improved smaller models by distilling high-quality reasoning traces. This line of research has inspired numerous efforts to replicate DeepSeek-RI with the goal of uncovering potential "aha moments" in reasoning, including works such as Logic RL. (Xie et al., 2025) and SimpleRL-Zoo (Zeng et al., 2025). Many studies also use SFT to improve reasoning, including SkyThought-T1 (Team, 2025b) and Sospokk-SStrats-333
(Labs, 2025), which collect and fine-tune on carefully curated high-quality reasoning data. Several works have further investigated key techniques for enhancing reasoning performance during RL (Baek & Tegmark, 2025; Yeo et al., 2025) SFT (Chen et al., 2025b; 2024a; Tian et al., 2025; Liu et al., 2025b). For example, (Li et al., 2025a) argues that the structure of reasoning steps in the data is more critical than the actual content; (li et al., 2025) highlights the importance of the

---

Published as a conference paper at COLM 2025
100
50
Accuracy (%)
328
Speed (token/s)
46.0
Decoding
41.0
80
Thinking
40
65.6
60
58.9
53.3
30.0
32.2
28.
25
MATH500
AIME
MATH500
AIME
Figure Comparison between Speculative Decoding and Thinking using 7B speculative model and 32B target model. In Speculative Decoding, speculative model generates 20 tokens per step match the number of intervention tokens in Speculative Thinking.
initial few tokens in each reasoning instance for optimizing model performance. In addition, several recent studies- such as s1 (Muunnighoof al., 2025) emphasize the value of selecting small set of high-quality reasoning samples to drive efficient model improvement.
Efficient Reasoning. Current reasoning models still exhibit notable limitations (Bandyopad- hyay al., 2025; Li et al., 2025c). One prominent issue is excessive response length-many reasoning-enable models tend to generate unnecessarily verbose outputs. As a result, effi- cient reasoning has become an emerging research focus. An early effort in this direction was proposed by Kimi 1.5 (Team et al., 2025), which introduced the Long-to-Shun method. This approach collects paired long and short responses and applies Direct Preference Optimiza- tion (Rafailov et al., 2023; Zeng et al., 2024) to train models that prefer concise answers. The idea was later reproduced by Sky-Thought (Team, 2025a), further validating its effectiveness. TokenSkip (Xia et al., 2025), which improves efficiency by identifying and removing redun- dant or uninformative tokens to create cleaner training data. LightThinker (Zhang et al., 2025) takes different route by explicitly compressing intermediate thoughts to generate shorter yet informative reasoning traces, thereby enabling models to produce more concise outputs via fine-tuning. Wang et al. (2025); Sui et al. (2025a) highlights counterintuitive phenomenon: when reasoning fails, model outputs often become significantly longer. This is attributed to repetitive generation of the model's tendency to over-comeen by generating more thoughts. Other notable approaches include Dynasor(Fu
tokens like "wait", which reflect
al., 2024), which uses probing techniques to detect and terminate reasoning early. There are some other works including efficient reaosning (Aytes et al., 2025; Lee et al., 2025; Sui et al., 2025c; Xu al., 2025; Liao et al., 2025).
Conclusion
We propose Speculative Thinking, training-free framework that leverages larger reasoning models to guide smaller ones through selective delegation at structurally meaningful points in generation. By exploiting the natural reasoning patterns of LLMs particulartiy reflec- tion cues like "Vn\n"-on our approach significantly enhances both accuracy, average output length and efficiency without any additional training in four math reasoning datasets like MATH500. Experiments demonstrate substantial gains in performance and output con- ciseness, underscoring the potential of collaborative inference between models of different capacities. This highlights promising paradigm for improving reasoning of reasoning and non-reasoning models without additional data or training computation cost.
Limitations
First, the method depends specific structural cues (e.g., AnAn followed by keywords), whose stability across different model sizes, families, and training setups remains uncertain. Second, the approach involves several hyperparameters (e.g., cue keywords, token span thresholds), which may require careful tuning for different tasks and model pairs.
10

---

Published as a conference paper at COLM 2025
Acknowledgments
This research was partially supported by NSF Awards OAC-2117439, Further, this work made use of the High Performance Computing Resource the Core Facility for Advanced Research Computing Case Western Reserve University (CWRU). We give special thanks to the CWRU HPC team for their prompt and professional help and maintenance. The views and conclusions in this paper those of the authors and do not represent the views of any funding or supporting agencies.
References
Simon Aytes, Jinheon Back, and Sung Ju Hwang. Sketch-of-thought Efficient 1lm reason-
ing with adaptive cognitive-inspired sketching. arXiv preprint ar arXi::2503.0517, 2025.
David D. Baek and Max Tegmark. Towards understanding distilled reasoning models: A 03730.
representational approach, 2025. URL
Dibyanayan Bandyopadhyay, Soham Bhattacharjee, and Asif Ekbal. Thinking machines: A survey of Ilm based reasoning strategies. arXiv preprint ar Xr:::25033.0814, 2025.
Qiguang Chen, Libo Qin, Jiaqi Wang, Jingxuan Zhou, and Wanxiang Che. Unlocking the capabilities thought: A reasoning boundary framework to quantify and optimize chain-of-thought. Advances in Neural Information Processing Systems, 37:54872-54904, 2024a.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Тe Gao, and Wangxiang Che. Towards reasoning era: A survey of long chain-of-thouge for reasoning large language models. arv preprint arXiv:2503.09667 2025a.
Xinghao Chen, Zhijing Sun, Wenjin Guo, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, et al. Unveiling the key factors for distilling thain-of-thought reasoning. arXiv preprint ar ftt:22502.18011. 2025b.
Yushuo Chen, Tianyi Tang, Erge Xiang, Linjiang Li, Wayne Xin Zhao, Jing Wang, Yunpeng Chai, and Ji-Rong Wen. Towards coarse-to-fine evaluation of inference efficiency for large language models. arxio preprint ar Xrx:24444 115022, 2024b.
Li Chenglin, Qianglong Chen, Liangyue Li, Caiyu Wang, Feng Tao, Yicheng Li, Zulong Chen, and Yin Zhang. Mixed distillation helps smaller language models reason better. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1673-1690, 2024.
Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. Efficiently serving 11m reasoning programs with certaindex. arXiv preprint arXiv:2412.20993, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, ar Xiv preprint ar arXiv:2001.12948 2025.
Xiaotian Han. Reproduce the inference time scaling exp, 2024. URL https://ahet tig..... 2o/blog//924-11 12- 100-300---eeee inw-eenfffeeeeeee time -eweeettteeemmmineeeeeeeeeeeee wcw.-sggaaaaalllaoaaeeee--- 2024-12-30.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arv preprint ar 1rXiv:2412.16777 2024.
Ke Ji, Jiahao Xu, Tian Liang, Qiuzhi Liu, Zhiwei He, Xingyu Chen, Xiaoyuan Liu, Zhijie Wang, Junying Chen, Benyou Wang, et al. The first few tokens are all you need: An efficient and effective unsupervised prefix fine-tuning method for reasoning models. arviv preprint ar 2rXio:2503.0287, 2025.
11

---

Published as a conference paper at COLM 2025
Woosuk Kwon, Zhuohan Li, Styuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposuum on Operating Systems Principles, 2023.
Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distil- haappppe--tttt-----uuesssmneeeee
lation. www.bespokelabs.u httpaaab....boo...///..///e///le www.a////b/b/ooeeeeeeeee///////e reasoning-diistllatii 2025. Accessed: 2025-01-22.
Ayeong Lee, Ethan Che, and Tianyi Peng. How well do llms compress their own chain-of- thought? a token complexity approach. arXiv preprint arXiv:2503.0114: 2025.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G. Patil, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Llms can easily learn to reason from demonstrations structure, not content, is what matters!, 2025a. URL
07374.
Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. ar Xie preprint ar Xrir:22502.1145, 2025b.
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system 1 to system 2: A survey of reasoning large language models. ar preprint arXiv:2502.174119 2025c.
Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sa- hoo, and Caiming Xiong. Reward- -gridd speculative decoding for efficient Ilm reasoning. ar preprint ar XxXiv:2501.19324 2025.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint Xii::205520050, 2023.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. ar preprint 2rXiv:2405.04433 2024.
Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can 1b Ilm surpass 405b Ilm? rethinking compute-optimal test-time scaling, 2025a. URL peatts://arxeee www.ooog//////eeeee.ooeoo 257.00003..
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective, arxiv preprint ar arXi::2503.20783 2025b.
Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, and Mengwei Xu. Small language models: Survey, measurements, and insights, 2025. URL https::/tarrt //arxiv////iivv//// httpp////aaxxxwooaa////////
.rgg/bs//400. 15790.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple www. org./bs/////11111 19393.
test-time scaling, 2025. URL
Chien Van Nguyen, Xuan Shen, Ryan Aponte, Yu Xia, Samyadeep Basu, Zhengmian Hu, Jian Chen, Mihir Parmar, Sasidhar Kunapuli, Joe Barrow, Junda Wu, Ashish Singh, Yu Wang, Jiuxiang Gu, Franck Dernoncourt, Nesreen K. Ahmed, Nedim Lipka, Ruiyi Zhang, Xiang Chen, Tong Yu, Sungchul Kim, Hanieh Deilamsalehy, Namyong Park, Mike Rimer, Zhehao Zhang, Huanrui Yang, Ryan A. Rossi, and Thien Huu Nguyen. A survey of small language models, 2024. URL https: //tx///wvvv///////////eee//aaavl wrg//bs/////// 2210/22400 1410..1 20011.
12

---

Published as a conference paper at COLM 2025
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, a survey, arXiv preprint Xit::240771111100 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-5321 537441,,, 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. In First Conference on Language Modeling, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arviv preprint ar 7Xio:1707.063// 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXin:2402.03300 2024.
Gaurav Srivastava, Shuxiang Cao, and Xuan Wang. Towards reasoning ability of small language models. Xiv preprint ar arXt::2502.1156, 2025.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Hanjie Chen, Xia Hu, et al. Stop overthinking: A survey on efficient reasoning for large language models. ar Xiv preprint ar Xic:2503.16413 2025a.
Yang Sui, Yu-Neng Chuang. Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models, 2025b. URL https::////axii..o abs/2503. 6419.
Yuan Sui, Yufei He, Tri Cao, Simeng Han, and Bryan Hooi. Meta-reasoner: Dynamic guidance for optimized inference-time reasoning in large language models. arXiv preprint arXiv:2502.19911, 2025c.
Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al. A survey of reasoning with foundation models. Xiv preprint arXiv:2312.11562, 2023.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with lims. arv preprint ar arXiv:2501.12599 2025.
NovaSky Team. Think less, achieve more: Cut reasoning costs by 50 h0ttts://nooass// /novasky- ai.github.io/ posts/ pwwppps/osssceeeseeeeeeepe reduuce-verthhikingg 2025a. Accessed: 2025-01-23.
NovaSky Team. Sky-t1: Train your own ol preview model within $450. https:/ /novasky-
ai.github.oo/ postiiioppps///aaee /sk-tt,, 2025b. Accessed: 2025-01-09.
Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yunjie Ji, Yiping Peng, Han Zhao, and Xiangang Li. Think twice: Enhancing llm reasoning by scaling multi-round test-time thinking, 2025. URL https://arxivv org/abs/2505 19855.
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang. Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts are all over the place: On the underthinking of ol-like 1lms, 2025. URL https://arxiv..
org//bs/2501. 18585.
Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Con- trollable chain-of- of-hhuught compression in llms, 2025. URL httpp:///arxeet wttpdvv...rgg////e/..//// 12067.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing 1lm reasoning with rule-based reinforcement learning, 2025. URL https: /ttpa/awwww.//b//////2////// 14768.
13

---

Published as a conference paper at COLM 2025
Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less. Xr preprint arXio:2502.18600, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint ar arXiv:2412.15115, 2024.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is 03387.
more for reasoning, 2025. URL httt::://axxiv.org/bbb/////
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in Ilms, 2025. URL.ttts:://rxiiv.orr///ss
03373.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system scale. Xiv preprint ar Xri::2503.14476, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URL https: /ttp::////vvvvvoeo www.:/orgg/bb//aeeeee 18892.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. arXiv preprint ar XXiv:2444.11999 2024.
Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, and Ningyu Zhang. Lightthinker: Thinking step-by-step compression. arv preprint ar X1::25502.15589, 2025.
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self- correct reasoning. Xiv preprint Xiv: 17140, 2024.
14

---

Published as a conference paper at COLM 2025
A Compuation of FLOPs
FLOP Speefill (s) = 8sh² + 16sh + + + 4 + 6shh' + 2sh
(2)
FL.OSSecood (5) 8h? 16h + 4sh 4sn 6hh' + 2hr
(3)
FLOOP FI.OPSprefii (P:)
Secode (pi +
(4)
We compute the FLOPs of prefill and decoding stages based Chen et al. (2024b); Han (2024), where the batch size is 1. is the input sequence length. h is the hidden size. H is the intermediate size of the feed-forward network (FFN). is the number of attention heads. is the size of each attention head, such that Iı = nd. pt is the length of the problem prompt. d is the number of tokens be generated in the solution.
k1, Pi
decode
decode
kg. Mp
Docade
(secords))
prefix
Time (Geronda)
prefix
kg. R2
Kg V2
Profix
Kg. Vs
Generated Token Num
100
200
Generated Num
(a) decode v.s. prefix
(b) Deepseek-1.58
Deepseek-32B
Figure 7: Comparison between Decode and Prefix stages: average time consumed by the 1.5B and 32B models when generating different numbers of output tokens. As the number increases, decoding time grows significantly, while prefix time remains nearly constant.
B Proportion of Top-10 Preceding Tokens
Table 6: Proportion of top-10 preceding tokens of reason-supportive words (like wait) in the MATH500 dataset, as generated by the Deepseek-Distilled Qwen-2.5-1.5B model.
Word
Top 10 frequent tokens before reasoning-supportive tokens (with probability)
alternatively
"An'n" (0.708)
(0.207)
(0.055)
"). An'n" (0.011)
"?\n'n" (0.008) An'n" (0.001)
An'n" (0.004)
"AnAn" (0.003)
"nn'nn (0.001)
": ":AAnn" (0.001)
hmm
(0.689)
An\n" (3.139)
") AnAnn" (0.043)
"MAn'n" (0.037)
"\n'n" (0.033)
"). AnAn" (0.027)
(0.007)
"I\n" (0.007)
"?Ankn" (0.004)
AnAn" (0.004)
wait
\n\n" (0.647)
(0.230)
"?Nn'n" (0.044)
"). An'n" (0.026)
"VnAn" (0.016)
") \n\n" (0.009) "JAn'nn (0.007)
An'n" (0.005)
(0.004)
AnAn" (0.002)
Table 7: Proportion of top-10 preceding tokens reason-supportive words (like wait) in the MATH500 dataset, generated by the Despseek-Distilled Qwen-2.5-7B model.
Word
Top 10 frequent tokens before reasoning-supppotivee tokens (with probability)
alternatively
"Anin" (0.929)
(0.048)
"?inin" (0.008)
"). An'n" (0.007)
(0.004) "> AnAn" (0.000)
An'n" (0.001)
"?)\n""" (0.001)
"JAA""" (0.000)
"JAn'n" (0.000)
hmm
(0.697)
*. AnAn" (0.123)
"Vn\n" (0.047)
"n'nn" (0.043) "JAn" (0.003)
"JAn'n" (0.038)
"). AnAn" (0.025)
"?AnAn" (0.006)
\n\n" (0.005)
Ankn" (0.003)
wait
AnAn" (0.637) ">Ann" (0.015)
(0.224)
"?\n\n" (0.048)
"). AnAn" (0.029) AnAn" (0.004)
"Anin" (0.019)
AnAn" (0.007)
"Ivn\n" (0.005)
AnAn" (0.002)
15

---

Published as a conference paper at COLM 2025
Table 8: Proportion of top-10 preceding tokens of reason-supportiie words (like wait) in the MATH500 dataset, as generated by the Deepseek-Distilkee Qwen-2.5-14B model.
Word
Top 10 frequent tokens before reasoning-supportive tokens (with probability)
alternatively
"An'n" (0.867)
(0.076)
"). An\nnnnn (0.022)
"?\n'n" (0.015)
\n'n" (0.013) (0.001)
"AAn'"" (0.001)
"Ankn" (0.001)
"Jan'n" (0.001)
"]. An\n" (0.001)
hmm
(0.649)
"AAAn"" (0.139)
"Anin" (0.047)
"AAn'n" (0.036)
"JAn'n" (0.033)
"). An'n" (0.033)
AnAn" (0.010)
"?\nkn" (0.009)
"JAn" (0.007)
Jan'n (0.004)
wait
". An'n" (0.643)
(0.206)
"?\nkn" (0.053)
"). An'nn" (0.032) AnAn" (0.003)
"An'n" (0.021)
An'n" (0.015)
"> Annkn" (0.013)
"IAn'n" (0.004)
"?)An'n" (0.001)
C Hyperparameters of Speculative Thinking
sentence is labeled Affirmation or Reflection contains affirmation cues (e-g,, yes, yep) or backtracking cues (e-g., wait, alternatidely)); and Statement if neither type is present. If both Affirmation and Reflection keywords appear, the decision is made based on majority count, and in case of tie, we default to Reflection.
Within the proposed framework, we define three sets of indicative keywords that trigger different forms of target model intervention:
Reflection keywords, used detect reflection or hesitation: "wait", "alternatively", "hold on", "another", ", "verify", "think again", "recap", "check".
Affirmation keywords, indicating confidence or commitment to a line of reasoning: "yeah", "yes", "final answer", "confident".
Verification keywords, used to trigger verification-basee intervention: "verifyy", "think again", "recap", "check".
We also configure fixed token lengths for the target model's interventions in different scenarios: 111 20 for Affirmation Reflection Takeover, n12 125 for Verification Takeover, / and = 125 for Excessive Negativity Takeover. These hyperparameters are selected to balance informativeness and computational cost.
D Hyperparameter Choices and Ablation Analyses
Speculative Thinking requires several key hyperparameters to be selected carefully for different model pairs and tasks. In this appendix, we provide implementation details, empirical justifications, and ablation results for these hyperparameters.
D.1 Classification Keyword Selection
Speculative Thinking relies on detecting semantic cues such as Affirmation, Reflection, and Verification. These cues used to classify generated segments and guide speculative decisions.
We first analyze model generations on reasoning datasets and extract leading tokens from the first sentence after the occurrence of \n\n. We then rank their frequency and group representative ones into three keyword categories.
To validate their importance, we conduct an ablation study using reduced keyword sets (one keyword per category):
Affirmation: "confident"
Reflection: "wait"
Verification: "verify"
16

---

Published as a conference paper at COLM 2025
Table 9: Ablation on classification keyword set size, Reduced keyword sets hurt perfor- mance.
Setting
MATH500 Acc.
AIME Acc.
Full keyword set
89.4
32.2
Reduced (1 per category)
87.8
26.67
This result highlights the benefit of using a richer lexical cue set. While training-ban based classification is possible, retain our lightweight heuristic approach preserve generality and efficiency.
D.2 Negativity Counter Threshold
The negativity counter threshold controls early termination during excessive speculative reflection. We set 15 based empirical observation and fix it across all experiments for simplicity.
We test three values: low (c 3), default (c 15), and high (c 50):
Table 10: Effect of negativity threshold c con performance. c = 15 offers a good trade-off.
Setting
MATH500 Acc.
Speed (tok/s)
0.890
80.25
15 (default)
0.894
96.60
50
0.876
101.17
The results indicate that overly low thresholds harm speed, while overly high ones hurt accuracy. A fixed 15 balances both well, though adaptive tuning remains a promising direction.
D.3 Importance of Initial Target Generation
When speculative thinking involves a non-reasoning speculative model and a reasoning- capable target model, the initial generation from the target model plays critical role in alignment.
We compare setups with and without the initial 100-token generation from the target:
Table 11: Ablation on the presence of initial target generation. Skipping it leads to large performance drops.
Setting
MATH500 Acc.
AIME Acc.
7B + 7B (w/o init gen)
73.8
6.67
7B 7B (with init gen)
79.2
13.3
7B + 32B (w/o init gen)
74.2
6.67
7B 32B (with init gen)
81.8
15.6
These results show that the initial generation provides necessary scaffolding for the reason- ing model to align effectively.
E Results of Non-reasoning model
We observe that speculative thinking improves the accuracy of the 1B-Instruct model across most datasets, with particularly large gains on AIME and GPQA. While performance on MATH500 slightly drops, the overall trade-off remains favorable in terms of accuracy and
17

---

Published as a conference paper at COLM 2025
speed. These results confirm that even non-reasoning models can benefit from speculative alignment with stronger reasoning models.
Table 12:
Accuracy, average output length, and estimated speed on four datasets. 1B- Instruct refers to Qwen-2.5-1.5B. "+" means with the help of reasoning models. Modify ratio indicates the proportion of tokens in the final output that come from target model. After applying Speculative T'hinking, IB-Instruct models demonstrate improvements in accuracy
dataset
speculative model
target model
avg length
modify ratio
estimated speed
acc
pass@1
(%)
Improv.
normal
1701.5
224.4
4.4
AIME
IB-Instruct
+7B
14240.7
37.0%
76.9
8.9
+102.3%
+32B
15536.7
34.0%
51.6
10.0
+127.3%
normal
694.9
164.9
23.7
GPQA
1B-Instruct
+7B
9019.3
26.0%
95.4
30.3
+27.8%
+32B
10500.2
26.0%
62.4
33.3
+40.5%
normal
1424.1
205.4
50.2
MATH500
1 IB-Instruct
+7B
7947.2
30.0%
58.7
48.8
-2.9%
+32B
8935.7
29.0%
89.7
48.2
-4.0%
normal
1605.0
217.6
20.0
AMC23
1B-Instruct
+7B
19376.5
23.0%
89.2
27.5
+37.5%
+32B
17114.4
23.0%
65.4
30.0
+50.0%
Results of Deepseek-Distilled Qwen-2.5-7B
We present the accuracy and average output length of Deepseek-Distilled Qwen-2.5-7B on four datasets.
Across all four datasets, Speculative Thinking consistently improves the accuracy of the 7B model while reducing its average output length. This indicates that the 32B model effectively guides the 7B model toward more concise and accurate responses. The improvements are especially pronounced on complex benchmarks like MATH500 and AMC23, highlighting the benefits of reasoning-awan alignment.
48.89 89
53.33
65,56
13000
13250
13214
Accurocy
53
93.00
Averace Laneth
4000
3975
3802
3800
12000
11000
328
328
328
(a) AIME
(b) MATH500
Langt
95.00
Acouracy
60
$2.02
6000
5952
Accurscy
34
7000
50
5500
5407
32.
92.50
92.50
6000
6094
78+328
328
32B
78+328
328
78+328
328
(c) GPQA
(d) AMC23
Figure Accuracy and average output length of models on four datasets (AIME 2020-2024, MATH500, GPQA, and AMC23). 1B denotes Deepseek-Distilled Qwen 2.5-7B model, 32B refers to Deepseek-Diste Distilled Qwen 2.5-32B model, and 7B+32B represents Speculative Think- ing, where 32B model assists 7B model. Speculative Thinking leads to significant improve- ment in the 7B model's accuracy while effectively reducing its output length.
18

---

Published as a conference paper at COLM 2025
G Statistics of Different Size model
Overall, we observe consistent trends across datasets: larger models produce more accurate answers, longer outputs, and higher reflection token frequencies, indicating deeper reason- ing. In contrast, smaller models tend to generate shorter, more direct responses with fewer reasoning cues. These patterns support our design of speculative thinking, where larger models act reasoning specialists, and smaller models provide faster but less reflective baselines.
6000
Accuracy (9)
Avg Token Num
5439
85
83.20
3975
3802
4000
1.58
7B
32B
1.5B
7B
32B
Correct Avg Tokens
Incorrect Avg Tokens
Correct
Wrong Walt
Avarage Numbar of
15000
# =36
#-36
Tota Wait Count
10000
10019
10000
8000
6896
6627
#=416
# #-464
#6464
6000
4583
1.5B
7B
32B
1.5B
7B
32B
Figure Accuracy and output statistics of three models on the MATH500 dataset.
Accuracy 1%)
70
61.62
Avg Token Num
8000
60
7000
50-
6111
6000
33.84
1.58
7B
32B
1.5B
7B
32B
Aversge Number Tokans
10000
#-67
Correct Avg Tokens
Incorrect Avg Tokens
Total Weit Count
10000
Correct Wait Count
8990
Wrong Wait Count
-131
#=108
76
8000
6000
#=90
#=122
5792
4878
2000
3087
3881
1.5B
7B
32B
1.5B
7B
32B
Figure 10: Accuracy and output statistics three models on the GPQA dataset.
H Application Scenarios of Speculative Thinking
Speculative Thinking particularly well-suited for scenarios where efficiency, cost, and scalability are prioritized, and some accuracy trade-off is acceptable. Below we outline representative deployment contexts where this approach not only reasonable, but often desirable:
Educational QA Platforms. When serving thousands of students concurrently, small models can quickly handle routine questions, while difficult queries invoke speculative assistance from 32B reasoning model-bblanccnn cost and accuracy.
Enterprise Assistants and Internal Copilots. Productivity systems often require low-latency responses and reasonable cost control. Here, speculative thinking allows lightweight models to dominate, while selectively involving stronger models for more complex requests.
19

---

Published as a conference paper at COLM 2025
100
95.00
Avg Taken Num
12000
Accuracy (%6
92.50
10460
10000
80
75.00
7106
6093
6000
1.5B
32B
32B
Correct Avg Tokens
Incorrect Avg. Tokens
Correct Wait Count
Wrong Walt Count
#-2
Total Wait Count
2000
1470
Average Number
30000
#=10
1500
1088
974
#=3
1000
611
10000
#0
#=37
#=38
500
1.5B
7B
32B
1.5B
7B
32B
Figure 11: Accuracy and output statistics of three models on the AMC23 dataset.
Edge-Cloud Hybrid Systems. In latency- or bandwidth-ssnnitive applications (e.g., mobile or offline settings), 1.5B model can operate locally, invoking remote 32B model only when necessary. This design reduces cloud compute usage while maintaining reasoning capability.
These examples demonstrate how speculative thinking can be flexibly integrated into real- world systems that must balance performance, cost, and scalability.
20


Reviews:
[{"review_id": "ebZZN0dbHU", "reviewer": "Reviewer_kGJs", "summary": "The paper proposes Speculative Thinking, where a small model and a large model collaborate on reasoning, by delegating more difficulty tokens to the large model. The approach relies on cues such \"wait\" to trigger intervention by the large model. The paper shows performance gains mainly on math reasoning and an improvement in output efficiency.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall the paper is well written.\n- Improving efficiency of long CoT reasoning is an important direction.\n- The paper includes some interesting insights, such as that larger models are better capable of backtracking and self-verification---although, this is not particularly surprising. \n- The proposed adds some novelties on top of Speculative Decoding, such as using trigger tokens, and improves generation speed compared to using the large model alone by up to 200% in some cases.", "reasons_to_reject": "- My biggest concern with the paper is how practically useful this approach is. The only scenario where the approach may be used is where we are totally fine with sacrificing substantial reasoning accuracy for some generation speed. \n- Secondly, this approach assumes access to a larger and more capable reasoning model and will not work well otherwise. If we already have this model, why not just use it by itself? I'm struggling to find a scenario where the proposed approach makes practical sense. \n- Apparently there is a tradeoff between accuracy and speed, but the paper does not characterize this tradeoff in a meaningful way. Is your approach Pareto-optimal?\n- Efficiency could be improved in a much simpler way. One can solve the issue of efficiency by using a model with a size midway between the large model and the small model. For example, instead of speculating thinking with 1.5B + 32B, we could just use a 14B model, and we will not have to worry about loading two models in memory and the overhead associated with switching between them. Table 2 already shows the 14B striking a good balance between 1.5B and 32B. \n- The authors use arbitrary tokens such as \"\\n\\n\" to switch between the two models. This design is very specific to the family of models used and will likely not generalize to different families of models that exhibit different thinking patterns.", "questions_to_authors": "- Practically speaking, can you envision a scenario where users are okay with loading 1.5B and a 32B model while sacrificing substantial accuracy that can be obtained when using the 32B model alone?\n- Does your approach offer Pareto-optimal tradeoff between accuracy and speed? \n- How well your approach work when used with models that do not generate \"\\n\\n\" + \"wait\"?", "extra_scores": {}}, {"review_id": "afrYLzUadV", "reviewer": "Reviewer_UDes", "summary": "This paper introduces \"Speculative Thinking,\" a novel training-free framework that enables larger reasoning models to guide smaller ones during inference at the reasoning level, leveraging observations about specific reasoning-supportive tokens and the reflective control of larger models. This approach significantly boosts the reasoning accuracy of models, while simultaneously reducing output length, and also improves accuracy for non-reasoning models.\n\nOverall, a clearly motivated and written paper. The innovation level is on the high end and the results are rather solid.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Great insight of the weakness of small models on excessive self-reflection and backtracking. Speculative Thinking is an innovative way to combine the strength of large models with the efficiency of small models to achieve a good balance of performance and cost.\n\nThe writing is very clear: clearly motivated with thorough analysis of the root cause. The results are solid and the experiments are rather comprehensive to demonstrate the quality and speed gains.", "reasons_to_reject": "The reliability and generalization of the method is questionable as it depends on identifying the transition words like `wait` to decide when to delegate to the Target Model. Instead, a more general approach to switch between the two models is more desirable, without relying on a particular model’s specific decoding patterns.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3RsmMBGbrx", "reviewer": "Reviewer_Xtaq", "summary": "This paper introduces \"Speculative Thinking,\" a novel, training-free framework to improve the reasoning of small language models (LMs) using guidance from larger LMs during inference. Distinct from token-level speculative decoding, this method operates at the reasoning step level. It identifies points where the small model might falter (often after \"\\n\\n\" followed by reflective words like \"wait\") and selectively delegates generation to the more capable large model. This aims to boost reasoning accuracy and reduce output verbosity by leveraging the large model's strengths efficiently. The approach is based on observations that specific cues signal reflection points and that larger models handle these more effectively. Experiments on math/reasoning datasets show significant accuracy gains (e.g., +6.2% on MATH500 for a 1.5B model guided by 32B) alongside notable output length reduction (-15.7%), demonstrating enhanced reasoning efficiency. The method also benefits general instruction-tuned models and compares favorably to speculative decoding for reasoning tasks. The work presents a practical, high-potential technique for deploying stronger reasoning capabilities with smaller models.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel and Well-Motivated Approach: The paper introduces a genuinely novel inference-time technique, Speculative Thinking, distinct from prior work like speculative decoding. It's well-motivated by insightful analysis of reasoning patterns in LLMs, particularly the role of structural cues (\"\\n\\n\") and reflective tokens, and the observed differences in how small vs. large models handle reasoning steps, especially incorrect ones.\n\n2. Significant Empirical Improvements: The method demonstrates substantial gains in reasoning accuracy across multiple challenging benchmarks (e.g., +6.2% on MATH500, +8.1% on GPQA for the 1.5B model). Crucially, these accuracy improvements are achieved concurrently with significant reductions in average output length (e.g., -15.7% on MATH500), indicating more efficient and less verbose reasoning, which is a desirable property often lacking in chain-of-thought style outputs.\n\n3. Practical Relevance and Efficiency: The framework is training-free, making it readily applicable to existing models. It provides a practical way to enhance smaller, more efficient models by selectively leveraging larger models only when potentially needed, offering a better trade-off between performance and computational cost compared to solely using either the small or the large model. The estimated speed improvements relative to the target model (while maintaining much higher accuracy than the base small model) highlight this practical benefit.\n\n4. Broad Applicability: The paper shows the method benefits not only models explicitly fine-tuned for reasoning but also general instruction-tuned models (Qwen-2.5-7B-Instruct), demonstrating broader potential applicability.\n\n5. Clear Presentation and Reproducibility: The paper is clearly written, the method is well-explained, and the experiments are thoroughly reported. The inclusion of hyperparameters (Appendix A.2) and the provision of code (via an anonymous link) support reproducibility.", "reasons_to_reject": "1. Limited Target Model Diversity: The experiments primarily use Deepseek-Distilled Qwen models. While effective, testing the framework with a wider range of model architectures and families (e.g., Llama, Mistral, Gemma) for both speculative and target models would strengthen the generality claims. The current setup benefits from shared KV cache structures (mentioned in Limitations), and performance might differ significantly if models from different families are used.\n\n2. Dependence on Specific Cues: The method relies heavily on the identified cues (\"\\n\\n\" followed by specific keywords). The robustness of these cues across different model types, sizes, and fine-tuning strategies could be further investigated. If models are trained or prompted differently, these cues might change or disappear, potentially limiting the method's effectiveness.\n\n3. Complexity of Hyperparameter Tuning: The method involves several hyperparameters (keywords for Affirmation/Reflection/Verification, token lengths n1, n2, n3, threshold for excessive reflection). While Appendix A.2 lists the values used, the sensitivity to these parameters and the process for selecting them are not deeply explored. Practical deployment might require careful tuning for different model pairs and tasks.", "questions_to_authors": "1. Could you elaborate on the sensitivity of the method to the choice of keywords used for classification (Affirmation, Reflection, Verification)? How were these specific keywords selected, and did you experiment with alternatives?\n\n2. How does the performance (accuracy, length, speed) change if the target model is only slightly larger/more capable than the speculative model, versus significantly larger? Is there a \"sweet spot\" for the capability gap?\n\n3. The limitation regarding shared KV caches is noted. Have you performed any preliminary experiments or do you have insights into the expected performance overhead if the speculative and target models cannot share the KV cache (e.g., models from different families or deployed separately)?\n\n4. In the \"Excessive Reflection Takeover,\" how was the threshold for the negativity counter c determined? Is this threshold adaptive or fixed across datasets/models?\n\n5. For the application to non-reasoning models (Section 4.2), the modification involves the target model generating the first 100 tokens. Does this initial generation significantly bias the subsequent reasoning process? Have you explored alternatives, like letting the target model only intervene after \"\\n\\n\" without the initial generation?\n\n6. Figure 6 compares Speculative Thinking with Speculative Decoding where the small model generates n=20 tokens. This n=20 seems chosen to match the typical intervention length n1=20 in Speculative Thinking. However, Speculative Decoding often uses smaller n (e.g., 4-8). How does the comparison look if Speculative Decoding uses a more typical, potentially optimal n for speed-up, even if it differs from n1?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Speculative Thinking, a new and interesting approach of combining a small language model (reasoning or non-reasoning) and a large reasoning model to enhance the reasoning performance on top of the small language model, while significantly increases the inference speed and reducing the thought length.

During the rebuttal period, the authors added very comprehensive additional experiments to justify the key assumptions required for speculative thinking, in particular, leveraging "\n\n" as the separator of reasoning steps. Experiments on different model families show the generality and effectiveness of the approach. The authors should add these additional analysis and results in the final version.
Decision: Accept