SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning
Authors: Rui Pan
Yinwei Dai
Zhihao Zhang
Gabriele Oliaro
Zhihao Jia
Ravi Netravali
Venue: NeurIPS 2025 poster ()
Keywords: Inference-Time Compute
Large Reasoning Models
Efficient Inference
Speculative Execution
Systems for Machine Learning
Primary area: infrastructure

Abstract: Recent advances in inference-time compute have significantly improved performance on complex tasks by generating long chains of thought (CoTs) using Large Reasoning Models (LRMs). However, this improved accuracy comes at the cost of high inference latency due to the length of generated reasoning sequences and the autoregressive nature of decoding. Our key insight in tackling these overheads is that LRM inference, and the reasoning that it embeds, is highly tolerant of approximations: complex tasks are typically broken down into simpler steps, each of which brings utility based on the semantic insight it provides for downstream steps rather than the exact tokens it generates. Accordingly, we introduce SpecReason, a system that automatically accelerates LRM inference by using a lightweight model to (speculatively) carry out simpler intermediate reasoning steps and reserving the costly base model only to efficiently assess (and potentially correct) the speculated outputs. Importantly, SpecReason's focus on exploiting the semantic flexibility of thinking tokens in preserving final-answer accuracy is complementary to prior speculation techniques, most notably speculative decoding, which demands token-level equivalence at each step. Across a variety of cross-domain reasoning benchmarks, SpecReason achieves 1.4-3.0$\times$ speedup over vanilla LRM inference while improving accuracy by 0.4-9.0%. Compared to speculative decoding without SpecReason, their combination yields an additional 8.8-58.0% latency reduction. We open-source SpecReason at \url{https://anonymous.4open.science/r/specreason/}.

SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning
Rui Pan
Yinwei Dai
Zhihao Zhang
Gabriele Oliaro
Zhihao Jia
Ravi Netravali
Princeton University
*Carnegie Mellon University
fruipan,
y (ruipan,yinweien wttpryyiiiiiddiiieeettttoooo edu, {zhihaoz3, goliaro)@es cmu. zhihao@cmu, edu, rnetravali@cs. princeton. edu
Abstract
Recent advances in inference-time compute have significantly improved perfor- mance on complex tasks by generating long chains of thought (CoTs) using Large Reasoning Models (LRMs). However, this improved accuracy comes at the cost of high inference latency due to the length of generated reasoning sequences and the autoregressive nature of decoding. Our key insight in tackling these overheads is that LRM inference, and the reasoning that it embeds, is highly tolerant of approxi- mations: complex tasks typically broken down into simpler steps, each of which brings utility based on the semantic insight it provides for downstream steps rather than the exact tokens it generates. Accordingly, we introduce SpecReason, system that automatically accelerates LRM inference by using lightweight model to (spec- ulatively) carry out simpler intermediate reasoning steps and reserving the costly base model only to assess (and potentially correct) the speculated outputs. Impor- tantly, SpecReason's 's focus on exploiting the semantic flexibility of thinking tokens in preserving final-answer accuracy is complementary to prior speculation tech- niques, most notably speculative decoding, which demands token-level equivalence at each step. Across a variety of cross-domain reasoning benchmarks, SpecReason achieves 1.4 3.0x speedup over vanilla LRM inference while improving accu- racy by 0.4 9.0%. Compared to speculative decoding without SpecReason, their combination yields an additional 8.8 58.0% latency reduction. We open-source SpecReason at httph::t/// //tg:://gttttttt com http///eeeeeeeeaaaaappeeeeooo swppeeerrppeeeeeeeeesoooooo
1 Introduction
Inference- tirrencce-imeeeeee compute has unlocked a new axis for scaling Al capabilities. Recent advancements in Large Reasoning Models (LRMs) such OpenAl ol/o3 (Jaech et al. 2024 ope 2025 and DeepSeck R1 [Guo et al. 2025) have demonstrated state-of-the-art performance across a wide range of complex tasks. Although these LRMs share the architectural backbones traditional large language models (LLMs), their inference behavior differs significantly: LRMs first "think" by generating internal thinking tokens tokens that decompose a task into a sequence composable reasoning steps via long
(CoT) WWi et al. 2022] before producing the final tokens that summarize the reasoning process.
Despite their promise, LRMs incur substantial inference latency due to the length of the reasoning sequences they generate. This challenge is primarily driven by the autoregressive nature of LLMs, where decoding time scales linearly with sequence length. As a result, final output generation can routinely take minutes, if not hours, answer single query: such delays far exceed those of typical LLMSs and are prohibitively slow for many interactive applications, ultimately degrading user experience (Fu al. 2024b
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Questron: Every morning Aya goss for a Sef-kllometer-iong walk and stops st coffee shop afterwards. When she walks at constant speed of SeS kllometeess per hour, the walk takes her hours, including StS minutes spent in the coffee shop. When she walks Sa+25 klometers per hour, the walk takes Vrrac( [2] kilometers per hour. Find the number of minutes the walk takes her, including the S15 minutes spent in the coffee shop.
hours and minutes, including $1$ minutes spent coffee shop. Suppose Aya
Small Model X
SpecFReason
Saln: km/hhr
The result (2.5 0.5) 60
minutes.
First, let me try parse the problem step by step in the first scenano. laking the postive
240 1- 9
most a 0.009 sm/h Thus a 240 - (68- 12"se 190 e 182 12 12's0rt(18) 139 6932.... The result. bbooed (050) minutes.
First, me try parse the problem step by step.
first acanario, 240 9
second ecenand: = // +
these equatione to
then Lden- 122 find the new walking time.
nead to sclva
varisble simesilly Lat call it - t/60.
Let dencta 1/50 sar
Solution: (bowedgg00))
[Score:
(Score:
[Score: B]
[Boore: BI
Base Model's y 7
First, Ined to determine Ayrt's wilking spsed, s, and the time sine apends the collee shop, So, equation
First, need to determine Ayú a walkinng speed, and the time sho spends in the coffee shop, I.
So, equation 1: B/s U800 4. Equilion 2:
t150 Equatiin sitevil 1800 2 Thus, 2.5. and in (boxed]204)
2/ + 1650 2.4.
reculf
Figure I: SpecReason leverages a smaller reasoning model to speculate individual reasoning steps, deferring to the base model only for assessment (and optionally fallback), enabling faster yet accurate reasoning. For illustration, we show a math question as an example; our evaluation includes more general reasoning workloads.
without compromising accuracy-is rooted in two fundamental properties of LRMs: (1) LRMs tackle difficult tasks by generating long CoT's that decompose them into many simpler, sequential steps. For example, in mathematical problem solving, a few key reasoning steps require complex long-term planning and have a major influence on downstream reasoning, while most subsequent steps simply execute the plan through straightforward calculations or case analyses (Fig- : (2) The utility of an individual reasoning step hinges less on the exact wording of the thinking tokens but more on the semantic insight it provides. That is, as long as step contributes meaningfully to advancing the CoT, it remains effective-even if phrased imprecisely or differently (Fig. 2). Moreover, LRMs possess self-reflectton capabilities that enable them to revise or correct occasional missteps from earlier steps.
Our approach to tackling reasoning
Taken together, these properties make the decoding of thinking tokens-thh the dominant source of inference latency in LRMs- L.MMs-ineerently more approximation tolerant than typical LLM decoding. large fraction of intermediate reasoning steps can be effectively handled by lightweight reasoning models, which both align with the nature of these steps and can tolerate minor inaccuracies. As shown in Fig. 3 this opens the door to significantly faster inference without sacrificing output quality.
Building on these insights, we propose SpecReason, a system for accelerating LRM inference by selectively offloading easier intermediate steps to be speculated by smaller model without compromising final output accuracy. SpecReason employs a lightweight reasoning model to generate individual reasoning steps, while reserving the slower but more capable base model to efficiently verify these speculated steps ($4.I) and guide the reasoning process along the correct trajectory (Fig. D. Consistent with prior findings [Song et al. 2023), we observe that base models can be prompted to act as critic models assessing the utility of intermediate steps and accepting or rejecting them as needed (Fig. 7
Speculative reasoning vs. speculative decoding. While SpecReason is conceptually related to speculative decoding [Leviathan et al., 2023). which accelerates L.M inference by using smaller draft model to predict future tokens, there are key distinctions between the two. Most notably, speculative decoding an exact optimization: it relies on token-level equivalence between the small and base models, i.c., focusing on typical LLM serving where all generated tokens are part of the final model output being assessed. In contrast, SpecReason explicitly leverages the approximation tolerance inherent in reasoning: it targets thinking tokens- inntsrmeditteeeeee steps in the reasoning process wheee semantic alignment, rather than token-level equivalence, is sufficient. This relaxation enables substantial latency savings during LRM inference, as semantically similar intermediate steps (Fig. 2 are often adequate to preserve end-task accuracy (Fig. 3). In many cases, SpecReason even improves final accuracy over the base model by generating fewer unnecessary tokens (Fig. 4.. To further address the high inference cost of LRMs, SpecReason also exposes a user-configurable knob that allows trading off accuracy for latency by adjusting the tolerance level
2

---

"9/s +1==4" (t' t/60)
"9/s
(s) Mrac
"9/s
Token-Level Equivalence
Semantic-Level
SernantccLevel Similarity
Factual/LLoicaa
Equivalence
Incorrectness
Figure 2: The spectrum of approximations one example reasoning step (equation in Fig. D SpecReason can control the exactness of reasoning approximations by adjusting its acceptance threshold to navigate through the accuracy-latency tradeoff space ($3.3).
for speculative approximations. Finally and most importantly, because speculative reasoning and speculative decoding operate at different levels, we show that they are complementary techniques (44.2) and when combined in hierarchical speculation framework, achieve even greater reductions in inference latency.
We evaluate SpecReason across wide range of reasoning workloads spanning tasks of varying complexity [aim] 2025 Hendrycks et al., 2021 Rein et al. [024]. Overall, SpecReason reduces end-to-end inference latency 1.4 3.0x compared to vanilla LRM inference while improving accuracy by 0.4 9.0%. Moreover, SpecReason can be combined with speculative decoding to provide additional 8.8 58.0% improvement speculative decoding alone.
2 Background
Inference-time scaling, LRMs introduce a structured problem-solving approach that breaks down complex problems into multiple simpler reasoning steps, commonly referred to long chain of thought (CoT) [Wei et al., 2022]. This enables the model to generate intermediate reasoning steps before progressing further, reflect, and backtrack
correct errors needed. LRMs that output long CoTs have been popular approach to scale inference-
compute [Guo et al.) [025 Jaech
al. 2024 ope 2025). and there also exist other schemes like Tree Thoughts Yao al.
2023
tree search (Lightman et al. 2023 Qi et al. 2024 Guan et al.
2025
and repeated sampling for scaling inference- compute Brown et al. 2024].
Speculative decoding. Speculation has long been a classic concept in the literature of computer architecture [Burton] 1985 Due the memory-bound nature of LLM decoding, recent work has also leveraged the technique of speculation to accelerate the decoding phase [Stern et al.) 2018 Leviathan al. 2023 Yan et al. [0224] of LLM inference. The speculative decoding process alternates between speculation and verification steps to ensure correctness while achieving speed-ups. The speculation
phase usually consists of either standalone draft model [Leviathan et 2023 Miao al. 2024), a trainable module on top of the base model (Cai et al.
2024 Li et al. 2025 a tree-based token 2024 an n-gram lookup table [Fu al.. cache OOliaro al. 2024 Luo al., 2024 Zhao al.
2024a], or a retrieval based data store He et 20023 to make efficient but less accurate speculations.
The verification process, on the other hand, a base model chunked- -prefil over the speculation
results, which usually consists of either single sequence of tokens in [Leviathan et al. 2023 or
tree-like structures to further boost the accuracy of speculation [Miao et al. 2024 Cai et al. 2024 Li
et al. 2025 Chen et al., 2024). The verification process then accepts the longest matched sequences on the token level from the speculation results and repeats the process. As result, the speculation length is usually conservative to maintain optimal trade-off between the speculation overhead and accuracy.
Existing approaches for reducing latency. Sky-TI-Flash Team 2025 reduces unnecessary thinking tokens by fine-tuning models to curb overthinking, thereby reducing the length of reasoning chains and, consequently, latency. Dynasor-CoT [Fu et al. 2024b) 2025) takes a different approach by probing intermediate model confidence and terminating the reasoning process early when the model exhibits sufficient confidence in its current output. LessIsMore (Yang et al. 2025 selects tokens for sparse attention by leveraging cross-head attention patterns reasoning and reduces the decoding latency.

---

3 Motivation
In this work, we show that reasoning workloads executed by LRMs exhibit unique opportunities for latency reduction due to their inherent tolerance to approximation- setting them apart from traditional generation tasks in L.MSs. We illustrate these properties using a representative example from the AIME dataset, selected for its clarity and ease of exposition.
Intermediate steps are easier than end-to-end reasoning. A key observation in LRM behavior is that reasoning difficulty is not uniform across the steps in a long chain-of-thoughe (CoT). As shown in Fig. while the overall task might be too challenging for small model to solve end-to-end, only a few critical steps steps-such as problem analysis, decomposition through formulations or case analyses, and high-level planning are critical the overall reasoning progress. In contrast, many other steps are significantly easier.
This behavior is intentional by design: LRMs are often trained with reinforcement learning to generate CoT's that decompose complex problems into sequences of simpler, more tractable reasoning steps. These intermediate steps often include routine reasoning such as arithmetic calculations, case enumeration, or basic logical deducctioss-opraatoo that are much easier to decode than synthesizing a full solution directly. This heterogeneity step difficulty and importance creates an opportunity for lightweight models to handle a substantial portion of the reasoning process both efficiently and accurately.
Reasoning progress depends on insights, not exact tokens. Another key takeaway from our work is that the utility of reasoning step lies in the semantic contribution it makes the overall reasoning process, rather than the precise tokens it uses. Unlike tasks like translation in traditional LLM inference, where fidelity to exact combinations of tokens matters more, reasoning CoT's within LRM's thinking tokens care more about the information that advances the reasoning chain. As illustrated in Fig. 2 a spectrum of valid phrasings often exists for given step: semantically equivalent or similar expressions can convey the same insight and lead to the same downstream reasoning trajectory. This semantic flexibility is key enabler for approximation-tolerte
inference.
Occasional mistakes can be corrected via self-reflection. LRMs exhibit strong self-reflection capabilities, enabling them to recover from earlier reasoning errors. Even when an earlier step contains a factual or logical mistake, the model often revises its trajectory in subsequent steps, marked by tokens like "Wait" or "Hmm".. Moreover, unlike LLM inference where all output tokens contribute to the final answer, in LRM inference, only the tokens generated after the thinking tokens determine the final outcome. Therefore, LRM inference reasoning phase, as the model can often identify and correct these mistakes during self-reflection. This inherent fault tolerance further underscores the viability and effectiveness of approximation-basee acceleration.
tolerate occasional mistakes during the
In summary, compared to traditional LLM inference, LRM inference is inherently more tolerant of approximations that do not require token-level equivalence as long as the overall reasoning trajectory preserved. This property is not limited to single, linear CoT; rather, it extends naturally to more general inference-time compute scaling paradigms such as tree-based search strategies and other structured reasoning approaches.
4 Method
4.1
Speculative Reasoning
Due to its reliance on autoregressive decoding, LRM inference incurs significantly higher latency than typical LLMs-often to the point of being prohibitively slow for interactive applications and degrading user experience [Fu et al. 2025]. Existing approaches for latency reduction include using a distilled version of the base model [uoo et al. (225). limiting the number of thinking tokens via a predefined token budget, or disabling the reasoning process altogether by omitting the thinking tokens (<think> and </think>) during generation (qwe) 2025). However, these approaches impose harshly trade-off between accuracy for latency: they either limit the model's capacity to reason or apply a lower-quality model uniformly across all reasoning steps. contrast, SpecReason takes a more fine-graine finengrained and adaptive approach. Instead of explicitly restricting output length, it selectively

---

offloads only the easier reasoning steps to lightweight model, preserving overall reasoning quality while substantially reducing inference latency.
The approximation-toleannl nature of LRM reasoning enables a new form of speculative execution: tentatively carrying out reasoning steps using a lightweight model, assessing their utility with stronger base model, and selectively accepting them. SpecReason leverages this flexibility to reduce decoding latency while preserving output quality. To achieve this goal, SpecReason offloads easier or less critical reasoning steps-defined as semantically self-contained units such as complete sentences or logical steps to smaller, faster speculator model. Each step is decoded in two stages: (1) the lightweight speculator proposes the next reasoning step based on the current context, and (2) the base model evaluates the proposed step for semantic utility. If the step is accepted, SpecReason proceeds to the next step; otherwise, SpecReason falls back to the base model to regenerate the step. While our implementation uses a simple matic-threshold mechanism for verification, the framework supports richer, customizable decision strategies. We outline key design principles below.
Navigating the Pareto frontier of the latency -acuuracy tradeoff. SpecReason expands the Pareto frontier of the latency-accuracy tradeoff by exposing fine-grained control knobs to navigate through this space. The key knob SpecReason employs is the acceptance threshold: after each speculated reasoning step, the base model is prompted to generate a single-token utility score (e.g., an integer from 0 to 9) indicating the quality of the step. the utility score above static acceptance threshold (e.g., score 2 7). the speculated reasoning step is accepted: otherwise, it is discarded and regenerated by the base model.
Adjusting this threshold allows users to control the strictness of speculation (Fig.5) a higher threshold requires speculated steps to be closer token-level equivalence the equivalence spectrum (Fig. [ improving accuracy but reducing the acceptance rate and thereby increasing latency. Conversely, a lower threshold increases speculation efficiency at the cost of potential accuracy degradation.
An additional knob involves forcing the first n reasoning steps to be decoded by the base model. Since LRMs often use the initial steps to analyze the problem and formulate high-level plan, assigning these initial steps to the base model can steer the overall reasoning trajectory toward higher quality. We show in Fig. 6 that this knob also allows SpecReason to manage the though with less impact than the acceptance threshold knob.
tradeoff,
While our current implementation uses simple, discrete threshold-based scoring scheme-offering only a coarse- coarse-grained configuration space-it establishes a lower bound on verification quality. Future work can explore more sophisticated strategies, such as logprob-based confidence estimates or dynamic thresholds, to enable finer-grained tradeoffs without incurring additional runtime cost, and may further improve overall performance.
Efficient verification. Because each step requires verification by the base model, it's crucial to keep verification overhead low to avoid compounding latency. Instead of autoregressively decoding or reranking multiple candidate steps, SpecReason evaluates each speculated step in a single prefill-only pass of the base model. The verification prompt is templated to reuse most of the CoT prefix, so each verification requires prefilling only ~0 new tokens. Since short-prefill forward passes are memory-bound, the overhead is comparable to decoding just 1-2 tokens, making verification highly efficient in practice.
Implementation details. Since the small model is lightweight, we colocate both the small and base models on the same GPU. The memory reserved for Key-Value caches [Kwon et al. 2023] is statically partitioned between the two models. They do not share any internal model states-only the token IDs of the generated reasoning steps are managed and shared by SpecReason. If speculative step is rejected, the corresponding KV cache entries are discarded.
Inference is performed sequentially: the small and base models take turns, avoiding kernel-level interference. In future work, we plan to explore pipelining to overlap the small model's decoding with the base model's inference. While this may introduce mild resource contention, it could further reduce end-to-end latency.
4.2 Hierarchical Speculation across Semantic Similarity and Token Equivalence
At high level, SpecReason's 's speculative reasoning resembles the philosophy behind traditional speculative decoding, but differs in two important ways. First, speculative decoding guarantees
5

---

token-level equivalence between draft and verified outputs, making it form of exact acceleration. In contrast, SpecReason targets semantic-level similarity, accepting steps that carry the same insight even if phrased differently, and exposes knobs to control the exactness of reasoning approximations. Second, speculative decoding typically applied to output generation tasks (e.g., text continuation or translation), where the fidelity of each token matters. SpecReason, on the other hand, is designed specifically for internal thinking tokens in reasoning tasks, where intermediate steps are approximate and interchangeable long they preserve the logical progression of thought.
Further, because SpecReason and speculative decoding operate at different levels (semantic-leve) similarity vs. token-level equivalence), these two approaches are complementary and can be combined into a unified, hierarchical system SpecReuson+Decodee first applies step-level speculative reasoning to draft and verify reasoning steps. If a step is rejected and regenerated by the base model, standard token-level speculative decoding can be applied during the base model regeneration to further accelerate decoding.
5 Evaluation
The overview of our evaluation results includes:
Reducing end-to-end latency. Because many intermediate steps are easier than end-to-end reasoning, many (up to 80%) of the speculated steps are accepted. SpecReason achieves 1.4 3.0x speedup over vanilla LRM inference. Additionally, when combined with speculative decoding. SpecReason further reduces latency by 8.8 58.0% over speculative decoding alone, highlighting the complementary nature of these optimizations.
accuracy. Beyond latency reduction, SpecReason also improves accuracy over the base model by 0.4 - 9.0% under the same token budget. We empirically find that small, lightweight models typically have shorter output sequence lengths meaning, they need fewer thinking tokens before deriving an answer. Thus, by accepting many small model's speculated reasoning steps, SpecReason reduces the token consumption compared to the base model's ss vanilla inference. When the token budget is low common setup to curb inference cost and latency - SpecReason helps improve accuracy as the base model would need more tokens to get to an answer (Fig. 4
Improving eoken-budget-aweer
5.1 Setup
Models. In our main results, we use two base models: QwQ-32B [qwq] 2025] and Skywork-OR!
We also two different small models for speculation: DeepSeek- RI-1.5B [Guo et al. 2025 and Zyphra's ZRI-1.5B (zyp) 2025) both of which are based on Qwen-2.5 Yang et al. 2024 and embed the capability of reasoming with long CoTs and evaluate all four different model combinations. We evaluate an additional base model with a different size and architecture, 71B70B Guo et al. 2025). distilled version of DeepSeek-Ri R1 onto Llama3.3- 70B GGratafiiri et al., 2024.. in I
Preview-32B [sky] 2025].
Datasets. We evaluate SpecReason three diverse reasoning benchmarks: AIME (aim, 2025 for high-school competition-level mathematical problems, MATH500 (Hendryeks et al. 202 for high-school competition-level mathematical problems sampled from AMC TO, AMC 12, and AIME, and GPQA Diamond [Rein et al., 2024) for graduate-level questions in general domains like biology, physics, and chemistry. The accuracy metric we evaluate on is pass @ 1. Similar to prior work [Guo et al.) 2025), we set k=16 when calculating pass@1 I i... we generate 16 responses with temperature=0.6 for every query and calculate the average accuracy and set the token budget to be 8192 tokens to ensure an
comparison between baselines.
Baselines. We run vanilla inference using the small and base models as the latency and accuracy baseline, respectively. Aside from SpecReason, we also run speculative decoding ("SpecDecode") with the smaller model the draft model, speculating five tokens at time. To demonstrate SpecReason's 's compatibility with speculative decoding, we also run a "SpecReason-Deccode baseline that employs the hierarchical speculation described in $4.2

---

model
Small model
SpecReason
SpecDecode
SpecReesoo++ Decode
MATH
AIME
GPQA
pass@1 (9)
30
60
40
Better
20
Better
Better
10
50
100
150
0
200
400
100
200 Latency (s)
300
Latency
Latency (s)
(a) QwQ-32B R1-1.5B
MATH
AIME
GPOA
70
40
pass@1 (%)
50-
65
30
Battor
Better
40
Better
60
20
30
50
100
150
0
200
400
100
200
300
Latency (s)
Latency (s)
Latency (s)
(b) QwQ-32B + Zyphra-
MATH
AIME
GPQA
pass@1 (%)
40
60
Better
20
Better
Better
20
50
10
*
50
100
200
100
200
300
400
Latency (s)
Latency (s)
Latency (s)
(c) Skywork-Previcw-33 Preview-322 R1-1.5B
MATH
AIME
GPQA
50
poss@1 (%)
65
30
Better
Bettior
Beiter
60
20
30
50
100
0
200
400
100
200
300
400
Latency (s)
Latency (s)
Latency (s)
(d) Seyywork-Previe-33133 Zyphra-1.5B
Figure 3: Comparison of the accuracy and latency of different schemes on different model combi- nations. SpecReason significantly reduces latency while improving accuracy over vanilla inference. When combined with speculative decoding, SpecReason outperforms speculative decoding in both latency and accuracy on all datasets and model combinations.
Hardware. We run our evaluations on two NVIDIA A6000-48GB GPUs. We use vLLM 0.8.2 as the underlying inference engine and enable prefix caching (Kwon et al. 2023 Zheng et al. 2023 Pan et al. 2025). Both models are served with a tensor parallelism degree of two.
5.2 Main Results
We compare SpecReason against baseline methods in Fig. 3 Across the four model combina- tions, SpecReason achieves 1.5x-2.5x, 1.6x-3.0x, 1.4x-2.5x, 1.7 reduction in latency, respectively, compared to vanilla inference with the base model.
Accuracy improvement. Alongside these efficiency gains, SpecReason also yields modest accuracy improvements of 1.3%-3.6%. 4.0%-9.0% 0.4%-1.7%, and 1.4%-5.0% compared to the base model. The key reason behind this accuracy improvement is the reduction in token consumption required for reasoning. In Fig. we focus on the model combination with the highest overall accuracy improvement, QwQ-32B Zyphra-1.5B, and compare the average number of thinking tokens needed to derive an answer between the base model, the small model, and SpecReason. As seen in Fig. [4a the small model is generally less verbose than the base model, and because SpecReason adopts many speculated steps from small model, its token consumption is also reduced by 1.2x-2.0x. We also focus on the AIME dataset and vary the token budget to study its effect on the difference in accuracy between SpecReason and the base model in Fig. 4bb] The effect of token reduction on accuracy the most significant for tighter output token budgets (16.2% at 4096 tokens) but shrinks as the base
7

---

Base Model
Small Model
SpecReason
Avg Thinking Token Count
Base Model
Small Model
SpecReason
Pass@ (%)
6000
20
4000
10
2000
0
2000
4000
6000
8000
MATH
AIME
GPQA
Output Token Budget
(a) Output length comparison. SpecReason reduces (b) [AIME] Accuracy gap under different token bud- gets.
token consumption needed to answer queries by adopt- ing speculated steps from small models that are less verbose.
Figure 4: [QwQ-32B + 1rphra-1.5B) Intuition behind SpecReason's 's accuracy improvement. See Fig. 9 in A for the full set of results.
Base model
Small model
SpecReason
SpecDecode
Spec OpecReason+Decode GPOA
MATH
AIME
pass@1 (%)
40
60
40
Better
Better
Beter
50
20
50
100
150
200
400
100
200 Latency (s)
300
Latency (s)
Latency (s)
Figure 5: [QwQ-32B R1-1.5B] SpecReason allows trading off latency for accuracy via adjusting the acceptance threshold (from left right, the thresholds are: 3, 5, 7, and out of 9).
model allowed to generate more thinking tokens (4.7% 8192 tokens). We also attribute these accuracy gains to SpecReason's explicit judgment and scoring mechanism at each reasoning step, which augments the model's internal self-reflection with more structured assessment.
When compared with speculative decoding. SpecReason lies on the Pareto frontier the accuracy- latency tradeoff. More importantly, combining SpecReason with speculative decoding (SpecRea- son+Decode) results in further latency reductions of 19.4%.44.2%% 30.8%-58..0. 58. 3.8%-52.2% and 25. %-5..8% over speculative decoding alone. The most significant performance gains for SpecReason when the base model is QwQ-32B occur on the MATH dataset, where both models achieve relatively high accuracies and the capability gap between the small and base models the narrowest. This makes intermediate steps easier for the small model to speculate correctly. increasing the acceptance rate of speculated steps and thereby lowering end-to-end latency. In comparison, Skywork- Seework-rreveww-.. Preview-32B is slightly inferior at instruction following, so SpecReason has to adopt higher threshold avoid an accuracy loss, reducing SpecReason's 's latency wins.
Finally, when comparing SpecReason+ Decode with SpecReason, SpecReason+ DeseReeason+Decode reduces latency by 1.7x-1.9x, 1.7x-1.8x, 1.6x-2.2x, and 1.6x-2.1xx demonstrating the difference in ease of speculation across varying tasks. On these three datasets, the ratio steps carried out by small models in SpecReason 38. 10.000% 36.5%-71.3% 39.3%-70.2%% and 41.4%-66. 66.6%, respectively.
5.3 Controlling the Accuracy-Latency Tradeoff
In Fig. []] we illustrate how SpecReason enables flexible control over the accuracy-latency tradeoff, using a representative, randomly selected subdataset from the full datasets $5.2 on QwQ-32B + R1-1.5B for ease of evaluation. During the base model's evaluation of each reasoning step, we vary the acceptance threshold for the utility score between 3, 5, 7, and 9, and report the resulting accuracy and latency.

---

Base model
SpecDecode
6.0
Small model
SpecReason++ Decode
Avg Utilty SpecReason
SpecReason
AIME
5.5
40
-.0
pass@1 (%)
4.3
20
Better
4.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
*
Binned PRM
100
200
300
400
500
Latency (s)
Figure 7: The utility scores in SpecReason closely re- flect the quality score judgements from process reward model. on the x-axis denotes PRM scores in the range [x, x 0.1).
Figure 6: Effect of the alternative knob: forcing the first n steps for base model decoding.
On the MATH subdataset, increasing the acceptance threshold from to results in fewer specu- lative steps from the small model being accepted. This leads latency increase from 35.7s to 69.2s, while accuracy improves from 59.4% to 63.7%, due to tighter control over the approximation level of intermediate reasoning steps. Notably, the gap between SpecReason+Decooee and SpecRea- son widens from 8.1s to 28.8s, since more reasoning steps delegated to the base model, and SpecReason+ reduces only the base model's decoding time compared to SpecReason.
A similar trend is observed on the AIME and GPQA subdatasets: as the acceptance threshold increases from 3 to 7, latency grows from 109.4s to 261.9s and from 72.7s to 223.0s, and accuracy improves from 22.3% to 39.3% and from 53.1% to 50.7%. However, the accuracy degrades less gracefully as the threshold relaxed compared the MATH subdataset. This because the small model exhibits larger performance gap relative to the base model on AIME and GPQA, making aggressive acceptance of its speculative steps more costly terms of accuracy.
In Fig- 6, we also study the effect of the alternative knob, forcing the first n reasoning steps to be decoded by the base model, the accuracy-latency tradeoff. As we change from to 10, 20, 30, and 40, SpecReason's accuracy increases from 33.2% to 37.3% while the latency increases from 270.4s to 292.6s, showcasing alternative approach improve accuracy with slight increase in latency.
5.4 Base Model's Judgement Capability
The base model's ability assess the quality of intermediate reasoning steps is crucial cornerstone of SpecReason's 's performance. In this experiment, we compare the scores generated by a process reward model (PRM) which assigns a reward score to each step within the solution to a math problem with those given by the QwQ-32B base model on the AIME dataset. Specifically, we use Math-Shepherd [Wang et al. 2023). PRM trained via reinforcement learning from the Mistral-7B base model math problems, to score each speculated step produced by the R1-1 small model.
In Fig. 7 we bin the reward scores (a float from 0 to 1) into ten bins. Within each bin, calculate the mean utility score given by the base model in SpecReason. This analysis demonstrates a strong correlation between the base model's and the PRM's assessments, particularly for lower-quality reasoning steps, where both models assign low scores. The results suggest that the base model can effectively approximate the PRM's judgments, making viable option for evaluating reasoning step quality in SpecReason.
Conclusion
In this work, we introduce SpecReason, novel approach that accelerates LRM inference by leverag- ing speculative reasoning. By offloading simpler intermediate reasoning steps a smaller, lightweight model and reserving the base model for assessment, SpecReason significantly reduces inference latency while maintaining or even improving accuracy. Our results demonstrate that SpecReason achieves a 1.4 - 3.0x speedup over vanilla LRM inference, with accuracy improvements ranging from 0.4 9.0%. Additionally, when combined with speculative decoding, SpecReason further reduces lateney by 8.8 58.0%. highlighting the complementary nature of these optimizations. We

---

believe this work opens up new angles for efficient LRM inference acceleration, making it especially valuable for scenarios that demand both high accuracy and low latency.
Acknowledgments and Disclosure of Funding
We thank Princeton's Systems for Artificial Intelligence Lab (SAIL) and Princeton Language and Intelligence (PLI) for providing the hardware resources for running experiments. Rui would like to thank Chongyi Zheng for sharing his experience on submitting to NeurIPS. This work was supported by NSF CNS grants 2147909, 2151630, 2140552, 2153449, and 2152313.
References
Aime 2024 dataset card.
2025.
Openai o3-mini system card. http:////e////a/aa../..//o////aa
2025.
Qwen3: Think deeper, act faster. http:///gggallnntt......//////// 2025. 2025.
Qwq-32b: Embracing the power of reinforcement httpaanngg...gggg///e///aaaaaaad 2025.
Skywork-orl (open reasoner 1).
2025.
Introducing 1r,5,, a small but powerful reasoning model for math and code ). post/introducing -eaeeiiinnnn------- 1-5b- a- mumalllbuutpooorriii matth-mah-hee code waweeerr-roooonnnmoooooo hagpperruuuuttttcccooeeeeaaaaaar
com/
model, 2025.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint 20X2:2440.217877 2024.
F Warren Burton. Speculative computation, parallelism, and functional programming. IEEE Transactions on Computers, 100(12):1190-11 1985.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng. Jason D Lee, Deming Chen, and Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads, arXiv preprint 24Xiv:2401.00774 2024.
Zhuoming Chen, Avner May, Ruslan Svirschevski, Yu-Hsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable and robust speculative decoding, Advances in Neural Information Processing Systems, 37: 129531-129563, 2024.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of inference using lookahead decoding. arXiv preprint arXiv:2402.02057. 2024a.
Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. Efficiently serving Ilm reasoning programs with certaindex. arXiv preprint urXiv:2412.2098) 2024b.
Yichao Fu, Junda Chen, Yonghao Zhuang, Zheyu Fu, Ion Stoica, and Hao Zhang, Reasoning without self-doubt: More efficient thain-of-thought through certainty prohing. ICLR 2025 Workshop on Foundation Models in Wild, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilaaa 3 herd of models. arXiv preprint 2XXv:2247..2/783 2024.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-mant math: Small 1mm can master math reasoning with self-evolved deep thinking. arXiv preprint 25l.:2500.045111 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in lms via reinforcement lcarning. arXiv preprint arXiv:2501. 12248, 2025.
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: Retrieval-based speculative decoding. arXiv preprint arXiv:2311.08252 2023.
10

---

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora. Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprini arXiv:2103.03874 2021.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai of system card, arXiv preprint 2tl::2222..162000 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1-626, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias, Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286. PMIR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference accelcration of large language models via training-time test. arXiv preprint arXiv:2503.01844 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Xianzhen Luo, Yixuan Wang. Qingfu Zhu, Zhiming Zhang. Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. Turning trash into treasure: Accelerating inference of large language models with token recycling, 2024. httpss//s///xxii org/ abs/2408. 08696
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang. Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree- based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 932-949) 949, 2024.
Gabriele Oliaro, Zhihao Jia, Daniel Campos, and Aurick Qiao. Suffixdecoding: model-free approach speeding up large language model inference, 2024. URL URL https::/arxix...ot org/ wwwpg/////////o/oeo/oooo/eeeee/e 2411.14199999
Rui Pan, Zhuang Wang. Zhen Jia, Can Karakus, Luca Zancato, Dao, Yida Wang. and Ravi Netravali. Marconi: Prefix caching for the era of hybrid 1lms. In Eighth Conference on Machine Learning and Systems, 2025.
Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller Ilss stronger problem-solvers. arXiv preprint urXiv:2408. 201699500000195 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. Prmbench: fine- rre-grained and challenging benchmark for process-level reward models. arXiv preprint arXiv:2501.03124 2025.
Mitchell Stern, Noam Shazeer, and Jakob Uszkorcit. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018.
NovaSky Team. Think less, achieve more: Cut reasoning costs by 50% without sacrificing accuracy. htw.::/www...aiiib..oooooooeeeee
2025. Accessed: 2025-01-23.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math- shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:2312.08935, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Fd Chi, Quoe L Denny 7hou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Minghao Yan, Saurabh Agarwal, and Shivaram Venkataraman. Decoding speculative decoding. arXiv preprint arXiv:2402.01522 2024.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li. Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 151155, 2024.
11

---

Lijie Yang, Zhihao Zhang, Arti Jain, Shijie Cao, Baihong Yuan, Yiwei Chen, Zhihao Jia, and Ravi Netravali. Less is Training-Iree free sparse attention with global locality for efficient reasoning. arXiv preprint arXiv:2508.7700, 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao. and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances neural information processing systems, 36: 11119-888122. 2023.
Yao Zhao, Zhitian Xie, Chen Liang, Chenyi Zhuang, and Jinjie Gu. Lookahead: An inference acceleration framework for large language model with lossless generation accuracy. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, page 6344 6355. Association for Computing Machinery, 2024. ISBN 9798400704901. doi: 01.1145/3637728.66677788
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody_ Yu, Shiyi Cao, Christos Kozyrakis, lon Stoica, Joseph E Gonzalez, al. Efficiently programming large language models using sglang. 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The design section discussed the simple nature of the default implementation the scoring mechanism. The evaluation section discussed the potential pitfalls, e.g., model assessment's quality degradation when the base model is weak at instruction following.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully discloses all the information needed to reproduce the main experimental results, and we open source the code in an anonymized GitHub repository to facilitate reproduction efforts.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We open-soureed our codebase and instructions on reproducing the results in an anonymized GitHub repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all details on the experiments. We also release the code for full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper uses k=16 when evaluating pass @ 1, which is consistent with the evaluation setup in prior work for statistical significance. We also run on multiple model combinations and multiple datasets for generalizability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient detail on the compute resources needed to reproduce the experiments. We also release our code for full reproducibility.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neuripss.
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed, as it aims to improve the efficiency of reasoning model inference and does not pose ethical concerns.
Guidelines:
16

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All models and datasets used in this work are open-source and have been properly referenced and cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release our codebase in an anonymized GitHub repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19

---

A
Appendix
A.1
Base Models of Varying Sizes and Architectures
Base model
Small model AIME
SpecReason
40
pass@1 (%)
30
Better
50
100
150 Latency (s)
200
250
300
Figure SpecReason's results on the model combination (R1-70B, R1-1.5B).
To demonstrate the generality of SpecReason, replace the QwQ-32B base model with DeepSeek's R1-70B and evaluate on the same representative subdatasets as in $5.3 Given the size the R1-70B model, we deploy it across four A100-80GB GPUs using tensor parallelism degree of 4.
On the AIME subdataset, SpecReason achieves 1. 1.5x latency reduction compared to vanilla R1- 70B inference. This speedup is smaller than the gains observed with the QwQ-32B model in our main results (1.9x) due to two key factors. First, the RI-70B model benefits from both stronger hardware and greater parallelism (4-way TP on A 100s), resulting in 1.5x lower time-per-token (TPT) compared to QwQ-32B (2-way TP on A6000s). In contrast, the smaller model R1-1.5B sees only modest 1.1 x TPT improvement stronger hardware, which narrows the performance gap between base and small models and thus diminishes latency savings. Second, QwQ-32B empirically stronger model outperforming R1-70B across many reasoning benchmarks qwq [2025] and this performance gap impacts their respective abilitics to assess intermediate steps. To maintain accuracy, adopt stricter acceptance threshold when using R1-70B the base model, which reduces the fraction of steps offloaded to the small model (23.2% compared to 40.8% in the main results).
A.2 Intuition behind Accuracy Improvement
Bese Model
Small Model
SpecReason
6000
Aug Thinking
4000
2000
MATH
AIME
GPQA
MATH
AIME
GPQA 2y5hra-1.58
MATH
AIME
GPQA
MATH
AIME
GPQA
(a) QwQ-328
R1-1.58
(b) QwQ-32B
(c) Skywork-328 + R1-1.58
(d) Skywork-328
2yphra-1.53 1155
Figure 9: Intuition behind SpecReason's accuracy improvement on all datasets and model combina- tions.
we evaluate the average thinking token count of SpecReason and two vanilla inference baselines on a wide range of datasets and model combinations. We observe that the small model is generally less verbose than the base model, and because SpecReason adopts many speculated steps from the small model, its token consumption is reduced by 1.0 1.3x, 1.2 2.0x, 1.0 1.8x, and 1.1 2.3x on the four model combinations, respectively.
In Fig. 9
20


Reviews:
[{"review_id": "A2ZkMWoUZJ", "reviewer": "Reviewer_D9dY", "summary": "The paper presents a technique called SpecReason for accelerating the inference of large reasoning models by using a comparatively smaller model as a draft model. The larger model verifies the small model’s proposed next steps by giving a discrete score from 0 to 9. The step is accepted or rejected based on a user-defined threshold.  In the evaluation, SpecReason shows 10%-50% improvement in speed compared to Speculative Decoding in MATH, AIME, and GPQA benchmarks.", "questions": "How can this technique be generalized to domains beyond mathematics?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. A simple and practical idea for a crucial problem of inference latency of large reasoning models.\n2. The evaluation results are consistent with 10-50% improvement over standard Math benchmarks. \n\n**Weaknesses:**\n1. The evaluation is only focused on Math tasks and does not show the generality of the technique on other domains. \n2. The motivation section alludes to several insights about LRMs. However, these insights are stated without any empirical evidence. Can authors include empirical analysis on the confidence scoring patterns that assess the difficulty of each step in the reasoning trajectory?\n3. The base model is prompted to generate the score draft reasoning step. The authors should include empirical analysis on using confidence based on the logits of the speculated step.\n4. Although the main experiments are performed with k=16 samples, the results do not report the error bars as stated in the Checklist. \n\n**Minor:**\n \nLine 50: shown in Fig. 3, this opens the door to significantly faster inference without sacrificing output quality. - Can you avoid pointing to Fig. 3 here\n\nSection 4.1: A use of small pseudocode may improve the readability of Section 4.1\n\nSection 4.2: it repeats what is discussed in the background section discussion on speculative decoding.\n\nWhile comparing with vanilla LRM, the evaluation uses the term “speedup” and comparison with speculative decoding term “latency” is used. I would recommend using single terminology for consistency.", "was_revised": true, "final_justification": "Authors have addressed my concerns and promised to include experiments that would improve the paper.", "rebuttal": "Thank you for your detailed feedback and suggestions on the presentation! We will incorporate them into the final version of the paper. We also appreciate that the reviewer finds our paper interesting and practical. Below, we respond to individual questions.\n\n> Question 1/Weakness 1: The evaluation is focused on math tasks and does not show the generality of the technique on other domains\n\nSpecReason is designed to generalize to a broad range of reasoning workloads like question answering (e.g., GPQA) and code generation (e.g., HumanEval). **We note that our evaluations included GPQA, a general-purpose question answering dataset with questions in biology, physics, and chemistry, and we found that SpecReason's wins generalize to domains beyond math workloads.** The three key insights presented in §3 are broadly applicable to any task where reasoning models – those that generate intermediate thinking tokens before summarizing and answering – are primarily used. To further strengthen our evaluation, if time permits, we plan to include representative datasets from additional workload categories (e.g., code generation) in the final version of our paper.\n\n> Weakness 2: Empirical analysis on the insights and the confidence scoring patterns\n\nOur first key insight, \"*intermediate steps are easier than end-to-end reasoning*,\" is the cornerstone assumption of chain-of-thought reasoning [1]. Our third key insight, \"*LRMs can correct occasional mistakes via self-reflection*,\" is a widely observed phenomenon in LRMs (e.g., DeepSeek-R1’s Aha moment [2] suggests that reflection behavior emerges during training). We will add the corresponding references to related work.\n\nOur second key insight, \"*reasoning progress depends on insights, not exact tokens*,\" is our novel contribution. It is supported by empirical results showing that relaxing token-level equivalence to semantic equivalence or similarity still preserves quality, as demonstrated in our main results (Fig. 3). We demonstrate SpecReason’s confidence score patterns in Fig. 1, but we will be happy to include additional example reasoning trajectories comparing vanilla inference with SpecReason (alongside the confidence score patterns) in the final version of our paper.\n\nIf the reviewer has specific suggestions for experiments to empirically analyze the confidence scoring patterns (e.g., how the average confidence scores across requests correlate with final answer accuracy, or how the acceptance rate correlates with the acceptance threshold), we will be happy to ask for clarifications during the reviewer–author discussion stage and address them in the final version of the paper.\n\n> Weakness 3: Empirical analysis on using confidence based on the logits of the speculated step rather than prompting the base model\n\nThis is an interesting idea – we speculate that SpecReason’s current approach will outperform this scheme, since our current scoring mechanism relies on a larger, more capable model for assessment rather than on the information (i.e., logits/confidence) within the smaller draft model. If time permits, we will include a microbenchmark of this in the final version of the paper.\n\n> Weakness 4: Error bars\n\nWe thank the reviewer for pointing this out, and we will add error bars to our main results.\n\n### References\n\n[1] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837.\n\n[2] Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948."}, {"review_id": "jgdLMkuxVC", "reviewer": "Reviewer_cSLs", "summary": "Inference-time compute has emerged as a key factor for scaling AI capabilities, particularly in Large Reasoning Models (LRMs), which perform complex tasks by generating long CoTs. However, these models face significant latency issues. To address this latency without compromising accuracy, the paper introduces SpecReason, an approach that offloads simpler intermediate reasoning steps to smaller, faster speculative models. SpecReason leverages two critical insights: first, reasoning difficulty is heterogeneous, meaning many intermediate steps are relatively simple and can be effectively handled by lightweight models; second, reasoning effectiveness depends primarily on semantic insight rather than exact token fidelity, allowing for approximate yet meaningful step generation. SpecReason employs a lightweight model to speculate reasoning steps, which are subsequently evaluated by a more capable base model. This method significantly reduces latency by selectively accepting semantically adequate speculative steps.", "questions": "Please check points 3, 4, and 6 in the weakness for missing related works and comparisons, support for motivation.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to follow.\n2. Achieving fast and accurate inference is important for both LLMs and LRMs.\n3. The extension of the \"speculative\" concept from decoding to reasoning is interesting and innovative.\n\nWeaknesses:\n\n1. It is unclear whether this method applies exclusively to LRMs or could also be effectively employed in LLMs. The paper presents it as potentially general, yet does not explicitly explore its broader applicability.\n2. In line 36, the authors claim \"without compromising accuracy.\" However, can this method guarantee 100% identical outputs compared to baseline methods? If not, this statement is imprecise. Unlike speculative decoding, which ensures lossless acceleration and perfect accuracy, this speculative reasoning approach may inherently introduce slight inaccuracies. The authors should clarify this distinction and avoid overstating the claim.\n3. Several relevant recent works [1, 2] addressing the \"overthinking\" problem [3] are omitted. Including these studies would strengthen the related work section and clearly situate the paper within existing research.\n4. In Section 3 Motivation, key assumptions such as \"intermediate steps are easier\" and \"reasoning progress depends on insights\" are presented without sufficient experimental evidence or external references. Additional experiments or citations are required to substantiate these claims effectively.\n5. The use of references formatted as \"[...]\" is somewhat unconventional.\n6. Consistent with point 3, the paper lacks direct comparisons with state-of-the-art methods [1, 2] addressing similar challenges. Including explicit comparative analyses would significantly bolster the validity and impact of the presented approach.\n\n[1] O1-pruner: Length-harmonizing fine-tuning for o1-like reasoning pruning.\n\n[2] L1: Controlling how long a reasoning model thinks with reinforcement learning.\n\n[3] Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models", "was_revised": true, "final_justification": "The authors addressed parts of concerns; there is no comparison with the current SOTA method, so I keep my score", "rebuttal": "Thank you for your detailed feedback! We appreciate that the reviewer finds our insights interesting and innovative. Below, we respond to individual questions.\n\n> Weaknesses 3 & 6: Related work on addressing the overthinking problem\n\nWe thank the reviewers for pointing us to the related work. A key distinction between SpecReason and recent work [1,2] is that SpecReason is a training-free approach that can be applied to any off-the-shelf model, whereas related work [1,2] requires fine-tuning or training LRMs, which is a time- and resource-intensive process. If time permits, we will add a baseline that runs vanilla inference using a reasoning model fine-tuned from one of the base models in our evaluation to reduce overthinking.\n\n> Weakness 4: Experimental evidence of our insights\n\nOur first key insight, \"*intermediate steps are easier than end-to-end reasoning*\", is the cornerstone assumption of chain-of-thought reasoning [4]. Our third key insight, “*LRMs can correct occasional mistakes via self-reflection*”, is a widely observed phenomenon in LRMs (e.g., DeepSeek-R1’s Aha moment [5] suggests that reflection behavior emerges during training). We will add the corresponding references to related work.\n\nOur second key insight, \"*reasoning progress depends on insights, not exact tokens*\", is our novel contribution. It is supported by empirical results showing that relaxing token-level equivalence to semantic equivalence or similarity still preserves quality, as demonstrated in our main results (Fig. 3). We will add additional explanations illustrating that token-level equivalence is indeed not achieved by the smaller reasoning model, yet this relaxation is sufficient from an accuracy perspective. For example, we will include example reasoning trajectories comparing vanilla inference with SpecReason in the final version of our paper.\n\n> Weakness 1: Extension to LLMs\n\nSpecReason targets LRM inference, which is the driving force behind inference-time scaling, making our focus relevant to a wide range of practical workloads. While our method is tailored to LRMs, it naturally extends to LLMs – particularly in scenarios that elicit chain-of-thought (CoT) decoding [4]. This is because our core insights – that intermediate steps are easier, and that reasoning progress depends on insights – are rooted in the CoT paradigm, where complex tasks are broken down into simpler, sequential subproblems. We will include a discussion on the applicability of SpecReason’s techniques in the final paper.\n\n> Weakness 2: Accuracy compromise\n\nWe agree with the reviewer that, unlike speculative decoding, SpecReason does not guarantee token-by-token equivalence. This is by design: our key insight is that relaxing the constraint for exact token matches enables the use of alternative, potentially imprecise reasoning tokens – relative to the base model – while still achieving the same final output accuracy. This flexibility opens up new opportunities for latency reduction. We thank the reviewer for pointing this out and will revise the paper to clarify this distinction and explicitly note that SpecReason's accuracy-preserving property is demonstrated empirically rather than guaranteed theoretically.\n\n### References\n\n[1] - [3]: See the reviewer's comment\n\n[4] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., ... & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837.\n\n[5] Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948."}, {"review_id": "S60x8Zm1a1", "reviewer": "Reviewer_k8Qf", "summary": "The paper introduces SpecReason, a novel framework designed to accelerate inference in Large Reasoning Models (LRMs) by leveraging the inherent approximation tolerance of intermediate reasoning steps. Unlike conventional speculative decoding approaches that require token-level equivalence between draft and base models, SpecReason operates at the level of semantic similarity, enabling a lightweight model to speculatively generate intermediate reasoning tokens that are subsequently assessed by the base model for their utility in advancing the reasoning trajectory. The method exposes configurable mechanisms to balance accuracy and latency, such as adjustable acceptance thresholds and selective delegation of initial reasoning steps to the base model. Extensive experiments on diverse reasoning benchmarks demonstrate that SpecReason achieves substantial reductions in inference latency—ranging from 1.4× to 3.0× compared to vanilla LRM inference—while also yielding accuracy improvements of up to 9%. Furthermore, the authors show that SpecReason is complementary to speculative decoding, and their combination yields additional latency gains, highlighting the practical relevance of the proposed approach in scaling efficient reasoning-centric applications.", "questions": "see weakness", "limitations": "see weakness", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1. SpecReason deals with the problem of high inference latency in LRMs by proposing an acceleration framework that does not compromise accuracy. The method is effective and readily transferable, demonstrating promising potential for practical deployment.\n\n\nWeakness\n\n1. SpecReason relies heavily on the base model’s ability to accurately assess the utility of speculative steps. While preliminary evidence is provided (e.g., correlation with process reward models), this dependency could limit robustness if the base model’s evaluation quality degrades in more diverse or noisy settings.", "was_revised": true, "final_justification": "The authors addressed my concerns in rebuttal, so I maintain my original score.", "rebuttal": "Thank you for your detailed feedback! We appreciate that the reviewer finds our insights interesting.\n\n> Weakness: SpecReason’s reliance on the base model’s assessment capability\n\nWe acknowledge that a robust scoring/assessment mechanism is indeed a requirement; however, we find that the latest reasoning models are indeed capable of making accurate assessments (see our results in Fig. 3). We are also happy to demonstrate that not all models, e.g., smaller reasoning models, can consistently make accurate assessments. Furthermore, SpecReason doesn’t hinge on the base LRM’s assessment capabilities; it generalizes to alternative approaches for evaluating the quality of speculated reasoning steps, such as a lightweight verification model (commonly used in other workloads like RL rollout scoring) or leveraging the confidence of tokens generated by the smaller draft model."}, {"review_id": "QyOksgP3NQ", "reviewer": "Reviewer_aS9u", "summary": "- This paper proposes SpecReason, a system to accelerate inference for Large Reasoning Models (LRMs).\n- It uses a lightweight model to speculatively perform intermediate reasoning steps, invoking the base LRM only for verification and correction.\n- Unlike prior speculative decoding, SpecReason exploits the semantic flexibility of reasoning steps without requiring token-level equivalence.\n- On multiple reasoning benchmarks, it achieves 1.4×-3.0× speedup and 0.4%-9.0% accuracy improvement over standard LRM inference. Combined with speculative decoding, it delivers an additional 8.8%-58.0% latency reduction.", "questions": "- In Line 200, the paper mentions generating “∼70 new tokens” per reasoning step. Is this number empirically determined from observations, or is it a fixed hyperparameter?\n\n- Have you considered or observed cases where the LRM produces repeated reasoning steps or revises earlier steps? Could SpecReason’s scoring or acceptance mechanism be extended to reuse previously generated steps to further improve efficiency?\n\n- Since it can be difficult to clearly delineate reasoning steps in many LRM outputs, how is step segmentation controlled in your implementation? Is it based on a fixed token length, semantic boundary detection, or other heuristics?", "limitations": "See \"Strengths And Weaknesses\".", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- **Strengths**:\n\n  (1) The paper makes an **interesting observation** in Section 3, identifying three key properties of LRM reasoning: Intermediate steps are easier than end-to-end reasoning; Reasoning primarily depends on semantic insight rather than exact token outputs; Occasional errors can be corrected through reflection. **This analysis is insightful and provides a strong motivation for the proposed method.**\n\n  (2) The paper is clearly written and well-structured, making the methodology and results easy to follow.\n\n  (3) As shown in Figure 3, the experimental results effectively demonstrate the advantages of SpecReason. **Moreover, the combination of SpecReason with speculative decoding achieves even better latency reductions, verifying the method’s compatibility with existing techniques.**\n\n----\n\n- **Weaknesses**:\n\n  (1) As described in Lines 196–202, the method uses DeepSeek-R1 1.5B as the lightweight model to generate 70 new tokens (one reasoning step), and a larger model (QWQ-32B or DeepSeek-R1-70B) to score the step. Steps with a score ≥7 are accepted, while others are regenerated by the large model. This **heavily relies on the small model’s acceptance rate** — if it is low, the overall efficiency gain diminishes. **Moreover, using the large model for step evaluation introduces potential additional latency**. It would be valuable to provide guidelines on suitable model size ranges. **Section A.1 addresses part of this concern, but it would be helpful to see corresponding SpecReason+Decoding results in that section as well.**\n\n\n  (2) Since many related speculative reasoning and decoding works have emerged recently, **it would be helpful if the authors could include a brief discussion in the Appendix comparing their approach with concurrent methods.**\n\n  (3) No other major weaknesses identified.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback! We appreciate that the reviewer finds our insights interesting and valuable. Below, we respond to individual questions.\n\n> Question 1: Prefilling 70 tokens in each verification step\n\nThe ~70 tokens come from the concise yet effective base model prompt that scores the speculated reasoning step, and this number is derived from our design choice. We will include the base model prompt in the final version of the paper. We note that the current implementation of SpecReason allows for extensive customizations in future work, such as alternative concise, yet effective, scoring prompts.\n\n> Question 2: Have you considered or observed cases where the LRM produces repeated reasoning steps or revises earlier steps? Could SpecReason’s scoring or acceptance mechanism be extended to reuse previously generated steps to further improve efficiency?\n\nEmpirically, LRM revising earlier steps is a widely observed phenomenon (e.g., DeepSeek-R1’s Aha moment [1] suggests that reflection behavior emerges during training; Sky-T1 [2] similarly observed redundant and excessive reasoning steps). Reusing scores from prior similar steps is indeed a viable way to reduce the prefilling time of the base model’s scoring prompts. However, we did not adopt this strategy because we observed that the end-to-end latency of SpecReason (and LRM inference in general) is dominated by the aggregated time-per-token (TPT) latency rather than the total prefilling time across verification rounds. Due to the sheer number of tokens required per request, score reuse would result in only marginal reductions in end-to-end latency.\n\n> Question 3: How are boundaries between steps defined\n\nIn our current implementation, we simply use each occurrence of a double newline character ('\\n\\n') as the divider between reasoning steps. SpecReason also supports other segmentation markers, such as ending a step after each period ('. ').\n\n> Weakness 1: Reliance on the small model’s acceptance rate\n\nWe acknowledge that although we did not observe low acceptance rates across the diverse workloads and model combinations we evaluated, the small model should be of a reasonable size to ensure it can generate a substantial portion of reasoning steps that preserve semantic meaning. In a real system deployment scenario, we foresee a runtime monitor that continuously tracks the acceptance ratio on a per-request basis. For requests that exhibit a consistently low acceptance rate, SpecReason would fall back to vanilla base model generation to avoid the latency overhead of drafting and verification.\n\n> Weakness 1: Latency overhead of base model verification\n\nSpecReason introduces only marginal latency overhead thanks to its efficient design. As noted in our response to Question 2, each verification is a single prefill, which efficiently parallelizes token processing and results in latency equivalent to the TPT of only 1–2 tokens. In contrast, the end-to-end latency of SpecReason (and LRM inference in general) is dominated by the decoding time of the thinking tokens.\n\n> Weakness 2: Related work on speculative reasoning\n\nWe acknowledge that many papers have been published on arXiv since we submitted our paper in April, and we will include a brief discussion of these works in the final version of our paper. As a preview, the primary differences between SpecReason and related work are:\n1. **Accuracy preservation**: SpecReason maintains accuracy (compared to the base model) for state-of-the-art models and datasets, whereas other approaches often exhibit accuracy degradation for speedups (e.g., Speculative Thinking [3]). \n2. **Training-free design**: SpecReason is a training-free approach that works with off-the-shelf models, whereas related work (e.g., SplitReason [4]) requires resource- and time-intensive retraining or fine-tuning.\n\n### References\n\n[1] Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.\n\n[2] Griggs, T., Cao, S., Li, D., Liu, S., Patil, S. G., Zaharia, M., Gonzalez, J., & Stoica, I. (2025). Think Less, Achieve More: Cut Reasoning Costs by 50% Without Sacrificing Accuracy. Blog post.\n\n[3] Yang, W., Yue, X., Chaudhary, V., & Han, X. (2025). Speculative thinking: Enhancing small-model reasoning with large model guidance at inference time. arXiv preprint arXiv:2504.12329.\n\n[4] Akhauri, Y., Fei, A., Chang, C. C., AbouElhamayed, A. F., Li, Y., & Abdelfattah, M. S. (2025). Splitreason: Learning to offload reasoning. arXiv preprint arXiv:2504.16379."}]

Author rebuttal:

Meta-review: This paper addresses the challenge of accelerating inference for Large Reasoning Models (LRMs) by leveraging inference-time computation. The proposed approach employs a lightweight model to speculate individual reasoning steps, which are then assessed by a stronger base model that assigns a score between 1 and 9. If the score exceeds a threshold (e.g., ≥7), the lightweight model’s output is accepted; otherwise, the base model regenerates the step. While the mechanism is conceptually simple, the paper builds on key insights into reasoning: (1) reasoning difficulty is heterogeneous—many intermediate steps are relatively simple and can be effectively handled by smaller models; and (2) reasoning quality depends more on semantic soundness than on exact token fidelity, enabling approximate but meaningful step generation. These are important contributions to the emerging and highly active area of LRMs.
The main weakness of the proposed approach is the reliance on prompting a stronger model for step scoring, which introduces additional latency, computational cost, and possible robustness issues. Moreover, the paper lacks sufficiently thorough comparisons with concurrent methods, which weakens the evaluation.
The authors have responded to the identified weaknesses and promised to update the draft with additional discussions, more comprehensive comparisons, and further clarifications. If these commitments are fulfilled in the final version, the paper will be substantially strengthened and could make a meaningful contribution to advancing efficient inference for LRMs.
Decision: Accept (poster)