Title: SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Inference-Time Compute

Large Reasoning Models

Efficient Inference

Speculative Execution

Systems for Machine Learning
Abstract: Recent advances in inference-time compute have significantly improved performance on complex tasks by generating long chains of thought (CoTs) using Large Reasoning Models (LRMs). However, this improved accuracy comes at the cost of high inference latency due to the length of generated reasoning sequences and the autoregressive nature of decoding. Our key insight in tackling these overheads is that LRM inference, and the reasoning that it embeds, is highly tolerant of approximations: complex tasks are typically broken down into simpler steps, each of which brings utility based on the semantic insight it provides for downstream steps rather than the exact tokens it generates. Accordingly, we introduce SpecReason, a system that automatically accelerates LRM inference by using a lightweight model to (speculatively) carry out simpler intermediate reasoning steps and reserving the costly base model only to efficiently assess (and potentially correct) the speculated outputs. Importantly, SpecReason's focus on exploiting the semantic flexibility of thinking tokens in preserving final-answer accuracy is complementary to prior speculation techniques, most notably speculative decoding, which demands token-level equivalence at each step. Across a variety of cross-domain reasoning benchmarks, SpecReason achieves 1.4-3.0$\times$ speedup over vanilla LRM inference while improving accuracy by 0.4-9.0%. Compared to speculative decoding without SpecReason, their combination yields an additional 8.8-58.0% latency reduction. We open-source SpecReason at \url{https://anonymous.4open.science/r/specreason/}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper proposes SpecReason, a system to accelerate inference for Large Reasoning Models (LRMs).\n- It uses a lightweight model to speculatively perform intermediate reasoning steps, invoking the base LRM only for verification and correction.\n- Unlike prior speculative decoding, SpecReason exploits the semantic flexibility of reasoning steps without requiring token-level equivalence.\n- On multiple reasoning benchmarks, it achieves 1.4×-3.0× speedup and 0.4%-9.0% accuracy improvement over standard LRM inference. Combined with speculative decoding, it delivers an additional 8.8%-58.0% latency reduction.\n\nSTRENGTHS AND WEAKNESSES:\n- **Strengths**:\n\n  (1) The paper makes an **interesting observation** in Section 3, identifying three key properties of LRM reasoning: Intermediate steps are easier than end-to-end reasoning; Reasoning primarily depends on semantic insight rather than exact token outputs; Occasional errors can be corrected through reflection. **This analysis is insightful and provides a strong motivation for the proposed method.**\n\n  (2) The paper is clearly written and well-structured, making the methodology and results easy to follow.\n\n  (3) As shown in Figure 3, the experimental results effectively demonstrate the advantages of SpecReason. **Moreover, the combination of SpecReason with speculative decoding achieves even better latency reductions, verifying the method’s compatibility with existing techniques.**\n\n----\n\n- **Weaknesses**:\n\n  (1) As described in Lines 196–202, the method uses DeepSeek-R1 1.5B as the lightweight model to generate 70 new tokens (one reasoning step), and a larger model (QWQ-32B or DeepSeek-R1-70B) to score the step. Steps with a score ≥7 are accepted, while others are regenerated by the large model. This **heavily relies on the small model’s acceptance rate** — if it is low, the overall efficiency gain diminishes. **Moreover, using the large model for step evaluation introduces potential additional latency**. It would be valuable to provide guidelines on suitable model size ranges. **Section A.1 addresses part of this concern, but it would be helpful to see corresponding SpecReason+Decoding results in that section as well.**\n\n\n  (2) Since many related speculative reasoning and decoding works have emerged recently, **it would be helpful if the authors could include a brief discussion in the Appendix comparing their approach with concurrent methods.**\n\n  (3) No other major weaknesses identified.\n\nQUESTIONS:\n- In Line 200, the paper mentions generating “∼70 new tokens” per reasoning step. Is this number empirically determined from observations, or is it a fixed hyperparameter?\n\n- Have you considered or observed cases where the LRM produces repeated reasoning steps or revises earlier steps? Could SpecReason’s scoring or acceptance mechanism be extended to reuse previously generated steps to further improve efficiency?\n\n- Since it can be difficult to clearly delineate reasoning steps in many LRM outputs, how is step segmentation controlled in your implementation? Is it based on a fixed token length, semantic boundary detection, or other heuristics?\n\nLIMITATIONS:\nSee \"Strengths And Weaknesses\".", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SpecReason, a novel framework designed to accelerate inference in Large Reasoning Models (LRMs) by leveraging the inherent approximation tolerance of intermediate reasoning steps. Unlike conventional speculative decoding approaches that require token-level equivalence between draft and base models, SpecReason operates at the level of semantic similarity, enabling a lightweight model to speculatively generate intermediate reasoning tokens that are subsequently assessed by the base model for their utility in advancing the reasoning trajectory. The method exposes configurable mechanisms to balance accuracy and latency, such as adjustable acceptance thresholds and selective delegation of initial reasoning steps to the base model. Extensive experiments on diverse reasoning benchmarks demonstrate that SpecReason achieves substantial reductions in inference latency—ranging from 1.4× to 3.0× compared to vanilla LRM inference—while also yielding accuracy improvements of up to 9%. Furthermore, the authors show that SpecReason is complementary to speculative decoding, and their combination yields additional latency gains, highlighting the practical relevance of the proposed approach in scaling efficient reasoning-centric applications.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. SpecReason deals with the problem of high inference latency in LRMs by proposing an acceleration framework that does not compromise accuracy. The method is effective and readily transferable, demonstrating promising potential for practical deployment.\n\n\nWeakness\n\n1. SpecReason relies heavily on the base model’s ability to accurately assess the utility of speculative steps. While preliminary evidence is provided (e.g., correlation with process reward models), this dependency could limit robustness if the base model’s evaluation quality degrades in more diverse or noisy settings.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInference-time compute has emerged as a key factor for scaling AI capabilities, particularly in Large Reasoning Models (LRMs), which perform complex tasks by generating long CoTs. However, these models face significant latency issues. To address this latency without compromising accuracy, the paper introduces SpecReason, an approach that offloads simpler intermediate reasoning steps to smaller, faster speculative models. SpecReason leverages two critical insights: first, reasoning difficulty is heterogeneous, meaning many intermediate steps are relatively simple and can be effectively handled by lightweight models; second, reasoning effectiveness depends primarily on semantic insight rather than exact token fidelity, allowing for approximate yet meaningful step generation. SpecReason employs a lightweight model to speculate reasoning steps, which are subsequently evaluated by a more capable base model. This method significantly reduces latency by selectively accepting semantically adequate speculative steps.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well-written and easy to follow.\n2. Achieving fast and accurate inference is important for both LLMs and LRMs.\n3. The extension of the \"speculative\" concept from decoding to reasoning is interesting and innovative.\n\nWeaknesses:\n\n1. It is unclear whether this method applies exclusively to LRMs or could also be effectively employed in LLMs. The paper presents it as potentially general, yet does not explicitly explore its broader applicability.\n2. In line 36, the authors claim \"without compromising accuracy.\" However, can this method guarantee 100% identical outputs compared to baseline methods? If not, this statement is imprecise. Unlike speculative decoding, which ensures lossless acceleration and perfect accuracy, this speculative reasoning approach may inherently introduce slight inaccuracies. The authors should clarify this distinction and avoid overstating the claim.\n3. Several relevant recent works [1, 2] addressing the \"overthinking\" problem [3] are omitted. Including these studies would strengthen the related work section and clearly situate the paper within existing research.\n4. In Section 3 Motivation, key assumptions such as \"intermediate steps are easier\" and \"reasoning progress depends on insights\" are presented without sufficient experimental evidence or external references. Additional experiments or citations are required to substantiate these claims effectively.\n5. The use of references formatted as \"[...]\" is somewhat unconventional.\n6. Consistent with point 3, the paper lacks direct comparisons with state-of-the-art methods [1, 2] addressing similar challenges. Including explicit comparative analyses would significantly bolster the validity and impact of the presented approach.\n\n[1] O1-pruner: Length-harmonizing fine-tuning for o1-like reasoning pruning.\n\n[2] L1: Controlling how long a reasoning model thinks with reinforcement learning.\n\n[3] Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models\n\nQUESTIONS:\nPlease check points 3, 4, and 6 in the weakness for missing related works and comparisons, support for motivation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a technique called SpecReason for accelerating the inference of large reasoning models by using a comparatively smaller model as a draft model. The larger model verifies the small model’s proposed next steps by giving a discrete score from 0 to 9. The step is accepted or rejected based on a user-defined threshold.  In the evaluation, SpecReason shows 10%-50% improvement in speed compared to Speculative Decoding in MATH, AIME, and GPQA benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. A simple and practical idea for a crucial problem of inference latency of large reasoning models.\n2. The evaluation results are consistent with 10-50% improvement over standard Math benchmarks. \n\n**Weaknesses:**\n1. The evaluation is only focused on Math tasks and does not show the generality of the technique on other domains. \n2. The motivation section alludes to several insights about LRMs. However, these insights are stated without any empirical evidence. Can authors include empirical analysis on the confidence scoring patterns that assess the difficulty of each step in the reasoning trajectory?\n3. The base model is prompted to generate the score draft reasoning step. The authors should include empirical analysis on using confidence based on the logits of the speculated step.\n4. Although the main experiments are performed with k=16 samples, the results do not report the error bars as stated in the Checklist. \n\n**Minor:**\n \nLine 50: shown in Fig. 3, this opens the door to significantly faster inference without sacrificing output quality. - Can you avoid pointing to Fig. 3 here\n\nSection 4.1: A use of small pseudocode may improve the readability of Section 4.1\n\nSection 4.2: it repeats what is discussed in the background section discussion on speculative decoding.\n\nWhile comparing with vanilla LRM, the evaluation uses the term “speedup” and comparison with speculative decoding term “latency” is used. I would recommend using single terminology for consistency.\n\nQUESTIONS:\nHow can this technique be generalized to domains beyond mathematics?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}