Title: Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
Venue: NeurIPS 2025
Accepted: no
Keywords: Large Language Models

Speculative Decoding

Inference System

Serving System
Abstract: LLMs have low GPU efficiency and high latency due to autoregressive decoding. Speculative decoding (SD) mitigates this using a small draft model to speculatively generate multiple tokens, which are then verified in parallel by a target model. However, when speculation accuracy is low, the overhead from rejected tokens can offset the benefits, limiting SD’s effectiveness, especially at large batch sizes.

To address this, we propose Speculative Verification (SV), an efficient augmentation to SD that dynamically predicts speculation accuracy and adapts the verification length to maximize throughput. SV introduces a companion model -- a small auxiliary model similar in size to the draft model -- to estimate the alignment between draft and target model distributions. \xxx{By maximizing the information gain from quantifying this alignment,} SV refines verification decisions, reducing wasted computation on rejected tokens and improving decoding efficiency. Moreover, SV requires no modifications to the draft or target models and is compatible with existing SD variants.

We extensively evaluated SV on publicly available LLMs across three NLP tasks using nine combinations of draft, companion, and target models, including 13B--72B target models and three types of variations: base (no finetuning), instruction-tuned, and task fine-tuned. Across all experiments and batch sizes (4--80), SV consistently outperforms both SD and standard decoding with the target model. It improves SD performance by up to 2$\times$, with an average speedup of 1.4$\times$ in large-batch settings (batch sizes 32–80). These results demonstrate SV’s robustness, scalability, and practical utility for efficient LLM inference.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes **Speculative Verification (SV)**, an extension of speculative decoding (SD) that dynamically adjusts the length of the speculative draft to improve efficiency. The authors identify two key signals that correlate strongly with the number of accepted draft tokens: (S) the distributional similarity between the draft and companion models, and (A) the acceptance probability of draft tokens estimated by the companion model. The core idea is to introduce a companion model (of similar size to the draft model), compute these two metrics, and accurately estimate an optimal draft length that minimizes speculation overhead while maximizing goodput.\n\nBy estimating alignment between the draft and companion models’ output distributions, SV adaptively selects the verification length to maximize throughput and reduce wasted computation from rejected tokens. The authors evaluate SV extensively across several large language model families (Qwen, LLaMA, and Layerskip) and task domains (chat, code generation, and math), while varying batch sizes, model sizes, and draft lengths. The proposed approach consistently outperforms baseline SD, achieving up to 2× speedup and an average of 1.4× improvement in high-throughput scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe research problem is well-motivated, and this paper proposes a plausible solution. The paper is well written, easy to read, and accompanied by well-designed and well-illustrated diagrams and figures. The method improves upon speculative decoding by up to 40% under large batch sizes when combined with a well-chosen companion model. Speculative decoding has been widely deployed, which makes the paper impactful. The method is clear and original; at least I have not seen something similar before. The experiments and discussion are comprehensive, covering many aspects such as self-speculative decoding, and providing a thorough picture of the method.\n\n## Weakness\n\nThe improvement is based on two preconditions: (1) a well-chosen companion model — as shown in Figure 6 (a-4), when pairing Qwen-0.5B with Unsloth-0.5B, a small degradation is observed under batch size 16; (2) a large batch size.  The prefill throughput is consistently lower than the baseline, but the overall throughput is higher under large batch sizes, and the improvement grows as batch size increases, so this is acceptable.\n\nOverall, I do not see any major weaknesses.\n\nQUESTIONS:\nI don't have questions for the author.\n\nThere is a small indexing error in figure 6 where the rightmost subfigure has the index \"d\" while it should be \"c\" I think.\n\nLIMITATIONS:\nThe authors acknowledge a limitation that the prefill throughput of their method is lower than the baseline speculative decoding due to the additional overhead of the companion model. They also describe a partial mitigation strategy, where logit computations in the draft and companion models are skipped during prefill, yielding a 3–5% reduction in overhead.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ​​Speculative Verification (SV)​​, an enhancement to speculative decoding that improves the efficiency of LLMs. SV employs a small companion model to estimate the draft acceptance likelihood of the target LLM, dynamically adjusting the verification length to reduce wasted computation from incorrect drafts. Experiments show that SV accelerates speculative decoding, particularly in large-batch scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. The motivation behind this paper is well-justified.\n\n2. As a plug-in technique, SV demonstrably improves speculative decoding performance.\n\n3. SV achieves significant speedup with large batch sizes, offering distinct advantages over alternative speculative decoding approaches.\n\n**Weakness**\n\n1. Limited real-world applicability due to dependency constraints. SV is inherently limited by its reliance on an intermediate-sized companion model, which may be unavailable or impractical to obtain in many scenarios. While the paper suggests model distillation as a solution, this approach remains infeasible when training data is inaccessible or computational resources are scarce, leaving SV restricted in practical deployments.\n\n2. Overly idealized alignment assumption. The paper assumes perfect alignment between the output distributions of the draft/companion models and the target model—an unrealistic premise, especially since experiments only use homogeneous LLMs (e.g., Qwen2.5-Instruct variants). Key unanswered questions remain: Does the performance hold with heterogeneous models? \n\n3. The evaluation settings lack clarity. While Figure 4 shows consistent speedup for SV across batch sizes, the paper does not specify whether the computational overhead of the companion model is included in these results. Neither the Implementation nor Evaluation sections address this, making explicit documentation necessary.\n\n4. A critical baseline is missing: directly employing the companion model as the draft model in speculative decoding. Although this approach does not incorporate dynamic verification-length adjustment, it could potentially result in higher draft acceptance rates and reduced draft costs (since only one model is active during the drafting phase). Including this comparison is essential to properly assess the contribution of SV.\n\n5. Lack of comparison with state-of-the-art.​​ While SV is presented as a plug-in technique, its value would be more compelling if demonstrated against recent advances in speculative decoding (e.g., self-drafting methods [1-3]), rather than just traditional speculative decoding. The paper should clarify: (1) how SV complements or improves upon these SOTA methods, or (2) what unique capability SV provides that existing approaches cannot achieve. Without this analysis, the claimed benefits of SV remain unclear in the context of current research.\n\n6. The paper reports performance gains on conventional benchmarks like ShareGPT, HumanEval, and GSM8K but omits several important standard speculative decoding benchmarks such as MT-Bench and Spec-Bench. Including a broader range of standard benchmarks would strengthen the evaluation.\n\n7. Single-query performance remains unexplored.​​ For comprehensive benchmarking, the performance in single-concurrency scenarios (batch size=1) should be included—even if they prove suboptimal—as they represent a critical real-world use case.\n\n[1] EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees\n\n[2] Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads\n\n[3] Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding\n\nQUESTIONS:\nPlease refer to **Weakness**.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Speculative Verification (SV), an augmentation for speculative decoding (SD) in Large Language Model (LLM) inference. SV incorporates a companion model, comparable in size (slightly larger) to the draft model, to dynamically estimate the alignment between the draft and target model output distributions. By doing so, it aims to predict speculation accuracy and adjust the verification length to maximize throughput (goodput). The method uses information-theoretic measures to guide this adjustment and minimize unnecessary verification on misaligned tokens, thus boosting decoding efficiency. Extensive experiments on various LLMs and tasks (dialogue, code, math) suggest SV consistently improves over both standard SD and baseline target decoding, especially at large batch sizes.\n\nSTRENGTHS AND WEAKNESSES:\nSummary Of Strengths\n- Novel Architectural Insight: This paper reveals an insightful observation: although the acceptance probability of draft tokens fluctuates sharply over time steps, it is somewhat predictable by introducing a third-party distribution. This finding can be effectively utilized to reduce the verification overhead of standard SD.\n- Thorough Empirical Validation: The work presents extensive experiments with varied LLMs (Qwen, Llama, CodeLlama), across base, instruction-tuned, and task-tuned settings, multiple batch sizes, and tasks (conversation, code generation and math). The evaluation, though domain-limited to available public LLMs, covers a range of architectures, tasks, and batch scales. The experimental analysis deals with the robustness of hyperparameters, datasets, draft/companion combinations and the compatibility of different SD variants.\n- Engineering Contributions: The authors report runtime and system-level optimizations (e.g., data-parallel drafting, overlapping drafting/verification, usage of NVIDIA MPS), highlighting the paper’s focus on real-world deployment. In addition, the author also focuses on the issue of balance in their method during batch inference, which enhances the practicality of the approach.\n\nSummary Of Weaknesses\n- Theoretical Rigor: While the information-theoretic framing is thoughtfully presented, some readers may find that the assumptions underpinning the strong correlation between draft-companion and draft-target alignments (Section 4.2) are mostly empirically motivated. As a result, the essential connection between the distributions of the draft model and the companion model with the final acceptance probability is not clear. \n  The methodological incompleteness of the paper leaves many questions unanswered. For example, the companion model, which plays a crucial role in the paper, is not clearly characterized by the authors. Is its slightly larger size the key, or could a model of the same size as the draft model but with a different distribution serve as the companion model?\n  I believe that an in-depth study of the coupling mechanism among the draft model, companion model, and target model would help enhance the contribution of this paper.\n-  Weak Baselines: Most empirical comparisons are to vanilla SD and target-only decoding.  There is a lack of head-to-head comparison with other strong SD extensions or multi-model pipeline designs. Specifically, the authors introduce an additional companion model, which is similar to multi-stage SD or multi-draft SD. The authors themselves mention staged SD (Section 4.1) but do not provide experimental evidence. Direct performance comparisons with these alternative pipelines would be useful.\n- Implementation Issue: Although the tables and figures show an overall improvement in goodput, it is difficult to attribute this purely to dynamic verification length rather than engineering optimizations. Specifically, in Figure 5 (right), OV and DP show significant improvements with SV. These techniques could theoretically also help improve SD, but in the authors' experiments, the opposite was true, and they did not provide sufficient explanation.\n\nQUESTIONS:\nSee the weakness part for more details.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces Speculative Verification (SV), which uses a companion model to predict token acceptance in speculative decoding by measuring the alignment between draft and companion model distributions. Through an information-theoretic framework, SV dynamically adjusts verification length to reduce wasted computation on rejected tokens.

STRENGTHS AND WEAKNESSES:
The paper effectively demonstrates the inefficiency of SD when speculation accuracy is unpredictable.

However, dynamic speculation length has been extensively explored: 
DISCO trains classifiers to decide token-by-token continuation.
AdaEDL uses draft model entropy to determine acceptance rate bounds without any training.
SVIP combines entropy and temperature for difficulty-aware draft lengths.
SmartSpec/Goodput Scheduler uses moving averages to predict acceptance and optimize batch-level throughput.
DySpec dynamically adjusts token trees in multi-branch SD.
Tetris greedily selects likely-accepted tokens across sequences in batched SD.
MagicDec provides theoretical formulas for optimal draft lengths in long-context scenarios.

The paper fails to demonstrate systematic advantages over these existing dynamic length methods.

QUESTIONS:
How does SV compare directly with AdaEDL, SVIP, or SmartSpec in terms of both accuracy of acceptance prediction and overall system throughput?

LIMITATIONS:
Not sufficient discussion and comparison with existing literature.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-07