Title: AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders
Venue: NeurIPS 2025
Accepted: yes
Keywords: Speculative Decoding

Large Language Models

Knowledge Distillation

Token Selection
Abstract: Speculative Decoding (SD) accelerates large language model inference by employing a small draft model to generate predictions, which are then verified by a larger target model. The effectiveness of SD hinges on the alignment between these models, which is typically enhanced by Knowledge Distillation (KD). However, conventional KD methods aim to minimize the KL divergence between the draft and target models across all tokens, a goal that is misaligned with the true objective of SD, which is to maximize token acceptance rate. Therefore, draft models often struggle to fully assimilate the target model's knowledge due to capacity constraints, leading to suboptimal performance. To address this challenge, we propose AdaSPEC, a novel method that incorporates selective token filtering into the KD process. AdaSPEC utilizes a reference model to identify and filter out difficult-to-fit tokens, enabling the distillation of a draft model that better aligns with the target model on simpler tokens. This approach improves the overall token acceptance rate without compromising generation quality. We evaluate AdaSPEC across diverse tasks, including arithmetic reasoning, instruction-following, coding, and summarization, using model configurations of 31M/1.4B and 350M/2.7B parameters. Our results demonstrate that AdaSPEC consistently outperforms the state-of-the-art DistillSpec method, achieving higher acceptance rates across all tasks (up to 15\%). The code is publicly available at \url{https://github.com/yuezhouhu/adaspec}.


Peer reviews:
comments: SUMMARY:
This paper introduces AdaSPEC, a novel method to improve the efficiency of Speculative Decoding (SD).
AdaSPEC addresses the problem of the small draft model struggling to fully learn the target knowledge by incorporating selective token filtering into the Knowledge Distillation (KD) process, using a reference model to identify and exclude diffucult tokens.
By focusing distillation on simpler tokens, AdaSPEC enables the draft model to better align with the target model, leading to higher token acceptance rates and improved inference speed without compromising generation quality.

STRENGTHS AND WEAKNESSES:
Strengths: 
1. AdaSPEC is simple to implement and is a plug-and-play method. 
2. AdaSPEC achieves much higher acceptance rate than DistillSPEC baseline. 

Weaknesses:
1. Only the vanilla SD is tested, so it is not clear whether AdaSPEC could generalize to more advanced SD methods like Eagle.
2. The only baseline is DistillSPEC. The evaluation is limited in scope. The discussion of related work is limited.
3. There is a huge gap between 3-epoch and optimal-epoch performance. A more advanced hyper-parameter selection strategy could have been proposed to optimize the performance.

QUESTIONS:
1. Can AdaSPEC generalize beyond vanilla SD to more advanced methods like Eagle or Medusa?

2. Could more relevant baselines be included, like Rho-1 mentioned in the paper?

3. Considering the significant gap between 3-epoch and optimal-epoch performance, why wasn’t a more effective hyperparameter selection strategy proposed to improve results?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-09

comments: SUMMARY:
This paper introduces AdaSPEC, an adaptive loss for model distillation for later use in Speculative Decoding settings. Based on the observation that a focus on simpler tokens should enhance the acceptance ratio during SD, the work proposes a two-step distillation, the first step being a normal KL-based distillation into a smaller reference model and during a second stage, another KL-based distillation of the final draft model on selected "simple" tokens derived by the KL-difference between reference and draft models. This approach can consistently improve acceptance scores over the DistillSPEC baseline.

STRENGTHS AND WEAKNESSES:
Strengths:
- Improved acceptance scores for Speculative Decoding

Weaknesses:
- Twice the distillation compute
- Limited to Speculative Decoding settings

QUESTIONS:
- Is there a possibility to train both reference and draft model at the same time? This would bring down the large cost of evaluating the big target model twice.

- Can you explain the only mild improvements of acceptance scores (often ~1%)? 

- Could you add in your discussion of SD alternatives like MoE, CoT and recurrent-depth approaches in l.25ff? What is your opinion on those compared to SD?

LIMITATIONS:
Within the narrow focus of this paper (Speculative Decoding setting), it addresses the main limitations.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the implications of the observation that Speculative Decoding should prioritize the accuracy of predicted tokens over optimizing the overall KL divergence between the target and the reference model. Consequently, it proposes AdaSPEC, which prioritizes distillation of the draft model on the subset of tokens that it is actually able to learn to predict properly, given its small scale. The heuristic used by AdaSPEC to choose this subset of tokens is to select those where the most learning progress has been made so far in terms of KL divergence with respect to the target model. Results show a modest but consistent gain in acceptance rate compared to the ablation of the selective-token filtering (corresponding to DistillSpec \\[1\\]).\n\n\\[1\\] Zhou, Y., Lyu, K., Rawat, A. S., Menon, A. K., Rostamizadeh, A., Kumar, S., Kagy, J., & Agarwal, R. (2023). DistillSpec: Improving Speculative Decoding via Knowledge Distillation. *ArXiv*. [https://arxiv.org/abs/2310.08461](https://arxiv.org/abs/2310.08461)\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* **Clarity.** I find the paper quite well-written and clear. The authors effectively motivate the problem and clearly explain their approach, and the paper’s structure is logical.  \n* **Problem significance**. The problem of increasing the computational efficiency of LLMs is very relevant and timely, and the paper’s results, although modest, could have a huge impact on the large scales of both usage and cost of LLM inference.  \n* **Originality**. Although token selection by difficulty \\[2\\], as discussed by the authors, or the hardness of examples in general \\[3\\] is not novel, the reason why this paper considers difficulty comes from a different perspective and has different implications. The observation that leads the author to adaptive token selection by difficulty, in my opinion, is where the originality of this paper lies, and it opens an avenue for future work to examine different designs to achieve a higher acceptance rate, beyond simply considering token difficulty.  \n* **Quality/Evaluation**. Evaluation is overall quite diverse, both in terms of architectures and datasets. Together with the analysis and ablation study experiments, the empirical results appear robust and sufficient \\[See questions below for further details\\].\n\n## Weaknesses\n\n* **Modest empirical results.** Although consistent, the effect of the method is modest, which reduces the impact of the work.\n* **Computation cost.** Although the implementation of token filtering is simple and should not incur a significant computational or memory cost, given that it involves the additional inference of the reference model, I would appreciate a comparison of the impact, given an equal number of tokens incorporated in the GD step.\n* **Reproducibility details.** Some minor implementation choices could be stated more clearly, see the questions below.\n\n---\n\nThe main limitation that stops me from increasing the score is the modesty of the empirical results, which I am unsure if this can be addressed during the rebuttal. However, I would greatly encourage the authors to consider and incorporate the other comments.\n\n\\[2\\] Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., & Chen, W. (2024). Rho-1: Not All Tokens Are What You Need. *ArXiv*. [https://arxiv.org/abs/2404.07965](https://arxiv.org/abs/2404.07965)   \n\\[3\\] Seedat, N., Crabbé, J., & Bica, I. (2022). Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular data. *ArXiv*. [https://arxiv.org/abs/2210.13043](https://arxiv.org/abs/2210.13043)\n\nQUESTIONS:\n* In L138, you state that the selected tokens are the ones that are easiest for the draft model to align with the target model. However, this motivation is slightly unclear to me. In particular, it seems that tokens maximizing (9) might satisfy a combination of “learnability”, related to your “ease”, but also “not yet learnt”, since tokens that are already predicted by the reference model well will be easy to predict, but will not satisfy (9). Would you agree?  \n* In L182, could you mention in the appendix and reference how you specifically chose the number of epochs?  \n* In L188-189, for reproducibility, it would be important to report how the hyperparameters and most importantly, the filter fraction were chosen. In particular, could you justify in L286 why you sometimes use k=0.8 instead of 0.4 everywhere? Is there any way to understand when we would want a higher k and when a lower one?  \n* In L198, could you please provide the respective plots for the other configurations in the appendix, given that these should be readily available already? That would aid in the completeness of the results.\n\n### Other suggestions\n\n* In the equation after L140, you mention selecting the top k tokens, but in the appendix pseudo-code, you appear to choose them based on quantiles instead. I would encourage addressing this inconsistency for reproducibility.  \n* I would encourage the authors in the Abstract and Introduction to focus more on the observation that simple KL optimization does not effectively maximize the acceptance rate, which is arguably the goal of SD. Therefore, in SD, we aim for high precision or maximum likelihood, not matching the distribution. As you rightly mention in the limitations, your heuristic is just a first approach to use this observation. Still, you could capitalize on this observation as a key contribution, which might enhance the impact of this work.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AdaSPEC, which demonstrates superior performance across diverse tasks (arithmetic reasoning, instruction-following, coding, summarization) and model configurations, consistently achieving higher acceptance rates than the state-of-the-art DistillSpec method Please refer to the strenths below.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\nConsistent and Robust Experimental Performance: AdaSPEC demonstrates superior performance across diverse tasks (arithmetic reasoning, instruction-following, coding, summarization) and model configurations (31M/1.4B, 350M/2.7B), consistently achieving higher acceptance rates than the state-of-the-art DistillSpec method. This broad validation across different domains and model scales highlights the generalizability of the approach .\n\nInnovative Selective Distillation Strategy: The introduction of selective token filtering via a reference model addresses a key limitation of existing knowledge distillation methods for draft models—their inability to fully align with target models due to capacity constraints. By focusing on \"easy\" tokens that the draft model can effectively learn, AdaSPEC improves token acceptance rates without sacrificing generation quality, representing a novel and intuitive solution .\n\nRigorous Ablation Studies: The paper conducts comprehensive ablation analyses on critical components, including token selection mechanisms, training methods, distillation objectives, and token selection ratios. These studies validate the effectiveness of each component (e.g., top 40% tokens outperform bottom 40%, forward KL is superior to TVD/RKL), enhancing the methodological credibility.\n\n## Weaknesses:\n\nInsufficient Support for the Reference Model: While the reference model is claimed to play a \"crucial role\" in filtering hard tokens, the paper lacks strong theoretical justification for why its perplexity-based filtering reliably identifies tokens beyond the draft model’s capacity. Additionally, there is a lack of direct experimental evidence (e.g., correlation between the reference model’s \"hard token\" labels and actual draft model errors) to validate its filtering accuracy .\n\nLimited Comparisons to Recent Methods: The method is only benchmarked against DistillSpec (2023), without comparisons to more recent advancements in speculative decoding or knowledge distillation . This narrow comparison weakens claims of state-of-the-art performance and reduces the work’s timeliness .\n\nUnderdeveloped Generalization Analysis: While AdaSPEC shows promise in tested configurations, its generalization to larger model gaps (beyond 40x) or more complex, low-resource tasks is not thoroughly explored. The limitations section only briefly mentions potential improvements to filtering strategies, without addressing how the method might scale or adapt to diverse real-world scenarios .\n\nQUESTIONS:\nPlease refer to the above weakness.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission introduces AdaSPEC, a selective knowledge distillation approach for optimizing efficient LLM inference via speculative decoding. AdaSPEC, initially relies on DistillSpec framework to distill a reference draft model from the original (verification) model. Subsequently, \"easy\" tokens are identified based on their loss, and selected to train the draft model of the resulting SD pipeline. The intuition behind the proposed approach is that by excluding the difficult tokens from the distillation process for fitting the small-capacity draft model, its training process will become less ambiguous and converge to a more robust and effective draft model (i.e. lead to higher acceptance rate and thus speed-up for the overall SD pipeline). The provided experimental results validate this assumption, showcasing consistent gains across different model pairs and datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed approach is very interesting and timely and exploits an interesting insight.\n- The manuscript is well written and easy to follow.\n- The effectiveness of the proposed methodology is demonstrated through extensive experiments across varying tasks and model pairs.\n- The conducted ablations offer potentially valuable insights for applying KD in related SD settings.\n\nComments:\n- It is unclear why the proposed filtering approach (eq 9) evaluates and compares the loss of the reference vs the draft model (Mq), instead of the correctness/distance of the reference (distilled) model to the original (Mp). Since Mq is the model being trained in step2, Mq's perception of easy samples is expected to change overtime, however the proposed approach does not appear to adapt to such changes. In contrast, evaluating the correctness of Mref (w.r.t Mp) may potentially form a more robust and stable proxy of the predictive difficulty of each token for the draft model. \n- It is unclear how the effectiveness of the proposed approach is affected by the learning capacity of the draft model (and whether the percentage of selected tokens should be scaled accordingly). Although the authors have conducted experiments between 2 different pairs of models, with varying relative parameter counts, more robust insights about the scalability of the proposed method will largely benefit the community and enhance the applicability of the proposed approach.\n\nQUESTIONS:\nPlease considering replying on the points raised in the comments section above.\n\nLIMITATIONS:\nSome limitations of the proposed approach are discussed at the conclusions section, along with proposed future work directions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}