AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders
Authors: Yuezhou Hu
Jiaxin Guo
Xinyu Feng
Tuo Zhao
Venue: NeurIPS 2025 spotlight ()
Keywords: Speculative Decoding
Large Language Models
Knowledge Distillation
Token Selection
Primary area: deep_learning

Abstract: Speculative Decoding (SD) accelerates large language model inference by employing a small draft model to generate predictions, which are then verified by a larger target model. The effectiveness of SD hinges on the alignment between these models, which is typically enhanced by Knowledge Distillation (KD). However, conventional KD methods aim to minimize the KL divergence between the draft and target models across all tokens, a goal that is misaligned with the true objective of SD, which is to maximize token acceptance rate. Therefore, draft models often struggle to fully assimilate the target model's knowledge due to capacity constraints, leading to suboptimal performance. To address this challenge, we propose AdaSPEC, a novel method that incorporates selective token filtering into the KD process. AdaSPEC utilizes a reference model to identify and filter out difficult-to-fit tokens, enabling the distillation of a draft model that better aligns with the target model on simpler tokens. This approach improves the overall token acceptance rate without compromising generation quality. We evaluate AdaSPEC across diverse tasks, including arithmetic reasoning, instruction-following, coding, and summarization, using model configurations of 31M/1.4B and 350M/2.7B parameters. Our results demonstrate that AdaSPEC consistently outperforms the state-of-the-art DistillSpec method, achieving higher acceptance rates across all tasks (up to 15\%). The code is publicly available at \url{https://github.com/yuezhouhu/adaspec}.

AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders
Yuezhou Hul", Jiaxin Guo Xinyu Feng", Tuo Zhao
University of California, Berkeley Tsinghua University Georgia Institute of Technology yuezhouhubberkellg edu jx-guo21 @mails tsinghua.... edu (xfeng300, tourzhao] @gatech. edu
Abstract
Speculative Decoding (SD) accelerates large language model inference by em- ploying small draft model generate predictions, which are then verified by a larger target model. The effectiveness of SD hinges on the alignment between these models, which is typically enhanced by Knowledge Distillation (KD). However, conventional KD methods aim to minimize the KI. divergence between the draft and target models across all tokens, a goal that is misaligned with the true objective of SD, which is to maximize token acceptance rate. Therefore, draft models often struggle to fully assimilate the target model's knowledge due to capacity con- straints, leading to suboptimal performance. To address this challenge, AdaSPEC, novel method that incorporates selective token filtering into the KD process. AdaSPEC utilizes reference model to identify and filter out difficult-to- tokens, enabling the distillation of a draft model that better aligns with the target model simpler tokens. This approach improves the overall token acceptance without compromising generation quality. We evaluate AdaSPEC across diverse tasks, including arithmetic reasoning, instruction-followiin coding, and summa- rization, using model configurations of 31M/1.44 and 350M/2.7B parameters. Our results demonstrate that AdaSPEC consistently outperforms the state-of-the-ar DistillSpec method, achieving higher acceptance rates across all tasks (up to 15%). The code publicly available https: //t: /gi thub. com/ com/ hwmm///eehouuuaaaao /daapec..
propose
Introduction
Large language models (LLMs) have revolutionized natural language processing, achieving im- pressive performance across a wide range of tasks. Models like GPT-4 [25] and Llama 3 [11] demonstrate -fate-of-the-art results in various natural language understanding and generation tasks [2, 27], including highly complex tasks such as summarization [23] and mathematical reasoning [9, 13]. However, as these models grow in size and complexity, their inference becomes increasingly computationally intensive, leading to practical challenges in deployment, including slow generation speeds and significant output latency.
To address these shortcomings, current approaches primarily focus on achieving a trade-off between efficiency and performance through two main strategies. The first involves compressing the model scale to enhance the capability of smaller models, often using techniques like Knowledge Distillation (KD) [14]. The second approach employs methods such quantization enable faster computation. However, these strategies inevitably lead to a sacrifice of performance, either due to a loss of representational capacity during compression reduced accuracy resulting from optimization for
Equal Contribution. Work done during an internship at Georgia Tech. *Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

speed. As a result, there is growing need for methods that can maintain the high performance of LLMs while significantly improving their inference efficiency.
Recently, Speculative Decoding (SD) [16, 7] has emerged as a promising paradigm for accelerating LLM inference without sacrificing performance. Unlike model compression or quantization, which modify the model architecture or parameters, SD accelerates generation by restructuring the decoding process itself. Specifically, it introduces a lightweight draft model that speculatively generates multiple candidate tokens, which are then verified by the larger target model. This paradigm preserves the target model's 's predictive quality while substantially reducing the number of expensive forward passes, offering a new efficiency--peformaaae performance trade-off.
The core of SD lies in the design of the draft model. This model is typically much smaller than the target model, even ranging from one-tenth to one-hundredth of the size, enabling faster token generation while maintaining certain level of capability. Consequently, the actual inference speed-up achieved by SI) relies on the draft model closely aligning its predictions with the target model's s output distribution. Typically, this alignment is achieved by pre-training and fine-tuning both models on the same datasets, yielding a pair of homogencous models from the same family, sharing the same architecture but differing in size. However, training two models on the same datasets does not necessarily produce optimal alignment, especially given the significant scale disparity between the draft and target models. This difference in scale makes the draft model prone to prediction errors. To address this challenge, -fate-of-the-aee methods employ KD techniques to refine the draft model, rather than relying solely on direct fine-tuning [32]. However, optimizing fidelity metric (e.g.. forward KI. divergence) does not necessarily lead to a high acceptance rate. Worse still, it may waste the draft model's limited capacity on tokens that are inherently hard to learn and unlikely to be accepted anyway. Additionally, these methods may encounter issues such as the loss failing to converge. Given these challenges, there is a critical need for SD-specific training regimes that effectively balance model capacity constraints with prediction accuracy requirements.
Fortunately, we observe substantial variation in the difficulty of learning individual tokens during KD. which has critical implications for transferring knowledge from the teacher (target) model to the student (draft) model. Instead of mimicking the full output distribution of the target model, the draft model only needs to produce correct predictions on the subset of tokens that is easy enough to propose. During the process of distillation, we identify a subset of "hard" tokens that pose particular challenges for the student model to learn and to predict accurately, regardless of training efforts. Conversely, other tokens are relatively easy to assimilate. both "easy" and "hard" tokens may be counterproductive. Attempting to reduce the loss on difficult tokens often comes at the expense of increasing the loss on easy tokens, resulting in suboptimal learning across both categories. To address this issue, we propose novel approach: deliberately excluding "hard" tokens from the training process. By focusing the loss function exclusively on "easy" tokens, we can more effectively utilize the limited capacity of the student model, thus achieving better alignment with the teacher model on these tokens. This strategic exclusion of hard-to-learn tokens allows the student model to concentrate its resources teacher's 's knowledge, potentially leading to improved overall performance in SD tasks. Our approach thus maximizes the alignment between the draft and target models within the constraints of the draft model's 's capacity.
We argue that uniformly emphasizing the loss on
mastering the more accessible aspect of the
In this study, we propose AdaSPEC, novel Knowledge Distillation method designed to bridge the capacity gap between the draft and target models SD. AdaSPEC operates in two phases:
1: Reference Model Distillation and Token Filtering: A reference model, initialized as copy of the draft model, is distilled using the target model as its teacher. For simplicity, we assume that the target model has been well fine-tuned to downstream tasks of our interests. Here the reference model serves a crucial role as a token filter. identifies "hard" vokens-those models to predict accurately- by comparing the perplexity differences between the reference and draft models on the training data.
those that are difficult for smaller
2: Selective Draft Model Distillation: Finally, the draft model undergoes distillation using a filtered dataset. The reference model removes the previously identified "hard" tokens, allowing the draft model to focus its limited capacity on learning to predict the remaining, more manageable tokens accurately.
2

---

We conduct extensive experiments on wide range of models and downstream tasks, where we benchmark AdaSPEC against DistillSpec and find that AdaSPEC sucessfully pushes the limit of SD across all tasks and model setups, AdaSPEC consistently achieves higher acceptance rates (up 15%; see Table I).
2
Preliminaries
In this section, we provide a formal overview of the foundational concepts. We begin with the mathematical framework of SD, followed by description of various evaluation metrics. Finally, we explore language model families and their significance in enabling techniques like SD to bridge performance gaps between models of different sizes.
Speculative Decoding. Speculative Decoding [7, 16, 29, 20, 32, 6, 17, 18, 30, 6, 21, 26, 32, 20] is originally proposed accelerate LLM inference by employing compact draft model to predict potential output sequences in advance and then verified by a larger target model. The typical framework of SD is formulated as follows. Let Mp and Mg denote the large target model and the compact draft model, respectively. SD leverages the draft model to autoregressively generate tokens which includes the prompt and previously generated tokens. The target model then verifies these proposed tokens by evaluating their probabilities [p(z: x, =r,<<))]== in parallel.
= {z, 1= qo(- x) based on the input x: = [x] r,,,..,, Jzl.
Both models generate probability distributions p(z++1 x, zc) and ((=++1 æ. z<c) for each token i = l,..... 7 in a single forward pass. Using greedy decoding strategy, only the tokens with the highest probabilities are selected for generation verification. The sampling functions are:
Sp(zz) = arg max maxp(Zi++1
22, Zcs),
#c+1
(,(=))) = arg g(i+1 a. z<(), f.11
(2)
for each = ,.. The complete sampling and verification process is detailed in Appendix A.1 Acceptance Rate. The acceptance rate, measures the accuracy of the draft model My compared to the target model Mp. calculated as:
accept
(3)
accept reject
Here, accept and reject are the count of tokens accepted and rejected by My, respectively. A higher indicates greater alignment between Mp and Mg. facilitating faster inference in practical scenarios.
Block Efficiency and Wall-time Improvement. Block efficiency [7, 16], т, quantifies the average number of tokens generated per iteration. It is defined as the expected number of accepted tokens per block, with maximum value of y + 1 for block size of The block efficiency can also be expressed in terms the acceptance rate a [16]:
7+1
T ( 1-a
4)
This metric evaluates how effectively M, approximates Mp. The speed-up factor for the total wall-time given by:
Speed-up T(x)
(5)
YC + 1'
where c is the cost coefficient, representing the ratio of the time taken by single execution of M, to that of Mp.
Language Model Families. Modern language models are often developed as part of family of models that share the same core architecture but differ scale, typically measured by the number of parameters or the size of the training dataset. These families, such as Llama [11], BERT [10], and Pythia [5], are designed to enable researchers and practitioners to balance computational efficiency and performance based specific use cases. Within family, smaller models are generally used for tasks requiring faster inference or lower computational cost, while larger models are leveraged for tasks demanding higher accuracy and richer representations. This structural consistency within family allows for techniques like Knowledge Distillation [14] and Speculative Decoding [7, 16) to transfer knowledge align predictions effectively between models of varying sizes.

---

Method
We introduce AdaSPEC, an adaptive distillation framework for SD that enhances the alignment between target model and a smaller draft model through selective Knowledge Distillation. Given target model Mp fine-tuned for a specific downstream task, AdaSPEC consists two key steps: (I) constructing reference model Mret and (2) selectively distilling knowledge from Mp and Myot to the draft model Mq.
Step 1: Constructing the Reference Model. The reference model Myer is constructed by distilling Mp on downstream task dataset D using the DistillSSpec framework [32]. The objective to minimize the forward KI. divergence between the target model and the reference model:
LKD = Em--Dyy-P(ja)) [K (P(y)=)) R(yl=)))
(6)
where æ represents the imput prefix, y denotes the generated context, '. denotes the forward KL diver- gence, P(y|æ) represents the probability distribution of the target model, and R(y (x) corresponds the probability distribution of the reference model.
Step 2: Selective Knowledge Distillation for the Draft Model. To identify learnable tokens for the draft model Mg. we compute token-wiss wise losses based on the predicted distributions of Mret and Mg- Specifically, for each token w, the token-wise KI. divergence losses are computed as:
Leef (w) = KC (P(w context) R[w context)),
(7)
Caraft (w) = (P(w) context) [[[[w context))
(8)
where Q(w) context) the probability predicted for token w by the draft model, given the context. Next, we calculate the difference in token-wise losses:
Ac(w) = (w) Cref
Tokens with higher (ww represent a larger performance gap between M, and Mref relative to Mp. suggesting that these tokens not yet well aligned but highly learnable for the draft model. Accordingly, we select the subset of tokens with larger A A(w) values, as they are most promising for improving the alignment between the draft and target models. Specifically, we denote
$ A (w)) is among the top A: 100% of all tokens }. E [0,
Therefore, the overall loss for training the draft model M, is:
lvl
Cdistill
I [ya E S)
Ldraft (y:),
(10)
k
where T[-] is the indicator function that equals 1 if the condition inside the brackets is satisfied, and 0 otherwise. It ensures that only the selected learnable tokens contribute the loss calculation. The whole filtering process is shown in figure 1.
Selected for training
Easy tokens
Emily then has 6+ 6+12=<<6+12=18>>16 marbles.
1
x2
x4
x5
X7
X8
AdaSPEC Loss
x2
x4
x8
Figure 1: Overview of AdaSPEC distillation process: AdaSPEC selects the training-effectinn tokens and distills on these tokens.

---

Experiments
We evaluate AdaSPEC through comprehensive experiments across diverse domains and conduct detailed ablation studies to analyze its impact on the acceptance rate
4.1 Experimental Setup
Our experimental framework employs GPT-like decoder-only Transformer models in two distinct configurations, designed evaluate performance across different parameter scales while maintaining tokenizer consistency for SD;
Small-to-Large Model Configuration: A draft model Pythia-3IM paired with target model Pythia-1.4B [5]. These models share architecture and tokenizer, providing an ideal test case for family knowledge transfer.
Medium-to-Laage Model Configuration: A draft model CodeGen-338 paired with target model Phi-2 [24, l]. While from different families, these models use an aligned tokenizer to ensure token-level consistency, allowing us to evaluate cross-family KD.
We test these two configurations on a diverse set of five tasks, each representative of specific domain provide robust evaluation framework for AdaSPEC: GSM8K [9] (A benchmark for multi-step arithmetic reasoning). Alpaca [27] (A comprehensive instruction following dataset), MBPP [3] (A Python programming challenge set for code generation). CNN/Daily Mail [22] (A long-fom form summarization task), and XSUM [23] (An extreme summarization challenge).
Reference Model Training. To ensure consistent starting point and fair comparison, both the draft model and the reference model are initialized from the same pre-trained model. For each task, we first fine-tune the target model the task-specific dataset establish strong baseline. The reference model is then trained using the method from DistillSpec [32].
4.2 Baseline Setup
We compare AdaSPEC against DistillSpec [32], the current state-of-the-ar method for SD. Although AdaSPEC builds upon DistillSpec's 's training framework for its reference model, it introduces novel token selection mechanism. To evaluate its effectiveness, we evaluate both methods under two settings: resource-efficien scenario with fixed training duration and scenario optimized for maximum performance:
Both reference and draft models are trained for exactly 3 epochs, standard practice LLM fine-tuning that balances task-specific performance with general capability retention [4]. This controlled training duration effectively prevents overfitting while ensuring adequate task adaptation. This setting evaluates model effectiveness under typical resource constraints and provides insights into rapid adaptation scenarios.
3- 3-Epoch Setting:
Models are trained for variable number of epochs, treated tunable hyperparameter, to maximize task-specific performance. While this approach may lead to overfitting the specific task at the expense of performance on other tasks, it allows us to thoroughly evaluate the upper bound of performance. The optimal number of epochs is determined empirically. Specifically, for GSM8K, the number target epochs is chosen according to validation accuracy, while for the rest of the experiments it is chosen according to validation perplexity. Afterwards, we distill the reference model and pick the one with highest a on validation set. Eventually, this model serves as reference only select the optimal epoch from 1, 3, 6. 10. 15, 20
Optimal-Epoch Setting:
to train our draft model. For robustness, and 30 (for XSUM and CNN/Daily Mail we select from 1, 3, 6, 10 for training efficiency). This configuration enables evaluation of both methods under less constrained scenarios, where achieving optimal task performance takes precedence over maintaining general capabilities.
While Zhou et al. [32] employs more extensive training schedule in their DistillSpec experiments, our study adopts more resource-efficcent approach due to computational constraints. In the Optimal- Epoch Setting, we limit training maximum of 30 epochs, striking balance between performance optimization and computational feasibility. Complete hyperparameter configurations and training specifications for both DistillSpec and AdaSPEC are detailed in Appendix A.2.
5

---

4.3 Main Results
We summarize the main experimental results in Table 1.
Table 1: Main experimental results for AdaSPEC compared to DistillSpec under two settings: 3- Metrics include acceptance rate (a).
Epoch and
Task
3-Epoch (a)
Optinal-Ppock (c)
Pythin-3IM -> 1.4B
CodeGen-3SUM
Phi-2
PYthia-3IM - 1.4
CodeGen-35OM
Phi-2
DistillSpee
AdaSPEC
DistillSpee
DistillSpee
AdaSPECC
DistillSpec
GSM8K
57.58%
62.63%
79.49%
82.79%
66.19%
68.28%
81.49%
83.48%
Alpaca
44.34%
47.25%
56.48%
58,80 %
65.41
65,79%
58.05%
600.66 %
MBPP
46.88%
47.73%
87.36%
88.76%
49,8
65.12%
87.70%
CNN/Daily Mail
73.05%
74.22%
79.33%
80.63%
80.15%
80.89 %
85.01%
86.29
XSUM
47.24%
49.11 %
58.88%
59.93%
56.11%
57.80%
66.78%
68.19%
Acceptance Rate Analysis. We evaluate performance using the acceptance rate a, defined as the proportion of traft-model--gneraa tokens validated by the target model. As shown in Table 1, AdaSPEC consistently achieves higher acceptance rates than DistillSpec across all tasks and model configurations, demonstrating superior draft-target model alignment.
4.4 Analysis
To provide detailed insights into AdaSPEC's effectiveness, we conduct in-depth analyses on two representative configurations:
This configuration examines performance on arithmetic reasoning under constrained training conditions, representing scenarios with limited computational resources and the need for generalization. Since reasoning is typically considered as an additional capability beyond general language modeling, this setup ensures that the model retains its core abilities while effectively handling arithmetic tasks.
Pythia-31M/ILAB on GSM8K (3-Epoch):
Pythia-31M/1.41 on CNN/Daily Mail (Optimal-Epoch): This setup investigates extractive summarization with extended training, demonstrating the model's ability optimize for task-specific objectives. In real-world applications, models are sometimes specifically deployed for summarizing forg-form& contents such as news reports, emails, or web pages, requiring dedicated fine-tuni fine-tuning. Thus, the Optimal-Epochh setting is chosen to maximize the model's summarization capabilities.
Task-Level Acceptance Rate Distribution. We first analyze the distribution of acceptance rates across tasks for both methods. As illustrated in Figure 2, AdaSPEC demonstrates consistently superior performance compared to DistillSpec. The acceptance rate histograms for both tasks exhibit a significant rightward shift under AdaSPEC, indicating more frequent successful draft predictions. This systematic improvement in acceptance rate suggests that AdaSPEC's selective distillation approach effectively enhances draft-target model alignment across diverse task contexts.
Logit Margin Distributions Across Tokens. Next, analyze the distribution of top-2 logit margins across tokens for both methods. The logit margin, defined as the difference between the logits the top-1 and top-2 predicted tokens, serves measure of prediction confidence. A positive margin indicates a correct draft model prediction, while a negative margin signifies an incorrect prediction that would be rejected in SD.
As shown in Figure 2, AdaSPEC demonstrates superior logit margin distributions compared to DistillSpec across both GSM8K and CNN/Daily Mail datasets. AdaSPEC exhibits:
Higher frequency and magnitude of positive margins, indicating more frequent and confident correct predictions.
Lower frequency and magnitude of negative margins, suggesting less frequent and less severe prediction emrors.
These patterns demonstrate that AdaSPEC achieves better draft-target model alignment through its selective distillation approach, enabling more effective knowledge transfer from the target model to the draft model.

---

KL-Divergence Distribution Across Tokens. We further analyze the Kullback-Leibler (KL) di- vergence between draft and target models' token prediction distributions on both GSM8K and CNN/Daily Mail datasets. As illustrated in Figure 2, AdaSPEC exhibits consistently lower KI. divergence values compared to DistillSpec across both tasks, demonstrated by significant leftward shifts in the distributions. This systematic reduction in KI. divergence across different tasks and tokens indicates that AdaSPEC's selective distillation approach achieves tighter alignment between draft and target model predictions, corroborating our previous findings on acceptance rates and logit margins.
120
Requancy
Nommlltzed
Acceptance Rate
Logit Margin
Tikan-leval KL Divergence
Normelizd Frequencyy 15
0.2
SPCC
0.09
0:44
Accegtiance Rate
Logit Margin
Token-level Divergence
Figure 2: Comparative analysis of AdaSPEC and DistillSpec performance across multiple metries GSM8K (a, c, e) and CNN/Daily Mail (h, d, datasets: (a-b) Task-level acceptance rate distributions showing AdaSPEC's superior performance across tasks. (c-d) Logit margin distribu- tions demonstrating AdaSPEC's improved prediction confidence with higher positive margins and lower negative margins. (e-f) Token-level KL divergence distributions indicating better draft-target model alignment for AdaSPEC with consistently lower divergence values. The results demonstrate AdaSPEC's s more effective knowledge transfer and improved draft-tane target model alignment compared to DistillSpec across different evaluation metrics.
Case Studies. We conduct detailed case studics on GSM8K and CNN/Daily Mail datasets. A consistent pattern emerges: AdaSPEC's prediction errors form nearly a subset of DistillSpec's 's errors, as illustrated in Figure 3. This pattern demonstrates the general effectiveness of AdaSPEC's targeted training approach in improving alignment and reducing inference discrepancies.
GSM8K, with its natural division between mathematical and non mathematical tokens, offers par- ticularly insightful analysis. During training. AdaSPEC predominantly selects mathematics-relater tokens for focused learning (see Appendix A.3). During inference, this selective approach translates into significantly improved prediction accuracy for mathematical tokens compared to DistillSpec, as shown in Figure 3. These results demonstrate AdaSPEC's s ability to identify and prioritize task-critical tokens during training, leading to more precise draft-targe model alignment.
4.5 Ablation Study
To systematically evaluate the effectiveness of different components in AdaSPEC, we conduct comprehensive ablation studies across the following four key dimensions. All experiments are conducted on GSM8K and MBPP with Pythia 1,4B (target) and Pythia 31M parameters (draft). All models are trained for epochs.
To evaluate our token selection strategy, compare models trained on the top 40% of tokens (selected based on KL-divergence margin) against those trained on the bottom 40%. As shown in Table 2, models trained on the top 40% tokens consistently outperform those trained on the bottom 40% with the latter performing even worse than the reference model. The improvement is particularly pronounced the MBPP dataset, where token selection yields up to a 6% performance gain. These results demonstrate that AdaSPEC effectively enhances model alignment by focusing more learnable tokens during Knowledge Distillation.
Token Selection Mechanism.
Training Method. To demonstrate that AdaSPEC's benefits extend beyond Knowledge Distillation, replace the distillation process for both reference and draft models with direct fine-tuming tunent Table 3
7

---

DistillSpeo
AdaSPEC
GSMBK
GSMBK
In the morning, afte gives her (0r225x1111225kk5 40 cupe In the anteroon, she gives (+22moe15A noadss chiiclens. 40<20mood0-
monning she gives 15 OLDS
glves
sape
15 25400 2542200 to give her crickens 00+22mmx000000000-
feed.
cups feed. The fhni answer 2022mmodl--2mmiooo 220m00o,
60 oups of food. The final answer is: 60
CNN/Dally Mail
CNIN/Dally Mall
Carsey Ho's new vicroo The Feefeet Body, is powsrrfll and
Cosdey Hp's now w deo, Perfect Body, powerfil and stancarde in the key to happiness. The video wse postee Blogii blog. recelveed overwtslimingly poethve feedstack"
soclety's
slanderda key happprreee The video poslted hor Blogiatee biloo, and has recolved overwieeiningyy
Figure 3: Comparison of prediction errors between AdaSPEC and DistillSpec on GSM8K and CNN/Daily Mail Datasets: Tokens highlighted in blue represent errors made by both methods, while tokens highlighted in red indicate errors unique to the corresponding method. As can be seen, AdaSPEC's errors form nearly subset of DistillSpec's errors, demonstrating the effectiveness of AdaSPEC's selective training approach in reducing inference discrepancies.
Table 2: Ablation study results for token selection strategies.
GSM8K
MBPP
Sub-Strategy
Reference
Draft
Reference
Draft
Top 40%
59.77% 59.77%
63.22% 49.03%
42.22% 42.22%
48.22% 39.75%
Bottom 40%
Table 3: Ablation study results for training methods.
GSMSK
MBPP
Sub-Strategy
Reference
Draft
Reference
Draft
Distillation
59.77%
63.22% 63.13%
42.22%
48.22% 45.61 %
Fine-tuning
59.64%
41.42%
Table 4: Ablation study results for distillation methods.
GSMSK
MBPP
Sub-Strategy
Reference
Draft a
Reference a
Draft a
KL
59.77%
63.22%
42.22%
48.22%
TVD
9.32%
9.09%
3.86%
6.76%
RKL
30.22%
30.05%
13.17%
15.61%
reveals two key findings: (1) fine-tuned draft models outperform the distillation baseline (reference model) across both datasets, confirming that our token selection process aids model convergence; and (2) fine-tuned draft models achieve up to 4% improvement over their reference counterparts, indicating that our token selection mechanism's 's benefits generalize beyond distillation to broader training scenarios.
Distillation Method. We expand AdaSPEC to more distillation approaches: Reverse KL (RKL) and Total Variation Distance (TVD) [28]. With k = 0.4 for all methods, we observe that token selection significantly improves the acceptance rate by 6% on MBPP when using forward KL. However, when using RKL. and TVD, the acceptance rate performance degrades. This is primarily attributed the inherent limitations of RKL and TVD as distillation objectives, which struggle to effectively align the draft and target models in the context of SD. It is worth noting that DistillSpec [32] uses TVD the distillation function with a batch size of 32 and a training step of 300,000. This prolonged training process only requires substantial computational resources but also results in the problem of overfitting, Considering these factors, we ultimately select forward KL divergence as our distillation objective.

---

Token Selection Ratio. To investigate the impact of token selection ratio, we vary & and compare the final accep- tance rate of the draft model. Results in Fig 4 show that typically, lower values result better final acceptance rate. To strike balance between training efficiency and performance, we finally choose 0.4 in most cases.
GSMBK
0.65
0.60
0.55
0.58.
4.6
Additional Experimental Results
Figure 4: Impact of token selection tio on acceptance rate for GSM8K: Results show a general trend where lower & values (0.2-0.4) yield higher ac- ceptance rates.
Wall Clock Speed-up. To investigate AdaSPEC's po- tential to accelerate end to end decoding in a real world setting, we use frontier inference engine vLLM [15] on one single A100 GPU and report speed-up in Table 5. Re- sults show that an expected 10~20% speed-up could easily achieved compared with DistillSpee, demonstrating the effectiveness of our approach.
Table 5: Generation speed for AdaSPEC with VLLM on one single A100 GPU. We use greedy decoding and report time to generate sentence and one token. We 1ythia-31M-+111 and the models are trained for epochs. On these tasks, AdaSPEC exhibits 10~20% speed-up.
Speed (s/sentence)
Speed (tokens/s)
DistillSpec
0.69
149.15
MBPP
AdaSPEC
0.57
181.67
DistillSpec
0.51
227.86
GSM8K
AdaSPEC
0.48
241.34
DistillSpec
0.76
248.49
CNN/DailyMail
AdaSPEC
0.67
283.50
Integration with Advanced SD Methods. To demonstrate the orthogonality and generalizability of AdaSPEC beyond vanilla speculative decoding (SD), we integrate our method with EAGLE [17], an advanced SD algorithm featuring tree attention and adaptive expansion strategies. Following the standard 3 3-Epoch training setup on the ShareGPT dataset, we evaluate both training accuracy and generation speed on MT-Bench. As shown in Table 6, AdaSPEC consistently improves both accuracy and decoding efficiency within the EAGLE framework.
Table 6: Vicuma-7B-v1.3 [8, 31] with 3- Epoch finetuning following original EAGLE recipe. Here, training accuracy refers to first- generated-token accuracy in the training set.
Table 7: Acceptance rate of larger model con- figuration with 3-Epoch GSMSK.
GSM8K
DistillSpec
84.43%
Eagle
Eagle + AdaSPEC
AdaSPEC
86.21 %
Training Accuracy t
75.3%
76.3%
Speed (s/sentence) |
8.85
8.06 (-8.9%)
Speed (tokens/s) 1
63.48
68.21 (+7.45 (555)
Results on Larger Models. We conduct an additional GSM8K evaluation using a combination of the Qwen2.5-0.5B and Qwen2.5-32B models. When trained with 3 epochs, AdaSPEC reaches acceptance rate of 86.21% 1% while DistillSpec achieves 84.43%, as shown in Table 7. This shows that our approach can easily scale up larger models.
Results on Mixed Dataset. To further validate AdaSPEC's ability to work on blended tasks, we mix GSM8K with MBPP training and validate a sepa- rately. Specifically, we first train the target on MBPP and then on GSM8K, each for epochs. The reference and draft models also follow the same process. The
Table 8: Performance on mixed datasets.
MBPP
GSM8K
DistillSpec AdaSPEC
69.63%
72.75% 78.41 %
70.69%

---

results in Table 8 reveals that AdaSPEC strives to retain the original model's capabilities as much as possible, with less forgotten knowledge.
Dicussion
Size Gap Between Target and Draft Models. In traditional SD settings, the size gap between the draft model and the target model is often within 10x, In this work, we demonstrate that AdaSPEC effectively bridges the performance gap, even when the size difference is substantial up to 64 times in our experiments. By leveraging selective token filtering and Knowledge Distillation, AdaSPEC can enhance token acceptance rates and help maintain generation quality, providing more opportunities to use significantly smaller draft models.
Model Size Gap and Performance Gains. From Table 1, we observe that the performance gain of AdaSPEC over DistillSpec becomes more pronounced as the size gap between the reference and target models increases (e.g., from CodeGen-350M - Phi-2 to Pythia-3 1M + 1.4B). This trend is consistent across both 3-Epoch and Optimal-Eppch settings. The result aligns well with our motivation: when the capacity discrepancy between models widens, direct Knowledge Distillation tends to suffer from representation mismatch, making harder for the smaller model to absorb all teacher signals uniformly. AdaSPEC's s adaptive mechanism mitigates this issue by selectively aligning easier tokens first, effectively narrowing the transfer gap. Consequently, the larger the size difference, the greater the relative improvement AdaSPEC achieves.
Connection with Lin al. [19]. A similar token selection method is proposed in Lin al. [19], which focuses on identifying and prioritizing harder-to-leant tokens (opposite to the motivation of AdaSPEC) during pre- tre-training. Different from their design, our approach focuses on addressing the limited capacity of the draft model in SD. Specifically, we focus on identifying and filtering out challenging tokens, allowing the draft model to concentrate on learning selective distillation process ensures that the draft model aligns more effectively with the target model on tokens that are more tractable, given its constrained capacity. By doing so, we maximize the draft model's is limited resources while maintaining high-quality predictions in SD tasks. Thus, the essential difference lies in the distinct objectives of pre-training and Speculative Decoding.
to-eediit- tokens. Our
Limitations. As preliminary study selective training for SD, we limit our study on simple loss-related token filter. In future work, one can design more adaptive filtering strategies as well as integrate AdaSPEC with free-based or multi-step verification frameworks to further improve both speed and quality of LLM inference.
Conclusion
We present AdaSPEC, a novel approach for training more efficient draft models for SD. AdaSPEC introduces selective token filtering based on reference model perplexity gaps, enabling draft models to focus limited capacity on tokens where alignment with the target model is most achievable. Experiments show it outperforms basclines in arithmetic reasoning, instruction following, code generation, and summarization with higher acceptance rates.
References
[1] Marah Abdin, Jyoti Aneja, Sebastien B, Caio Mendes, Weizhu Chen, Allic Giorno, Ronen Eldan, Sivakanth Gopi, Suriya Gunasekar, Mojan Javaheripi. Piero Kauffmann, Yin Tat Lee. Yuanzhi Li, Anh Nguyen, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Michael Santacroce, and Yi Zhang. Phi-2: The surprising power of small language models, 12 2023.
[2] Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, Michael Noetel, and Andreas Stuhlmüller. Raft: Á real-world few-shot text classification benchmark, 2022. URL https: 14076.
//arxiv. wwtervv..rggab/////0
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint 1xXiv:2108.07733 2021.
10

---

[4] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.0295. 2024.
[5] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' BrBien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430.. PMLR, 2023.
[6] Tianle Cai, Yuhong Li, Zhengyang Geng. Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple IIm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10772 2024.
[7] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving. Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.003318 2023.
[8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023, URL https://lmsy::
htt///llgg//222/// 30- cuna/.
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[10] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018
[11]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. The Ilama 3 herd of models, 2024. URL https: httpss://araa h/tp:///axxvvviooee org/ abs/ 2407.21783.
[12] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour, 2018.
[13] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https: //trxxaa.vvvv//////////ea
wwwaaaiiv.../raaaaeeeaa org/abs/2103 03874403744......
[14] Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint 2015.
[15] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL httph::t////.ttte wttpr/ii..rr//b///339//// 06180.
[16] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference Machine Learning, pages 19274-19286. PMLR, 2023.
[17] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024.
[18] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406. 16858, 2024.
[19] Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. Rho-1: Not all tokens are what you need, 2024. URL https: //txxv/a/vvv///a////a//eaa/aaaaa org/ abs//444. 07965.

---

[20] Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, lon Stoica, Zhijie Deng. Alvin Cheung, and Hao Zhang. Online speculative decoding. arXiv preprint arXhv:2310.07177, 2023.
[21] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference Architectural Support Programming Languages and Operating Systems. Volume 3, pages 932-949, 2024.
[22] Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text sum- marization using sequence-to-sequence mns and beyond. arXiv preprini 1rXiv:160.006023 2016.
[23] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization, 2018. URL attps://arxiv org/ aaaaab///// 1808..880...0 08745.
[24] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 13474, 2022.
[25] OpenAL. Gpt-4 technical report, 2024. URL httts::t///ttttteee org/
[26] Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec: Massively parallel speculative decoding for interactive Ilm inference on consumer devices. arXiv preprint arXiv:2406.0m 02532, 2024.
[27] Rohan Taori, Ishaan Gulrajani. Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follooing llama model. https://gg gi thub. ctm/ hatm//ttsullab/aaaa stanf ord aapppaaaaapaacaa 2023.
[28] Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation, 2023. URL https://arxi: org/ 2230/ 15190.
[29] Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909-3925, 2023.
[30] Jun Zhang, Jue Wang. Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft& verify: Lossless large language model acceleration via self-speculative decoding. In Lun-We Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11263-1122 11822, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: / /////alallttt aaclaaaclanntoyy...a. 202/ acl aonllong. 607.
[31] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https: org/ abs/2306. 05685.
[32] Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Ros- tamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint 10.0022310.08461, 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Both the abstract and beginning part of introduction indicate the scope of the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in "conclusions" chapter.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
This article is based on experiments and does not require strict theoretical
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Results be reproduced according to "method" and "experiments" chapters. Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: Please refer to the "abstract" part.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
public/guides Saaiidee/Codeuuuisssiiii. weriisssssooolilioooooo oolicy)yyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //niss.. wtt.cc/ppuuc wwtppcccp//aaaecccccccccccccelle http:////uuddee/oeeeuubbiiiiiiia htte//CooSSuiiieeeeeeeee cesssonPollle onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to "experiments" chapter.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The experiments related to our method is relatively costly so it is difficult to
repeat each setting. However, we testify our method on a wide range of settings and tasks, which exhibits the feasibility of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question:
Does the research conducted in the paper conform, in every respect, with the cstuidelslllleiccccccccc dines???? ines?
NeurIPS Code of Ethics https: http///eerrpp.eeeeee httpp:://aarrpp..ccpllttt/////// tttp:///c..pppittttttttt...eedd.
Answer: [Yes]
Justification:
The authors have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper doesn't 't release any new models, generators or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer the "experiments" chapter.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ h/tp://ies..../eeeee httpe//rip.cc//ooeeeeeeeeeeeeeee http:////cceeeeeeee////////11//0 for what should or should be described.
19

---

A Technical Appendices and Supplementary Material
A.1 Full Algorithms for AdaSPEC
Algorithm 1 Greedy Speculative Decoding
1: Input: target model Mp draft model Mg. input sequence a
2: accept 0, reject 0, len(æ))
3: & Sample y tokens 21,.... Zy from Mg autoregressively.
4: for i 1 to 7 do
(= 6: end for
5:
24
7: >> Verify in parallel
8: z' - (Sp(=<)), ( Sp(zz2), Sp(zct) _ 12 22.,, 2n]]
9: for 1110 y do
10: 2 2; then
11: reject +- reject + 1; break
12: end if
13: x - x 2, accept t- accept + 1
14: if 2, <eos> then
15: return x, accept, reject
16: end if
17: end for
18: x (p(zz1); goto
Algorithm 2 AdaSPEC: Selective Distillation for Speculative Decoding
1: Input: dataset D, target model Mp, draft model Mq. fraction k:
2: Step 1. Fine-tune Mp:
3: Train Mp on D (via standard LM fine-tuning) to obtain Mp.
4: Step 2. Reference Model Training:
5: Initialize Mret Mg-
6: Distill Myet from M; on D (e.g. forward KL).
7: Step 3. Selective Distillation:
8: (a) Compute losses: For each token w and context,
Lree K. (P(w) context)] R(w context))
Láraft (w) KC (P(w context)]lQ((vr context)),
A Aw( Ldran (w)
9: (b) Filter tokens:
$ w Ac(w) is among the top x 100% of all tokens)) ), k E (0, 1].
(c) Distill on filtered set:
lvv
1
min Caistill
I [at
S]
(ys
1: 1
1=1
11: return M, (draft model in SD)
A.2 Implementation Details
We use the hyperparameters in Table 9. For 3-Epoch setting, both reference and draft model are distilled for epochs. For Optimal-Epoch setting, the target model is first fine-tuned to maximize performance validation set. Specifically, for GSM8K, the number of target epochs is chosen
20

---

according to validation accuracy, while for the rest of the experiments it is chosen according to validation perplexity. Afterwards, we distill the reference model and pick the one with highest a on validation set. Eventually, this model serves the reference model to train our draft model. For robustness, we only select the optimal epoch from I, 3, 6, 10, 15, 20 and 30 (for XSUM and CNN/Daily Mail we select from 1, 3, 6, 10 for training efficiency). Note that when performing token selection, we apply the linear scaling rule for learning rate adjustment [12].
Table 9: Experimental hyperparameters.
Optimal-Fpoch Codegen-350M-PPicn
Task
Hyperparameter
Pythia 31M-1.48
Codegon-35IM-PPPM 350M-+Phi-2
Pythia 31M-1144
Batch size
16
16
16
Leaming rate
3e-4
3e-4
3e-4
3e-4
Epochs target model
Epochs for reference model
Epochs for draft model
3
3
30
30
Filter fraction
Batch size
16
16
16
16
Leaming rale
3e-1
3e-4
3e-4
3e-4
Alpaca
Epochs target model
Epochs for reference model
3
15
20
Epochs for draft model
3
30
Filter fraction
Batch size
8
8
Leaming
1o-5
1c-4
Epochs for target model
MBPP
Epochs for reference model
30
Epochs draft
Filter fraction
0.4
0.4
0.4
0.4
Batch
16
16
Leamning rate
1o-4
1c-4
1o-4
1o-4
CNN/Daily Mail
Epochs for target model Fpochs for reference model
3
3
Epochs for draft model
10
Filter fraction &
0.4
0.4
0.4
04
Ratch
16
Learning rafe
3e-4
1c-1
3-4
1e4
XSUM
Epochs for target model
3
Epochs reference
Epochs for draft model
3
3
10
10
Filter fraction
0.4
A.3 AdaSPEC Example Tokens
Here, we showcase some example tokens (Listing 1) that AdaSPEC selects while training on GSM8K. These selected tokens are typically mathematical related tokens, such as digits and operators.
{ "sscore", "8", "in", "thus", "9", "x", "1", "1", ">>", "g", "19", "Em", "because", "28", "28", "\+", "8", "28", "4" "90", "18",
"<<", "g",
"91",
"19",
"g", "18", "18", "18", "_" "8", "=" "99", "99" "99" "4" "100" "The" final", "answer", "100", "equal", "12" " "7", "=" "19", ">>" "19", "packs "19", "5", "24", "total", "(", "24", "*(", "2", ")=", "16", ">>", "16", "J", "spends", "inside", "because", "_", "(", "inside", "16", "iley", "3", "18", spenda", "12", "In", "total", "they", "4" "12", "=", "<<" "8" " "=", "20", "/" "=", "10", "10", "The", "earned", "final", "answer", "difference", "_" "=" "13", "*", "2", "26", ">>", "26", "twice", "26", "18", "=", "26", "18", "8", "The", "final", "answer", ":", "8",
Listing I: Selected tokens during GSM8k training.
A.4 Minimal Code Implementation of AdaSPEC
The core of AdaSPEC could be implemented with 100 lines of code. We show it in Listing 2.
def compute_loss(
self,
model,
inputs,
21

---

num num_items in batch-None
):
labels inputs (" 1:]
outputs model mode((*+)))ts)
with torch.no_gtt :::
target_outputs self taagetttrtttteee contt_mmdellc (*einputs)
ref_outputs relf.ref_modell (*+inuut)
logits
target_ logget_ parget__outuue ("logitta]
ref ref_logits ref_outputa ["logits"]
fct "IDivlosss(raducciooo=="ooo
shift logits = logits :-1, :] contiguous ()
shift target_logita logits =
[.... :-1, :]
contiguous()
Saiftreffloose = (ref logits[.... :-1, :]
contiguous()
shift_ shift_logits shift lorttooogiis.ieerr
-1, model.config........... wewavoocabssaeeee size)
shift_target__ogiit.. (thistt_target_logiia
vvie(-11,
shift_rre__loitte
(shift_ref_logitz
view(-1, model. config. vocab vocab_eize)) eize))
confi..oofig. vocaig..occb....omm.
mask labels (IGNORE flatten() unsqueeze (-1)
shift faifft_ogitt--- =
torch. masked seaetedd_seeetttthhheeeeeee (shiit_logitta,,,, mask=mask)
.view(-1, model. config.vocab_..........
ahift_ ahift_target, target_logita =
(torch.
wtt.mmassddeeeeeeeee (shift (shift, target_ _ogits, mask-nask)
.vieu(11, model.confiae vocab vocab_size))
shift_ref ree_logits= =
mask-mask)
(-ie model.confis (oocab_size))
p soffmaa taagstttrttttteee...... lorggtt dim=-1)
sof (shift_logits, dim=-1) q_log F. actual (q_ p)
9_log F.log_ (shiifi _ee logits, dim=-1) ref loss_ lose_fct(q_log, p)
actual actual. aecual.sum/fe sdmddi==1)) ref = ref. sddi==-1d)
k self.k
delta = actual - ref
mask delta >= torch turch.quantile(
delta, k, dim=0, keepdin-True)
if num ietittem_iii binbtchhh_atchhhhh is None:
loss Bascchmasked__dase (actual, mask=mask) sum loss loss num_itema, in batch
else:
marked. faaeked_selectte (actual, mask=mask) .eaan ()
return (loss, outputa) if return_outpine else loss
Listing AdaSPEC implementation with PyTorch.
22

---

A.5 Broader Impact
AdaSPEC can be used mainly improve generation speed LLMs, which positive influence to reduce potential electric energy consumption for serving LLMs. However, this technique may also be used for some models that non-compliance with regulations and ethics, such as models that generate discriminatory contents.
A.6 Experiments compute resources
Here list the estimated GPU hours; see Table 10.
Table 10: GPU hours of training models on A100 GPUs.
Task
3-Epoch
Optimal-Epoch
Pythia-31M + 1.4B
CodeGen-350M Phi-2
Pythin-3IM
CodeGen-350M
GSMSK
Alpaca
15
50
MBPP
15
50
CNN/Daily Mail
60
200
200
700
XSUM
700
23


Reviews:
[{"review_id": "7t1vXUjC0X", "reviewer": "Reviewer_fMLq", "summary": "This submission introduces AdaSPEC, a selective knowledge distillation approach for optimizing efficient LLM inference via speculative decoding. AdaSPEC, initially relies on DistillSpec framework to distill a reference draft model from the original (verification) model. Subsequently, \"easy\" tokens are identified based on their loss, and selected to train the draft model of the resulting SD pipeline. The intuition behind the proposed approach is that by excluding the difficult tokens from the distillation process for fitting the small-capacity draft model, its training process will become less ambiguous and converge to a more robust and effective draft model (i.e. lead to higher acceptance rate and thus speed-up for the overall SD pipeline). The provided experimental results validate this assumption, showcasing consistent gains across different model pairs and datasets.", "questions": "Please considering replying on the points raised in the comments section above.", "limitations": "Some limitations of the proposed approach are discussed at the conclusions section, along with proposed future work directions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed approach is very interesting and timely and exploits an interesting insight.\n- The manuscript is well written and easy to follow.\n- The effectiveness of the proposed methodology is demonstrated through extensive experiments across varying tasks and model pairs.\n- The conducted ablations offer potentially valuable insights for applying KD in related SD settings.\n\nComments:\n- It is unclear why the proposed filtering approach (eq 9) evaluates and compares the loss of the reference vs the draft model (Mq), instead of the correctness/distance of the reference (distilled) model to the original (Mp). Since Mq is the model being trained in step2, Mq's perception of easy samples is expected to change overtime, however the proposed approach does not appear to adapt to such changes. In contrast, evaluating the correctness of Mref (w.r.t Mp) may potentially form a more robust and stable proxy of the predictive difficulty of each token for the draft model. \n- It is unclear how the effectiveness of the proposed approach is affected by the learning capacity of the draft model (and whether the percentage of selected tokens should be scaled accordingly). Although the authors have conducted experiments between 2 different pairs of models, with varying relative parameter counts, more robust insights about the scalability of the proposed method will largely benefit the community and enhance the applicability of the proposed approach.", "was_revised": true, "final_justification": "Increasing my score from 3 to 4, as the raised concerns have been satisfactorily addressed by the authors during the rebuttal phase.\nThe proposed methodology is demonstrated to have notable contribution, novelty and effectiveness and its publication can benefit the community, in my opinion.", "rebuttal": "Dear Reviewer fMLq,\n\nThank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.\n\n---\n\n>**W1**: It is unclear why the proposed filtering approach (eq 9) evaluates and compares the loss of the reference vs the draft model (Mq), instead of the correctness/distance of the reference (distilled) model to the original (Mp). Since Mq is the model being trained in step2, Mq's perception of easy samples is expected to change overtime, however the proposed approach does not appear to adapt to such changes. In contrast, evaluating the correctness of Mref (w.r.t Mp) may potentially form a more robust and stable proxy of the predictive difficulty of each token for the draft model.\n\n**Reply**: \n\nAs we propose in \"introduction\" chapter (line 57), our goal here is to identify tokens that are **more learnable** at the current training stage of the draft model Mq. Compared to the absolute value of per-token loss/accuracy, **the relative loss change is more indicative of ease of “learnability”**. This is because the potential to learn each token varies, and **a high loss/low accuracy does not necessarily equate to ease of learning**. Evaluating the difference between Mref and Mp, as suggested, would instead reflect how closely the reference approximates the teacher model, but may not align with the objective of selecting *learnable* tokens for improving Mq, leading to suboptimal results.\n\nWe think that possibly the reviewer might be referring to the initial (untrained) version of Mq when mentioning Mp. Interpreted this way, comparing to the base model could provide an alternative view on token difficulty, and we agree that this is an interesting direction worth exploring in future work. That said, this may be more likely to be viewed as an alternative design choice rather than a clear weakness.\n\n---\n\n>**W2**: It is unclear how the effectiveness of the proposed approach is affected by the learning capacity of the draft model (and whether the percentage of selected tokens should be scaled accordingly). Although the authors have conducted experiments between 2 different pairs of models, with varying relative parameter counts, more robust insights about the scalability of the proposed method will largely benefit the community and enhance the applicability of the proposed approach.\n\n**Reply**: \n\n**The first question is the capacity of the draft model.** Technically, the capacity of the draft model is relative to the target model, which can be depicted by the size gap between them. The two pairs of draft/target size ratios are representative in contemporary research : $\\sim$1/40 for Pythia setting, and $\\sim$1/8 for Phi-2 setting. This holds practical significance in real-world applications.\n\nWhen practical applications require larger target and draft models, our method is designed to be implementation-friendly and readily scalable. It can be integrated seamlessly into existing training pipelines without architectural changes, making it suitable for scaling to larger models and datasets.\n\nIn **Section 4.6**, we conducted further GSM8K tests using a combination of Qwen2.5-0.5B and Qwen2.5-32B models ($\\sim$1/64 size ratio), which demonstrate that our approach remains effective when scaling up to larger models. We again present those results below.\n\n*Table R1.* Acceptance Rate of Qwen2.5-0.5B$\\rightarrow$32B with 3-epoch GSM8K.\n\n|       | **DistillSpec** | **AdaSpec** |\n| ----- | --------------- | ----------- |\n| GSM8K | 84.43%          | 86.21%      |\n\n**The second question is how to scale k.** This hyperparameter appears to be **relatively stable with respect to model scale**, and we did not find strong evidence that it needs to be adjusted as the model size increases. Thus, we keep it to 0.4 for all shapes of the models, even for the Qwen2.5-32B experiment.\n\nWhile we acknowledge that experiments on even larger models (e.g., >70B) would be valuable, such experiments are unfortunately beyond the computational affordability of academic research. Nonetheless, our method poses no intrinsic barrier to such scaling, and we believe it can be readily adopted in industrial-scale settings.\n\n**The third question is the generalizability of AdaSpec on broader settings and applications.** To further validate AdaSpec on more advanced approaches beyond vanilla SD, we have now included additional experiments combining our method with Eagle, a popular decoding strategy utilizing tree attention for multi-token speculation. Following the standard practice in the Eagle paper [1], we trained the Eagle decoding head on the ShareGPT dataset using the 3-epoch setting from our work and evaluated generation speed on MT-bench.\n\n*Table R2.* Vicuna v1.3 7B with 3-epoch ShareGPT. Here, train accuracy refers to first-generated-token accuracy in the training set.\n\n|                                 | **Eagle** | **Eagle + AdaSpec** |\n| ------------------------------- | --------- | ------------------- |\n| Train ACC ↑                     | 75.3%     | **76.3%**           |\n| Generation Speed (s/sentence) ↓ | 8.85      | **8.06** (-8.9%)    |\n| Generation Speed (tokens/s) ↑   | 63.48     | **68.21** (+7.45%） |\n\nThe results in the table above demonstrate AdaSPEC's effectiveness and generalizability beyond vanilla SD to more advanced SD methods like Eagle.\n\n[1] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty"}, {"review_id": "jtAgscpxnq", "reviewer": "Reviewer_ocoD", "summary": "This paper proposes AdaSPEC, which demonstrates superior performance across diverse tasks (arithmetic reasoning, instruction-following, coding, summarization) and model configurations, consistently achieving higher acceptance rates than the state-of-the-art DistillSpec method Please refer to the strenths below.", "questions": "Please refer to the above weakness.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths:\n\nConsistent and Robust Experimental Performance: AdaSPEC demonstrates superior performance across diverse tasks (arithmetic reasoning, instruction-following, coding, summarization) and model configurations (31M/1.4B, 350M/2.7B), consistently achieving higher acceptance rates than the state-of-the-art DistillSpec method. This broad validation across different domains and model scales highlights the generalizability of the approach .\n\nInnovative Selective Distillation Strategy: The introduction of selective token filtering via a reference model addresses a key limitation of existing knowledge distillation methods for draft models—their inability to fully align with target models due to capacity constraints. By focusing on \"easy\" tokens that the draft model can effectively learn, AdaSPEC improves token acceptance rates without sacrificing generation quality, representing a novel and intuitive solution .\n\nRigorous Ablation Studies: The paper conducts comprehensive ablation analyses on critical components, including token selection mechanisms, training methods, distillation objectives, and token selection ratios. These studies validate the effectiveness of each component (e.g., top 40% tokens outperform bottom 40%, forward KL is superior to TVD/RKL), enhancing the methodological credibility.\n\n## Weaknesses:\n\nInsufficient Support for the Reference Model: While the reference model is claimed to play a \"crucial role\" in filtering hard tokens, the paper lacks strong theoretical justification for why its perplexity-based filtering reliably identifies tokens beyond the draft model’s capacity. Additionally, there is a lack of direct experimental evidence (e.g., correlation between the reference model’s \"hard token\" labels and actual draft model errors) to validate its filtering accuracy .\n\nLimited Comparisons to Recent Methods: The method is only benchmarked against DistillSpec (2023), without comparisons to more recent advancements in speculative decoding or knowledge distillation . This narrow comparison weakens claims of state-of-the-art performance and reduces the work’s timeliness .\n\nUnderdeveloped Generalization Analysis: While AdaSPEC shows promise in tested configurations, its generalization to larger model gaps (beyond 40x) or more complex, low-resource tasks is not thoroughly explored. The limitations section only briefly mentions potential improvements to filtering strategies, without addressing how the method might scale or adapt to diverse real-world scenarios .", "was_revised": true, "final_justification": "I have read the feedback from the authors. The issues seems to be resovled. Considering the responses from the authors and the comments from the other  reviewers, I would like to upgrade my rating to BA, the authors are encouraged to fully incorprate the concerns and improve the quality of the content in the paper during finalisation.", "rebuttal": "Dear Reviewer ocoD,\n\nThank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.\n\n---\n\n>**W1**: Insufficient Support for the Reference Model: While the reference model is claimed to play a \"crucial role\" in filtering hard tokens, the paper lacks strong theoretical justification for why its perplexity-based filtering reliably identifies tokens beyond the draft model’s capacity. Additionally, there is a lack of direct experimental evidence (e.g., correlation between the reference model’s \"hard token\" labels and actual draft model errors) to validate its filtering accuracy .\n\n**R1**: Take GSM8K as an example, we argue that the reference model can actually prune out the hard tokens for the draft model:\n\n1. An observation is that the hard tokens are typically math related tokens, as depicted in Appendix A.3, these tokens are difficult to predict as they involve mathematical logic, whose complexity is often beyond the draft model’s capacity.\n2. Case Studies in Section 4.4 reveals that our token selection mechanism can successfully filter out hard tokens, especially those mathematical tokens.\n\nBesides, we need to point out that many researches on large models are driven by empirical results, with theoretical studies or justification lagging far behind empirical research. Our method has demonstrated strong results in practice and empirically supported our statement in how tokens are filtered, which we believe sufficiently substantiates our conclusions.\n\n---\n\n>**W2**: Limited Comparisons to Recent Methods: The method is only benchmarked against DistillSpec (2023), without comparisons to more recent advancements in speculative decoding or knowledge distillation. This narrow comparison weakens claims of state-of-the-art performance and reduces the work’s timeliness .\n\n**R2**: As of now, DistillSpec is the only existing work that applies model distillation to train the draft model in the speculative decoding (SD) framework, and it is widely regarded as the state-of-the-art method in this line of research. Therefore, we believe our comparison to DistillSpec is both appropriate and meaningful within the scope of distillation-based SD methods.\n\nMoreover, our results demonstrate not only superior decoding efficiency under the standard SD architecture but also stronger generalization across models of different sizes. We have also highlighted that **our approach is orthogonal to other classes of SD techniques** (e.g.adaptive rejection, or sampling strategies), and can in principle be combined with them to further enhance performance. \n\nTo further validate our proposed method on frontier SD approaches, we implemented AdaSPEC with Eagle—a representative SD method featuring tree attention that demonstrates superior performance to Medusa. Following the standard practice in the Eagle paper [1], we trained the Eagle decoding head on the ShareGPT dataset using the 3-epoch setting from our work and evaluated generation speed on MT-bench.\n\n*Table R1.* Vicuna v1.3 7B with 3-epoch ShareGPT. Here, train accuracy refers to first-generated-token accuracy in the training set.\n\n|                                 | **Eagle** | **Eagle + AdaSpec** |\n| ------------------------------- | --------- | ------------------- |\n| Train ACC ↑                     | 75.3%     | **76.3%**           |\n| Generation Speed (s/sentence) ↓ | 8.85      | **8.06** (-8.9%)    |\n| Generation Speed (tokens/s) ↑   | 63.48     | **68.21** (+7.45%） |\n\nThese results demonstrate AdaSPEC's effectiveness and generalizability beyond vanilla SD to more advanced SD methods like Eagle, which enhances the practical utility and impact of our work.\n\n---\n\n>**W3**: Underdeveloped Generalization Analysis: While AdaSPEC shows promise in tested configurations, its generalization to larger model gaps (beyond 40x) or more complex, low-resource tasks is not thoroughly explored. The limitations section only briefly mentions potential improvements to filtering strategies, without addressing how the method might scale or adapt to diverse real-world scenarios .\n\n**R3**: We would like to clarify that **we have already evaluated AdaSpec under a draft/target size ratio of 1/60** in our additional experiments on the **Qwen2.5-0.5B → Qwen2.5-32B** setting, which goes beyond the 1/40 ratio raised in the question. These results confirm that AdaSpec remains effective even under such extreme scale gaps.\n\nWhile we agree that analyzing even larger gaps could be interesting, we believe that in **general-purpose applications**, going beyond a 40× size ratio is rare and often **unrealistic**—particularly in settings where the draft model must still perform useful computations (e.g., general text generation). In practice, such extreme ratios may only arise in **domain-specific** or highly constrained deployment scenarios, which are somewhat outside the scope of our current work.\n\nIn terms of **diversity**, we have already included a wide range of domains in our evaluation—spanning **mathematical reasoning, instruction following, code generation, text summarization**, and **multi-task mixtures** (as discussed in §4.6). In addition, our **dual setting design (3-epoch and optimal-epoch)** is intended to simulate **realistic training budgets and performance goals**, further supporting the robustness of AdaSpec across practical scenarios.\n\n[1] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty"}, {"review_id": "gR6H7YtPDb", "reviewer": "Reviewer_TYdk", "summary": "This paper examines the implications of the observation that Speculative Decoding should prioritize the accuracy of predicted tokens over optimizing the overall KL divergence between the target and the reference model. Consequently, it proposes AdaSPEC, which prioritizes distillation of the draft model on the subset of tokens that it is actually able to learn to predict properly, given its small scale. The heuristic used by AdaSPEC to choose this subset of tokens is to select those where the most learning progress has been made so far in terms of KL divergence with respect to the target model. Results show a modest but consistent gain in acceptance rate compared to the ablation of the selective-token filtering (corresponding to DistillSpec \\[1\\]).\n\n\\[1\\] Zhou, Y., Lyu, K., Rawat, A. S., Menon, A. K., Rostamizadeh, A., Kumar, S., Kagy, J., & Agarwal, R. (2023). DistillSpec: Improving Speculative Decoding via Knowledge Distillation. *ArXiv*. [https://arxiv.org/abs/2310.08461](https://arxiv.org/abs/2310.08461)", "questions": "* In L138, you state that the selected tokens are the ones that are easiest for the draft model to align with the target model. However, this motivation is slightly unclear to me. In particular, it seems that tokens maximizing (9) might satisfy a combination of “learnability”, related to your “ease”, but also “not yet learnt”, since tokens that are already predicted by the reference model well will be easy to predict, but will not satisfy (9). Would you agree?  \n* In L182, could you mention in the appendix and reference how you specifically chose the number of epochs?  \n* In L188-189, for reproducibility, it would be important to report how the hyperparameters and most importantly, the filter fraction were chosen. In particular, could you justify in L286 why you sometimes use k=0.8 instead of 0.4 everywhere? Is there any way to understand when we would want a higher k and when a lower one?  \n* In L198, could you please provide the respective plots for the other configurations in the appendix, given that these should be readily available already? That would aid in the completeness of the results.\n\n### Other suggestions\n\n* In the equation after L140, you mention selecting the top k tokens, but in the appendix pseudo-code, you appear to choose them based on quantiles instead. I would encourage addressing this inconsistency for reproducibility.  \n* I would encourage the authors in the Abstract and Introduction to focus more on the observation that simple KL optimization does not effectively maximize the acceptance rate, which is arguably the goal of SD. Therefore, in SD, we aim for high precision or maximum likelihood, not matching the distribution. As you rightly mention in the limitations, your heuristic is just a first approach to use this observation. Still, you could capitalize on this observation as a key contribution, which might enhance the impact of this work.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n* **Clarity.** I find the paper quite well-written and clear. The authors effectively motivate the problem and clearly explain their approach, and the paper’s structure is logical.  \n* **Problem significance**. The problem of increasing the computational efficiency of LLMs is very relevant and timely, and the paper’s results, although modest, could have a huge impact on the large scales of both usage and cost of LLM inference.  \n* **Originality**. Although token selection by difficulty \\[2\\], as discussed by the authors, or the hardness of examples in general \\[3\\] is not novel, the reason why this paper considers difficulty comes from a different perspective and has different implications. The observation that leads the author to adaptive token selection by difficulty, in my opinion, is where the originality of this paper lies, and it opens an avenue for future work to examine different designs to achieve a higher acceptance rate, beyond simply considering token difficulty.  \n* **Quality/Evaluation**. Evaluation is overall quite diverse, both in terms of architectures and datasets. Together with the analysis and ablation study experiments, the empirical results appear robust and sufficient \\[See questions below for further details\\].\n\n## Weaknesses\n\n* **Modest empirical results.** Although consistent, the effect of the method is modest, which reduces the impact of the work.\n* **Computation cost.** Although the implementation of token filtering is simple and should not incur a significant computational or memory cost, given that it involves the additional inference of the reference model, I would appreciate a comparison of the impact, given an equal number of tokens incorporated in the GD step.\n* **Reproducibility details.** Some minor implementation choices could be stated more clearly, see the questions below.\n\n---\n\nThe main limitation that stops me from increasing the score is the modesty of the empirical results, which I am unsure if this can be addressed during the rebuttal. However, I would greatly encourage the authors to consider and incorporate the other comments.\n\n\\[2\\] Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., & Chen, W. (2024). Rho-1: Not All Tokens Are What You Need. *ArXiv*. [https://arxiv.org/abs/2404.07965](https://arxiv.org/abs/2404.07965)   \n\\[3\\] Seedat, N., Crabbé, J., & Bica, I. (2022). Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular data. *ArXiv*. [https://arxiv.org/abs/2210.13043](https://arxiv.org/abs/2210.13043)", "was_revised": true, "final_justification": "After reconsidering the paper in light of the other reviews, the author’s rebuttals, and the additional experiments. I choose to maintain my original score.\n\nNonetheless, I would encourage the authors to strengthen their related work discussion if the paper is accepted, potentially using an additional page or an appendix to discuss the broader literature in speculative decoding and how their approach is orthogonal to some of the approaches, and for what methods it might not be readily applicable.\n\nI would also appreciate a more specific measurement or an in-depth discussion of the particular impact of the additional inference step compared to the vanilla cost in the paper, even if the focus of SD is on the inference cost. I believe it would be of interest to practitioners and for comparison with future works.", "rebuttal": "Dear Reviewer TYdk,\n\nThank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.\n\n---\n\n>**W1**: Although consistent, the effect of the method is modest, which reduces the impact of the work.\n\n**R1**: The reason about modest improvement in our main results is mainly due to small size gap between draft and target model. In Pythia settings, where the draft/target size ratio is about 1/40, AdaSpec introduces up to 8% accuracy improvement. When it comes to the Phi settings, whose draft/target ratio is about 1/8, that advantage is weakened to up to 3%. This shows the advantage of our method on extreme draft/target ratios: the larger the size gap is, the more effective our method is.\n\nFurthermore, we would like to highlight that the advantage of AdaSpec is in inference acceleration but not training. Although AdaSpec introduces **minimal overhead to the standard distillation framework**, it is actually very effective in **inference-heavy scenarios**. Compared with the marginal computation cost of distillation, AdaSpec can lead to **10–20% speedup** (as shown in Section 4.6), making it highly practical for real-world deployment.\n\nFinally, to further enhance the practical utility and impact of our work, we integrate AdaSpec to Eagle, a frontier method that combines tree attention with SD. Following the standard practice in the Eagle paper [1], we trained the Eagle decoding head on the ShareGPT dataset using the 3-epoch setting from our work and evaluated generation speed on MT-bench. The results below demonstrate AdaSPEC's effectiveness and generalizability beyond vanilla SD to more advanced SD methods like Eagle.\n\n*Table R1.* Vicuna v1.3 7B with 3-epoch ShareGPT. Here, train accuracy refers to first-generated-token accuracy in the training set.\n\n|                                 | **Eagle** | **Eagle + AdaSpec** |\n| ------------------------------- | --------- | ------------------- |\n| Train ACC ↑                     | 75.3%     | **76.3%**           |\n| Generation Speed (s/sentence) ↓ | 8.85      | **8.06** (-8.9%)    |\n| Generation Speed (tokens/s) ↑   | 63.48     | **68.21** (+7.45%） |\n\nThese results demonstrate that AdaSpec is not only scalable and lightweight, but also delivers **substantial practical gains** in scenarios where inference cost dominates—underscoring its relevance for efficiency-critical applications.\n\n---\n\n>**W2**: Although the implementation of token filtering is simple and should not incur a significant computational or memory cost, given that it involves the additional inference of the reference model, I would appreciate a comparison of the impact, given an equal number of tokens incorporated in the GD step.\n\n**R2**: We argue that the extra computation cost would be marginal and can be ignored.\n\n1. The extra computational cost mainly involves one forward pass of the reference model. Since the reference model and draft model share the same architecture and size, this would introduce about 25% extra computation (the backward pass of the draft model is twice as much as the forward pass, and gradient checkpointing introduces an extra forward pass).\n2. In distillation we also need to run the target forward pass, whose computation is tens of the forward pass as the draft model.\n\nMost importantly, the **core goal of speculative decoding is to accelerate inference**, not to reduce training cost. Our method offers substantial decoding speedup during inference with only a small, one-time overhead during distillation training. We believe this trade-off is both reasonable and practical.\n\n---\n\n>**Q1**: In L138, you state that the selected tokens are the ones that are easiest for the draft model to align with the target model. However, this motivation is slightly unclear to me. In particular, it seems that tokens maximizing (9) might satisfy a combination of “learnability”, related to your “ease”, but also “not yet learnt”, since tokens that are already predicted by the reference model well will be easy to predict, but will not satisfy (9). Would you agree?\n\n**R1**: Thank you for raising the question. Your understanding is correct. We will make this clear with more explanation at L138.\n\n---\n\n>**Q2**: In L182, could you mention in the appendix and reference how you specifically chose the number of epochs?\n\n**R2**: For GSM8K, the number is chosen according to accuracy on validation set. For the rest of the settings, that number is chosen according to Cross Entropy loss on validation set.\n\n---\n\n>**Q3**: In L188-189, for reproducibility, it would be important to report how the hyperparameters and most importantly, the filter fraction were chosen. In particular, could you justify in L286 why you sometimes use k=0.8 instead of 0.4 everywhere? Is there any way to understand when we would want a higher k and when a lower one?\n\n**R3**: We naturally choose some batch size and learning rate that appears reasonable. Alpaca, XSUM and CNN/DailyMail datasets are relatively large, so we modify some of the settings accordingly. We further tune the epochs to get optimal results.\n\nFor Alpaca, we first train a set of ablations including different k, learning rate and batch size. Then we select the optimal result which appears to be 0.8. For the rest of the experiments, since 0.4 is a feasible choice for most of the settings, we keep k=0.4 and no longer tune this parameter.\n\n---\n\n>**Q4**: In L198, could you please provide the respective plots for the other configurations in the appendix, given that these should be readily available already? That would aid in the completeness of the results.\n\n**R4**: Yes, we will provide more plots. Since NeurIPS 2025 forbids plots in author reviewer discussion, we will update this in the final version of our paper.\n\n---\n\n>**S1**: In the equation after L140, you mention selecting the top k tokens, but in the appendix pseudo-code, you appear to choose them based on quantiles instead. I would encourage addressing this inconsistency for reproducibility.\n\n**R1**: Sorry for the misunderstanding. The \"top k tokens\" refers to a ratio of the selected tokens, for example, top 40% tokens. Here, k $\\in$ [0,1]. We will clarify this in the paper.\n\n---\n\n>**S2**: I would encourage the authors in the Abstract and Introduction to focus more on the observation that simple KL optimization does not effectively maximize the acceptance rate, which is arguably the goal of SD. Therefore, in SD, we aim for high precision or maximum likelihood, not matching the distribution. As you rightly mention in the limitations, your heuristic is just a first approach to use this observation. Still, you could capitalize on this observation as a key contribution, which might enhance the impact of this work.\n\n**R2**: Thank you for pointing this out. We will emphasize this insight in the introduction and abstract sections.\n\n[1] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty"}, {"review_id": "xoTdpad7gu", "reviewer": "Reviewer_qrev", "summary": "This paper introduces AdaSPEC, an adaptive loss for model distillation for later use in Speculative Decoding settings. Based on the observation that a focus on simpler tokens should enhance the acceptance ratio during SD, the work proposes a two-step distillation, the first step being a normal KL-based distillation into a smaller reference model and during a second stage, another KL-based distillation of the final draft model on selected \"simple\" tokens derived by the KL-difference between reference and draft models. This approach can consistently improve acceptance scores over the DistillSPEC baseline.", "questions": "- Is there a possibility to train both reference and draft model at the same time? This would bring down the large cost of evaluating the big target model twice.\n\n- Can you explain the only mild improvements of acceptance scores (often ~1%)? \n\n- Could you add in your discussion of SD alternatives like MoE, CoT and recurrent-depth approaches in l.25ff? What is your opinion on those compared to SD?", "limitations": "Within the narrow focus of this paper (Speculative Decoding setting), it addresses the main limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- Improved acceptance scores for Speculative Decoding\n\nWeaknesses:\n- Twice the distillation compute\n- Limited to Speculative Decoding settings", "was_revised": true, "final_justification": "I want to recommend this paper for acceptance as it provides an elegant approach to improve acceptance scores of speculative decoding, accompanied with sound experimentation.", "rebuttal": "Dear Reviewer qrev,\n\nThank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.\n\n---\n\n>**W1**: Twice the distillation compute\n\n**R1**: While it is true that our approach introduces additional compute during the distillation stage, the core objective of speculative decoding is to accelerate inference. We argue that a moderate increase in training cost is acceptable, especially given the substantial inference speedups achieved. Moreover, compared to pretraining large language models, the overall cost of distillation on downstream tasks is relatively small and typically affordable in practice.\n\n---\n\n>**W2**: Limited to Speculative Decoding settings\n\n**R2**: Our method targets the standard speculative decoding (SD) setting, which is a widely adopted and practically relevant scenario for accelerating inference in large language models. In this context, our approach achieves state-of-the-art performance and outperforms DistillSpec in both efficiency and generalization. \n\nTo further validate AdaSpec on more advanced approaches beyond vanilla SD, we have now included additional experiments combining our method with Eagle, a popular decoding strategy utilizing tree attention for multi-token speculation. Following the standard practice in the Eagle paper [1], we trained the Eagle decoding head on the ShareGPT dataset using the 3-epoch setting from our work and evaluated generation speed on MT-bench.\n\n*Table R1.* Vicuna v1.3 7B with 3-epoch ShareGPT. Here, train accuracy refers to first-generated-token accuracy in the training set.\n\n|                                 | **Eagle** | **Eagle + AdaSpec** |\n| ------------------------------- | --------- | ------------------- |\n| Train ACC ↑                     | 75.3%     | **76.3%**           |\n| Generation Speed (s/sentence) ↓ | 8.85      | **8.06** (-8.9%)    |\n| Generation Speed (tokens/s) ↑   | 63.48     | **68.21** (+7.45%） |\n\nThe results in the table above demonstrate AdaSPEC's effectiveness and generalizability beyond vanilla SD to more advanced SD methods like Eagle.\n\nCould you please clarify which specific scenarios are considered missing? We believe the standard SD setting and the additional experiment above already provides a strong and representative benchmark for evaluating speculative decoding methods.\n\n---\n\n>**Q1**: Is there a possibility to train both reference and draft model at the same time?\n\n**R1**: Yes. Although the reference model is limited to a roughly pre-finetuned speculative decoder in our study, it can be replaced by any small model that aligns with the target model. One possible approach to train both reference and draft model simultaneously is to dynamically assign the reference model to be an earlier checkpoint of the draft model. We find this idea quite interesting and believe it opens up a new direction for adaptive speculative decoding, which we plan to explore in future work.\n\n---\n\n>**Q2**: Can you explain the only mild improvements of acceptance scores (often ~1%)?\n\n**R2**: The reason for modest improvement in our main results is mainly due to the small size gap between draft and target model. In Pythia settings, where the draft/target size ratio is about 1/40, AdaSpec introduces up to 8% acceptance rate improvement. When it comes to some of the Phi settings, whose draft/target ratio is about 1/8, that advantage is weakened to up to 3%. This shows the advantage of our method on extreme draft/target ratios: the larger the size gap is, the more effective our method is.\n\n---\n\n>**Q3**: Could you add in your discussion of SD alternatives like MoE, CoT and recurrent-depth approaches in l.25ff? What is your opinion on those compared to SD?\n\n**R3**: We thank the reviewer for the suggestion. However, we would appreciate further clarification on the intended comparison, as the listed approaches seem to target different aspects of model design and usage.\n\nTo our understanding:\n\n- **MoE (Mixture of Experts)** accelerates inference by activating only a subset of model parameters during each forward pass, providing efficiency gains through activation sparsity. In contrast, speculative decoding is model-agnostic and focuses on speeding up generation by parallelizing the decoding process.\n\n  \n\n- **Chain-of-Thought (CoT)** and **recurrent-depth** methods aim to improve reasoning capabilities or model expressivity, often at the cost of dynamically expanding inference computation. These approaches are typically used to enhance output quality rather than inference efficiency.\n\n[1] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty"}, {"review_id": "XpP2L6ICA2", "reviewer": "Reviewer_5tif", "summary": "This paper introduces AdaSPEC, a novel method to improve the efficiency of Speculative Decoding (SD).\nAdaSPEC addresses the problem of the small draft model struggling to fully learn the target knowledge by incorporating selective token filtering into the Knowledge Distillation (KD) process, using a reference model to identify and exclude diffucult tokens.\nBy focusing distillation on simpler tokens, AdaSPEC enables the draft model to better align with the target model, leading to higher token acceptance rates and improved inference speed without compromising generation quality.", "questions": "1. Can AdaSPEC generalize beyond vanilla SD to more advanced methods like Eagle or Medusa?\n\n2. Could more relevant baselines be included, like Rho-1 mentioned in the paper?\n\n3. Considering the significant gap between 3-epoch and optimal-epoch performance, why wasn’t a more effective hyperparameter selection strategy proposed to improve results?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. AdaSPEC is simple to implement and is a plug-and-play method. \n2. AdaSPEC achieves much higher acceptance rate than DistillSPEC baseline. \n\nWeaknesses:\n1. Only the vanilla SD is tested, so it is not clear whether AdaSPEC could generalize to more advanced SD methods like Eagle.\n2. The only baseline is DistillSPEC. The evaluation is limited in scope. The discussion of related work is limited.\n3. There is a huge gap between 3-epoch and optimal-epoch performance. A more advanced hyper-parameter selection strategy could have been proposed to optimize the performance.", "was_revised": true, "final_justification": "The additional experiments during rebuttal (on SOTA speculative decoding methods like Eagle and Medusa) prove that adaspec can generalize to arbitrary speculative decoding methods. Hence, my main concerns are addressed.\n\nThis work is an orthogonal work to most of the existing speculative decoding methods, targeting at the pruning difficult tokens to improve acceptance rate. I appreciate the novelty and significance of the proposed method, and believe it will be of interest to a wider range of audience.", "rebuttal": "Dear Reviewer 5tif,\n\nThank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.\n\n---\n\n>**W1**: Only the vanilla SD is tested, so it is not clear whether AdaSPEC could generalize to more advanced SD methods like Eagle.\n\n**R1**: Thank you for your insightful comment. To address this, we implemented AdaSPEC with Eagle—a representative SD method featuring tree attention that demonstrates superior performance to Medusa. Following the standard practice in the Eagle paper [1], we trained the Eagle decoding head on the ShareGPT dataset using the 3-epoch setting from our work and evaluated generation speed on MT-bench. The results below demonstrate AdaSPEC's effectiveness and generalizability beyond vanilla SD to more advanced SD methods like Eagle.\n\n*Table R1.* Vicuna v1.3 7B with 3-epoch ShareGPT. Here, train accuracy refers to first-generated-token accuracy in the training set.\n\n|                                 | **Eagle** | **Eagle + AdaSpec** |\n| ------------------------------- | --------- | ------------------- |\n| Train ACC ↑                     | 75.3%     | **76.3%**           |\n| Generation Speed (s/sentence) ↓ | 8.85      | **8.06** (-8.9%)    |\n| Generation Speed (tokens/s) ↑   | 63.48     | **68.21** (+7.45%） |\n\n---\n\n>**W2**: The only baseline is DistillSPEC. The evaluation is limited in scope. The discussion of related work is limited.\n\n**R2**: We appreciate the reviewer’s feedback and the opportunity to clarify. As of now, DistillSpec is the only existing work that applies model distillation to train the draft model in the speculative decoding (SD) framework, and it is widely regarded as the state-of-the-art method in this line of research. Therefore, we believe our comparison to DistillSpec is both appropriate and meaningful within the scope of distillation-based SD methods.\n\nMoreover, our results demonstrate not only superior decoding efficiency under the standard SD architecture but also stronger generalization across models of different sizes. We have also highlighted that **our approach is orthogonal to other classes of SD techniques** (e.g.adaptive rejection, or sampling strategies), and can in principle be combined with them to further enhance performance. For example, our experimental results in R1 above demonstrate that our methods can be effectively integrated into the EAGLE framework.\n\nRegarding the discussion of related work, we have included comparisons to both speculative decoding and distillation literature in Section 2. We agree that additional connections—such as to broader acceleration strategies—may be further explored in future iterations of this work.\n\n---\n\n>**W3**: There is a huge gap between 3-epoch and optimal-epoch performance. A more advanced hyper-parameter selection strategy could have been proposed to optimize the performance.\n\n**R3**: Please refer to Q3 for this.\n\n---\n\n>**Q1**: Can AdaSPEC generalize beyond vanilla SD to more advanced methods like Eagle or Medusa?\n\n**R1**: Yes. Please refer to W1 for the corresponding results. \n\n---\n\n>**Q2**: Could more relevant baselines be included, like Rho-1 mentioned in the paper?\n\n**R2**: Though AdaSpec resembles Rho-1 in the token selection stage, the two studies are different. Specifically, Rho-1 focuses on the pretraining of language models, which is not relevant to our study. We have no idea how AdaSpec could be compared with Rho-1.\n\n---\n\n>**Q3**: Considering the significant gap between 3-epoch and optimal-epoch performance, why wasn’t a more effective hyperparameter selection strategy proposed to improve results?\n\n**R3**: First, we have already tuned our hyperparameters to their optimal as much as possible within each setting. The hyperparameters can be found in the appendix.\n\nImportantly, the goal of our work is not to maximize absolute performance via exhaustive hyperparameter tuning, but rather to evaluate and compare the effectiveness of AdaSpec across different realistic training scenarios. As discussed in Section 4.2, we deliberately introduce two representative training settings to simulate common use cases:\n\nThe **3-epoch setting** simulates a realistic case where the draft model has been *partially trained* and already performs reasonably well on the target domain. This reflects a typical ablation setup, where forgetting is limited and generalization is already strong.\n\n\n\nThe **optimal-epoch setting** aims to approximate the *best achievable performance* on the target domain with further training, serving as an upper bound for model capability.\n\nIn addition, the **3-epoch setting also aligns with our mixed-task experiments in Section 4.6**, where we train models sequentially on MBPP and then GSM8K (each for 3 epochs) to test AdaSpec’s ability to retain previously acquired knowledge. As shown in Table 6, AdaSpec is effective in preserving the model’s original capabilities under such limited training regimes, further supporting the design choice of the 3-epoch setting.\n\nFinally, we note that AdaSpec consistently improves performance under both settings, demonstrating strong generality. Importantly, these gains are achieved **without task-specific hyperparameter tuning**, highlighting the robustness and practicality of our approach.\n\n[1] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty"}]

Author rebuttal:

Meta-review: This paper introduces AdaSPEC, a novel selective knowledge distillation method for optimizing speculative decoding in large language models. The key contribution is a two-phase distillation process that uses a reference model to identify and filter out "hard-to-learn" tokens, allowing the draft model to focus its limited capacity on mastering "easier" tokens. This approach leads to better alignment between the draft and target models, resulting in higher token acceptance rates and, consequently, faster inference speeds. The method is evaluated across a range of tasks, including arithmetic reasoning, instruction-following, coding, and summarization, demonstrating consistent improvements over the state-of-the-art DistillSpec method.

The paper received consistently positive reviews, with final ratings of 5, 5, 5, 4, 4. Reviewers generally expressed high confidence in their assessments, with scores. There were no significant conflicting viewpoints among reviewers; initial concerns were addressed during rebuttal, and several reviewers raised or confirmed positive scores. All agreed on the paper's strengths and the importance of its contribution.

All reviewers pointed out the novelty and timeliness of the proposed approach, the clarity of the writing, and the extensive experimental validation across various tasks and model configurations.
 
Still, there were some initial concerns included the modest empirical results in some settings, the additional computational cost of the two-phase distillation process, and the need for a more in-depth comparison with other speculative decoding methods.

While not a theoretical paper, the underlying intuition of focusing on "learnable" tokens is well-motivated and supported by the empirical results. The experimental evaluation is thorough and well-executed, with a diverse set of tasks and model pairs. The ablation studies provide valuable insights into the effectiveness of each component of AdaSPEC. The paper also provides sufficient detail for reproducibility, including hyperparameter settings and a clear description of the methodology.

This work has the potential to significantly impact the field of efficient LLM inference by providing a practical and effective method for improving speculative decoding. Furthermore, AdaSPEC is a plug-and-play enhancement to existing speculative decoding pipelines, making it highly practical for real-world deployment.

This is a technically solid paper with a novel and impactful contribution to the field of efficient LLM inference. The proposed method, AdaSPEC, is well-motivated, thoroughly evaluated, and demonstrates consistent improvements over the state-of-the-art. The authors have done an good job of addressing the reviewers' concerns and have made a strong case for the practical applicability of their work. 

This paper will be of great interest to the NeurIPS community and I am pleased to recommend it for a spotlight presentation.
Decision: Accept (spotlight)