OmniDraft: A cross-vocabulary, online adaptive drafter for on-device speculative decoding
Authors: Ramchalam Kinattinkara Ramakrishnan
Zhaocong Yuan
Shaojie Zhuo
Chen Feng
Yicheng Lin
Chenzheng Su
Xiaopeng Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Speculative Decoding
LLM
Online Adaptation
Distillation
Tokenization
Inference Optimization
Fine-tuning
LoRA
Primary area: deep_learning

Abstract: Speculative decoding generally dictates having a small, efficient draft model that is either pretrained or distilled offline to a particular target model series, for instance, Llama or Qwen models. However, within online deployment settings, there are two major challenges: 1) usage of a target model that is incompatible with the draft model; 2) expectation of latency improvements over usage and time. In this work, we propose OmniDraft, a unified framework that enables a single draft model to operate with any target model and adapt dynamically to user data. We introduce an online n-gram cache with hybrid distillation fine-tuning to address the cross-vocabulary mismatch across draft and target models; and further improve decoding speed by leveraging adaptive drafting techniques. OmniDraft is particularly suitable for on-device LLM applications where model cost, efficiency and user customization are the major points of contention. This further highlights the need to tackle the above challenges and motivates the “one drafter for all” paradigm. We showcase the proficiency of the OmniDraft framework by performing online learning on math reasoning, coding and text generation tasks. Notably, OmniDraft enables a single Llama-68M model to pair with various target models including Vicuna-7B, Qwen2-7B and Llama3-8B models for speculative decoding; and additionally provides up to 1.5-2x speedup.

OmniDraft: A cross-vocabulary, online adaptive drafter for on-device speculative decoding
Ramchalam Kinattinkara Ramakrishnan" Zhaocong Yuan". Shaojie Zhuo', Chen Feng', Yicheng Lin², Chenzheng Su Xiaopeng Zhang?
Qualcomm Al Research f ('rkinatti, ?zhaocong, ?shaojiez, 'chenf, "yichengl, cchnnzhen, 'xiaopeng) qumiimm.coomm.. com
Abstract
Speculative decoding generally dictates having small, efficient draft model that is either pretrained or distilled offline to a particular target model series, for instance, Llama Qwen models. However, within online deployment settings, there are two major challenges: 1) usage of target model that is incompatible with the draft model; 2) expectation of latency improvements this work, we propose Omnilraft, a unified framework that enables a single draft model operate with any target model and adapt dynamically user data. We introduce an online n n-gram cache with hybrid distillation fine-tuning to address the cross-vocabulary mismatch across draft and target models; and further im- prove decoding speed by leveraging adaptive drafting techniques. OmniDraft is particularly suitable for on-device LLM applications where model cost, efficiency and user customization are the major points of contention. This further highlights the need tackle the above challenges and motivates the "one drafter for all" paradigm. We showcase the proficiency of the OmniDraft framework by perform- ing online learning on math reasoning, coding and text generation tasks. Notably, OrniDraft enables single Llama-68M model to pair with various target models including Vicuna-7B, Qwen2-7B and Llama3-8B models for speculative decoding; and additionally provides up to 1.5-2x speedup.
usage and time. In
Introduction
Unlike traditional auto-regressiv generation LLMs, speculative decoding (SpD) 25 9 offers a unique advantage accelerate LLM inference by decoupling the generation phase and verification phase. Speculative decoding generally requires a small but efficient draft model and a large target model. The draft model generates sequence of proposed tokens to be verified by the target in one shot, amortizing the target model's 's memory bottleneck in batch inference and attaining better tokens per second throughput. The speedup factor relies not only on the predictability of the generated text like commonly occurring phrases, but also on the alignment between draft and target model. As such, common practice is to utilize draft and target models from the same model family given their consistency in pretrained data, tokenization and training configurations. Alternatively, one might consider distilling a target model into smaller model serve the drafter [53] 30), which still follows the same principle of better alignment leading to greater speedup.
The tight coupling of draft and target models limits flexibility of model selection and creates additional overhead for draft model distillation and maintenance, especially when deploying LLMSs at scale
"Contributed equally
'Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

'Snww'
"kee
liss
'llkee
(d 92)
(d,,9))
(1lp.9))
Cross- vocabalary Translator
<bos>
(Ga-th) (u.qi) (5x.05)
Deshar
Draiter
Drafter
Target
N-gaam
Y, Tal, Na lake'
Croek-socetbuuee distllaiinn
(GaPP) (4s.Pi) (CaiP2) (fsiPs)
'Snow'
'Ilake' 'stamps"
Figure 1: Overview of the OrniDraft framework: during cross- vross-vocabulary speculative decoding, the drafter (Llama-68M) generates multiple tokens d; with corresponding distributions d-- Cross- vocabulary translator then converts the drafter tokens into tokens the vocabulary of the target model (Llama3-8B). In this example, token (o""now") and da i are directly mapped target tokens to and f2, while token (Cf"), d2(")a") and da("ke") are merged into a single target token & ( ("flake"), since there is a mapping item the n-gram cache. The translated proposal 6, along with combined probabilities of is verified by the target model, resulting in to and t being accepted while ty being rejected and replaced by tz2. The target outputs tokens and their probabilities pt are translated into drafter tokens and sent back to drafter for next round of drafting. The gram cache is updated by inserting new unseen item ('st', 'mmms's >>'ssmmps') Meanwhile, the accepted and corrected tokens from the target model are used to align the drafter through online cross-voca distillation.
across diverse hardware platforms and updating target models overtime. It is compelling use universal lightweight model on-device to draft tokens for a broad range targets models. This universality greatly simplifies deployment, facilitates easy optimization, and allows for rapid model updates. Furthermore, the heterogeneity of on-device and cloud hardware presents an opportunity for hybrid speculative decoding. This enables users choose between running any target model locally or on the cloud, balancing model performance, inference cost, and privacy concerns.
However, building universal drafter for speculative decoding presents several unique challenges. Firstly, different family of target models might use tokenizers with different vocabularies. This is natural since target models typically trained with massive pretrained data and hence larger vocabulary is needed to include higher-order n-grams or BPE 7 merges. As a result, the vocabulary mismatch breaks the speculative decoding formulation where the draft and target model need to evaluate probabilities over the same set of tokens. A previous work UAG (45) addresses vocabu- lary mismatch with intermediate translation layer, but it primarily deals with tokens within the intersection of the drafter and target vocabularies which might "falsely" reject good tokens from the drafter. Secondly, independent training of the drafter and target models can result in misaligned pre- dicted token distributions, which reduces the acceptance rate during verification. This misalignment diminishes the efficiency benefits of speculative decoding. Although existing research [53] 30 (3]
has explored techniques to improve alignment between the drafter and target, the alignment is often done offline with assumption of a fixed target model. practice, the target model may change due to user personalization or tasks switching, further complicating the aligmment issue. Additionally. edge devices usually have limited memory, compute capacity, and power budget. The drafting process must therefore be efficient to maximize the benefits of speculative decoding.
To address the challenges, we propose scalable speculative decoding framework, named OmniDraft, centered on an on-device universal drafter that generates draft tokens for a wide variety of target models. An overview of OmniDraft is shown in Figure [ To tackle the vocabulary mismatch issue and enable cross-vocabulary speculative decoding, we introduce an n-gram cache to store cross-vocabulary mappings from draft tokens to target tokens. By integrating the n-gram cache into the speculative decoding algorithm, we alleviate vocabulary mismatches and achieve higher acceptance rate for future queries. We further employ online knowledge distillation to improve alignment between the drafter and target. A hybrid distillation loss, combining token-level and distribution level objectives, updates the drafter using the target's accepted and corrected outputs. This enables continuous alignment during speculative decoding, even when the target model changes due to personalization or task switching. To further improve runtime efficiency, we incorporate online adaptive drafting, where the drafter dynamically adjust the number of tokens it proposes

---

based on predicted confidence. The adaptive drafting balances generation cost and acceptation rate, maximizing throughput under device constraints.
Overall, our OmniDraft framework enables a robust, efficient and flexible speculative decoding system with universal drafter for on-device applications. Through extensive experiments, show that a single Llama-68M draft model can be paired with various target models including Vicuna-7B, Qwen2-7B and Llama3-8B models for voossboocab speculative decoding and provides up to 1.5-2x speedup on reasoning, coding and text generation tasks.
vocabulary speculative decoding via online n- n-gram cache that translates between drafter and target vocabularies, enabling speculative decoding across models with different tokenizers; (2) We introduce online knowledge distillation with hybrid alignment loss, which updates the drafter using accepted and corrected outputs from the target model to improve alignment and acceptance rate over time; (3) We integrate alignment training with adaptive drafting, allowing the drafter to dynamically adjust draft length based on alignment confidence for improved efficiency and speedup.
Contributions (I) We propose
Related work
Speculative decoding The idea of speculative decoding is proposed and formalized in the pioneer works of [25] (]. Subsequent works have centered around optimizing different components of the speculative decoding framework. Some have highlighted tree attention to facilitate simultancous verification of multiple draft sequences such SpecInfer [32), Medusa [8] and Sequoia [Л. There is also focus on more efficient drafting by using retrieval- based methods as in Lookahead [16] REST [19 NEST 26). RASD (38] or by using dynamic length drafting in BiLD (2). DISCO [1].
AdaEDL 2 SpecDec++ 21, EAGLE2 [28].
More recent works explore speculative decoding in the context of vocabulary adaptation like UAG
4 and Adapti Vocab (35), long-context tasks like LongSpec [50]. MagicDee [41], or more efficient
draft models as in EAGLE 27,, Speculative Streaming [5], and self speculative decoding as in [51]
14
48
Distillaton
Model distillation in LLMs is crucial for speculative decoding since quality of the draft
model dictates the final speed-up. Early works on sequence model distillation include [ 23 46) There are also recent works that have specific focus on LLM distillation such as MiniLLM [8 and
GKD []. Most closely aligned to our setting are the works of DistillSpec [33 and OSD 30 that aim
towards training better draft model to given target. Another related line of works is distillation on different tokenizers as in 6
Online adaptation Compared to model finetuning on a fixed offline training set over multiple epochs, online adaptation focuses on continual learning or few-shot generalization to new data, Frameworks such as ProtoNet [42 and MAML [15] address the few-shot learning problem. Robotics and reinforcement learning also offer insights with works like DAGGER 40, RI.22 [3] and PEARL 33 for continuous adaptation to new tasks. There are also works that emphasize online adaptation
for LLMS as 20 43 29.. Lastly speculative decoding, OSD (30 explicitly optimizes for draft
model online adaptation which is closest to ours.
3 Methodology
Notation Assume small draft model M, and a large target model Mp. let p(y x Wee) and q(yelz æ. Veer) be the distributions of next-token predictions at time step t for M, and M, respectively, where represents the prompt prefix and x, yet represents the context time step For convenience, we use p(ya) and (()e) as shorthands for p( (/l12 (2, Vee) and q(ve (x, y<e) for the remaining sections. denotes the number of tokens proposed by the drafter M, and p (3(+4) norm norm(max(() p(y++s)
(()++))) is the residual distribution for the resampling, as per the original SpD algorithm 25
53 30 shows that better alignment between the draft and target model gives higher acceptance rate and hence higher speedup speculative decoding. To align them, some common distillation losses include supervised finetuning (FT) sequence level Knowledge Distillation (KD) (SFr(0)
3

---

((x.))-(x.)) log qo [)], and supervised KD (0)( = (...)) (P(M, (MM with selected choice of divergence metric D.
3.1 Cross-vocabulary N-gram Cache
Normal speculative decoding is infeasible when the draft vocabulary VG and target vocabulary Vp are different, since the rejection scheme relies on acceptance ratios evaluated on the same token -(
min(1, p(yt++)/ (3(((+)))))))) and the residual distribution norm(max(( per-token probability differences. This foreshadows two issues: 1) tokens without direct mapping between the drafter vocabulary and target vocabulary cannot be handled i.e. the drafter can propose tokens not recognized by the target or vice-versa. 2) target can "falsely" reject good tokens from the drafter. This occurs when the drafter proposes sequence of (sub-)toker token/n-grant -raangramm in target, but target rejects the sequence since it prioritizes the merged token over the prefix sub-token. This a byproduct of the tokenization process that optimizes for the longest token in the vocabulary or sequentially applies merge rules to get the longest possible token (147) 24] 77).
also requires
that constitute merged
UAG [55 addresses the first mismatch with translation layer between the drafter and target and derives the intersection of draft and target vocabularies, where tokens have direct mappings. During proposal stage, UAG suppresses tokens outside of the intersection and converts drafter token ids to target ids with the mapping. After verification, if a token without direct mapping is sampled, the translation layer will invoke the target and draft tokenzier map the target token to sub-tokens in draft vocabulary. However, UAG cannot solve the second mismatch meaning it only guarantees feasibility of cross-vocabulary speculative decoding but lacks optimality.
To overcome this, we propose to build a cache n-grams C that tracks the instances of target-draft token translations. Denote target tokens E Vp and drafter tokens d; E Vg. then the n-grams cache is C {(4, [d)]j=i=))] where each element represents a mapped n-gram instance given the matching context so far ctzy di. ................. d' tokenize, (detokenize, (ctz,, tt)). In inference time, we add a postprocessing (pp) stage at the translation layer, where we scan over the proposed draft tokens de,- .dckk-1 and merge sub-tokens that hit the n-gram cache. The resulting new sequence le, ... femm and their draft probabilities of (4e),.-,;)) (f44)) will be under the target vocabulary space and follow the mapping rule
t, (t.))tt)= g(d:) 1o) C), g(d))
if direct mapping, 4 <> d, otherwise
(1)
This cross- vosabuoocyy mapping translates the draft sequence to the target vocabulary space, ensuring speculative decoding still functions well with per-token acceptance ratios min(1, p(t2+:)/q / (te+:)). From the perspective of the final matched text, p(4) and 11, q(d)) would be the probability of producing that specific chunk of text in their respective tokenization space.
However, in the correction stage we require the full distribution for the residual distribution instead of point-wise evaluation of probabilities, which is infeasible since p(-). q(-) work on different space. Hence enhance the mapping rule as
II ((d)) vt E Vp, d((1) = 9(d)) II; a(d)) g(t)
if n n-gram mapped, t ++ [d]ii=ım prefix sub-token of n-gram, t dj otherwise
(2)
We use the mapped probability for the selected n-gram but adjust the prefix sub-token probability by subtracting the n-gram probability. This can be seen as approximately re-allocating the original probability mass assigned prefix sub-token di under the draft distribution q(-), between the "new"
g-gaam token and the prefix sub-token under the modified draft distribution g' (-).. This is also related to the known problem of tokenization bias (37]. Using mapping [ we least ensure point-wise approximate correctness of the n n-gram token for the residual distribution norm (max(0, p(te+s)
())) Note that we can apply to the n-grams sampled and matched from the current speculative round. Evaluating all other n-grams require re-running drafter at the step of rejection which is impractical. We summarize the modified speculative decoding with our proposed mappings in Algorithm

---

E X Algorithm Cross-vocabulary Speculative Decoding
1: Given draft model q()), target model p(-). n-gram cache C
2: Given draft length max length T, prompt a
3: Initialize 1 <- 0, clayg clup <-
4: while <
5: for : do
6: Sample draft auto-regressively de+s ~ ((dets)ctxy det+s)
7: end for
8: Apply translation mapping to get m proposed tokens and draft probabilities target space
9:
f..... ft+m-1+ o (te) q' (f++m-1)
10:
In parallel, compute target probabilities on mapped tokens
11:
p(4+), ... ,(((++m)
12:
Apply rejection sampling with acceptance ratios min(1,p((4+:))) p(le+1)// (4++4) and correction
residual distributions norm(max(() p(12+1) (1()))))) to get accepted tokens
13:
4. ... 4+n-1 for some accepted length
if m then
15:
Sample free token 4+mm p((++m) and add to accepted tokens, n + m +
end if
17: Apply reverse translation to get accepted tokens in draft space
ctTy de, ditp-1 tokenize, (adenniez(detokennize,, (detokenize,) (ctz,, fer , ,t(nn-1)
19: Add to n-gram cache C if there exists unseen n-gram instance
<- + n, update clitgg, clap
21:
end while
22:
Return results
3.2 Cross-vocabulary Distillation
Using the n-gram cache with the approximate distribution mapping helps draft and verify n-gram tokens as if operating under the target vocabulary directly. To extend for online adaptation, we propose hybrid distillation framework that progressively aligns the draft and target model on both direct mapping tokens and n-gram tokens. Given the online setting, we have limited access to the target model we distill on the draft model generated data, simply on-policy data similar to GKD [. We employ reverse KL on direct mapping tokens for richer supervision signals, but use maximum log-likelihood (NLL) on n-gram tokens since we only have reliable point-wise evaluation of probabilities on those tokens. Overall, our proposed hybrid distillation loss is
Leross, vocab, vistb_distil (0) (D) + NC (0)
(3)
DKL (p)(tl//)(m/(] A log qo (d. (di)
(4)
1.44 mappingg(d..g)
where IDM, In-gram the indicator functions to identify if current token is part of direct mapping or n-gram; this implemented as binary masks in practice. Note that the KL loss term corresponds to direct mapping token ++ d, and divergence is computed in the target vocabulary space given q()) is elevated to y' (-) via the translation mapping. The NLL loss term however operates in drafter vocabulary space to increase likelihoods of drafter tokens which constitute n-gram accepted by the target during inference.
The parameter À can either be hyperparameter to account for ratio imbalance between direct mapping tokens and n-gram tokens, or can be a dynamic weight such as the verified target probability of the n-gram. The latter leads to a loss term of X (!) = -p(;;) log go (d.(z), which can be
treated as the point-wise KL evaluated on the n-gram token.
Moreover, it is possible to extend the NLL or point-wise KL loss to an approximate KL loss using mapping as LN. = Dx r. (g/ (4+))) [[(])())). This is equivalent to using KL on the intersection tokens plus the n-gram and it's first sub-token. Due to the additional components on the intersection tokens and the fist sub-token, this approximate KI. loss can provide richer learning signal. Empirically only observed minimal improvement from the approximate KL loss, as shown in section 4.3.4
5

---

3.3 Online Adaptive Drafting
One observation in performing cross-vocabure vocabulary speculative decoding is that we are implicitly shortening the proposal draft length, since multiple sub-tokens map to single n- n-gram token. We then explicitly incorporate adaptive drafting to gain even better speedup for on-device speculative decoding focusing on the same vocab setting. We adopt the framework in SpecDec++ [21] where a lightweight head network f+(+) predicts the acceptance rate of the current proposed token. The acceptance prediction head takes the embedding of the proposed token e; as input, and is trained using weighted BCE loss aadp with acceptance ratios min(1, p(u:)/ //(()) as labels. then controls if to early exist based on the cumulative probability of at least one proposed token getting rejected and a given stopping threshold 7.
P(y accepted| y<r accepted) = sigmoid(fo(e:))
(5)
c
P(31 sisk, s.t. Bi rejected) = 1 II
P(y. accepted yei accepted)
(6)
:=1
P sisk, s.t. Y: rejected) >> exit
(7)
However in online adaptation, the labels are subject to change since the draft model is continuously finetuned with distillation loss Ldistil to align with the target, which could cause distribution shift for dynamic drafting.
We propose two variants of online adaptive drafting. The first one performs draft model alignment and acceptance prediction head training jointly at each update step Ljoin = Caisun + Ladapt. The second one interleaves two trainings such that we perform multiple acceptance prediction updates per draft model alignment update, aiming to mimic slowly moving labels to reduce distribution shift. Unlike the first variant that performs both updates jointly using the online data batch, for the second variant we keep a larger buffer for the acceptance prediction updates which includes data from previous batches. This helps to enhance training stability and adaptation speed for the acceptance prediction.
4 Results
Models To show the efficacy of OmniDraft on the setting of a single drafter for multiple targets, we fix the drafter to be Llama-68M [32) and the target model to be Llama3-8B [17], Qwen2-7B [49] for the cross vocabulary results, as well as Vicuna-7B (52) for the same family vocabulary evaluation.
Tasks We perform online distillation across 4 tasks: GSM8K [12, Alpaca 44] XSum [36]and a combined Human 4 10 datasets. Each task has dedicated train and test or we slice out the a portion of the train set as the test set. For the MrPPP+HumnEval,, we combine the two datasets to add some more diversity to the data for the coding tasks. The training conducted for specific number of steps (<1 epoch) across each of the tasks as per general online adaptation setting. All the tokens will contribute the loss calculation including the tokens that were accepted by the target as they would provide option for improved alignment between the two models. Moreover, all experiments are performed with temperature 0.01, unless specified otherwise. We also include the setting for online adaptation of the drafter using LoRA across all the tasks. Using dynamic adapter switching we can pair the same drafter with any target across any of the task which is the ideal scenario for on-device online speculative decoding.
Evaluation Metrics
Speedup- Walltime acceleration rate, measures the improvement in throughput. We follow Medusa's 's [8] convention.
Acceptance Rate- Ratio of accepted tokens proposed tokens averaged over speculative decoding steps, measures alignment between draft and target model.
Notation We refer to SpD dM as the baseline speculative decoding which uses direct mapping be- tween vocabularies. N-gram postprocessing (pp) refers to using the N-gram eache as a postprocessing technique without directly training on it as we proposed in Algorithm | N-gram hit refers to the average number of successful N-gram cache lookups that were accepted by the target per speculative decoding step.

---

4.1 Cross-vocabulary Online Distillation
Table shows the results on the test set after training. Across all the tasks, LDM + gr. approach perform better than training only on LDM- Overall, this indicates that the additional LN gram plays significant role in improving the acceptance rate. Moreover, LD LN-gram with LoRA finetuning performs reasonably well across all of the tasks when compared to the baseline. The largest speedup is obtained for the GSM8K dataset for both the target models, with XSum being the least improved task. We observe XSum could also be improved by increasing the number of training samples scaling effect. Figure portrays the training dynamics across all tasks for both the acceptance rate and speedup metrics. For all the experiments, The instability seen in some of the LoRA curves could indicate that training apper-parameter's are not fully optimized or that training plateaus quicker for certain tasks, which could also explain why there still a small gap between the LoRA and the full finetuned model performance.
the metries improve training proceeds.
Table 1: Performance Cross vocabuaoyyy Distillation with Llama-68M, and two different targets
GSMBK
MBPP-HumanEval
Alpaca
XSum
Target
Method
Acc Rate
Speedup
Acc Rate
Speedup
Acc Rate
Speedup
Acc Rate
Speeduup
SpDpM
0.10
0.94x
0.09
1.03x
0.09
0.96x
0.11
0.91x
Llama3-8B
0.32
1,58x
0.22
1,26x
0.16
1.25x
0.20
1.20x
Com +hCNzman
0.42
1,70x
0.27
1,33x
0,20
1.30x
0.24
1.24x
ANNigon LoRA
0.37
1,59x
0.19
1,28x
0.17
1,21x
0.23
1.21x
SpDpx DM
0.14
104x
0.09
0.91X
0.13
1.01x
0.12
0.96x
Qwen2-78
Lus
0.33
1.50x
0.22
1.29%
0.17
1.25%
0.19
1.16x
Casa ACNgrm
0.37
1661x
0.26
1,36x
0.20
1,30x
0.22
1.22%
Cos + +ACr
LoRA
0.31
1.41x
0.21
1.21x
0.18
1.25x
0.22
1.21X
GSMMIK
MBPP Humant
Aloace
X5um
Spcco.o
1.5
1.6
D60
DM ngnem-cacte
Loca DM rggaamcccchee
500
Figure Cross-vocabulary SpD online distillation on Llama-68M with Qwen2-7B as target
4.2 Online Adaptive Drafting
Table shows the test results of online adaptive drafting with ablations to the different training variants. Across all tasks we observe a consistent increase acceptance rate. This is the combined effect of better model alignment due to distillation, and also the acceptance prediction head learning to exit early when there is a higher chance for proposal rejection
In terms of speedup, we can also see improvement in most tasks except on GSM8K where the distill-only baseline outperforms both adaptive drafting variants. This could be due to the latency reduction over the number of the proposed tokens are not sufficient in comparison to the increase in successful accepted tokens with longer draft length. We hypothesize it could be due the difficulty in training the acceptance prediction head in online setting. This involves optimizing with respect to

---

changing labels and training only on incoming data without stable corrective feedback compared to offline training. By end of the one epoch training. the acceptance prediction head might not have converged well to align with the current distilled drafter, leading to sub-optimal performance.
Finally, we also observe the interleaved variant has better speedup than the joint variant on average (similar in Alpaca and Xsum, larger increment in GSM8K and MBPP++HumanEva)) But the joint variant has higher acceptance rates over all tasks, indicating it might be underestimating the acceptance probability, leading to wrongful early exit. This could be a direct consequence of the previously mentioned training challenges, which alleviated with the interleaved variant with more stable training.
Table 2: Performance Online Adaptive Drafting with LJama-68M and Vicuna-7B.
GSMBK
MBPE-HumanEval
Alpaca
XSum
Target
Method
Ace Rate
Speedup
Acc Rate
Speedup
Acc Rate
Speodup
Acc Rate
Spoodup
SpD
0.21
1.4xx
0,20
1.4xx
0.20
1.42x
Vicuna-7B
Adapt Only
0.38
1.50x
0.28
1.34k
0.38
1.52x
0.38
1.51x
Distill Only
0.42
2.20x
1.92x
0.25
1.57x
0.23
1.53x
Joint Distill Adapt
0.61
2,08x
0.51
1.918
0.44
1.61x
0.42
1.59x
Interleaved Distill Adapt
0.52
2.15x
1.9x
1.60x
1.58x
4.3
Ablations Studies
4.3.1
Scalability to Larger Target and Draft Models
We perform an ablation on scaling the target and draft model and compare the various metrics and whether it follows similar trend. As shown in Table we used a new family of LLMs, the Qwen2.5 series of models of larger sizes including 7B, 14B and 32B parameters. Datasets used are GSM8k and HImPP+HumanEyall We also performed analysis a larger drafter of size 160M parameters. The performance on these larger models are consistent with our previous results, with larger target models resulting in 2x improvement strong scalability as the size of the target LLM increases, as shown in the table. While the gap between the drafter and the target model grows with model size, the drafter remains the limiting factor: During training, the drafter progressively aligns with the target; however, once it reaches its alignment capacity, further increase in target size continues to yield speedup improvements, while the acceptance rate plateaus. Regarding the larger drafter, the Llama- 1aama-160M parameter model was chosen after taking into consideration the increased latency of around 3x of the drafter model compared to Llama-68MM 1.... Overall, the larger drafter model does introduce a reduction in the overall speedup across all target models, however, due its enhanced potential capability, the acceptance rate does improve across all the target models.
the GSM8K dataset. The Omnidraft framework demonstrates
4.3.2 N-gram Cache Memory Footprint
We also analyzed the various cache size at the end of training for each of the reported tasks as shown in Table Overall n-gram cache size across tasks remain relatively small compared to the size of the draft or target model. This indicates potential feasibility for on-device setting given the small overhead.
4.3.3 Effectiveness of N-gram Cache
In this section, we focus on the impact the N-gram eache to different training variants as per Table 5) We perform ablation on subset of GSM8K dataset (4k samples) across multiple techniques. It can be seen that the baseline SpD performs poorly which indicates the mismatch between the drafter and the target model alignment for the pretrained models. We also capture all the possible N-gram matches during the SpD pM baseline for the train set, and then use the cache as a lookup for an improved baseline in the SpD SpDpM* DM + N-gram pp As such there is a cache hit of 0.87, which indicates that the baseline model without any pretraining can still benefit with the N-gram cache. We can
also train the model to improve the alignment using Lum and although without the N-gram, still see a large improvement over the baseline. Moreover. when we train with N N-gram pp> there is a further improvement on the overall speedup. Finally, we train using both Lpm + AC which provides the best results across all techniques different draft length k.
8

---

Table 3: Comparison of Cross-vocabulary Distillation Performance with Llama-68M and Alama-160M,
three target models
GSM8K
MBPP+HumanEval
Target
Method
Acc Rate
Speedup
Acc Rate
Speedup
SpD, DM (68M)
0.15
1.02x
0.10
0.94x
LDM +NLN gram (68M)
0.401
1.66x
0.27
1.33x
Qwen2.5 7B
SpD DMM (160M)
0.18
0.70x
0.13
0.60x
LDM +NLN gram (160M)
0.47
1.12x
0.32
0.90x
SpD DM (68M)
0.15
1.17x
0.10
1.14x
LDM +NCN gram (68M)
0.407
1.92x
0.272
1.57x
Qwen2.5 14B
SpD DM (160M)
0.178
0.89x
0.13
0.84x
LUM +X gram (160M)
0.472
1.40x
0.33
1.19x
SpD DM (68M)
0.153
1.30x
0.10
1.23x
LDM +NLN-gram (68M)
0.42
2.05x
0.274
1.71x
Qwen2.5 32B
SpD DM (160M)
0.187
1.03x
0.133
0.97x
LDM +XLN gram gram (160M)
0.49
1.62x
0.335
1.40x
Table Summary of final N-gram cache size across tasks with Llama-68M drafter Qwen2-7B target after online inference and distillation. Cache memory (MB) is derived from pymplerasizzo...szzzlll.
GSM8K
MBPP-HumanEval
Alpaca
XSUM
Training Samples
7473
910
8000
4000
Cache Size (#n-grams)
5569
2238
20339
17013
Cache Memory (MB)
1.372
0.501
4.569
3.924
4.3.4 Distillation Loss Comparisons
The different loss variants also provide diffferent levels of performance on the test set as shown in Table 6 When trained only on the Lx-gram- the training is very unstable. This could either be due to the number of n-grams being substantially smaller than the direct mapping tokens within most datasets, or the training requires additional constraints to direct towards minima. Consequently. training on the combined loss provides better performance metrics across temperatures as well. The final LPM KL + XLN KL, 2 provides slightly better results on temperature = indicating the impact of the additional KL over the intersection vocabulary which is beneficial for the sampling process. However, we noticed tuning the scaling factor becomes critical and hence we use Equation for all of the experiments since it was much more stable across all tasks. We fix a a 0.2 for all the tasks, across all experiments.
4.3.5 Adaptive Drafter Initialization and Thresholds
Two additional aspects in training and using adaptive drafting are the acceptance prediction head initialization and the stopping threshold for early exit. Before the joint training of model distillation and adaptive drafting, we can pretrain the acceptance prediction head on offline dataset, while keeping the drafter fixed. This should idcally capture some priors the draft draft-target alignment and provide better initialization for the online joint training. To verify this, we pretrain the acceptance prediction head on the Alpaca dataset for and epochs respectively, and use the two checkpoints as initialization for online training. We observe mixed results where the pretrained initializations harm performance on GSM8K and XSum, and only improve on MBPP+HumanEva by small margins. We suspect MumP+HumanEva has closer affinity data distribution to Alpaca while the other two do not, which leads to negative transfer with pretraining.
The stopping threshold in adaptive drafting also plays key role for speedup performance. In Alpaca and XSu we observe a conservative threshold 7 0.3 sufficient to attain better speedup than basclines, while in GSM8K and MBPP+ HupannEaaaEval need more relaxed threshold

---

Table 5: Effect of n-grams cache with Llama-68M and Llama3-8B GSM8K (subsel)
Metrics
SpD
SpD N-gram pp
LDM
LDM + N-gram
LDM + XCX-gram
=
Acc Rate
0.16
0.20
0.40
0.42
0.46
Speedup
1.04x
1.16x
1.59x
1.61x
1.66x
Avg n-gram hit
0
0.87
0
0.48
2.40
Acc Rate
0.12
0.16
0.32
0.35
0.41
Speedup
1.01x
1.11x
1.51
1,54x
1.61x
Avg n-gram
1.49
0
0.75
3.66
Table 6: Training Loss comparisons with Llama-68., and Llama3-8B on GSM8K (subset)
Metrics
LN gma NLI.
LDM NII. + La NLI. LDM KT. +CN- NLI
LOM + Lx.
Temperature = 0.01
Ace
0.090
0.368
0.375
0.376
Speedup
0.86x
1.51x
1.57x
1.57x
Temperature =
Acc Rate
0.070
0,271
0.273
0.275
Speedup 0.79x
1.23x
1.27x
1.31x
y = 0.7 to achieve comparable speedup. We also observe applying a relaxed threshold on model trained on stricter threshold improves speedup performance, indicating adaptive drafting could be prone to underestimating the maximum acceptance draft length. It also suggests the actual stopping threshold to be used for inference should be chosen based on the task and then adjusted based online performance.
4.4 Limitations
While most of our results indicate the potential of our methodology and the OmniDraft framework. some potential limitations still require additional research. 1) Although we are training on incoming single iteration of the data stream, there
data for online adaptation of the drafter, since it is limited still potential for instability on new unseen data. 2) We currently use the full n-gram cache per task per target model, however, should memory become bottleneck, would require optimized cache eviction policy to cater to edge devices. 3) Special tokens that do not have direct mapping currently would require some additional effort handle. Consequently, this would make it less seamless to integrate multi-modal tasks. As part of our future plan of action, although cross-vocabulary speculative decoding already implicitly includes the adaptive proposal length due n-gram merge, we are working towards incorporating an explicit adaptive head in the cross-vocabulary setting.
Conclusion
In this work we propose the OmniDraft framework that leverages n-gram cache and hybrid distillation
loss to enable vross-vocabulary speculative decoding. We show how the draft model can be aligned to different target models with different vocabulary space online adaptation, and we further showease online adaptive drafting to get additional speedup. Our empirical results also show good performance across all metrics. Overall, OmniDraft shows great potential and could pave way to new on-device LLM applications.
10

---

References
[1] Rishabh Agarwal. Nino Vieillard, Yongchao Zhou, Piotr Stanczyk. Sabela Ramos Garea, Matthiu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations, 2024.
[2] Sudhanshu Agrawal, Wonseok Jeon, and Mingu Lee. Adaedl:
Early draft stopping for speculative decoding of large language models via an entropy-hased lower bound on token acceptance probability. In NeurlPS Efficient Natural Language and Speech Processing Workshop, pages 355-369. PMLR, 2024.
[3] Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackic Chi Kit Cheung. Why exposure bias matters: An imitation learning perspective of enror accumulation in language generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 700-710, 2022.
[4] Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models, arXiv preprint or 10v: 07732, 2021.
[5] Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Mohammad Rastegari, and Mahyar Najibi. Speculative streaming: Fast Ilm inference without auxiliary models. In NeurlPS Efficient Natural Language and Speech Processing Workshop, pages 395 413. PMLR, 2024.
[6] Nicolas Boizard, Kevin El Haddad, Céline Hudelot, and Pierre Colombo, Towards cross-tokenizer distillation: the universal logit distillation loss for llms. arXiv preprint 2XX::2242.112300 2024.
[7] Kaj Bostrom and Greg Durrett. Byte pair encoding is suboptimal for language model pretraining. arXiv preprint arXiv:2004.00 03720, 2020.
[8] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning, pages 5209-5235. PMLR, 2024.
Charlic Chen, Sebastian Borgeaud, Geoffrey Irving, lean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelcrating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01. 2023.
[10] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Flizaheth Barnes, Ariel Herhert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders. Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodci, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
[11] Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding. CoRR, 2024.
[12] Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.1110..... 2021.
[13] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI 2: Fast reinforcement learning via slow reinforcement learning. 2016.
[14] Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, al. Layerskip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12622 12642, 2024.
[15] Chelsea Finn, Picter Abbecl, and Sergey Levine. Model-agnostic meta- veta-lcarrnng for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
[16] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang, Break the sequential dependency of I1m inference using lookahead decoding. In International Conference on Machine Learning, pages 2024.
PMLR,
11

---

[17] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint X11:22407221783, 2024.
[18] Yuxian Gu, Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations.
[19] Zhenyu Hc, Zexuan Zhong, Tianle Cai, Jason Lec, and Di Hc. Rest: Retrieval-basee speculative decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 1582-1595, 2024.
[20] Nathan Hu, Eric Mitchell, Christopher D Manning, and Chelsea Finn. Meta-leame leta-learnnng online adaptation of language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 44 4418-4432, 2023.
[21] Kaixuan Huang, Xudong Guo, and Mengdi Wang. Specdec++: Boosting speculative decoding via adaptive candidate lengths. In Workshop on Efficient Systems for Foundation Models II@ ICML2024.
[22] Schoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 39236 39256, 2023.
[23] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 conference empirical methods in natural language processing, pages 1317-1327, 2016.
[24] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: 1808::800000626 2018.
[25] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. International Conference on Machine Learning, pages 19274-19286. PMIR, 2023.
[26] Minghan Li, Xilun Chen, Ari Holtzman, Beidi Chen, Jimmy Lin, Wen-tau Yih, and Xi Victoria Lin. Ncarest neighbor speculative decoding for I1m generation and attribution. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems.
[27] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421-7432, 2024.
Faster inference of language
[28] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning, pages 28935 22948,, 2024.
[29] Juhao Liang. Ziwei Wang, Zhuoheng Ma. Jianquan Li. Zhiyi Zhang, Xiangbo Wu, and Benyou Wang. Online training large language models: Learn while chatting. arXiv preprint arXiv:2403.04790, 2024.
[30] Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, lon Stoica, and Hao Zhang. Online speculative decoding. In Forty-first International Conference on Machine Learning.
[31] Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Naday Timor, Moshe Wasserblat, and Roy Schwartz. Dynamic speculation lookahead accelerates speculative decoding of large language models. In NewrlPS Efficient Natural Language and Speech Processing Workshop, pages 456 456-467. PMLR, 2024.
[32] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rac Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv preprint arXiv:2305.0978) 2023.
[33] Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vulié. Zero-shot tokenizer transfer. arXiv preprint arXiv:240.007883 2024.
[34] Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vulić. Cross-tokenizer distillation via approximate likelihood matching. arXiv preprint arXiv:2503.2008) 2025.
[35] Itay Nakash, Nitay Calderon, Eyal Ben David, Elad Hoffer, and Roi Reichart. Adaptivocab: Enhancing llm efficiency in focused domains through lightweight vocabulary adaptation. arXiv preprint 2rXi::2503.19693 2025.
12

---

[36] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1808.08745 2018.
[37] Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, and Karen Ullrich. Exact byte- level probabilities from tokenized language models for fim-tasks and model ensembles. arXiv preprint arXiv:2410.0988 09303, 2024.
[38] Guofeng Quan, Wenfeng Feng, Chuzhan Hao, Guochao Jiang, Yuewei Zhang, and Hao Wang. Rasd: Retrieval-augmennee augmented speculative decoding. arXiv preprint arXiv:2503.03-4 2025.
[39] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforceemnt learning via probabilistic context variables. International conference on machine learning, pages 5331-5340. PMLR, 2019.
[40] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627 635. JMLR Workshop and Conference Proceedings, 2011.
[41] Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-HSu Yen, Avner May. Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency- latency-throughpu tradcoff for long context generation with speculative decoding. arXiv preprint arXiv:2408. 11049, 2024.
[42] Jake Snell. Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[43] Jihoon Tack, Jachyung Kim, Eric Mitchell, Jinwoo Shin, Yee Whye Teh, and Jonathan Richard Schwarz. On- line adaptation of language models with memory of amortized contexts. arXiv preprint 2rX1v:2403.04317, 2024.
[44] Rohan Taori, Ishaan Guirajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Stanford alpaca: An instruction-ooloviii Ilama model. https:///itthb com/tatsu-Dae wawlllabbatatatea wtwtaaanrrddddaaaeaeeaaeaaaeaaae 2023.
[45] Nadav Timor, Jonathan Mamou, Daniel Korat, Moshe Berchansky, Oren Pereg, Gaurav Jain, Roy Schwartz. Moshe Wasserblat, and David Harel. Accelcrating Ilm inference with lossless speculative decoding algorithms for hetcrogencous vocabularics. 2025.
[46] Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence- knowledge distillation. In Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 10817-10834, 2023.
[47] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's 's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint 16Xiv:1 2016.
[48] Heming Xia, Yonggi Li, Jun Zhang, Cunxiao Du, and Wenjie Li. Swift: On-the-fly self-speculative decoding for Ilm inference acceleration. arXiv preprint 10X.::2410.06916 2024.
[49] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li. Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. technical report. arXiv preprint 24X 2024.
[50] Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, and Bo An. Longspee: Long-context speculative decoding with efficient drafting and verification. arXiv preprint arXiv:2502.174:3 2025.
[51] Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft& verify: Lossless large language model acceleration via self-speculative decoding. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1263-11282. 2024.
[52] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Ii, Dacheng Li, Fric Xing, et al. Judging -lu-as-a-judge with mt-bench and chathot arena. Advances in Neural Information Processing Systems, 36:46595-46666 2023.
[53] Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, -Fanççoao Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation, The Twelfth International Conference on Learning Representations.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer:(Yes
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
Section 4.4 discusses the limitations of our methodology.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sele--peciiiatinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper focuses on enabling cross vocabulary online speculative decoding and providing analysis and insights the performances. There is associated theoretical proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper provides all the details of the experiments and the necessary information for reporducibility. The details provided in the appendix.
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
The paper uses open-source datasets, which are all available on huggingface. Links to all datasets are provided. We definitely want to provide access to code. However, it takes time for corporate legal team review and approve. If reviewers feel necessary, we will try our best to accelerate the process of releasing code
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/ boblic/guides htte://uddioo/ooouuiiieisssse
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /niiss cc/ wwwpcc/pplcc....c hoooll/cgguides/oodesbbbbii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
16

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:| Yes]
Justification:
All experimental details and results are provided
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We have a multitude of experiments done across different models and across different tasks. However for the main tables in the graph, we have done the evaluation by averaging 3 runs each with different seed and reported the number in the main table. We have also done some ablations on some experiments for further analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
17

---

Answer: [Yes]
Justification: All experiements done on single NVIDIA-A100 40GB RAM GPU, for both training and inference.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/ http:///earrpp.c.pppppc/ccaeaaaa poubblic/Ehiccstuidddiieee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper provides research results and analysis impact of cross vocab online speculative decoding to speed up LLM inference. We do not see any negative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
18

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
All the experiments conducted the paper are based on open source datasets
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All references on data source are provided
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
19

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development this research does not involve LLMS as any important, original, or non-standard components
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
20

---

Please refer to our LLM policy http:s//heerriigccc///oorrrrrres
for what should or should be described.
21

---

A
Implementation Details
This appendix provides detailed information about the implementation and training setup used in our experiments.
A.1 Model and Hardware Details
Throughout our work, we use the environment setup with NVIDIA A10D GPU (40/80GB), PyTorch 2. 2.1.0 framework, CUDA version 12.1, and Ubuntu 22.04 LTS. Our models used include Llama-68M B2]. Llama3-8B T77. Qwen2-7B 49 and Vicuna-7B [52]. We train the Llama-68M model in our experiments. Their model details are listed in the following.
Table 7: Model hyperparameters Hyperparameter
Value
model
Llama-684 -6M
layers
2
hidden size
768
attention heads
12
activation function
SiLU
Table 8: Model latency and vocabulary statisties
Model
Wall-time per Step (s)
Vocabulary Size Intersection with Llama-68M
Llama-68M
0,00203
32000
32000
Qwen2-7B
0.02667
152064
22275
Llama3-88
0.02846
128256
22416
Vicuna-7B
0.02683
32000
32000
A.2 Training Details
Our experiments GSM8K 112 Alpaca 44), XSum 36 and combined MBPP+HunanEEval
4 10 datasets. We also show the major hyperparameters used across the experiments. For evaluation, we perform three runs with different seeds and report the average in the main results sections.
Table Dataset details
Dataset
GSM8K
MBPP+ Evaan
Alpaca 8K
XSum
train
8K
IK
4K/8K
200
228
100
100
Table 10: Training hyperparameters Hyperparameter
Value
batch size
learning rate (LR)
1e-4/2e-5
LR scheduler optimizer
constant
AdamW
B1 B2 weight decay epochs
0.9
0.999
0.01/0
(online)
mixed precision
FP16
LoRA rank
32
temperature
0.01
22

---

B Additional Experiment Details
B.1 Speedup Metric
follow the convention Medusa [8] to define the speedup metric. Given acceleration rate as the average number of tokens decoded per decoding step and overhead as the average per step latency of proposed model divided by that of the vanilla model, speedup refers to the wall-time acceleration rate and can be computed with speedup acceleration rate/ ovverhead.
B.2 Adaptive Drafting Training
We show the training curves for adaptive drafting across all tasks. It can be seen that our proposed variants of joint or interleaved align + adapt training have the best performances in both average acceptance rate ans speedup.
Figure 3: Online Adaptive Drafting training plot of Llama-68M Vicuna-7B
B.3 Adaptive Drafting LoRA Training
To showcase the idea of "one drafter for all", we further show results of online adaptive drafting with LoRA used in distillation in Table I1 We observe slightly lower performances across tasks compared to distillation with full fine-tuning as expected. But each of the distillation or distillation with adaptive drafting baseline with LoRA still outperforms normal speculative decoding. And distillation with adaptive drafting outperforms the distillation only baseline, except on GSM8K where the gap is minimal.
By demonstrating the compatibility of LoRA with our proposed online distillation and adaptive drafting training, we empower single draft model to serve as the backbone and selectively fine-tune LoRA modules and acceptance prediction heads for any potential target model. This would strike good balance to keep minimal memory overhead while retaining the benefits of greater speedup and flexibility, which are ideal for on-device applications.
Table 11: Performance on Online Adaptive Drafting using LoRA with rank 32 with Llama 68M and Vieuna- 7B.
Target
GSMBK
MBPP-HumanEval
Alpaca
XSum
Acc Rate
Speedup
Acc Rate
Speedup
Acc Rate
Speedup
Acc Rate
Speedup
SpD LoRA Distill Only
1.44x
1.22x
1.42x
Vicuna-BB
0.37
1.95x
0.24
1.52%
0.25
1.54x.
0.23
1.49%
Joint LoKA Distill
0.49
0.39
0.39
1.58x
0.41
Interleaved LoRA Distill Adapt
0.49
1.94x
0.38
1.61x
0.42
1.58x
0.39
1.55x
23

---

B.4 Cruss-vocabulary SpD online distillation on Llama-68M with Llama3-8B as target
Figure showeasee the training dynamics of Llama-68M with Llama3-8B as the target model. Similar to the previous Figure 2 the pattern is very much matched. Overall, across all the tasks, our methods are able improve upon the SpD baseline significantly as training progresses.
CSMBK
HEPP HumanEval
Alpaca
XSum
113
Speedup
437
444
440
376
100
Acceptare 1ltte
630
334
535
012
100 760
Figure 4: Cross-voca SpD online distillation on Llama-68M with Llama3 8B as target
B.5 Rank ablation for Cross-vocabulary online distillation of LoRA
We also ablate on the rank for the LoRA experiments, specifically focussed on the LDM + + LoRA method. As shown in the Table 12 it is clear that as the rank increases, performance also improves, but beyond rank 32, there we choose conservative rank of 32 across all our experiments, tasks and methodologies. Moreover, for the "one drafter for all" setting, folding of the LoRA weights infeasible and as such having lower rank would benefit the overall memory and compute required for the drafter model on-device.
dimmishing return in terms of overall improvement. As such
Table 12: Effect of rank for Lum +XCN gam + LoRA for Llama-68M and Llama3-8B GSM8K
Metrics LoRA Rank
8
16
32
64
128
Ace Rate
0.30
0.34
0.37
0.38
0.38
Speedup
1.478x
1.543x
1.595x
1.625x
1.632x
B.6
Distribution Shift
B.6.1
Heterogeneous Dataset Shift
We also ablated dataset distribution shift and data heterogeneity. We analyzed our training framework by starting the training on the GSM8K dataset and after around 1 steps, shifting to new data distribution of MBPP-HumanEvall Mix. As shown Figure [5] initially, both speedup and acceptance rate show a steady upward trend, indicating that the model is lcarning effectively and becoming more efficient. At step 1000, we introduce new dataset with a different distribution which causes a sharp drop in both acceptance rate and speedup, highlighting the impact of this shift. However, following this disruption, both metrics begin gradually recover, demonstrating the model's 's ability to adapt to the new data distribution over time. Additionally, when using our n-gram cache, the recovery seems be much better and larger when compared to no cache. Finally, when comparing to the model that was solely trained on MBPP+HumanEyal mix dataset, the model is able to slowly reach similar performance on both metrics. Overall, using our n-gram cache and despite the distribution shift, the model is showing resilience and adaptability to the new dataset.

---

Speedup
1206
Wibiuut N-grar Cacho gram Crche
Figure 5: Dataset Drift tracking during training by switching the dataset from GSM8K to MBPP+HumanEval at Step 1000 for Llama-60M vs Qwen-2.5-7B model
B.6.2 Target Model Shift
To further understand the impact of target model drift during training, We conducted empirical analysis target switching between Qwen2.5 7B and 14B at arbitrary step intervals (e.g.. 14B - 7B - 14B) As shown in Figure our observations indicate that the drafter model is able to re-align seamlessly after each switch. Notably, switching to the 7B model yields reduced speedup due to its lower drafter latency, while maintaining alignment quality. This behavior is largely attributed to the fact that the target models belong to the model family and such the drafter is able continually align during training. We also evaluated the impact of enabling the n-gram cache during target switching. Results show that the drafter recovers alignment quickly when the cache is enabled. However, in scenarios involving cross-family model swaps, a new n-gram cache is preferred due differences tokenization. In such cases, simple distribution shift is insufficient maintain alignment.
Specoup
3.14
200 wicogteetee
WitHLE Nigram: Cacro With Nerrmmm Cache
Figure 6: Model Drift tracking during training by switching the target model from 14B to 7B and back to 14B at step 300 and step 600, with Llama-68M as drafter
25

---

C Additional Algorithm Details
We show the detailed algorithm for cross-vocabulary distillation in Algorithm The online distillation training leverages our proposed cross-vocabulary speculative decoding in inference data collection, and uses the hybrid distillation loss to update the draft model.
Algorithm 2 Cross-vocabulary Distillation
1: Given target p(-). drafter go (-), online data stream S, data buffer Q, update interval /.
2: 2 + 0, (}
3: while True do
4: 2 S i +1
5: Get response y with cross-vocabulary speculative decoding in Algorithm |
6: Append (z, y) to Q
7:
i mod == then
Update qo on Q with hybrid distillation loss Cenoss, vocab disiill (0) 9: Q
8:
10: end if
end while
We show the detailed algorithms for the two variants of online adaptive drafting training in Algorithm 3 and 4
The distillation loss Luisiul (0) uses KL divergence on generated response tokens. The adaptive drafting loss (o) is weighted BCE loss between the acceptance prediction head outputs and acceptance ratio labels constructed from the target model p and current draft model qa-
Algorithm 3 Online Adaptive Drafting- loatingg-ooli Joint Training
1: Given target p(-), drafter go(-), acceptance prediction head fo(+), online data stream S, data buffer 2, update interval I.
2: i, Q 4 0, {
3: while True
4: æ ~ S, < i ++1
5: Get response y with speculative decoding using adaptive drafting
6: Push (zr to Q
7:
if mod = then
8:
Update qe Q with distillation loss Laisun (0) Compute labels Update fo on {(Ix y. ()e) with adaptive drafting loss Ladapt (Ø)
9:
min(1, p(y)/ (1/l)///////))))))))) (x)) on (z, y) Q
10:
11: Q Q-( 0
12:
end if
13:
end while
D Cross-vocabulary N-gram Cache Ablations
D.1 N-gram distributions
To showcase the captured n-grams during our cross-vocabulary speculative decoding, we collect the n-gram cache across tasks after online training with the hybrid distillation losses. We plot the distribution of the n-gram counts with respect to the frequencies they are encountered in Figure 7 We include n-grams with maximum frequency of 3000 and use a log-scale of the n-gram counts for cleaner visualization. N-grams with higher frequencies over 3000 typically represents a group of delimiters such as spaces and newline characters that are used for formatting. Besides these, we can observe a clear long-tail distribution for more frequent n-grams. Although the majority of n-grams have low frequencies which are not guaranteed to re-appear future data stream, long-tail frequency n-grams still have non-trivial contribution and can be utilized to speed up the cross-vocabulary speculative decoding process. We also notice the frequent n- n-grams constitute higher percentage in the
domain, despite having smaller dataset size. It indicates
26

---

Algorithm 4 Online Adaptive Drafting - Interlevved Training
1: Given target p()) drafter (-).), acceptance prediction head (.), online data stream S, distillation data buffer Q, adaptive drafting data buffer with max size N, batch size B, update interval 7.
2: i, 2, R <- 0. 0. O
3: while True do
4: 2 1~ S,
+1
5: Get a response y with speculative decoding using adaptive drafting
6: Push (z to Q
7: i mod == then
8:
Update qe Q with distillation loss Laisuu
9:
Push 2 to R
if [R] N then
11:
Evict early data from R.
12:
end if
13:
2 ++
else
15:
Sample batch Rg E R
Compute labels I min(1, p(y)/go()) on (z. v) RB
17:
Update fø on {[z, y. h) with adaptive drafting loss Latap (6)
end if
19:
end while
the n-gram cache technique for vroas-vocayylary speculative decoding can be more effective in well-structured domain like coding.
GSMBK
MBPP+HuuanEEval
(Log Scere)
Negrerm Court:
87
V/e
Alpaca
X5um
Regrems Count (Log Scale)
30
2002
Frequency
Prequency
Figure 7: Cross-vocabulary N-gram Distributions of Llama-68M and Qwen2-7B
D.2 N-gram inference examples
Next in Table 13 show examples of the n-grams being used from the cache during inference in different tasks. The n-gram examples are extracted from actual samples in the datasets with Llama-68M and Qwen2-7B, given draft length of 4 each proposal or speculative decoding step. Pink tokens represent tokens with direct mapping between drafter vocabulary and target vo- cabulary. Yellow tokens represent sub-tokens in the drafter space and the corresponding n-gram tokens in the target space, which are mapped via cache lookup. Lime tokens represent those that are accepted by the target model after verification. Note that even with the mapping, the proposed n-gram tokens are not guaranteed be accepted and require cross-occabulary distillation further align their distributions with the target.
27

---

most common Imp act * Proposed Tokens (Drafter Vocab)
Mapped Tokens (Target Vocab)
Accepted Tokens
GSM8K
There fore there
Therefore there
appoint ments Since
appointments Since of quizzes
appointments
of qu izz
distance tra led
distance traveled
distance
traveled
MBPP+HumanEval
to Sw
to
SW
asc ending order
ascending order Counter
ascending
order
ounter
Counter
test sol ution
test solution
Alpaca
Future A
Future of AI of Intelligent
of Int ellig ent most common Impact ** XSum an offender
most common Impact
an off ender
an
offender
on going
is ongoing
ongoing
pect or David
Inspector David
Inspector
and thank ed the
and thanked the
Table 13: Examples of N-gram Cache Across Tasks. Each row represents one cross- vosss-occbbllrry speculative decoding step/proposal extracted from the inference results on the data test set (no sequential order between rows). Drafter proposes 4 tokens each time, which are mapped to the target space either direct mapping tokens
merged n n-gram tokens Target model then verifies
the mapped tokens get the final accepted tokens, plus any correction token from the residual distribution or sampled free token if all mapped tokens are accepted. We also denote no token being accepted as "_"
28

---

D.3 N-gram learning
Accrepaance Ratio
1empanatow-8.01
Impprratue-0.33
Figure 8: Evolution of n-gram tokens learning with Llama-68M and Llama3-8B on MBPP+HumanEyal dataset. N-gram hit refers to the number of n-grams proposed by drafter in each response, averaged over query samples. Acceptance ratio refers to the quantity min(1, p/g) in speculative decoding, it is averaged over proposal steps with an n-gram token and indicative of the distribution alignment between the drafter and target. These two metrics show cross-vocabulary distillation learns to slowly align n-gram distributions to the target and utilize them more in inference.
From Figure it can be seen that during the training of LDM +XLN MBPP+ dataset, the average acceptance ratio for the n-grams is steadily improving, showing the impact of the loss function for better alignment of the n-grams. Furthermore, the impact is clearly visible even across different temperatures. Similarly as training progresses, the average number of successful cache hit for n-gram tokens per query is also improving.
D.4 N-gram Cache Growth Rate
GAMBK
240
1909
150
106
300
E11
Growth
603
2012
Figure 9: N-gram cache growth rate for Llama-68M drafter and Qwen2-7B target on GSM8K (4K subset) and FaPpaHumanFyal datasets.
We can define the growth rate of the N-gram cache to be growth_rai cache_size giche_size/trainne with number tokens processed in online learning. The growth rate of the N-gram cache can also be seen to continually decrease.
From Figure 9
From the tables, we can observe that the cache grows sub-linearly
29

---

D.5
N-gram Cache Scalability
Table 14 indicates how speedup is impacted by different max cache size and cache eviction policies: Least Recently Used (LRU) that evicts the n-gram that hasn't I been accessed for the longest time, Least Frequently Used (LFU) that removes the n-gram with the lowest access frequency. "1/4" and "1/2" refer to using max cache size quarter and half of the full final cache size that we previously trained.
Among all eviction policies and max cache sizes, both evaluation metrics are better than using no cache. Specifically LRU shows diminishing returns over the growth of the cache. Meanwhile LFU plateaus at an earlier stage, with performance matching the full cache even with a quarter of the size. This could be due to LFU capturing most of the high frequency and important n-grams, compared to LRU which only captures the recent n g-grmms. In the context of on-device deployment, this implies with proper selection of cache size and eviction strategy, we can potentially attain robust performance given only constrained memory resources, highlighting the practical feasibility of our approach.
Table 14: Cache size scaling on MBPP-HumanEval
Cache Size
Cache Type
Acc Rate
Speedup
No Cache
0,219
1.29x
LRU
0,254
1.33x
1/4
LFU
0.259
1.36x
LRU
0,261
1.35x
1/2
LFU
0.263
1.36x
Full Cache
0.267
1.36x
30


Reviews:
[{"review_id": "uvuR9y0E4G", "reviewer": "Reviewer_8Gpq", "summary": "This paper introduces OmniDraft, a unified speculative decoding framework that allows a single, efficient draft model to work seamlessly with any target language model, even in online and on-device settings. The approach combines an online n-gram cache and hybrid distillation fine-tuning to resolve vocabulary mismatches between draft and target models, and uses adaptive drafting techniques to further accelerate decoding. The authors demonstrate that OmniDraft can pair a single small draft model with multiple large target models across diverse tasks achieving notable speedups of up to 2x without sacrificing flexibility or performance.", "questions": "1. In terms of scalability, could you share any intuition of how the performance of this new speculative decoding method changes as the size of LLMs gets larger?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength\n\ns1: This paper is well written and easy to follow\n\ns2: This paper comes up with the novel idea of solving incompatibility issue between draft and target model by integrating an online n-gram cache\n\ns3: This paper experimentally shows that their method achieves better performance and speed ups compared to previous baselines.\n\nWeakness\n\nw1: I recommend authors to do more experiments with more downstream tasks and different/larger LLMs\n\nw2: It would be better if authors can add more baselines other than SpD.", "was_revised": true, "final_justification": "I raise my score to 5. The authors addressed my comments well with additional experiments.", "rebuttal": "We would like to express our gratitude to the reviewers for their careful review of our paper and constructive feedback. We appreciate the opportunity to clarify the contributions, novelty and limitations of our work. We are grateful to Reviewer 8Gpq for recognizing the novelty of our method and acknowledging the strength of our experimental results. \n\n\n### Q1: I recommend authors to do more experiments with more downstream tasks and different/larger LLMs; in terms of scalability, could you share any intuition of how the performance of this new speculative decoding method changes as the size of LLMs gets larger?\n\n\n#### Different and Larger LLMs\n\n- **Clarifications**: Thank you for the question. As per the request, we performed additional experiments on larger LLMs.\n- **Experiments**: We used Qwen2.5 series of models of larger sizes including **7B, 14B and 32B** parameters. Datasets used are GSM8k and MBPP+HumanEval. As an additional ablation, we also performed analysis on a larger drafter of size **160M** parameters. The performance on these larger models are consistent with our previous results, with larger target models resulting in **2x** improvement on the GSM8K dataset.\n- **Reasoning**: Due to time and GPU constraints, we were able to perform these ablations on 2 datasets and only up-to 32B parameter model. The Llama-160M parameter model was chosen as the larger drafter after taking into consideration the increased latency of the drafter model compared to Llama-68M. Overall, the larger drafter model does introduce a reduction in the speedup across all target models, however, due to its enhanced potential capability, the acceptance rate does improve across all the target models. Due to time constraints, we are unable to perform further experiments on more downstream tasks. \n\n\n#### Scalability\n\n- **Clarification**: The Omnidraft framework demonstrates strong scalability as the size of the target LLM increases, as shown in the `Table 1` below.\n- **Reasoning**: The key intuition is that while the gap between the drafter and the target model grows with model size, the drafter remains the limiting factor. During training, the drafter progressively aligns with the target; however, once it reaches its alignment capacity, further increase in target size continues to yield speedup improvements, while the acceptance rate plateaus. As such, when we increase the drafters capacity to 160m, the acceptance rate does indeed improve, although speedup does take a hit due to the increased latency of the drafter. Due to time and hardware constraints, we were unable to conduct experiments with the Qwen2.5 72B target model.\n\nTable 1. Performance with Large Models. Due to time and GPU constraints, scaling experiments on the GSM8K dataset are only using a 4K subset.\n\n| Target        | Method                                          | GSM8K Acc | GSM8K Speedup | MBPP+HumanEval Acc | MBPP+HumanEval Speedup |\n|---------------|-------------------------------------------------|-----------|----------------|---------------------|--------------------------|\n| Qwen2.5 7B    | SpD_UAG-DM (68M)                             | 0.15      | 1.02x          | 0.10                | 0.94x                    |\n|               | *L_DM + λ L_N-gram* (68M)                   | **0.401** | **1.66x**      | **0.27**            | **1.33x**                |\n|               | SpD_UAG-DM (160M)                            | 0.18      | 0.70x          | 0.13                | 0.60x                    |\n|               | *L_DM + λ L_N-gram* (160M)                  | **0.47**  | **1.12x**      | **0.32**            | **0.90x**                |\n| Qwen2.5 14B   | SpD_UAG-DM (68M)                             | 0.15      | 1.17x          | 0.10                | 1.14x                    |\n|               | *L_DM + λ L_N-gram* (68M)                   | **0.407** | **1.92x**      | **0.272**           | **1.57x**                |\n|               | SpD_UAG-DM (160M)                            | 0.178     | 0.89x          | 0.13                | 0.84x                    |\n|               | *L_DM + λ L_N-gram* (160M)                  | **0.472** | **1.40x**      | **0.33**            | **1.19x**                |\n| Qwen2.5 32B   | SpD_UAG-DM (68M)                             | 0.153     | 1.30x          | 0.10                | 1.23x                    |\n|               | *L_DM + λ L_N-gram* (68M)                   | **0.42**  | **2.05x**      | **0.274**           | **1.71x**                |\n|               | SpD_UAG-DM (160M)                            | 0.187     | 1.03x          | 0.133               | 0.97x                    |\n|               | *L_DM + λ L_N-gram* (160M)                  | **0.49**  | **1.62x**      | **0.335**           | **1.40x**                |\n\n\n### Q2: It would be better if authors can add more baselines other than SpD.\n\n#### Additional Baselines\n- **Clarifications**: Thank you for the suggestion. Our current baseline mentioned in all our results is SpD_DM, which is our implementation of SpD with direct mapping tokens only and is equivalent to UAG.\n- **Reasoning**: Baselines such as Medusa are orthogonal to our approach and can be integrated on top of our current framework, like multi-head decoding, tree-based attention, and sparsification (SWIFT). In contrast, techniques like EAGLE rely on a shared language model head, which assumes a consistent model architecture and vocabulary. This dependency makes EAGLE unsuitable for our cross-vocabulary setting.\nAs there are only a few existing methods that explicitly address cross-vocabulary scenarios, we primarily compare against the UAG approach, which is most relevant to our setup.\n- **References**: UAG - reference [45], Medusa - reference [8], SWIFT - reference [48], EAGLE - reference [27] from the main paper."}, {"review_id": "LTqMNcybQc", "reviewer": "Reviewer_NjJe", "summary": "This paper presents OmniDraft, a speculative-decoding framework that lets a single 68 M-parameter drafter accelerate multiple large-model back-ends (Vicuna-7B, Qwen2-7B, Llama-3-8B) even when their tokenizers differ. Three technical pieces enable this:\n\n1. Cross-vocabulary n-gram cache. During inference, the system merges sequences of draft sub-tokens into target tokens via a cache that is populated on-the-fly, allowing correct acceptance-ratio tests across vocabularies.\n\n2. Online hybrid distillation. A token-level KL term aligns directly mapped tokens, while an NLL term aligns cached n-grams, continually fine-tuning the drafter as it serves requests.\n\n3. Online adaptive drafting. A lightweight head predicts token-acceptance probabilities and shortens or lengthens drafts to maximise speed per wall-time budget.", "questions": "1. End-to-end latency & energy. Please supply absolute wall-time and energy figures for each target on a mobile GPU or CPU to confirm that 1.6–2 × speed-up translates into real user-perceived gains.\n\n2. Cache footprint & scaling. How large does the n-gram cache grow after, say, 10 k prompts, and what is its lookup/update time under concurrency? A memory/throughput chart would clarify on-device feasibility.\n\n3. Robustness to target drift. If the cloud back-end swaps from Qwen2-7B to Qwen2-14B mid-session, how quickly does the drafter re-align, and does the cache hurt or help during the transition?\n\nAddressing (1), (2), (3) with concrete measurements would raise Quality and Significance and push the paper into clear-accept territory.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. Quality – The empirical study spans four tasks and three heterogeneous targets, with clear ablations on cache utility, loss choices and drafting heads. However, all timing figures are relative; absolute wall-clock latency, GPU utilisation and memory footprint of the cache are absent. Online adaptation is evaluated for only a single epoch and on 68 M→7 B/8 B scales; it remains unclear whether gains persist for larger drafters or longer streams.\n\n2. Clarity – The methodology is carefully derived and Algorithm 1 is explicit. Yet crucial implementation numbers (cache size limits, eviction policy, adaptive-head thresholds) are scattered or deferred to the appendix, and the main text exceeds typical concision norms.\nSignificance – Decoupling drafter and target tokenizers is an important step toward real on-device speculative decoding. Still, reported speed-ups hover around 1.6 ×; rival approaches such as UAG or AdaptiVocab are mentioned but not compared directly, leaving the incremental benefit uncertain.\n\n3. Originality – The live n-gram cache that learns cross-tokenizer merges during inference is novel, and combining it with continuous distillation and adaptive draft length is an elegant unification.", "was_revised": true, "final_justification": "The clarifications around cache scaling, target drift, and preliminary latency measurements — particularly on mobile hardware — are appreciated and address key concerns.\n\nWhile some results are still early-stage or pending (e.g., concurrent cache access and full on-device deployment), I find the methodology sound and the direction promising. Given the additional evidence, I am raising my score from 3 to 4, though not strongly. Stronger comparisons and more mature system profiling would further strengthen the case.", "rebuttal": "We would like to express our gratitude to the reviewers for their careful review of our paper and constructive feedback. We appreciate the opportunity to clarify the contributions, novelty and limitations of our work. We are grateful to Reviewer NjJe for recognizing the technicality of our methodology, novelty, algorithm and acknowledging the various empirical ablations done. \n\n### Q1: End-to-end latency on mobile GPU or CPU\n\n- **Clarifications**: Thank you for the suggestion. Regarding the system's end-to-end latency, we plan to implement the entire pipeline on-device as part of our future work. Due to current time constraints, we have profiled both the drafter and target models on both CPU (AMD Ryzen Threadripper PRO 5975WX) and mobile GPU (Apple M4 Pro) using llama-bench from llama.cpp with the best setting we figured out, and then computed the overall speedup based on the offline recorded steps of the SpD process for each proposal in the evaluation set. Please note that this speedup does not account for the ngram-cache lookup, as its on-device implementation is still pending.\n\n- **Experiments**: As shown in `Table 1`, the results on the CPU are promising, demonstrating a potential speedup of approximately 2.5×. On the mobile GPU, however, the performance is slightly less favorable. This is primarily due to the latency of the target model on the mobile GPU, where the AR4 (latency of a verification step with K=4) is ~ 2 * AR1 (latency of a generation step). This is not exactly aligned with speculative decoding's assumption, resulting in reduced overall speedup. This could also potentially be due to an optimal setting that we might have missed in llama-bench for the mobile GPU.\n\nTable 1: Wall-clock latency for drafter and target models on CPU and mobile GPU and the speedup on different datasets\n|Platform|Model|Wall-clock Latency (AR1/AR4, in ms)|GSM8K Speedup|MBPP+HumanEval Speedup|\n|-|-|-|-|-|\n|CPU|Llama-68M|0.37/-|-|-|\n||Qwen2.5 7B|58.57/59.13|2.47x|1.99x|\n||Qwen2.5 14B|115.71/116.29|2.50x|2.00x|\n||Qwen2.5 32B|255.41/255.65|2.61x|2.04x|\n|Mobile GPU|Llama-68M|0.62/-|-|-|\n||Qwen2.5 7B|34.39/46.18|1.81x|1.45x|\n||Qwen2.5 32B|149.46/268.06|1.45x|1.13x|\n\n### Q2: Cache Footprint, Scaling & Growth Rate\n\n- **Clarifications**: Thank you for pointing this out. Our current n-gram cache is implemented using a Python dictionary (hash table-based). This approach provides average-case O(1) time complexity for both insertion and lookup, which ensures high throughput for real-time insertion and lookup. However, our current implementation is not thread-safe for concurrent writes, and thus we are not able to provide the latency with concurrency, which requires an implementation with read-write lock to allow concurrent lookups and serializing writes. And we expect the insertion latency will increase due to the locking mechanism. We'll be looking into that in our on-device implementation.\n\n- **Reasoning**: `Table 2` shows the overall N-gram cache size across tasks remains relatively small compared to the size of the draft or target model. This indicates potential feasibility for on-device setting given the small overhead. In addition, we are also working towards a more compact and efficient implementation of the N-gram cache specifically for on-device deployment as part of our future work.\n\nTable 2. Final N-gram cache size\n*Cache memory (MB) is derived from `pympler.asizeof.asizeof()`.*\n\n||GSM8K|MBPP+HumanEval|Alpaca|XSUM|\n|-|-|-|-|-|\n|**Training Samples**|7473|910|8000|4000|\n|**Cache Size (#n-grams)**|5569|2238|20339|17013|\n|**Cache Memory (MB)**|1.372|0.501|4.569|3.924|\n\n- **Reasoning**: `Table 3` indicates the growth of the n-gram cache size as training progresses. Growth rate is defined as `growth_rate = cache_size / training_tokens`. We can observe the cache size increases sub-linearly with respect to the amount of training tokens, and should eventually plateau as the growth rate decreases. This is likely the result of the cache capturing most of the high frequency n-grams. We tested various eviction policies across cache sizes and observed diminishing returns, suggesting cache memory can be further reduced with minimal performance loss.\n\nTable 3. N-gram cache growth rate for MBPP+HumanEval\n\n|Training Step|100|200|300|400|500|600|700|800|900|\n|-|-|-|-|-|-|-|-|-|-|\n|**Cache Size (#n-grams)**|667|1021|1256|1466|1628|1791|1929|2089|2208|\n|**Training Tokens (#tokens)**|26K|57K|85K|113K|141K|167K|195K|224K|252K|\n|**Cache Growth Rate**|0.0247|0.0177|0.0147|0.0128|0.0114|0.0106|0.0098|0.0093|0.0087|\n\n### Q3: Robustness to Target Drift\n#### Target Model Drift\n- **Clarification**:Thank you for the suggestion. We conducted empirical analysis on model switching between Qwen2.5 7B and 14B at arbitrary step intervals (e.g., 7B → 14B → 7B)\n- **Experiments 1**: As shown in `Table 4`, our observations indicate that the drafter model is able to re-align seamlessly after each switch. Notably, switching to the 14B model yields improved speedup due to its higher latency, while maintaining alignment quality. This behavior is largely attributed to the fact that the target models belong to the same model family and as such the drafter is able to continually align during training.\n- **Experiments 2**: We also evaluated the impact of enabling the n-gram cache during target switching. Results show that the drafter recovers alignment quickly when the cache is enabled. However, in scenarios involving cross-family model swaps, a new n-gram cache is preferred due to differences in tokenization. In such cases, a simple distribution shift is insufficient to maintain alignment. \nSince we are unable to include plots directly, we have provided below a tabular summary of the key results to convey the main trends. The corresponding plots will be included in the final version of the paper.\n\nTable 4. Summary of Speedup Transitions Across Training Steps and Model Switches (7B → 14B → 7B)\n\n|Step Range|Target Model|Speedup Trend|Speedup Value|Notes|\n|-|-|-|-|-|\n|0–300|7B|Gradual increase|~0.8 → ~1.05|Initial training phase|\n|300 (Switch)|→ 14B|Immediate jump|~1.4|Transition to larger model|\n|300–600|14B|Slight increase|~1.45|System adapts to new model|\n|600 (Switch)|→ 7B|Drop in speedup|~1.15|Reverting to original model|\n|600–1000|7B|Slight increase|~1.2|Final phase with stable performance|\n\n### Additional Responses\n#### Larger drafter and target scalability \n- **New Experiments**: Due to time and GPU constraints, we were able to perform these ablations on 2 datasets and up-to 32B parameter model only. We used Qwen2.5 series of models with 7B, 14B and 32B parameters. Datasets used are GSM8k and MBPP+HumanEval. As an additional ablation, we also performed analysis on a larger drafter of size 160M parameters (`Table 5`). The performance on these larger models are consistent with our previous results, resulting in 2x improvement on the GSM8K dataset.\n- **Larger Drafter**: The Llama-160M parameter model was chosen as the larger drafter after taking into consideration the increased latency of the drafter model compared to Llama-68M. Overall, the larger drafter model does introduce a reduction in the overall speedup across all target models. However, due to its enhanced potential capability, the acceptance rate does improve across all the target models.\n- **Larger Targets**: On scalability, as the gap between the drafter and target model grows with model size, the drafter remains the limiting factor. During training, the drafter progressively aligns with the target; however, once it reaches its alignment capacity, further increase in target size continues to yield speedup improvements, while the acceptance rate plateaus. For the larger drafter, the acceptance rate does indeed improve, although speedup does take a hit due to the increased latency of the drafter.\n\nTable 5. Performance with Large Models. \nDue to time and GPU constraints, scaling experiments on the GSM8K dataset are only using a 4K subset.\n\n|Target|Method|GSM8K Acc|GSM8K Speedup|MBPP+HumanEval Acc|MBPP+HumanEval Speedup|\n|-|-|-|-|-|-|\n|Qwen2.5 7B|SpD_UAG-DM (68M)|0.15|1.02x|0.10|0.94x|\n||*L_DM + λ L_N-gram* (68M)|**0.401**|**1.66x**|**0.27**|**1.33x**|\n||SpD_UAG-DM (160M)|0.18|0.70x|0.13|0.60x|\n||*L_DM + λ L_N-gram* (160M)|**0.47**|**1.12x**|**0.32**|**0.90x**|\n|Qwen2.5 14B|SpD_UAG-DM (68M)|0.15|1.17x|0.10|1.14x|\n||*L_DM + λ L_N-gram* (68M)|**0.407**|**1.92x**|**0.272**|**1.57x**|\n||SpD_UAG-DM (160M)|0.178|0.89x|0.13|0.84x|\n||*L_DM + λ L_N-gram* (160M)|**0.472**|**1.40x**|**0.33**|**1.19x**|\n|Qwen2.5 32B|SpD_UAG-DM (68M)|0.153|1.30x|0.10|1.23x|\n||*L_DM + λ L_N-gram* (68M)|**0.42**|**2.05x**|**0.274**|**1.71x**|\n||SpD_UAG-DM (160M)|0.187|1.03x|0.133|0.97x|\n||*L_DM + λ L_N-gram* (160M)|**0.49**|**1.62x**|**0.335**|**1.40x**|\n\n#### Online learning and Speedup numbers\n- **Clarification**:  Speedup metrics were measured using absolute wall-clock latency on a single 40GB/80GB NVIDIA A100 GPU, averaged over three independent runs on the evaluation set. Due to the online nature of our setting, training was limited to a single epoch or a subset of samples. However, we performed experiments across four different tasks, each with varying sample sizes, providing a broader perspective on the observed speedup trends.. We observe that as we increase the target models, we get a larger speedup of  ~2x. We expect even larger speedup if we use the 70B target model. \n\n#### Comparisons to baseline\n- **Clarification**:\nOur baseline comparison for SpD-DM is equivalent to the UAG method. On all our results, our approach consistently outperforms the UAG baseline.\nRegarding the AdaptiVocab method, it is less suitable for on-device deployment, as it requires modifying the embedding matrix by adding new token embeddings i.e. effectively altering the architecture of the draft model. Other techniques, such as Medusa, are orthogonal to our setting and could be integrated on top of our framework to further enhance performance."}, {"review_id": "oJCAkAdfQ3", "reviewer": "Reviewer_F753", "summary": "This paper proposes OmniDraft, a framework designed to shift the speculative decoding paradigm for on-device applications from a static, model-specific setup to a universal and adaptive one. The central idea is to enable a single, lightweight draft model to not only serve a variety of target models with incompatible vocabularies but also to continuously improve its performance through online learning from user data. To achieve this, OmniDraft introduces two core interacting mechanisms: an online n-gram cache to resolve cross-vocabulary tokenization mismatches on-the-fly and a hybrid online distillation process that continually aligns the draft model with the target model's verified outputs. These components, combined with adaptive drafting techniques for efficiency, are shown to successfully allow a single Llama-68M to accelerate various models like Qwen2-7B and Llama3-8B by up to 1.5-2x.", "questions": "•\tThe paper correctly identifies the N-gram cache's memory footprint as a potential limitation for on-device use. Could the authors provide a brief empirical analysis of the cache's growth rate and final memory size observed during the experiments? This would be very helpful for understanding the practical scalability of the approach.\n•\tThe experiments effectively demonstrate OmniDraft's performance with target models in the 7-8B parameter range. Have the authors conducted any preliminary experiments, or do they have insights on how the framework would perform with significantly larger target models? The alignment difficulty and potential speedup could change substantially at that scale.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n•\tSignificance: Addresses a critical and practical problem in LLM deployment: enabling a single draft model to work with any target model.\n•\tOriginality: The \"one drafter for all\" online adaptive framework is a novel paradigm, using an online n-gram cache for cross-vocabulary translation and hybrid distillation for live adaptation.\n•\tQuality: Presents a solid empirical evaluation across various incompatible models and tasks, supported by relevant ablation studies.\n•\tClarity: The paper is clearly written, with its methodology clearly outlined in pseudocode and its experimental results presented effectively. \n\n\nWeaknesses:\n•\tN-gram Cache Scalability: The scalability and memory footprint of the online n-gram cache, critical for on-device use, are not fully analyzed.\n•\tOnline Learning Stability: Online adaptation can be unstable, as shown by training curves (including in the appendix), raising long-term stability concerns.\n•\tModest Speedup: The reported speedup (up to 1.5-2x) is valuable but more modest compared to results from other specialized systems.", "was_revised": true, "final_justification": "After carefully reading the response, I found that most of my concerns have been addressed, and I decided to keep my positive rating.", "rebuttal": "We would like to express our gratitude to the reviewers for their careful review of our paper and thoughtful feedback. We appreciate the opportunity to clarify the contributions, novelty and limitations of our work. We are grateful to reviewer F753 for recognizing our one-drafter-for-all novel paradigm and the methodology.\n\n\n### Q1: N-gram Cache Footprint, Growth Rate and Scalability\n\n\n#### Memory Size or Footprint\n\n- **Clarifications**: Thank you for the question. ``Table 1`` below indicates the final N-gram cache memory sizes across tasks.\n- **Reasoning**: Overall N-gram cache size across tasks remain relatively small compared to the size of the draft or target model. This indicates potential feasibility for on-device setting given the small overhead. In addition, we are also working towards a more compact and efficient implementation of the N-gram cache specifically for on-device deployment as part of our future work.\n\nTable 1. Final N-gram cache size across tasks with Llama-68M drafter and Qwen2-7B target. The N-gram cache is implemented as a normal python dict.\n*Cache memory (MB) is derived from `pympler.asizeof.asizeof()`.*\n\n||GSM8K|MBPP+HumanEval|Alpaca|XSUM|\n|-|-|-|-|-|\n|**Training Samples**|7473|910|8000|4000|\n|**Cache Size (#n-grams)**|5569|2238|20339|17013|\n|**Cache Memory (MB)**|1.372|0.501|4.569|3.924|\n\n\n#### N-gram Cache Growth Rate\n\n- **Clarifications**: ``Table 2 & 3`` indicates the growth of the n-gram cache size as training progresses on GSM8K and MBPP+HumanEval datasets. Growth rate is defined as ``growth_rate = cache_size / training_tokens``\n- **Reasoning**: We observe that the cache size increases sub-linearly with respect to the amount of training tokens, and should eventually plateau as the growth rate decreases. This is likely the result of the cache capturing most of the high frequency n-grams.\n\nTable 2. N-gram cache growth rate during online learning for GSM8K\n\n|**Training Step**|500|1000|1500|2000|2500|3000|3500|4000|\n|-|-|-|-|-|-|-|-|-|\n|**Cache Size (#n-grams)**|1097|1717|2263|2692|3078|3430|3734|4009|\n|**Training Tokens (#tokens)**|83K|166K|253K|336K|418K|501K|583K|668K|\n|**Cache Growth Rate**|0.0130|0.0102|0.0089|0.0080|0.0073|0.0068|0.0064|0.0060|\n\n\nTable 3. N-gram cache growth rate during online learning for MBPP+HumanEval\n\n|Training Step|100|200|300|400|500|600|700|800|900|\n|-|-|-|-|-|-|-|-|-|-|\n|**Cache Size (#n-grams)**|667|1021|1256|1466|1628|1791|1929|2089|2208|\n|**Training Tokens (#tokens)**|26K|57K|85K|113K|141K|167K|195K|224K|252K|\n|**Cache Growth Rate**|0.0247|0.0177|0.0147|0.0128|0.0114|0.0106|0.0098|0.0093|0.0087|\n\n\n#### N-gram Cache Scalability \n\n- **Clarifications**: We implemented 2 baseline cache eviction policies with varied max cache size. Due to time constraints, we perform experiments on a single dataset.\n- **Experiments**: `Table 4` indicates how speedup is impacted by different max cache size and cache eviction policies: Least Recently Used (LRU) that evicts the n-gram that hasn’t been accessed for the longest time, Least Frequently Used (LFU) that removes the n-gram with the lowest access frequency. `1/4` and `1/2` refer to using max cache size as a quarter and half of the full final cache size that we previously trained. \n- **Reasoning**: Among all eviction policies and max cache sizes, both evaluation metrics are better than using no cache. Specifically LRU shows diminishing returns over the growth of the cache. Meanwhile LFU plateaus at an earlier stage, with performance matching the full cache even with a quarter of the size. This could be due to LFU capturing most of the high frequency and important n-grams, compared to LRU which only captures the recent n-grams. In the context of on-device deployment, this implies that with proper selection of cache size and eviction strategy, we can potentially attain robust performance given only constrained memory resources, highlighting the practical feasibility of our approach.\n\n\nTable 4. Cache size scaling v.s. Speedups on MBPP+HumanEval\n\n|Cache Size|Cache Type|Accept Rate|Speedup|\n|-|-|-|-|\n|No Cache|-|0.219|1.29x|\n|1/4|LRU|0.254|1.33x|\n||LFU|0.259| 1.36x|\n|1/2|LRU|0.261|1.35x|\n||LFU|0.263|1.36x|\n|Full Cache|-|0.267|1.36x|\n\n\n\n\n\n### Q2: Performance with Larger Models.\n\n\n\n#### Different and Larger LLMs.\n- **Clarification**:  Thank you for the suggestion. As per the request, we performed additional experiments on larger LLMs as shown in `Table 5`.\n- **New Experiments**: We used Qwen2.5 series of models of larger sizes including **7B, 14B and 32B** parameters. Datasets used are GSM8k and MBPP+HumanEval. As an additional ablation, we also performed analysis on a larger drafter of size **160M** parameters. The performance on these larger models are consistent with our previous results, with larger target models resulting in **2x** improvement on the GSM8K dataset.\n- **Larger Drafter**: Due to time and GPU constraints, we were able to perform these ablations on 2 datasets and up-to 32B parameter model only. The Llama-160M parameter model was chosen as the larger drafter after taking into consideration the increased latency of the drafter model compared to Llama-68M. Overall, the larger drafter model does introduce a reduction in the overall speedup across all target models, however, due to its enhanced potential capability, the acceptance rate does improve across all the target models.\n- **Larger Targets**: The Omnidraft framework demonstrates strong scalability as the size of the target LLM increases, as shown in the table. While the gap between the drafter and the target model grows with model size, the drafter remains the limiting factor. During training, the drafter progressively aligns with the target; however, once it reaches its alignment capacity, further increase in target size continues to yield speedup improvements, while the acceptance rate plateaus. For the larger drafter, the acceptance rate does indeed improve, although speedup does take a hit due to the increased latency of the drafter.\n\n\nTable 5. Comparison of Cross-vocabulary Distillation Performance with Llama-68M and Llama-160M across Three Target Models. Due to time and GPU constraints, scaling experiments on the GSM8K dataset are only using a 4K subset.\n\n|Target|Method|GSM8K Acc|GSM8K Speedup|MBPP+HumanEval Acc|MBPP+HumanEval Speedup|\n|-|-|-|-|-|-|\n|Qwen2.5 7B|SpD_UAG-DM (68M)|0.15|1.02x|0.10|0.94x|\n||*L_DM + λ L_N-gram* (68M)|**0.401**|**1.66x**|**0.27**|**1.33x**|\n||SpD_UAG-DM (160M)|0.18|0.70x|0.13|0.60x|\n||*L_DM + λ L_N-gram* (160M)|**0.47**|**1.12x**|**0.32**|**0.90x**|\n|Qwen2.5 14B|SpD_UAG-DM (68M)|0.15|1.17x|0.10|1.14x|\n||*L_DM + λ L_N-gram* (68M)|**0.407**|**1.92x**|**0.272**|**1.57x**|\n||SpD_UAG-DM (160M)|0.178|0.89x|0.13|0.84x|\n||*L_DM + λ L_N-gram* (160M)|**0.472**|**1.40x**|**0.33**|**1.19x**|\n|Qwen2.5 32B|SpD_UAG-DM (68M)|0.153|1.30x|0.10|1.23x|\n||*L_DM + λ L_N-gram* (68M)|**0.42**|**2.05x**|**0.274**|**1.71x**|\n||SpD_UAG-DM (160M)|0.187|1.03x|0.133|0.97x|\n||*L_DM + λ L_N-gram* (160M)|**0.49**|**1.62x**|**0.335**|**1.40x**|\n\n\n\n\n### Q3: Online Learning Stability and Modest Speedup\n\n\n#### Online Learning\n- **Clarifications**: Regarding online learning, the observed instability is an inherent characteristic of the setting, which occurs as fluctuations (e.g., zig-zag patterns) in the training curves. However, as training progresses and the n-gram cache becomes more populated and captures a broader set of in-distribution data, the instability can be potentially mitigated by training for much larger samples. We can also incorporate other techniques such as replay buffer or other aggressive regularization techniques for online learning instability mitigation.\n\n#### Speedup \n- **Clarifications**: In terms of speedup, we observe a consistent improvement as the target LLM size increases from 7B to 32B. Additionally, our framework could potentially benefit from incorporating advanced SpD techniques such as Medusa (which introduces multiple token prediction heads on the drafter) and various tree-based sampling strategies. However, these methods are orthogonal to our approach and are expected to provide further gains when layered on top of our current system."}, {"review_id": "QWHkHERd51", "reviewer": "Reviewer_PzLn", "summary": "This paper proposes a new framework for speculative decoding (termed as OmniDraft) that addresses two major challenges in online deployment settings - cross-vocabulary incompatibility between draft and target models, and the need for dynamic latency improvements over usage time. OmniDraft proposes an online n-gram cache with hybrid distillation fine-tuning to handle vocabulary mismatches and leverages adaptive drafting techniques to further improve decoding speed. \n\nThe overall motivation of this work is to obtain a \"one ring rules all\" type framework - enable a single, small draft model to work effectively with various larger target models for speculative decoding in on-device LLM applications. The authors show empirically that their generalized idea indeed works in practice and gives 1.5-2x speedup on tasks like math reasoning, coding, and text generation.", "questions": "- Beyond the mentioned future work of incorporating an explicit adaptive head in the cross-vocabulary setting, what other avenues do the authors have in mind to handle special tokens?\n\n- Could you elaborate on the potential strategies for optimizing the n-gram cache for memory-constrained edge devices? (e.g. offload, caching, indexing, etc)\n\n- Do the authors see any other drawbacks with the \"dynamic adapter switching\" approach e.g. for LoRA work in practice?", "limitations": "N/A (no data or models released).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper tackles two significant practical challenges in speculative decoding: cross-vocabulary mismatch and online adaptation for latency improvement, both of which are crucial for real-world on-device LLM deployment. In this regard, this is a good contribution.\n- There are some technical strong ideas in this paper such as the introduction of an online n-gram cache for cross-vocabulary translation which I find a very effective solution to the vocabulary mismatch problem - perhaps there are more efficient ones but this is a reasonable practical approach imo.\n- Integration of online adaptive drafting is another further improvement in the proposed method - this makes their approach more flexible by allowing the drafter to dynamically adjust proposal length.\n\nWeaknesses:\nFirst of all, I applaud the authors for thinking through and bringing out several possible limitations of their work. Many of my thoughts are in the same direction, although I will re-emphasize:\n- The paper acknowledges that online adaptation, being limited to a single iteration of the data stream, still has potential for instability on new unseen data. This is an inherent challenge of online learning but remains a practical limitation.\n- Is the draft model indeed universal? How does it work in the presence of distribution shifts and data heterogeneity? I feel the discussion on this is limited in this paper but compelling direction to make such methods work in practice.", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewers for their careful review of our paper and thoughtful feedback. We appreciate the opportunity to clarify the contributions, novelty and limitations of our work. We are thankful to reviewer PzLn for recognizing the novelty and practicality of our approach.\n\n\n### Q1: Instability from Online Adaptation, Performance on Distribution Shifts and Data Heterogeneity\n\n#### Online Learning\n- **Clarifications**: Regarding online learning, the observed instability is an inherent characteristic of the setting, which occurs as fluctuations (e.g., zig-zag patterns) in the training curves. However, as training progresses and the n-gram cache becomes more populated and captures a broader set of in-distribution data, the instability can be potentially mitigated by training with more samples. We can also incorporate other techniques such as replay buffer or other aggressive regularization techniques for online learning instability mitigation.\n\n#### Distribution Shifts\n- **Clarifications**: Regarding the dataset distribution shift and data heterogeneity, we analyzed our training framework by starting the training on the GSM8K dataset and after around 1k steps, shifting to a new data distribution of MBPP+HumanEval Mix. \n- **Reasoning**: As shown in `Table 1`, initially, both speedup and acceptance rate show a steady upward trend, indicating that the model is learning effectively and becoming more efficient. At step 1000, we introduce a new dataset with a different distribution which causes a sharp drop in both acceptance rate and speedup, highlighting the impact of this shift. However, following this disruption, both metrics begin to gradually recover, demonstrating the model's ability to adapt to the new data distribution over time. Additionally, when using our n-gram cache, the recovery seems to be much better and larger when compared to no cache. Finally, when comparing to the model that was solely trained on MBPP+HumanEval mix dataset, the model is able to slowly reach similar performance on both metrics. Overall, using our n-gram cache and despite the distribution shift, the model is showing resilience and adaptability to the new dataset. We will be updating the final paper with these additional plots in the appendix.\n\nTable 1. Performance with Data Distribution Shift\n\n|**Phase**|**Step Range**|**Model**|**Acceptance Rate**|**Speedup**|\n|-|-|-|-|-|\n|Initial Training (GSM8K)|0 – 1000|With Cache|Increasing ~0.15 - ~0.35|Increasing ~1.0x - ~1.45x|\n|||No Cache|Increasing ~0.14 - ~0.32|Increasing ~0.96x - ~1.42x|\n|Dataset Switch (MBPP+HumanEval)|1000|With Cache|Sharp Drop ~0.11|Sharp Drop ~0.87x|\n|||No Cache|Sharp Drop ~0.09|Sharp Drop ~0.86x|\n|Recovery (MBPP+HumanEval)|1000 – End|With Cache|Gradual Increase to ~0.28|Gradual Increase to ~1.22x|\n|||No Cache|Gradual Increase to ~0.22|Gradual Increase to ~1.16x|\n\n\n### Q2: Special Tokens Handling\n\n- **Clarifications**: Thank you for the question. Regarding special tokens across models with different vocabularies, we treat them as regular text string during the processing in the drafter. For example, tokens like `<im_start>` and `<im_end>` are handled using the drafter's tokenizer and encoded as part of the n-gram sequence. As training progresses, the drafter learns to associate these token patterns with their intended semantics, effectively learning the appropriate mappings.\n- As part of future work of extending our technique to multimodal models which might have a larger number of special tokens, if the draft model has unused reserved tokens, we can repurpose them to map to the target's special tokens and adapt their semantics through online learning. Also, we could consider incorporating separate embeddings for these special tokens and training them along with the drafter similar to AdaptiVocab (reference [35]), although this might not be optimal for on-device deployment. \n\n\n### Q3: Strategies for N-gram Cache Optimization on Edge Devices\n\n#### Current Cache Footprint & Growth Rate \n\n- **Clarifications**: Thank you for the question. ``Table 2`` shows the n-gram cache memory footprint across tasks, and ``Table 3`` indicates the growth of the n-gram cache size as training progresses on the MBPP+HumanEval dataset. The growth rate is defined as ``growth_rate = cache_size / training_tokens``\n- **Reasoning**: Overall, n-gram cache size across tasks remains relatively small compared to the size of the draft or target model. This indicates potential feasibility for on-device setting given the small overhead. We can also observe the cache size increases sub-linearly with respect to the amount of training tokens, and should eventually plateau as the growth rate decreases. This is likely the result of the cache capturing most of the high frequency n-grams. \n- Considering the cache overhead is minimal in our selected model pairs, it may not be the bottleneck for edge deployment. That is one of the reasons for our current Python dictionary based n-gram cache implementation. It offers average-case O(1) time complexity for both insertion and lookup, making it highly efficient for real-time access, at the cost of higher memory usage, primarily due to the overhead associated with hash table storage.  \n\nTable 2. Final N-gram cache size across tasks with Llama-68M drafter and Qwen2-7B target. The N-gram cache is implemented as a normal python dict.\n*Cache memory (MB) is derived from `pympler.asizeof.asizeof()`.*\n\n||GSM8K|MBPP+HumanEval|Alpaca|XSUM|\n|-|-|-|-|-|\n|**Training Samples**|7473|910|8000|4000|\n|**Cache Size (#n-grams)**|5569|2238|20339|17013|\n|**Cache Memory (MB)**|1.372|0.501|4.569|3.924|\n\nTable 3. N-gram cache growth rate during online learning for MBPP+HumanEval\n\n|Training Step|100|200|300|400|500|600|700|800|900|\n|-|-|-|-|-|-|-|-|-|-|\n|**Cache Size (#n-grams)**|667|1021|1256|1466|1628|1791|1929|2089|2208|\n|**Training Tokens (#tokens)**|26K|57K|85K|113K|141K|167K|195K|224K|252K|\n|**Cache Growth Rate**|0.0247|0.0177|0.0147|0.0128|0.0114|0.0106|0.0098|0.0093|0.0087|\n\n#### Cache Optimization Strategies  \n\n- **Clarifications**: In the case that memory is a limiting factor, one alternative is to use the prefix tree (Trie). Tries are particularly well-suited for n-gram storage because they exploit common prefixes among n-grams, leading to better memory efficiency. While Tries typically have slightly slower insertion and lookup times compared to hash maps, this trade-off can be acceptable when memory is a primary concern. Additionally, compression techniques such as radix trees (compressed tries) can be applied to further reduce memory usage.\n- Beyond structural changes, we can also consider implementing eviction policies to control the size of the cache, either by limiting the number of cached n-grams or by capping total memory usage. Common strategies include: Least Recently Used (LRU) that evicts the n-gram that hasn’t been accessed for the longest time, Least Frequently Used (LFU) that removes the n-gram with the lowest access frequency and hybrid policies that combine recency and frequency to better reflect usage patterns.\n- **Experiments**: `Table 4` indicates how speedup is impacted by different max cache size and cache eviction policies: LRU and LFU. `1/4` and `1/2` refer to using max cache size as a quarter and half of the full final cache size that we previously trained. `No Cache` and `Full Cache` are the same as $L_{DM}$ and $L_{DM}+\\lambda L_{N-gram}$ in Table 1 in the original paper.\n- **Reasoning**: Among all eviction policies and max cache sizes, both evaluation metrics are better than using no cache. Specifically LRU shows diminishing returns over the growth of the cache. Meanwhile LFU plateaus at an earlier stage, with performance matching the full cache even with a quarter of the size. This could be due to LFU capturing most of the high frequency and important n-grams, compared to LRU which only captures the recent n-grams. In the context of on-device deployment, this implies that with proper selection of cache size and eviction strategy, we can potentially attain robust performance given only constrained memory resources, highlighting the practical feasibility of our approach.\n\nTable 4. Cache size scaling v.s. Speedups on MBPP+HumanEval with Llama-68M as the drafter and Qwen2-7B as the target model.\n\n|Cache Size|Cache Type|Accept Rate|Speedup|\n|-|-|-|-|\n|No Cache|-|0.219|1.29x|\n|1/4|LRU|0.254|1.33x|\n||LFU|0.259| 1.36x|\n|1/2|LRU|0.261|1.35x|\n||LFU|0.263|1.36x|\n|Full Cache|-|0.267|1.36x|\n\n\n### Q4: Drawbacks with the \"dynamic adapter switching\" approach (LoRA)\n\n- **Clarifications**: Thank you for bringing up the question. We can identify two immediate challenges arising from the LoRA approach in OmniDraft\n    - Challenge 1: Depending on the hardware and framework used, there could be some overhead with dynamically switching LoRA adapters depending on the logic used. Frequent switching could also impact memory movement and maybe caching systems. It all depends on the final implementation of these frameworks on the hardware.\n    - Challenge 2: Regarding fixed-point quantization, which would become a necessity depending on the edge-device used, the different adapters could induce varying floating-point activation ranges, which could pose challenges for effective quantization. This can be overcome by using adapter-specific quantization parameters and dynamically switching these parameters based on the target or task.\n- Overcoming these practical challenges should bring us closer to a truly universal drafter.\n\n\n### Formatting Issues\n\nWe sincerely thank the reviewers for highlighting the typos and formatting issues. We apologize for these oversights and will address them in the revised version."}]

Author rebuttal:

Meta-review: This paper proposes OmniDraft, a unified framework enabling a single draft model to work with any target model and adapt dynamically to user data for on-device speculative decoding. Some reviewers expressed positive comments highlighting its novelty and practical contributions, while others raised concerns about aspects like cache scalability and online learning stability. After discussion, the authors addressed major concerns through additional experiments, including larger model evaluations, cache scaling analysis, and robustness checks on distribution shifts.

Therefore, the AC recommends to accept this paper.
Decision: Accept (poster)