Title: OmniDraft: A cross-vocabulary, online adaptive drafter for on-device speculative decoding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Speculative Decoding

LLM

Online Adaptation

Distillation

Tokenization

Inference Optimization

Fine-tuning

LoRA
Abstract: Speculative decoding generally dictates having a small, efficient draft model that is either pretrained or distilled offline to a particular target model series, for instance, Llama or Qwen models. However, within online deployment settings, there are two major challenges: 1) usage of a target model that is incompatible with the draft model; 2) expectation of latency improvements over usage and time. In this work, we propose OmniDraft, a unified framework that enables a single draft model to operate with any target model and adapt dynamically to user data. We introduce an online n-gram cache with hybrid distillation fine-tuning to address the cross-vocabulary mismatch across draft and target models; and further improve decoding speed by leveraging adaptive drafting techniques. OmniDraft is particularly suitable for on-device LLM applications where model cost, efficiency and user customization are the major points of contention. This further highlights the need to tackle the above challenges and motivates the “one drafter for all” paradigm. We showcase the proficiency of the OmniDraft framework by performing online learning on math reasoning, coding and text generation tasks. Notably, OmniDraft enables a single Llama-68M model to pair with various target models including Vicuna-7B, Qwen2-7B and Llama3-8B models for speculative decoding; and additionally provides up to 1.5-2x speedup.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new framework for speculative decoding (termed as OmniDraft) that addresses two major challenges in online deployment settings - cross-vocabulary incompatibility between draft and target models, and the need for dynamic latency improvements over usage time. OmniDraft proposes an online n-gram cache with hybrid distillation fine-tuning to handle vocabulary mismatches and leverages adaptive drafting techniques to further improve decoding speed. \n\nThe overall motivation of this work is to obtain a \"one ring rules all\" type framework - enable a single, small draft model to work effectively with various larger target models for speculative decoding in on-device LLM applications. The authors show empirically that their generalized idea indeed works in practice and gives 1.5-2x speedup on tasks like math reasoning, coding, and text generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper tackles two significant practical challenges in speculative decoding: cross-vocabulary mismatch and online adaptation for latency improvement, both of which are crucial for real-world on-device LLM deployment. In this regard, this is a good contribution.\n- There are some technical strong ideas in this paper such as the introduction of an online n-gram cache for cross-vocabulary translation which I find a very effective solution to the vocabulary mismatch problem - perhaps there are more efficient ones but this is a reasonable practical approach imo.\n- Integration of online adaptive drafting is another further improvement in the proposed method - this makes their approach more flexible by allowing the drafter to dynamically adjust proposal length.\n\nWeaknesses:\nFirst of all, I applaud the authors for thinking through and bringing out several possible limitations of their work. Many of my thoughts are in the same direction, although I will re-emphasize:\n- The paper acknowledges that online adaptation, being limited to a single iteration of the data stream, still has potential for instability on new unseen data. This is an inherent challenge of online learning but remains a practical limitation.\n- Is the draft model indeed universal? How does it work in the presence of distribution shifts and data heterogeneity? I feel the discussion on this is limited in this paper but compelling direction to make such methods work in practice.\n\nQUESTIONS:\n- Beyond the mentioned future work of incorporating an explicit adaptive head in the cross-vocabulary setting, what other avenues do the authors have in mind to handle special tokens?\n\n- Could you elaborate on the potential strategies for optimizing the n-gram cache for memory-constrained edge devices? (e.g. offload, caching, indexing, etc)\n\n- Do the authors see any other drawbacks with the \"dynamic adapter switching\" approach e.g. for LoRA work in practice?\n\nLIMITATIONS:\nN/A (no data or models released).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes OmniDraft, a framework designed to shift the speculative decoding paradigm for on-device applications from a static, model-specific setup to a universal and adaptive one. The central idea is to enable a single, lightweight draft model to not only serve a variety of target models with incompatible vocabularies but also to continuously improve its performance through online learning from user data. To achieve this, OmniDraft introduces two core interacting mechanisms: an online n-gram cache to resolve cross-vocabulary tokenization mismatches on-the-fly and a hybrid online distillation process that continually aligns the draft model with the target model's verified outputs. These components, combined with adaptive drafting techniques for efficiency, are shown to successfully allow a single Llama-68M to accelerate various models like Qwen2-7B and Llama3-8B by up to 1.5-2x.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n•\tSignificance: Addresses a critical and practical problem in LLM deployment: enabling a single draft model to work with any target model.\n•\tOriginality: The \"one drafter for all\" online adaptive framework is a novel paradigm, using an online n-gram cache for cross-vocabulary translation and hybrid distillation for live adaptation.\n•\tQuality: Presents a solid empirical evaluation across various incompatible models and tasks, supported by relevant ablation studies.\n•\tClarity: The paper is clearly written, with its methodology clearly outlined in pseudocode and its experimental results presented effectively. \n\n\nWeaknesses:\n•\tN-gram Cache Scalability: The scalability and memory footprint of the online n-gram cache, critical for on-device use, are not fully analyzed.\n•\tOnline Learning Stability: Online adaptation can be unstable, as shown by training curves (including in the appendix), raising long-term stability concerns.\n•\tModest Speedup: The reported speedup (up to 1.5-2x) is valuable but more modest compared to results from other specialized systems.\n\nQUESTIONS:\n•\tThe paper correctly identifies the N-gram cache's memory footprint as a potential limitation for on-device use. Could the authors provide a brief empirical analysis of the cache's growth rate and final memory size observed during the experiments? This would be very helpful for understanding the practical scalability of the approach.\n•\tThe experiments effectively demonstrate OmniDraft's performance with target models in the 7-8B parameter range. Have the authors conducted any preliminary experiments, or do they have insights on how the framework would perform with significantly larger target models? The alignment difficulty and potential speedup could change substantially at that scale.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents OmniDraft, a speculative-decoding framework that lets a single 68 M-parameter drafter accelerate multiple large-model back-ends (Vicuna-7B, Qwen2-7B, Llama-3-8B) even when their tokenizers differ. Three technical pieces enable this:\n\n1. Cross-vocabulary n-gram cache. During inference, the system merges sequences of draft sub-tokens into target tokens via a cache that is populated on-the-fly, allowing correct acceptance-ratio tests across vocabularies.\n\n2. Online hybrid distillation. A token-level KL term aligns directly mapped tokens, while an NLL term aligns cached n-grams, continually fine-tuning the drafter as it serves requests.\n\n3. Online adaptive drafting. A lightweight head predicts token-acceptance probabilities and shortens or lengthens drafts to maximise speed per wall-time budget.\n\nSTRENGTHS AND WEAKNESSES:\n1. Quality – The empirical study spans four tasks and three heterogeneous targets, with clear ablations on cache utility, loss choices and drafting heads. However, all timing figures are relative; absolute wall-clock latency, GPU utilisation and memory footprint of the cache are absent. Online adaptation is evaluated for only a single epoch and on 68 M→7 B/8 B scales; it remains unclear whether gains persist for larger drafters or longer streams.\n\n2. Clarity – The methodology is carefully derived and Algorithm 1 is explicit. Yet crucial implementation numbers (cache size limits, eviction policy, adaptive-head thresholds) are scattered or deferred to the appendix, and the main text exceeds typical concision norms.\nSignificance – Decoupling drafter and target tokenizers is an important step toward real on-device speculative decoding. Still, reported speed-ups hover around 1.6 ×; rival approaches such as UAG or AdaptiVocab are mentioned but not compared directly, leaving the incremental benefit uncertain.\n\n3. Originality – The live n-gram cache that learns cross-tokenizer merges during inference is novel, and combining it with continuous distillation and adaptive draft length is an elegant unification.\n\nQUESTIONS:\n1. End-to-end latency & energy. Please supply absolute wall-time and energy figures for each target on a mobile GPU or CPU to confirm that 1.6–2 × speed-up translates into real user-perceived gains.\n\n2. Cache footprint & scaling. How large does the n-gram cache grow after, say, 10 k prompts, and what is its lookup/update time under concurrency? A memory/throughput chart would clarify on-device feasibility.\n\n3. Robustness to target drift. If the cloud back-end swaps from Qwen2-7B to Qwen2-14B mid-session, how quickly does the drafter re-align, and does the cache hurt or help during the transition?\n\nAddressing (1), (2), (3) with concrete measurements would raise Quality and Significance and push the paper into clear-accept territory.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper introduces OmniDraft, a unified speculative decoding framework that allows a single, efficient draft model to work seamlessly with any target language model, even in online and on-device settings. The approach combines an online n-gram cache and hybrid distillation fine-tuning to resolve vocabulary mismatches between draft and target models, and uses adaptive drafting techniques to further accelerate decoding. The authors demonstrate that OmniDraft can pair a single small draft model with multiple large target models across diverse tasks achieving notable speedups of up to 2x without sacrificing flexibility or performance.

STRENGTHS AND WEAKNESSES:
Strength

s1: This paper is well written and easy to follow

s2: This paper comes up with the novel idea of solving incompatibility issue between draft and target model by integrating an online n-gram cache

s3: This paper experimentally shows that their method achieves better performance and speed ups compared to previous baselines.

Weakness

w1: I recommend authors to do more experiments with more downstream tasks and different/larger LLMs

w2: It would be better if authors can add more baselines other than SpD.

QUESTIONS:
1. In terms of scalability, could you share any intuition of how the performance of this new speculative decoding method changes as the size of LLMs gets larger?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-04