Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs
Authors: Hongyi Liu
Jiaji Huang
Zhen Jia
Youngsuk Park
Yu-Xiang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Online algorithm
Speculative decoding
efficient LLM
Primary area: generative models

Abstract: Speculative decoding is widely used in accelerating large language model (LLM) inference. In this work, we focus on the online draft model selection problem in speculative decoding.  We design an algorithm that provably competes with the best draft model in hindsight for each query in terms of either the token acceptance probability or expected acceptance length. In particular, we show that we can accurately evaluate all draft models, instead of only the chosen model without incurring additional queries to the target model, 
which allows us to improve exponentially over the existing bandit-based approach as the number of draft models increases.
Our approach is generically applicable with any speculative decoding methods (single draft, multi-drafts and draft-trees). Moreover, we design system-efficient versions of online learners and demonstrate that the overhead in computation and latency can be substantially reduced. We conduct extensive experiments on open-source LLMs and diverse datasets,
demonstrating that our methods substantially outperform the state-of-the-art EAGLE3 and the BanditSpec baseline in a variety of domains where specialized domain-expert drafters are available,
especially when long reasoning chains are required.

Published as a conference paper at ICLR 2026
PROVABLY NO-REGRET DRAFTER SE- LECTION IN SPECULATIVE DECODING FOR LLMs
Hongyi Liu
Jiaji Huang², Zhen Jia", Youngsuk Park2 Yu-Xiang Wang?.39
"iiee University, 2 "Amazon Web Services, University of California, San Diego hongyi.liu @ @iuce.edu, {jajjhug, zhenj, pyoungsu) @ @amazon.com yuxiangw @ uesd.edu
ABSTRACT
Speculative decoding is widely used in accelerating large language model (LLM) inference. In this work, focus the online draft model selection problem in speculative decoding. We design an algorithm that provably competes with the best draft model hindsight for each query in terms of either the token acceptance probability or expected acceptance length. In particular, we show that we can accurately evaluate all draft models, instead of only the chosen model without incurring additional queries to the target model, which allows us to improve exponentially over the existing bandit-based approach as the number of draft models increases. Our approach is generically applicable with any speculative decoding methods (single draft, multi-drafts and draft-trees). Moreover, we design system-efficient versions of online learners and demonstrate that the overhead in computation and latency can be substantially reduced. We conduct extensive experiments open-source LLMs and diverse datasets, demonstrating that our methods substantially outperform the of EAGLE3 and the BanditSpec baseline in variety of domains where specialized domain-expert drafters are available, especially when long reasoning chains are required.
INTRODUCTION
Speculative decoding (Chen et al., 2023; Sun al., 2023; Li et al., 2025b) is widely adopted for accelerating large language models (LLMs). uses a smaller surrogate model-refenree to as a draft model simply drafter-to predicr the sequence that a larger target model would generate. These predictions are then verified in parallel by the target model. When the drafters make correct guesses, single expensive target pass is leveraged produce multiple tokens, reducing per-token latency.
However, a single drafter may perform well on certain tasks but fail badly on others, leading to inconsistent quality of service and long-tail latency for certain queries. For example, retrieval-based drafter works well when outputs closely match the input but falters elsewhere (Hou et al., 2025). Similarly, domain-speciff drafters (e.g.. for code, scientific writing, or summarization) excel their own fields yet perform poorly outside them (Liu et al., 2023a; Kim et al., 2024; Yi et al., 2024). This raises a natural question: given access to multiple candidate drafters, how can we dynamically select the best one for each incoming query? Formally, we study the problem of online drafter selection:
Given a pool of N drafters, can we design an algoritlum whose performance nearly matches that of adopting the best drafter in hindsight, for every user query?
This problem was originally posed in MetaSD (Kim et al., 2024), who framed bandit task. Their method, together with the more recent BANDITSPEC (Hou et al., 2025), balances exploration (trying different drafters) and exploitation (using the empirically best drafter).
malti-armed
In this paper, we make the surprising observation that exploration is unnecessary. By carefully leveraging the structure speculative decoding, we show that is possible to efficiently compute feedback for all drafters- just the one selected-withond incurring additional calls to the target model. This transforms the problem from bandit setting into a full-information online learning
Work done during internship AWS. 'Correspondence Hongyi I.., Youngsuk P., Yu-Xiang W.

---

Published as a conference paper at ICLR 2026
Selagg
Prefxx: HedgeSpec chooses
with no remorse regret Evaluation
Drafting
on diellerge
selecled
Verification
Cheristry
dretter
others performen?
Tange: cesornatioon
Python
Drafhon dleritutoo with the vorting traedwar
Fool cheesse dicftors
Medica
des
Full Information feedback
Next speculative round
UC3SPEC
CXPSSppe:
CML EH
Lolden
2044
Matth:ON Coding-4
Mathh drafter)
Coding drafter]
Requests
Quean
Qwan
Figure 1: Overall workflow of HedgeSpec. Left: Acceptance rates across domains for Qwen-3-8B highlight each drafter's s strong in-domain performance but steep decline outside its expertise, posing challenges for effective drafter selection. HedgeSpec achieves performance close to that of the best- performing expert. Right-top: Workflow of HedgeSpec. During the verification phase, lightweight evaluation collects panoramic feedback from drafters by estimating the distribution gap between each drafter and the target. Right-bottom: Speedup ratio across baselines. HedgeSpec benefits from holl-infommatio feedback, achieving the highest speedup ratios.
problem. Building on this insight, introduce HEDGESPEC, which uses algorithms such as Hedge (Littlestone & Warmuth, 1994; Vovk, 1995) or NormalHedge (Chaudhuri et al., 2009) to identify the best drafter exponentially faster in N than bandit-based approaches.
Our key contributions are as follows:
Tull-information framework for multi-drafter decoding: We designed novel method that accurately evaluates all drafters - not only the one that is chosen with very low overhead, hence enabling full-information online drafter selection.
Theoretical guarantees: We formulate and analyze the drafter selection problem under both ac- ceptance probability and expected acceptance length objectives, establishing no-regret guarantees.
Empirical validation: We conduct extensive experiments LLaMA-3.1-88B and Qwen-3- 8B/32B reasoning model each with seven drafters, demonstrating that HEDGESPEC consistently outperforms EAGLE (up to 83.7% token/s gain in single domain and up to 46. 1 % average) and bandit baselines (up to 49% MAT gain) in both acceptance rate and reduced per-token latency.
Related work and novelty. To the best of our knowledge, we the first show that full- information online learning is possible for drafter selection. Prior art (Hou et al., 2025) modeled the problem as a bandit problem. HedgeSpec could work with any given set of drafters on any speculative decoding method, thus making our contribution orthogonal to methodological innovation in speculative sampling and drafter curation (Sun et al., 2023; Cai al., 2024; Li al., 2024a; 2025b).
While we do not invent any new online learning algorithm, it is highly nontrivial to apply existing algorithms correctly to this problem. The core of our innovation is an off-policy estimator that returns correct acceptance length in expectation, as well as to address subtle censoring issue using learning from delayed feedback (Joulani al., 2013).
On the system front, we designed a practical version HedgeSpec that carefully balances the computational/latnnyy cost of the evaluation and the statistical efficiency. As for the evaluation, curated twenty-one drafters: seven dedicated to each target model, with every drafter specialized in particular domain. We demonstrated that HedgeSpec with these drafters (a collection of specialists) outperforms the fate-of-the-art EAGLE3 (a strong generalist) by a sizable margin.
2

---

Published as a conference paper at ICLR 2026
2
PRELIMINARIES ON SPECULATIVE DECODING
2.1 SYMBOLS AND NOTATION
We denote the vocabulary by V, and generated sequence by 21:7, where each token 24 V. The target language model defines probability distribution p(-), while the draft (or surrogate) model defines q(-). A speculative decoding method M, uses q generate K speculative tokens e.+ 1-4 at chunk onset index ls which are then verified in parallel using p. We use to index multiple draft models q; / for the global token position, for chunk indices, and k [K] for token positions within a chunk. The token acceptance probability given that all previous tokens are correct is denoted by y [i]. We adopt standard probability notation p(-), conditional probability p(- +), and conditional expectation [-| 1]. with all probabilities assumed to be discrete.
2.2
BASIC SPECULATIVE DECODING METHOD
The basic speculative decoding algorithm proceeds as follows:
Generate a sequence of K draft tokens 844144 using a draft model q, i.c., 444144 ~ ( [<+++++<<//)))) (xgse)
p( 2. For the target model, compute pl paap://allaaaa......cc.cccce.ee.
in parallel.
3. For k: = 1,2. K, if Z ~ Uniform((0, 1)), if 2 p assign 2+44 < fe+* and continue; clse draw Ze+k ~ (res(z) x max p(2)0<((8-))) -((00((((xx1)1) and break.
all K tokens pass, assign T++ p(
Theorem 1 (Leviathan et al., 2023, Theorem 3.5). The samples generated from the speculative decoding algorithm with any draft model are drawn from the same distribution as The acceptance probability of the above algorithm is Exey min min(p( q(x)}
TV q) where TV TV(p,)) ErEv (p(x) - ((x)) denotes the total variation distance of two probability distributions.
2.3 EAGLE AND DRAFT TREE
The EAGLE family is the most widely deployed speculative decoding models (Li et al., 2024a;b; 2025b). EAGLE-3 introduced multi-layer feature fusion with a training-time test mechanism. EAGLE- 2 expands the draft tree with high-confidence tokens and prunes low-probability branches, improving efficiency by increasing the likelihood that more tokens are accepted per cycle. Specifically, instead of sampling from g.:K recursively like language model, it generates a deterministic (*) draft-tree with Depth K and branching factor L. From the root node (æz²), EAGLE model q generates L children as possible choice of rt+1- These are chosen to be the tokens with largest probabilities according to q' (+lagge). Then from each child, another L descendants (subsequent tokens) are generated by sorting y² (-lxggr11). Different from speculative decoding, EAGLE drafters are not sensitive to actual numerical values in q. Instead, only the relative ranking of the tokens matter."
Theorem 2. EAGLE with any draft model q returns samples from the correct target distribu- tion Moreover, the probability of accepting token at step k given all previously accepted tokens is EveTopyiot (p:seee-)) P(u)zg(4k-1). If dynamic pruning is used, then the candidate set Top, (q(- ((((1=)++)1)) should be replaced with the pruned set of descendants of I-
We defer the proof of Theorem to Appendix A.1. All references to FAGLEE our paper refer to EAGLE-3 unless specified. We defer more related work discussion regarding speculative decoding, adaptive drafting and online algorithms in Appendix C.
2.4 PERFORMANCE METRICS FOR SPECULATIVE DRAFTER EVALUATION
We key metrics evaluate speculative decoding method Ma.pi the Expected Token Accep- tance Probability (ETAP) and the Expected Acceptance Length (EAL). These capture complementary notions of per-token accuracy and end-to-end efficiency.
E-2 implementation focused on growing a deterministic greedy tree. TThe dependence on the values of g becomes relevant when the dynamic pruning approach from EAGLEE-2 is used, but does not change the temperauur-invariincee of the original draft tree.
It could also randomized,
3

---

Published as a conference paper at ICLR 2026
Token Acceptance Probability (TAP). At decoding step t, given prefix z<t. the drafter q(-)
generates a speculative token ft. The token is accepted it matches the target model p(-):
Tt
Pr
[ 3t [=a)-
Ma.
The ETAP is the expectation of this probability over distribution of prefixes D;
ETAP(M D) E [
ETAP measures the average per-token reliability of the drafter.
Expected Acceptance Length (EAL). Speculative decoding drafts K tokens per chunk starting at index la which then verified by the target model. The acceptance length AcceptL.nngthe ( the number of consecutive tokens accepted before the first mismatch. Even given a fixed prefix Egt, AcceptLength remains random variable because My often involves sampling. The EAL is:
EAL(M D) Exgg, ~ (wcp._AAcceeLenggh(reeee..
xgg,,]]-
EAL reflects how many tokens are accepted per chunk and thus directly determines decoding efficiency: higher EAL means fewer calls to the target model, reducing time per token. We will later show how these two metrics are used to evaluate drafters.
The Role of the Prefix Distribution. Here prefix 2SG,, refers only to the initial prompt but also to the generated tokens that extend it up to step t. is inherently random, determined jointly by the prompt, the stochastic rollout of the target model p(-). and the "tempo" of speculative decoding (i.e., where mismatches occur and chunks restart). The distribution 'D over prefixes provides a useful abstraction: encapsulates all these factors into a single probabilistic view, allowing evaluate methods without conditioning specific prompts or model trajectories.
3 HEDGESPEC: METHOD AND THEORY
We present HedgeSpec, a full-information online learning framework for drafter selection. By leveraging lightweight evaluation for panoramic feedback, HedgeSpec adaptively identifies most effective drafters, improving acceptance rates and end-to-end efficiency.
3.1 A NO-REGRET ONLINE LEARNING APPROACH TO DRAFT MODEL SELECTION
We are given N draft models .1...., QN- every time i, we decide which draft model to use to generate the next draft tokens. Our goal is compete favorably with the best draft model that gives us either (a) the highest acceptance probability overall or (b) the highest expected accepted length per chunk. In this section, we will formulate this problem as no-regret online learning problem, design appropriate loss functions that align with the two optimization objectives above, and develop algorithms that come with provable guarantees.
The performance guarantee stated as a regret:
T Regret, Σ Sildd
min NEN
Σ fl
where fe the loss function time determined by how well draft model q can perform time t. For example, ft can measure the probability of not accepting tokens at time or how far the expected accepted length from K where K is the depth of the drafts.
3.2 FULL-INFORMATION -AAAMMMAATTO-EAAAATIIOOO HEDGESPEC NOT A BANDIT
Our main algorithmic idea is to add evaluation phase between token verification and drafting. This evaluation phase was introduced in the BanditSpec paper. Differently, our work is motivated by that can get feedback for all draft models, i.e., the full-information feedback, rather than only the draft model that we choose to play, i.c., the Bandit-feedback model. Adapting to different drafters changes generation speed, but thanks Theorem and 2, such adaptation does not change the distribution of
4

---

Published as a conference paper at ICLR 2026
the output tokens. Moreover, it will become clear, the additional evaluation drafters does not require extra queries to the expensive target model, and can be carried out in parallel.
How does this evaluation phase work? naive way is to roll out each drafter explicitly against the target, which is costly and infeasible. Instead, our key idea is that a single verified trajectory from the target serve counterfactual evidence for evaluating all drafters. Concretely, after the chunk of new tokens F++1:++ are verified for (.,, we prefill them into for each of the other i E [M] This yields feedback from the trajectory, summarized by sequence of feedback vectors:
7 [ P; [z is accepted| for i [M] E [T]
For each method of speculative decoding, the acceptance probability is calculated differently. For example, for the standard speculative decoding with a single draft, 73.4 = 1--VV [I<(+j-1)] by Theorem 1. For EAGLE's Greedy- -Drfft Tree approach 7i is the total probability of children nodes parent Ige+j-1 on the draft tree (constructed and pruned using qi in Theorem 2).
TV - (=
Once these feedbacks collected, they form the basis for computing losses for each drafter, which will then be used to update the drafter-selection strategy. Crucially, our first result (Theorem 3) shows that the acceptance probabilities derived from the above verified trajectory are sufficient to construct an unbiased estimator of the acceptance length for any drafter.
Theorem 3. Let M be speculative decoding method and K be the depth of its drafts. We write I++1itK ~ p(-(ag() where p refers to the target model. We denote "Ih = accepted, x=g++k1]-- Also, define Yk+1 0 for notational con-
P PM is accepted venience. The "one-step counterfactual" estimator:
K+1 Σ
AcceptLength, [|M)
k(1
7h)
II
%
j=1
satisfies that EM AcceptLength,,tr x [M]
TSe = EM [ of accepted tokens
The proof of Theorem 3 is included in Appendix A.2. This result is non-trivial because the probability 7ik is not the probability of accepting the specific realized token Ze++ given E=+1--k-1 but rather the probability of accepting any token Te+* ~ 00-10++1:++==
Yi
even if Æ # Tr+-k. We cannot compute the # of accepted tokens di- rectly because that would require access all possible (combinatorial many) ways the target language model rolls out (see the illustration in Figure 2). The theorem proves that as long as we have single trajectory rolled out by the target model (which we do since speculative decoding is lossless), we can counterfactually compute an unbiased estimator of the acceptance length for any alternative drafters.
Y2
Y3
Figure 2: Illustration of the conditional acceptance probability 7 for the EA- GLE model. Observe that T is not the probability of accepting 2g only, but rather the total probability of accepting a token at level Tt- Also note that for evaluating the estimator, never need to compute the probabilities the target model for the possible alternative trajec- tories that the draft models generate.
Variance of our estimator. Observe that our length esti- mator has bounded range [1, K 1]. thus by Popoviciu's
inequality Var AcceptLength,,cr [ [M] E&Ct < K2/4
In comparison, the estimator from BanditSpec (EXP3- Spec) is ONNK2). which grows with N.
Connection to Experience Replay in Reinforcement Learning. Our evaluation phase is similar to Experience Replay in the RL literature at glance. However, we do not have the additional challenges in solving the distribution-shiie induced by the policy used to collect the experience. For this reason, our "experience replay" for other draft models can substantially more effective than that in RI.
5

---

Published as a conference paper at ICLR 2026
3.3 HEDGESPEC: LOss FUNCTIONS AND DELAYED FEEDBACK
We have seen that a single verified trajectory suffices to estimate drafter's EAL. The remaining step formulate the online learning game: each round = ... nature generates loss vector f E [0, 1J8h and the algorithm selects a drafter in and incurs a loss of [e(
We just need figure out how to instantiate it in our problem. There are several options. Each round of the game can be one token or one chunk of tokens. Also, we can choose the loss functions to optimize acceptance-length
acceptance probability. Pros and cons of these decisions described in the Appendix B.3, but we find that the most natural setting is to directly optimize the acceptance length token-level game. The loss function is then chosen to be:
K+1
A-1
Σ
h(1 - - (+) II 7
measures how much room the inh drafter has to improve in its acceptance length from the maximal chunk length K + if it starts at predicting token l. If we to optimize acceptance probability instead, then we would choose fr = %/1..
By Theorem 3, the expected loss [ [x]
Chunts of ofaccopted collons from the cheson drafters
Chunis of acceored tollons from the chosen diafters
x:
P[Acrept, I
Pau
P[Acrepty Ixx
Ysx
Yau
AecopELen
Accipt
Feedback
Feedback
Acceppt
Draftari
Accepi-ten
Chunk-
Accept
level
Ascept-Lats 44.4
leaming
learning
(oo w't
Accept Los a 3.34
Acceps Len 3.34
chosen
tm f..... 5)
Ancupt Latt 2.58
chosen t- 1,4,6)
Arrept Jast 2,98
Accipi Len 3.42
AccipE 8.42
Not dats,
Weitandiee-14
Figure 3: Illustration of the counterfactual feedback for Drafter j (per-token per-chunk) while the model keeps collecting new data. In this example, the generated token length K = 5. For the token-level learner, the feedback is delayed for K steps, while for the chunk-level learner, the delay depends on the average number of chunks to collect K K-tokens.
Readers with sharp eyes may notice a problem the standard online learning game is not applicable! The accepted tokens are not observed immediately after token I, they are revealed in chunks after each speculative decoding chunk completed. Moreover, unless the chosen drafter maxes out the
acceptance length, there is not sufficient information to compute our estimator Accept. Length, K- This is because alternative drafters may end up accepting more tokens than the drafter we chose. We refer to this problem censoring issue. If we simply ignore the issue and compute the truncated version of the losses, we may end up getting stuck at suboptimal solution (see the Appendix B.4 for an example). In other words, the learner may need to wait until multiple chunks of data to become available before getting the loss vectors for each time (see Figure for an illustration).
for ...., Tioken
1. Nature chooses loss vector 10
2. Player chooses Drafter it for token t and incurs loss fijee]-
3. If is the end of chunk, Player observes f&e (if optimizing acceptance prob) or observes fsi-K (if optimizing acceptance length).
Figure 4: Token-Lever Game with Delay.
These issues can be modeled as "delayed feedback" (Weinberger & Ordentlich, 2002; Joulani et al., 2013; Van Der Hoeven & Cesa- Cesa-Bianchi, 2022), which can be handled through blackbox reduction to the standard online learning without delayed feedback with a mild increase in the regret as a function of the expected maximum delay.

---

Published as a conference paper at ICLR 2026
Theorem 4. Assume Trkken > 2K + 1. Let A be (Joulani al., 2013, Algorithm 1) instantiated with Hedge or NormalHedge as BASE. Then it chosen by A in the game defined in Figure satisfies that:
When we optimize acceptance probability:
Flime P PAla is accepted) Z mmxx [[[]
Towes E/ P.- (zg is accepted) }
Olv
Klog T
Tasb
2.
When we optimize accepted length:
Tales E/er ExlAccepiLength,, (ill 2 max
{ Tua Eer ElAccepiLength, xx("1]]]]]]]]
O(v
K+11 Tates
The proof (rigorously written in Appendix A.3) applies the reduction to the regret bounds without delay in Theorem of (Joulami et al., 2013) by noting that the max delay 2/. Then the result follows by taking expectation (over the distribution induced by the target model) on both sides and applying the definition of "/e [i] (or the unbiasedness of the length estimator Theorem 3).
The result says that the choices made by the online learner is nearly as good as the optimal choice in terms of either the average acceptance probability, or the expected accepted length.
3.4 PRACTICAL ALGORITIMMS AND EXTENSIONS
In this section, briefly describe the online learning algorithms that chose implement for the experiments. More detailed algorithms blocks can be found in the cited references therein. In our paper, adopt NormalHedge (Chaudhuri et al., 2009) our base algorithm (details in Figure 7). For discussion regarding more hedging variants, please refer to D.6 for detailed discussion.
Handling delays. We the algorithms in (Joulani et al., 2013) for handling delay. In particular, the theoretical results above can be obtained by using (Joulani et al., 2013, Algorithm 1), which handles bounded but non-constant delay (WWeinberger & Ordentlich, 2002) requires constant delay). Moreover, since our problem is not really adversarial, but rather a Markov process induced by the target LLM, we found that (Joulani al., 2013, Algorithm 2) that operates the "stochastic setting" works the best for us in practice. The algorithm is stated for the more general "partial monitoring" setting, but we are instantiating queue of the feedback and keep applying the next available actions generated by the base algorithm. In the iid setting, the regret is Regret,- + (MaaxDely with Regret, being the regret achieved by the base algorithm. We apply this algorithm heuristic (despite that the settings are iid) for the efficiency of learning in practice. More practical implementations heuristics can be found B.5.
the full-information setting. Basically, the idea to maintain a
4 EMPIRICAL EVALUATION
In this section, we provide comprehensive evaluations HedgeSpec by conducting the following analysis: 1. does HedgeSpec yield better end-to-end performance; 2. in-depth analysis of hedge based selection process. 3. How does HedgeSpec perform relative to offline based method?
4.1 EXPERIMENT SETUP AND BASELINES
We use Llama-3.1
(Dubey et al., 2024), Qwen-3-8B and Qwen-3-32B (Yang al., 2025) reasoning model as target models. In our main paper, all drafters in this paper are implemented with EAGLE-3, widely used framework for speculative decoding. More experiments and discussion regarding the integration of HedgeSpec to broader speculative decoding framework can be found in Appendix D.9. We adopt EAGLE's default generation configuration (see Section D. 1). For hedge algorithm update,
expected length to compute the loss and more detailed setup can be found in B.5. We compare against the state-of-the-an drafter selection framework, BanditSpec (Hou et al., 2025), including both Exp3Spec and UCBSpec variants.
We report the Mean Number of Accepted Tokens (MAT) along with the wall-clock time required to complete each request which
to compute token per second. All models are served using FP16 precision with batch size of 1, following standard practices in latency- focused studies (Fu et al., 2024; He al., 2023; Cai al., 2024). Specifically, Token/s reflects the end-to-end latency
7

---

Published as a conference paper at ICLR 2026
Datasets
Python
Math
Biology
Chemisrry
MedQA
CNN.DM
SQL.
Drafter domains MAT
TToeenks
MAT
Token's
MAT
Taken's
MAT
Token's
MAT
Tokenes
Token's
Tokenls
Avg
Avg Token's
EAGLE
6.48
87.37
5.88
76.35
5.95
71.20
5.28
71.28
4.96
66.48
5.31
67.26
5.99
90.41
5.69
74.34
Python
7.89
106.99
5.05
67.50
2.86
36.63
3.64
19.87
2.65
33.77
2.94
31.12
4.87
65. 65.45
4.27
55.90
4.52
62.77
8,03
106.43
3.07
4.15
3,02
3,32
44.79
60.16
58.79
Biology
3.29
35.34
3.91
47.46
7.27
96.10
4.35
56.42
4,42
56.44
3.17
36.19
2.72
28.78
4.16
50.96
Chamisry
3,80
$1.84
6.46
86.28
4.45
59.93
7.39
96.28
3.82
50.42
5.09
32.56
3.76
46.90
4.68
60.60
MedicalQA
3.98
4.49
4,95
3.38
ENN_DM
2,07
26.85
2,89
38.79
3.02
40.55
2,99
99.21
3.11
42.22
6.20
77.19
2.14
28.
3.20
41.85
SQL
3.71
44.16
3,66
46.47
2.64
36.34
2,96
31,84
2.62
34.15
2,90
37.94
8.49
114.52
3.85
49.35
Table 1: Statistics of the curated drafters with Llama-3. 3.1-BB-I. the target (Bold best). Each drafter performs strongly in-domain (diagonally) but suffers noticeable inefficiency when applied outside, and on average performs worse than the vanilla EAGLE model. Similar trends are observed in Qwen's 's drafters in Table 6. These drafters form realistic evaluation pool for evaluating HedgeSpec, and we will see in Table 3, HedgeSpec orchestrates them to jointly accelerate the LLM inference.
during inference. These metrics are widely used (Hou et al., 2025; Li al., 2025b) in speculative decoding and are positively correlated: longer accepted lengths typically lead to better throughput. All experiments are conducted on nodes with NVIDIA A A100 GPUs connected via NVLink. We defer jointly trained drafter discussion D.5 and hedging algorithm variants D.6 due to space limit.
4.2 CURATING DIVERSE DRAFTERS FOR LARGE- EVALUATION
To thoroughly evaluate the effectiveness of our framework at scale, build 21 drafters upon the official EAGLE-3 models (Li et al., 2025b; Tengyunw, 2025; Contributors, 2025) and finetune them on seven open-sourced datasets spanning multiple domains: Python (jtatman, 2025), Math (Toshniwal et al., 2024), Biology (Wesney, 2025). Chemistry (mlfoundations dev, 2025), MedicalQA (Chen et al., 2024), CNN DM (Nallapati et al., 2016 and SQL (Meyer et al., 2024). The statisties of the resulting Llama and Qwen drafters are summarized in Table 1, and 7. We observe that finetuning the generic EAGLE on specific domain can greatly enhance its in in-domain ability. We use SpecForge (Li et al., 2025a) as the training pipeline. In the meantime, no single model performs well across all domains, often exhibiting noticeable efficiency drops outside specialization, and on average performs worse than the vanilla EAGLE. These drafters constitute realistic experimental pool for evaluating our framework, and we will see that SedgeSpec orchestrates multiple drafters without prior knowledge to jointly accelerate the LLM inference process.
4.3 END-TO-END EFFICIENCY ANALYSIS
We report the efficiency evaluation of HedgeSpec. We first conduct overhead break down in HedgeSpec, followed by the end-to-end evaluation comparing with EAGLE and BanditSpec based drafting. Overall, HedgeSpec outperforms all other baselines with significant margin consistently across every single domain, demonstrating its effectiveness.
4.3.1 EVALUATION OVERHEAD BREAKDOWN
We analyze the evaluation overhead induced by incorporating global feedback for faster adapta- tion. This overhead comes from two major com- ponents: (i) prefilling drafters and (ii) comput- ing losses and updating hedge weights. Table reports the breakdown. Here, 'Llama" and 'EA- GLE forward' denote forward passes through the target and drafter, respectively, while 'hedge update' includes both loss computation and Nor- malHedge weight updates. Evaluating drafter costs roughly 1/25 of a target forward, since EAGLE's drafter is a lightweight one-layer transformer compared to the 32-layer 8B Llama target. Under ideal assumptions, it implicates that if HedgeSpec secures one additional MAT, the gain offsets the cost of evaluating up to 25 drafters in sequence. In practice, the overhead is even smaller since drafter evaluations are independent and can be run in
Table Overhead comparison for drafter evalua- tion for Llama-3.1-8B-1 Compared to expen- sive target call, even a small boost in acceptance from Spdg is enough to offset the cost of evaluating all drafters.
Major Compunents
Forward
EAGLE
Hedge Update
Timelms
75.709
2.497
0.413
8

---

Published as a conference paper at ICLR 2026
Datasets
Python
Math
Biology
Chemistry
McdQA
CNNNUMM
SQL
Methods
Token/s
Tokene's
Tokesls
MAT
Taken's
Tokends
MAT
TTokess
MAT
Token's
Arg
Avg Token's
LLOMA-3.1-8BJ1
1.00
17.03
1.00
18.87
1.00
18.44
1.00
18.25
1.00
18:57
1.00
17.86
1.00
17.86
1.00
18.13
Fagle
6.48
87.37
5.88
76.35
5.95
71.20
5.28
71.28
4.96
66.48
5.31
67.26
5,99
80.41
5.00
74.34
UCBSpec
74.39
77.57
70.27
5.11
4.46
3.94
50.47
5.71
5,09
EXPSSSpe
5.16
69.29
5.59
74.21
4.93
67.70
4,93
64,25
4.25
58.13
3.81
49.70
5.37
73.29
4,86
65.
HedgeSpec
7.69
99.58
7.69
98.63
7.18
93.78
7.10
89.65
6.47
77.26
5.58
70.68
8.06
103.31
7.15
90.41
Queen-
14.55
14.52
1.00
14.59
1,00
1.00
1,00
14,57
Eagle
4.96
55.84
4.52
50.88
4,05
46.22
3.98
44.95
3.84
44.13
4.00
46,20
4.20
44.60
4.23
47.53
UCBSpec
4.75
51.80
5,30
61,28
4.16
47.29
4.73
51.37
4,25
48.96
3.58
41.07
5.28
61.28
4.58
$2.72
EXPeSppec
4.54
52.77
5.10
59,06
4,03
46.50
4.46
50.66
4.07
45.02
3,39
37.53
4.97
56.74
4.37
49.75
HedgeSpec
6.32
68.52
7.27
79.55
5.66
62.08
6.61
73.18
6,08
65.82
5.10
54,97
7.52
61.94
6.37
69.44
Qwen-3-328
1.00
8.13
1.00
7.96
1.00
8.43
1.00
8.33
1,80
8:20
1,00
8.06
1.00
8.55
1,00
8.21
Engle
3.02
21.98
3.36
24.16
2.62
19.30
3.01
21.53
2.57
18:33
2.59
18.67
3.00
21.34
2.88
20.76
UCRSppe
4.34
31.42
5.24
38.6 60
4.25
31.73
4.76
35.58
4.19
50.99
3.64
26.93
5.07
37.36
4.50
33.24
EXP3Spec
30.33
3,96
3.50
4.88
31.55
HodgeSpee
5.82
38.44
6.96
45.14
5.93
39.13
6.50
42.86
5.92
38.60
5.89
33.38
7,16
45.30
6.21
40.41
Table 3: MAT (Mean Accepted Tokens) and Token/s (token generation rate) across datasets for each method. Bold indicates the best. Results of GSMSK and HumanEval and more other datasets, which is out of the training distribution, is in Table 12 showing the same trend. HedgeSpec consistently outperforms all baselines across domains. Its adaptive orchestration of expert drafters improves MAT and throughput, while full- iull-information feedback delivers substantial gains over bandits, translating into real efficiency and highlighting HedgeSpec's effectiveness.
parallel. Thus, HedgeSpec is highly worthwhile: improved acceptance rates reduce costly target calls, outweighing the added cost of drafter evaluation.
4.3.2 HEDGESPEC ACHIEVES CONSISTENT END-TO-END GAINS ACROSS DOMAINS
We further present the end-to-end results of HedgeSpec against EAGLE and BanditSpec across seven testsets in Table 3. For GSM8K (Cobbe et al., 2021) and HvmanEval (Chen et al., 2021) which are outside the training domains, similar trends hold and we defer details to Section D.8. The experiments presented here are conducted under the greedy inference setting of the target model. Results under non-greedy inference exhibit a similar trend, and a detailed discussion deferred to Table D.4.
Overall, HedgeSpec consistently outperforms both EAGLE and bandit-based methods across all domains. Compared to EAGLE, its adaptive selection of speculative drafters enables effective orches- tration, yielding faster responses. Notably, on SQL requests with Qwen, HedgeSpec improved MAT from 4.2 7.52 (an impressive 79% gain) and token/s from 44.6 to 81.94 (a 83.7% gain). Across all mixed queries, HedgeSpec achieved a strong 46. % average improvement. While individual domain- specialized drafters perform worse on average than EAGLE, our results show that orchestrating them with HedgeSpec effectively leverages their strengths, leading to substantial efficiency gains.
HedgeSpec also surpasses bandit methods by a wide margin (up to 49% MAT gain and 41% token/s gain). Its advantage comes from the panoramic feedback: all drafters evaluated, enabling faster convergence, higher acceptance, which finally leads to fewer target calls and better overall efficiency. In contrast, bandit learners adapt slowly because they only observe feedback from the chosen drafter, often converging to suboptimal choices and resulting in lower acceptance and throughput.
Interestingly, we also observe that bandit methods generally underperform EAGLE on LLaMA-3.1 1- 8B-I'T, whereas they outperform on Qwen-3-8B. This is hecause the Qwen-3 series reasoning model tends to produce longer outputs than LLaMA-3.1ID (i.e. 1.64x length in Math workload), providing more time for bandits to converge. The same phenomenon applies to HedgeSpec, where the longer reasoning chains in Qwen amplify its efficiency advantage, highlighting HedgeSpec's superiority in long- oong-generation scenarios.
4.4 A DEEPDIVE INTO PULL-INFORMATION BENEFITS
We have seen HedgeSpec delivers better efficiency than EAGLE and bandit-based methods. In this section, we dive deeper into the benefit of full information on two aspects: (i) how their regret develops over time, and (ii) how each method scales with increasing number of drafters.

---

Published as a conference paper at ICLR 2026
HedgeSpec settles faster to near-zero (average) regret. Figure (a) shows cumulative regret on Llama-3.1 I with python workload, measured by normalized EAL. Bandit methods accumulate regret quickly due to slow adaptation with only partial feedback, wasting exploration weak drafters. In contrast, HedgeSpec rapidly converges to near-zero regret (detailed discussion in B.2) within a handful of steps, demonstrating the benefit of full information.
HedgeSpec scales effectively with larger drafter pools. We further plot MAT w/ number of candidate drafter increasing in Figure (b). Scalability matters because larger pools raise the chance of includ- ing strong specialists, of course only works if drafter collaboration effec- tive. The upper and lower lines mark the best drafter and EAGLE's perfor- mance. Bandit methods deteriorate sharply with more drafters, as their regret grows quickly and exploration becomes much more costly. In con- trast, HedgeSpec scales gracefully, be- ing nearly unaffected because the global information helps to rapidly adapt on the best drafter, remaining effective even with large pool.
Hedgelipac UK Kiidpnt EepSSpac
Teaa (WAC)
Medy:Sox
Pepisger
DAGE
Tene (wwerookeckkiccooo Maad Cumuuatti Regret
Number of Drahhrr
B. wwt.inionssennggg. www.oeeennnnnmbrroooo drofter
Figure 5: Cumulative regret and MAT vs. number of drafters. HedgeSpec quickly settles with near-zero regret, and can scale up with larger drafter pool. Qwen-3 results show similar trend in Appendix D.7, highlighting HedgeSpec's robustness.
4.5 OFFLINE ROUTER COMPARISON: HEDGESPEC'S ROBUSTNESS IN DISTRIBUTION SHIFT
In this section, we compare HedgeSpec with a static offline router. An offline router classifier trained on the request distribution and applied at serving time to dispatch queries to the best drafter. For fairness, we finetune BERT (Devlin et al.. 2019) classifier on the aggregated seven-domane dataset used to construct the drafters (hyperparameter in D.2). The classifier achieved 100% test accuracy across all domains, indicating that under closed-wond world assumptions, a lightweight classifier can reliably route requests to domain-specializzn drafters.
tilam
1-BB--1
Oven-3
Math
MAT
Token's
MAT
Token's
Engle
78.05
Statie Ronter
5.28
70.40
4.05
55 18
HedgeSpec
6.94
90.15
7.12
77.45
MedQA
MAT
Tokews
MAF
Tokeo's
Hagle
5.23
07.67
3.9
43.54
Router
28.18
HedgeSpee
5.56
70.38
6,04
65.99
However, this approach critically assumes that all runtime queries remain in training distribution. In practice, cloud serv- ing frequently encounters O.O.D prompts (Liu et al., 2023b; Chao et al., 2025; Cao al., 2024). We came across simple yet revealing case when attempting to elicit longer reasoning with the instruction: 'Please carefully read the question. After that, please generate two answers to validate it. Output the one you think works well.' This minor prompt variation caused catastrophic failures: the classifier misrouted 98% of MedQA queries and 90% of Math queries. This was not even an adversarial attack but a natural prompt variation, under- scoring the fragility of static routing. Such misrouting is costly as query dispatched to unsuitable drafter incurs long-tail overhead. As shown in Table 4, HedgeSpec remains robust under distribution shift, achieving up to 2.34x gains over the offline router. Its adaptive online learning leverages runtime feedback to identify the best drafter on-the-fi fly, offering three key advantages for real deployment: (i) no reliance on prior knowledge, (ii) resilience to O.O.D queries as long as experts remain useful, and (iii) adaptability when the prompt does not explicitly reveal the best drafter. Finally, we note that offline routing could complement HedgeSpec by providing a "warm start' in the initial steps, which we leave for future investigation.
Table 4: HedgeSpec vs. offline router under O.0.D (Bold-best). The router suffers and mis-routes requests to suboptimal drafters. HedgeSpec adapts through runtime feedback, and remains robust.
5 CONCLUSION
We present HedgeSpec, full-information online drafter selection framework for speculative decoding. By leveraging the structure of speculative decoding, HedgeSpec evaluates all candidate drafters without incurring additional calls to the target model, enabling panoramic feedback. We establish theoretical guarantees under this setting and demonstrate substantial empirical gains, highlighting HedgeSpec's robustness and practical effectiveness in real-world L.M serving.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank Yida Wang from AWS for helpful discussion at an early stage of the project. We thank the anonymous reviewers and meta-reviewers for helpful feedback that led improvements the paper.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0m 08774,, 2023.
Peter Auer, Nicolo Resa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256, 2002a.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(():48-777 2002b.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao.
Medusa: Simple IIm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774 2024.
Chentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, and Bo Han. Envisioning outlier exposure by large language models for out-of-distribution detection. arXiv preprint arXiv:2406. 00806, 2024.
Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
Nicolo Disa-Bianchi, Yishay Mansour, and Gilles Stoltz. Improved second-order bounds for predic- tion with expert advice. Machine Learning, 66(2):321-352, 2007.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 23 42. IEEE, 2025.
Kamalika Chaudhuri, Yoav Freund, and Daniel Hsu. parameter-free hedging algorithm. Advances in neural information processing systems, 22, 2009.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, fean-Baptise Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01331 01.00, 18, 2023.
Jian Chen, Yesheng Liang, and Zhijian Liu. Dilash: Block diffusion for flash speculative decoding. arXiv preprint arXiv:2602.060066, 2026.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang. Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-ol, towards medical complex reasoning with 1lms, 2024. URL
https: /aaxivivvvv.. wrg/abss////ooooooooooooooo.o / 18925.......
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.0337. 0033474 2021.
Zhuoming Chen, Avner May, Ruslan Svirschevski. Yuhsun Huang. Max Ryabinin. Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding, 2025, URL htttps://arxiei
abs 2402 12374.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñchhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14163 2021.
Tencent AngelSlim Project Contributors. Official Eagle model for Qwen-3-32B from AngelSlim, 2025. URL https: //gt thub. wwm comm/Tencett/Angell.... im.
11

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Ming-Wei
Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. arXiv preprint arXiv:2402.02008) 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-240X) 2024.
Yoav Freund. Open problem: Second order regret bounds parametrized by variance across actions and top percentile. In Conference on Learning Theory, pp. 1651 PMLR, 2016.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. arXiv preprint 21Xiv:2402.02057.. 2024.
Aurélien Garivier and Olivier Cappé. The kl-ucb algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th annual conference learning theory, pp. 359-376. JMLR Workshop and Conference Proceedings, 2011.
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: Retrieval-based speculative decoding. arXiv preprint arXiv:2311.08252 2023.
Yunlong Hou, Fengzhuo Zhang, Cunxiao Du, Xuan Zhang, Jiachun Pan, Tianyu Pang, Chao Du, Vincent YF Tan, and Zhuoran Yang. Banditspec: Adaptive speculative decoding via bandit algorithms. In International Conference on Machine Learning, 2025.
Zhengmian Hu, Tong Zheng, Vignesh Viswanathan, Ziyi Chen, Ryan A. Rossi, Yihan Wu, Dinesh Manocha, and Heng Huang. Towards optimal multi-draft speculative decoding, 2025. URL https: aarxiv.vvvvv..////vv//aa.///avvv org/ abss 2202.11777
Kaixuan Huang, Xudong Guo, and Mengdi Wang. Specdec++: Boosting speculative decoding via adaptive candidate lengths, 2025a. URL https: http:://wwwwwo..aaaaeee /twpg///a///.....//o//// 2405.19715
Kaiyu Huang, Hao Wu, Zhubo Shi, Han Zou, Minchen Yu, and Qingjiang Shi. Specserve: Efficient and slo-aware large language model serving with adaptive speculative decoding, arXiv preprint arXiv:2503. 25000,,00300006, 2025b.
Pooria Joulani, Andras Gyorgy, and Csaba Szepesvári. Online learning under delayed feedback. In International conference machine learning, pp. 1453 1453-1461. PMLR, 2013.
jtatman. jaw.:aaaaapppttooooooodddeeeeeee dataset. https: http::::///ogggggacc.cc///eeeae 2025.
Ashish Khisti, M. Reza Ebrahimi, Hassan Dbouk, Arash Behboodi, Roland Memisevic, and Christos Louizos. Multi-draft speculative sampling: Canonical decomposition and theoretical limits, 2025.
URL https: /ttxxvaiivv/v////////ee//eaaaal/ org/ abs/2410. 18234.
Tachyeon Kim, Hojung Jung, and Se- Young Yun. A unified framework for speculative decoding with multiple drafters bandit. 2024.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Shenggui Li, Yikai Zhu, Chao Wang, Fan Yin, Shuai Shi, Yubo Wang, Yi Zhang, Yingyi Huang, Haoshuai Zheng, and Yineng Zhang, Specforge: Train speculative decoding models effortlessly. https: //ttt/gittt gightuu..b....gggg com/ www-ppojeec.eeeeoeeoeooooooooose spw.forgeeepeeeeeeeeeeeeee 2025a. GitHub repository.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, 2024a.
12

---

Published as a conference paper at ICLR 2026
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. Empirical Methods in Natural Language Processing, 2024b.
Yuhui Li, Pangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-timm test. In Annual Conference Neural Information Processing Systems, 2025b.
Nick Littlestoone and Manfred Warmuth. The weighted majority algorithm. Information and computation, 108(2):212-261, 1994.
Sijia Liu. Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu. Rethinking machine unlearning for large language models, 2024. URL https: //arxiv. org/ abs 2402. 08787.
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung. Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. arXiv preprint arXiv:2310007177 2023a.
Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Kailong Wang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint 2r35::2305. 2023b.
Haipeng Luo and Robert Schapire. Achieving all with no parameters: Adaptive normalhedge. arXiv preprint 1.0.:0:5002.09344 2015.
Alexandre Marques, Dipika Sikka, Eldar Kurtić, Fynn Schmitt-Ulms Megan Zhao, Helen Flynn, Rahul Tuli, and Mark Kurtz. Speculators: Standardized, production-ready speculative decoding. co/
https: hettps:/huggingteee htwp://ee.eeeeeeeenn//e//e/eeeee
3wwp://wuwwwww.....mm3 _b___alee,,___________________e_ 2025.
Yev Meyer, Marjan Emadi, Dhruv Nathawani, Lipika Ramaswamy, Kendrick Boyd, Maarten Van Segbroeck, Matthew Grossman, Piotr Mlocek, and Drew Newberry. Synthetic-Text-TD SQL: synthetic dataset for training language models generate sql queries from natural lan- guage prompts, April 2024. URL https: Shuggingfacee co datasets. gretelai// Syythetic-text-tcoe
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang. Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference Architectural Support for Programming Languages and Operating Systems, Volume 3. pp. 932-9499 2024.
mlfoundations dev. Organic chemistry qa dataset. molfouddaiion -ons-dev and SCP wtt__unniieeedddrooooe strerreesrrrroncccccmcssssrr.. quest sressiion,, 2025.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using eequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.0660 011660200002.,.. 2016
Rui Pan, Zhuofu Chen, Hongyi Liu, Arvind Krishnamurthy, and Ravi Netravali. Fail fast, win big: Rethinking the drafting strategy in speculative decoding via diffusion 1lms. arXiv preprint arXiv:2512.20573. 2025.
Ramchalam Kinattinkara Ramakrishnan, Zhaocong Yuan, Shaojie Zhuo, Chen Feng, Yicheng Lin, Chenzheng Su, and Xiaopeng Zhang. Omnidraft: A cross-vocabulary, online adaptive drafter for on-device speculative decoding, 2025. URL https : araiv. org abs /tb./250...m 02659.
Herbert Robbins. Some aspects the sequential design of experiments. 1952.
Apoorv Saxena. Prompt lookup decoding, November 2023. URL https: /tl:::/giitti wigggttthhuu.... com/ apoorvumang/proogge http:p/roppttoooooooooooooooppop decoding/.
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36:30222-3022 2023.
13

---

Published as a conference paper at ICLR 2026
Bangsheng Tang, Carl Chengyan Fu, Fei Kou, Grigory Sizov, Haoci Zhang, Jason Park, Jiawen Liu, Jie You, Qirui Yang, Sachin Mehta, et al. Efficient speculative decoding for llama at scale: Challenges and solutions. arXiv preprint arXiv:2508.08/ 2025.
Tengyunw. Official eagle model for qwen-3-8b from tengyunw. https: https://huggingtace.coo Bangyynn//wwen.. __________e Bee_eeeeeeeaaaeeeee 2025.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstmuct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01566 2024.
Dirk Van Der Hoeven and Nicolo Desa-Bianchi, Nonstochastic bandits and experts with arm- dependent delays. In International Conference on Artificial Intelligence and Statistics. PMLR, 2022.
Vladimir G Vovk. A game of prediction with expert advice. Proceedings of the eighth annual conference on Computational learning theory, pp. 51-60, 1995.
Marcelo J Weinberger and Erik Ordentlich. On delayed prediction of individual sequences. IEEE Transactions on Information Theory, 487):999977 2002.
Matthew Wesney. Tot-biology.
2025.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion Ilm by enabling kv cache and parallel decoding. arXiv preprint 250k:::2005.2261. 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Euiin Yi, Tachyeon Kim, Hongseok Jeung, Du-Seong Chang, and Ye-Young Yun. Towards fast multilingual Ilm inference: Speculative decoding and specialized drafters. arXiv preprint arXiv:2406. 16758, 2024.
14

---

Published as a conference paper at ICLR 2026
A PROOFS OF TECHNICAL RESULTS
A.1 THEOREM PROOF
Proof. EAGLE generates the draft tree greedily, thus the children at Node T5t+k-1 is Topı (f(-(==44x11)) The next token is accepted when one of the children is sampled by the target model p.
A.2 THEOREM PROOF
Proof of Theorem 3. First note that the expectation is over the distribution of F++1:++K from the target distribution as well as the randomness of the speculative decoding method. In particular, 7M (even the drafts are determinstic, e.g., EAGLE3) random because 1:44k-1 is random.
I # of accepted tokens|aga]
K+1
kP 01+1444-11 is accepted, To+h is not accepted|
kB P [ee4 is not cccepted/ is accepted, #gd
AE P [ not uccepted 141144-111111 accepted, [g (0-- accepted a 255
K+1
kE (1 TaPP [1++1:+A-1 is accepted|[s++a-]]] [ss
(1)
Now by the Law of Iterated Expectation again
PP [244144-11 is accepted] [ss]
R8
PP [Tenk-1 is accepted Fe 144k-2 Jgenk Plaraaek is accepted| S014 E5A
=R
70))0-11 2 is accepted| #5t
- 71) (esiatt-a secepted|= E<t
B (1 n 17 72P (1 is accepted [g
=B (1 nh 2 age
In Line 4 onwards, we are repeatedly applying the same arguments from Line 1-3 which "peels off" one random token j at time and applying the definition of 7j-
The proof is complete by substituting into (1).
A.3 THEOREM PROOF
Proof of Theorem 4. First, by the Hedge algorithm, the hypothetical online learning without delay enjoys regret bound of / log N for any choice of T and N.
By Theorem of (Joulani al., 2013), the blackbox reduction shows that when the delay is smaller than 2K, there is an algorithm that gives a regret of
2T log N
2(2K + +1)
2K +1
long T > (2K + 1). This yields the following regret bounds
Taken
Tiske
log N
reli reloe] 2 max
me[="]
O(1
T Troken
Tinken
Troken
15

---

Published as a conference paper at ICLR 2026
and
Takes AcceptLength,,kle
Tikees AcceptLength,
K log N Tiken
max EIN]
OVy
T ioken
K ++1
T Twken
K+1
A=1
h=1
for the two different loss functions respectively. It remains to take expectation over all random variables on both sides for each 1" [N] separately. For the acceptance probability, notice that
E[>:[te]] E. [r, accepted [r-ge-]]]
E
Σ
Pali [| is accepted [øg(-1]]
iE|N]
Σ
EiPli = B] is accepted [[(-]]]]
kE[M]
Σ iE[N]
FAl ijP. is accepted] Fal is accepted]
where the third line uses the conditional independence of i, and given T-t- I. which follows because algorithm A decides on i, before Ta and that x, determined by the target model no matter which & is chosen.
For the accepted length,
AcceptLength,,nr x [ia]
EE
E[ AcceptLennth, . ( sce]]
E
E |AAccptLLnng,h, [ie]lie,
=E
= [lec=elikcc xec) real
[E|N]
=E
AcceptLengthh..' [i:]]
where the second identity follows from Theorem 3 and the third-identity uses the conditional independence i, and H, given # as before. The claim is proven by multiplying both sides by
K
B FURTHER ALGORITHM AND IMPLEMENTATION DETAILS
B.1 ONLINE LEARNING WITH DELAYED FEEDBACK IN STOCHASTIC SETTING
In this section, we clarify how Algorithm 2 of (Joulani et al., 2013), stated in the more general partial monitoring setting, can instantiated for the full information setting.
This is the algorithm that we used in the experiment.
Setting. There are experts (actions) A = fl....,
N). On each round t 1,2,..., T the environment associates loss vector l [o, 1]. to the experts, but & may be revealed after arbitrary delay. At time f the learner receives a (possibly empty) set
He = {(s, Ea): the full vector l becomes available at time t).
When making the round-f decision, the learner can use all E, that have already been revealed, not those still pending.
Base learner. BASE is any standard full-information online algorithm (e.g., Hedge) designed for an immediate. delay-free stream of loss vectors. We assume BASE supports:
PREDICT() and UPDATE((),
16

---

Published as a conference paper at ICLR 2026
where PREDICT returns either a distribution over experts single expert index, and UPDATE feeds BASE one full loss vector & [0. 1]
Specializing QPM-D full information collapses the per-action queues into single FIFO queue of unprocessed loss vectors, because any revealed &g is usable regardless which expert was played.
Algorithm 1: Queued Full-Information with Delays (QFI-D)
Data: A FIFO queue Q of unprocessed loss vectors
Input:
(Optional) horizon 7 (not needed if the BASE is an anytime algorithm).
Initialize Q <- 0.
Initialize BASE; let < BASE.PREDICTT).
for \ , 1......,. do
// Pt phase: catch BASE up with all arrived feedback while Q + do
f POPFRONT(
BASE. UPDAT())
p BASS..PREDICT))
/ / Make the real-world choice for round /
Play action a, according to
// e.g., sample from or take arg max arg min as appropriate
for BASE
// Update phase: record any feedback that arrives now Observe the (possibly empty) set H, {(s,E, (,) of loss vectors revealed at time foreach (s, (.) H, do (USH)BACK(Q,(.)
11
Remarks.
This makes BASE experience delay-free stream its own clock: whenever a vector arrives, immediately fed to BASE before the next real prediction is made.
To instantiate with Hedge, UPDATE applies the usual weight update w; w. exp(-n E,) and PREDICT returns the normalized weights.
multiple loss vectors arrive at the same time, they are queued in arrival order and processed FIFO.
Corollary 5 (Regret of QFI-D). Assume the delay is bounded by Tmax and the loss is bounded by 1. Let Base be any full-information online learner analyzed in the same stochastic environment without delays, with expected regret Regret Basr Then the expected regret of QFI-D satisfies
Base.
E| Reggett
< E
Regret
Base
Base
+
Tmax-
Proof. This is an instantiation of Theorem 6 of (Joulani et al., 2013).
The implementation for HedgeSpec with delayed feedback in the stochastic setting is particularly simple.
1. Keep two pointers tupdated and where tupdared S t and all necessary statistics
2. After every chunk, process every batch of available loss vectors by updating the weights of the BASE learner, before taking the next action, then update tupdased to the last frontier.
Note that the hypothetical token-level game without delay, there are several updates within each chunk, but due to the delay, none of those updates will actually occur. This means that the weights on the drafters the delayed case will not be updated within each speculative decoding chunk is complete.
We could either sample independent sample from the drafter weights for each new token as the drafter roll out stick to the same drafter. Both approaches enjoy the same regret guarantees in Corollary 5.
17

---

Published as a conference paper at ICLR 2026
B.2 FIRST ORDER AND SECOND ORDER REGRET BOUNDS
We observe that in the experiments (Figure 5 and Figure 8), HEDGESPEC appears to have a regret that stops growing after learning for a few iterations, instead of the ((vT) predicted by the worst-case bound in Theorem 4 and Corollary 5.
We believe this is due to the adaptivity of NormalHedge (Chaudhuri et al., 2009).
Cesa-Bianchi et al. (2007) established that when the learning is optimally tuned for Hedge, the method enjoys both first order (small loss) and second order (small variance) regret bounds:
T Σ
Regret O
fi log N L
and Regret =0
Σ Var [fc[i]] log
In particular, the best drafter i" has very small losses or after a while the learner's weights pt concentrates on fixed drafter, then the regret bound will not grow with T. This is the case when there is clear winner among all drafters.
NormalHedge was proven to enjoy these strong adaptive regret bounds, though there conjecture that it does (Freund, 2016), and modified version of normal hedge algorithm known as AdaNormalHedge (Luo & Schapire, 2015) which does enjoy first order regret bounds.
Our experiments seem to support the conjecture.
In practice, we find that NormalHedge often quickly converges the optimal drafter, while still enjoy the worst-case -type regret when clear winner exists.
B.3 CHUNK-LEVEL TOKEN- ONLINE LEARNING AND REGRET GUARANTEES
There is more than one way set up the regret minimization game. In the main paper, we discuss the loken-level online learning game, Here, we further discuss the token-level online game.
Chunk-Level Games and Loss functions We it up online learning problem where each chunk is one round of the game. This choice natural because the action to choose drafters is made for each chunk.
for 1, v.ny T Thhnkk
1. Nature chooses loss vector Sn E [0, 1|
2. Player chooses Drafter in and incurs loss fn [ix]
3. Player observes fn-
Figure 6: Chunk-Level Game
It remains to design the loss functions. Let the token index right before the 4h chunk be fy. and the length of chunk returned by the chosen drafter be of length Ki. Let
Nh. [i] P. is accepted Jt,-1:t
accepted, x<n+t]h
namely the probability that 7h token of chunk h from drafter is successfully validated given that the first (j - 1) tokens are validated.
If optimize the average acceptance probability within each chunk, we can
ky ki Σ (1 1--m)()))
[
we optimize the expected chunk length, then we can apply the expression in Theorem with K ky. estimate the expected accept length of Drafter i. The resulting loss function is the
18

---

Published as a conference paper at ICLR 2026
normalized distance from the max length.
kn +x+1
fu =1
AcceptLengghh,.y A,, [1] =
Σ 1=1
j(1 - --mmm
II
melli
kp +
kn + 1
KEG--1
where Ah,++11 [i] assigned be for notation convenience.
Theorem 6. There is algorithm A that chooses in in the game in Figure 6, such that
1. When we optimize acceptance probability
Tiuue -inm The [in] max
Tarnt 2im mhs 1 - 2/ hml
los N Tihess
Tiues
Kh.
Torett
Kn
2. When we optimize Accept Length
Toust P==1
Tiant AccepiLength, My [6d
t1+1
maX]
Tohat
Tohat
AccepiLength, [F']
ku+1
2/
log N Tave
The algorithm that achieves this includes Hedge and many of its variants. the experiments, we test out both the original Hedge and more adaptive, and parameter-free variants of Hedge. Compared to BanditSpec, the regret improves exponentially in the number of drafters N.
Note that the actual accepted length for the chosen drafter model in is kn, but the expected value can be bigger or smaller than ky Other draft models will have an expected accepted length between 1 and ly 1. It is capped at kn + 1 due to censoring effect. The censoring effect may lead to an underestimation consequences of the censoring issue in the appendix.
the performance of alternative draft models. We will elaborate on these
Another issue of censoring is that it makes the regret guarantees Theorem somewhat difficult to interpret. In particular, it might be suprising to some readers that for any fixed
E |ccceptLLengthy, [t]] A E[AcceptLengthy,.u [:]]
in general. This is because hy is random, and when we condition on ka, it changes the distribution of tokens 4e, +1:4 +ki in this chunk, thus rendering Theorem inapplicable. Similar issues arise for the expected value of "h..-
B.4 TIME-INHOMOGENEITY IN EAGLE MODELS AND CONSEQUENCE OF CENSORING.
EAGLE draft models are special in that they belong to broader family of time-inhomogenous draft models. Let's denote EAGLE models by qr the ith choice and the corresponding model used for kth relative position.
As an example, if Draft model models being called are q1.1 for the first token, and 91 for the second. Support Draft model i = 2 was chosen instead, then it could've verified tokens.
chosen time it rolls out with tokens verified. Then the
Naively, this can give us feedback for 4;,1 45,2 for i = 2 too, but will have to wait until the next chunk
before we can get feedback for Q2.3 and 92,4- What if the onset is now + 1 instead?
The execution of the target model actually has provided us all the information needed to provide feedback for every [N] and ie E [k:
Why do we need to handle these complications? Let us inspect the following two examples that highlight the surprising fallure mode we do not handle these issues appropriately.
Evaluating EAGLE models off-policy is somewhat tricky because the best model is not fully captured by the acceptance probability.
Example 7 ""censoring" causes worse draft model win). Draft model q1,1 1 has acceptance probability 100% but q1,2 is terrible, it has acceptance probability of only 0%. The expected verified length is always 2 for draft model q1:- If q1,1 played first, the chunk length will converge to 2 throughout.
19

---

Published as a conference paper at ICLR 2026
Now let 92.k 0.r k 1,2 but 0.9 k 2.
Average acceptance probability for draft model is 0.5 and 0. for draft model 2 under the distribution of draft model 1. However, if we roll-out draft model 2 long enough, we get higher acceptance probability on average.
The expected acceptance length for the second draft model is 1 0.6/ - = 2.2, i.e. it is better than the first.
However, since draft model I is only generating two-token chunks, and we can only evaluate the first two steps of the draft model 2.
The expected length is now 0.4 + 0.6.4 2 0.6 1.96 < 2.
This example illustrates that for drafters that are not invariant to relative index (e.g., EAGLE models), it is perhaps better to use token-level online learning (with delay) to avoid getting stuck at a suboptimal drafter.
In practice we found that the censoring effect is not detrimental for EAGLE models. Chunk-level online learning works well token-level online learning with delay, and it incurs smaller system overhead.
B.5 PRACTICAL HEURISTICS: "HYBRID LOSSES" "SKIPPING UPDATE"
As now evaluate all drafters, even though this does not increase the total number of calls to the target model, does slightly affect the amount of compute needed for evaluation and increase the latency in practice in the inference system.
We propose a technique called "skipping updates" which reduces the number of times the evaluation phase need to be called. The idea is that simply grouping couple of updates together and apply them once in a batch once in a while.
In practice, we observed that our hedging algorithm quickly reaches stable performance plateau with near-zero regret, consistent with the first- and second-order regret bounds (see discussion B.2). To further reduce update latency, we experimented with skipping updates by a fixed number of tokens and combining this with batched feedback, treating both as tunable hyperparameters. Empirically, after a short warm-up period (e.g.. rounds of full-information updates), using delayed batched feedback (e.g., 12 rounds per batch) together with skipped updates (c.g., 6 tokens per update) continued to yield strong performance for the tested experiments. Overall, these results indicate that moderate delays in feedback and updates can maintain good mean accepted token length while providing practical tradeoff between accuracy and efficiency.
The other trick that we proposed is "hybrid losses". This involves starting the learner by choosing the first few loss vectors, e.g., fI, J2, Ja. ... to be based on acceptance probabilities, then switching to acceptance-length loss, later. The reason is that the delay is generally higher for acceptance length.
For example, if K = and after the first chunk, tokens are accepted. The acceptance rate loss would be computable after the first chunk. By the end of the second chunk, we can already update the learner by 4 times before the decision for the second chunk is due.
The loss based acceptance probability - even though not what we ultimately wanted to optimize can be used as a surrogate loss and help mitigating the "cold-start" problem.
Both tricks used in experiments.
C RELATED WORK
C.I THE ADVANCE OF SPECULATIVE DECODING
Speculative decoding is a pivotal way for optimizing LLM inference latency. This technique was first introduced with ctrin-structured drafts, where the draft model generates single sequence of tokens verified sequentially by the target model (Leviathan et al., 2023; Chen et al., 2023). Subsequent work generalized this into structurdd drafts, organizing draft tokens connected tree increase
20

---

Published as a conference paper at ICLR 2026
acceptance opportunities (Chen et al., 2025; Miao et al., 2024; Cai al., 2024; Du et al., 2024; Li et al., 2024b). Recent works extend speculative decoding to the multi-draft setting, where multiple candidate tokens are proposed in parallel at each step. Sun et al. (2023) casts draft selection into an optimal transport framework with efficient approximation schemes. Khisti et al. (2025) show that the optimal solution admits canonical two-step decomposition and provide exact acceptance characterizations in the two-draft case. Hu et al. (2025) derive tractable methods to compute theoretical upper bounds on acceptance rates, demonstrating practical benefits of sampling strategies such as without--eplacemmnt. The most recent advances leverage diffusion LLMs (dLLM) (Wu et al., 2025) as drafters (Pan al., 2025; Chen al., 2026), further reducing drafting overhead and improving overall efficiency. While prior work advances speculative decoding by improving how single drafter generates candidates-rangien from chain- to tree-based structures and multi-draft extensions -urroons-uurr work addresses the orthogonal level of challenge in multi-drafter selection, where diverse speculative decoding methods can all potentially serve as drafters the pool, and we dynamically evaluate and select among them with provable no-regret guarantees.
C.2 ADAPTIVE SPECULATIVE DECODING
Another line of work focuses adapting speculative decoding during inference incoming requests. OSD and OmniDraft (Liu et al., 2023a; Ramakrishnan et al., 2025) adapt the drafter on-the-fly to the target distribution via online knowledge distillation, improving token acceptance rate. Our method is training-free free and operates at a different level: candidates in the drafter pool can themselves adopt such adaptive mechanisms, while our contribution lies in selecting among them in the multi multi-drafter setting. SpecDec+++ (Huang et al., 2025a) instead adapts the speculation length, stopping drafting once the predicted rejection probability exceeds a threshold, which differs from our goal of multi-drafter selection. SpecServe (Huang et al., 2025b) takes system-level perspective, adapting speculative decoding configurations (c.g. resource allocation) at runtime to meet latency and throughput SLOs, rather than focusing acceptance probability. Among these works, MetaSD (Kim et al., 2024) first poses the problem and BanditSpec (Hou et al., 2025) represents the state-of-the-aa for adaptive multi-drafter speculative decoding. In contrast, we advocate different paradigm that cheaply exploits global information, achieving higher token acceptance rates and lower per-token latency.
C.3 ONLINE LEARNING ALGORITHMS
Hedge and multi-armed bandits capture the full-information and partial-informen settings, re- spectively, and have inspired numerous variants. The Hedge algorithm (Cesa-Biande & Lugosi, 2006) provides a classic framework for expert weighting; NormalHedge (Chaudhuri et al., 2009) removes the need for tuning learning rates; and AdaNormalHedge (Luo & Schapire, 2015) further improves adaptivity. In parallel, the stochastic K K-armed bandit problem was introduced by Rob- bins (Robbins, 1952), leading to a wide family of exploration-exploiaaee
algorithms. Canonical examples include UCB (Auer al., 2002a) and KL-UCB (Garivier & Cappé, 2011), which provide confidence- based exploration guarantees, and EXP3 (Auer et al., 2002b), which provides regret guarantees in the adversarial setting. As compared in Section D.6, adopting Hedge and its variants consistently achieves strong performance in our experiments, surpassing bandit-style algorithms. This highlights the effectiveness of leveraging global loss information across experts.
D ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS
D.1 EAGLE'S DEFAULT GENERATION CONFIGURATION
In our experiments, we directly adopt EAGLE's generation configuration as our framework focuses on the drafter selection problem. By default, EAGLE uses exploration depth of 7, resulting in a speculative decoding length of 9 tokens per chunk, with the top-k value equals to 10 during its expanding and reranking phase.
21

---

Published as a conference paper at ICLR 2026
D.2 BERT CLASSIFIER TRAINING DETAIL
Table 5 records the training hyperparameters of the offline BERT classifier for statically routing the requests to the corresponding drafter. The datasets used for training the classifier the aggregation of data across all 7 domains.
Hyperparameter
Value
learning rate
2c-5
wwr_device_train baiih_atch__siz
128
per_device_
128
cummtrrin_epochs scheduler type warmup ratio optimizer
3
linear
0.1
adamw
Table 5: Training hyperparameters for BERT classifier.
D.3. STATISTICS FOR THE CURATED QWEN DRAFTERS
Table 6 and 7 shows statistics of the 7 curated drafters with Qwen-3-8B and Qwen-3-32B as the target (bold indicates the best). Each drafter performs well in-domain (diagonal) but degrades when applied out-of-domain for Qwen-3-8B model, and average is weaker than the vanilla EAGLE model. Similar patterns are shown in Table 1. Together, these drafters provide realistic evaluation pool for Hpeg,Spee, which orchestrates them to jointly accelerate serving.
Datusels
Python
Math
Biology
Chemistry
McdQA
CNN_DM
SQL
Drafter domalus
MAT
Token's
MAT
Taken's
MAT
Taken's
MAT
Token''
MAT
Token's
MAT
Token's
MAT
Tokea's
Token's
Vanilla Eagle
4.96
55.84
4.52
50.88
4.06
46.22
3.98
44,85
3.84
44.13
4,07
46.20
4.20
44.60
4.23
47.53
Python
6.43
73.73
1.50
51.82
2.46
26.85
2.97
34.41
2.36
26.81
2,45
28:26
3.85
44.42
3.57
40.90
Math
7.39
2.66
3,46
32.93
Bielogy
3,20
37.25
3.61
43.01
5.70
67.39
3.74
43.90
3,80
45.11
2.70
31.50
2.50
33.99
3.65
43.16
Chamisry
3.79
5.40
3.79
6.69
3.54
2.74
20.14
3.36
38.
47.39
MedicalQA
3,66
42.85
1.05
46.23
4.07
46.86
4,01
46.66
6.17
73.51
2.94
33.38
3.42
39.65
4.05
47,02
CNN JM
2.56
29.80
2.56
30.40
2.45
78.49
2.45
28.61
2.55
90.21
5.15
60.70
2.52
70
2.89
33.95
3.71
3.37
2.36
2.68
7.60
Table 6: Statistics of the 7 curated drafters with Qwen-3-8B the target. Bold indicates the best. Each drafter shows strong in-domain performance (diagonally strong) but suffers noticeable inefficiency when applied outside, and average performs worse than the vanilla Eagle model. Similar trends are observed in Table 1. These drafters form a realistic evaluation pool for evaluating HedgeSpec, which orchestrates them to jointly accelerate the serving process.
Datasets
Python
Mash
Biology
Chemistry
MedQA
CNNLDM
SQL
Drafter domnins
MAT
Token's
MAT
Token's
MAT
Token's
MAT
Thken's
MAT
Trken''
MAT
Token''
MAT
Token's
Arg MAT
Aug Token's
Engle
3.36
2.62
19,30
3.01
21.53
2.57
2.59
Python
6.00
43.67
1.66
33.63
2.52
18.20
3.16
23.30
2.42
17.78
2.58
18.94
3.36
25.22
3.60
26.25
Math
4.04
29.75
7.12
48.67
2.75
20.10
3.64
26.54
2.74
20.33
3.00
21.98
3.79
27.61
3.87
27.85
Biology
3,88
27.81
Chemistry
3.29
23.71
5.32
37.27
3.91
78.72
6.61
47.26
3.56
25.66
2,66
1863
2.96
21.06
4.04
28.90
MedicalQA
3.56
25.97
1.18
32.85
4.24
30.36
4,21
31.25
5.95
43.34
3.10
22.42
3.52
26.83
4.11
30.43
CNN_DM
2.54
2.56
2.56
SQL
3.58
20.69
3.54
25.98
2.42
17.73
2,80
20.26
2.43
17.84
2.75
20.09
7.25
52.73
3.54
25.04
Table 7: Statistics of the 7 curated drafters with Qwen-3-32B as the target. Bold indicates the best.
D.4 EVALUATING HEDGESPEC UNDER THE NON-GREEDY INFERENCE SETTING
Table 8 shows the MAT (Mean Accepted Tokens) and Token/s (token generation rate) across datasets for each method under non-greedy target model inference. The temperature is set to be 0.5 in
22

---

Published as a conference paper at ICLR 2026
Datasets
Python
Math
Biology
Chemistry
McdQA
CNNNUMM
SQL
Token/k
Tokeress
Tokeals
MAT
MAT
MAT
Taken's
Arg
Avg Token's
LLOMA-J.1-8G
17.77
1.00
17:97
1.00
18.03
1.00
17.35
1.00
18.81
1.00
18.41
1.00
18.25
1.00
18.08
Vanilla Eagle
6.48
83.75
5.78
74.52
5.74
74.41
5.04
62.61
4.65
58,82
5.06
62.55
5,93
77.99
5.53
70.66
UCBSpec
72.61
4.11
3.72
EXPSSppe
4.93
62.63
5.33
69.60
4.67
00.28
4.76
62.62
4,11
52.23
3
46,25
5.07
64.066
4.67
59.67
HedgeSpec
7.16
87.38
7.32
88.58
6.67
$2.22
6.63
80.05
6.01
72,65
5.58
65.11
7.89
95.81
6.75
81.09
Qveen
15.33
1,00
1.00
1.00
Vanilla Eagle
4.88
53.03
4.58
50.59
4,07
47.13
3.90
44,10
3.76
42.89
4.05
45.47
4.18
48.06
4.20
47.32
UCBSpec
4.45
50.84
5.25
56.77
4.07
45.50
4.54
47.68
4.12
41.97
3.52
39.13
5.19
56.55
4.45
48.43
EXPsSppec
4.34
48.00
5.20
55.19
3.88
43.07
4.32
44.75
3.95
40.96
3.33
36,94
4.95
54,09
4.28
46,14
HedgeSpex
5.82
62.63
7.09
76.90
5.56
60.50
6.40
68,77
5.84
63.67
4.96
53.50
7.12
76.75
6.11
66.15
Qwen-3-328
1.00
8.52
1.00
8.57
1.00
8.35
1.00
8.56
1,80
8.38
1,00
8.51
1.00
8.82
1.00
8.53
Vanilla Engle
3.03
22.91
3.36
25.18
2.52
19.25
2.99
22,40
2.55
18.68
2.58
19.31
3.06
23.04
2.87
21.54
UCBSpec
4.12
10.36
5.22
38.04
4.08
30.14
4.60
31.76
4.06
29.09
3.45
24.97
4.78
34,90
4.33
31.32
EXP3Spec
28.27
36.22
3.37
29.77
HodgeSppe
5.31
36,86
6.71
47.37
5.44
38.54
6.23
42.77
5.46
39.39
4.78
33.48
6.58
46.24
5.79
40.65
Table 8: MAT (Mean Accepted Tokens) and Token/s (token generation rate) across datasets for each method under non-greedy target model inference. Bold indicates the best. The trend is consistent with results in greedy setting showing in Table 3.
Datasets
Python
Math
Biology
Chemistry
MedQA
CNN_DM
SQL
Joint trained model
7.03
7.76
6.63
7.16
6.22
5.35
7.39
Eagle
6.48
5.88
5.95
5.28
4.96
5.31
5.99
HedgeSpec
7.69
7.69
7.18
7.10
6.47
5.88
8.06
Table 9: Comparison of MAT across domains for a jointly trained model, the vanilla EAGLE, and HedgeSpec. Bold indicates the best performance in each domain.
this experiment. The trend is consistent with Table 3. HedgeSpec consistently outperforms other baselines across different model sizes and domains, demonstrating improved acceptance efficiency and throughput through adaptive orchestration of expert drafters.
D.5 DISCUSSION ON HEDGESPEC vs. JOINTLY TRAIEND DRAFTER
In this section, we study the effect of aggregating data from all domains and jointly finetuning a EAGLE model. This effectively turns the original EAGLE into another "generic" drafter. Results are presented Table 9, where HedgeSpec outperforms most domains terms of MAT. In the meantime, several additional observations emerge:
First, joint training empirically improves EAGLE's performance across multiple domains, which is expected. Second, the jointly trained drafter performs comparably expert drafters trained in single domain such as math, biology, chemistry, and MedQA, while we do observe performance drop in domains like python, CNN DM, SQL in this experiment. The reasons require further investigation, but it is also reasonable to expect this since datasets from different domains may not be fully aligned, with some pushing learning in the same direction while others pull in the opposite (Liu al., 2024); meanwhile, performance is also constrained by scaling laws and the model's capacity to digest
experimenne
given that EAGLE is only one-layer transforner. As more data is introduced, it is uncertain whether the performance in those domains will continue to boost or decline. Third, even when proper joint training can yield synergistic benefits, is often infeasible because parties may not release their training data due to confidentiality concerns (Achiam et al., 2023). In such cases, only the trained drafters might be available, making joint training impossible. This underscores HedgeSpec's advantage: it requires no access to training data and can operate directly on a pool of expert drafters. Moreover, a generically trained model can itself serve as candidate within this pool alongside specialized drafters. In short, HedgeSpec addresses the higher-level challenge of orchestrating expert drafters. With only a collection of such models, it can significantly improve serving efficiency.
23

---

Published as a conference paper at ICLR 2026
Initially: Set R,,0 Pı,1 1/N for each
For 1 .,....
1. Each action incurs loss li,e-
2. Learner incurs loss f. = Ei=1Pi,li,
Update the cumulative regrets: RA, = R1 + (la, - leee) for each
4. Find Ct that satisfying
([Rg,e++) 2cr
exp
= e.
5. Update distribution for round
1:
RR,,++
[[Rual+))
Pi1++1
exp
for each i.
Ct
20
Figure 7: The workflow of NormalHedge algorithm.
Datasets
Python
Math
Biology
Chemistry
MedQA
CNN_DM
SQL
HedgeSpec
7.69
7.10
6.47
5.88
8.06
HedgeSpec w/ Acc. rate loss
7.62
7.47
7.02
6.94
6.44
5.88
8.08
HedgeSpec w/ Standard Hedge
6.90
6.67
6.53
6.19
5.38
4.68
7.05
HedgeSpec w/ AdaNormalHedge
7.36
7.30
7.05
6.66
5.85
5.40
7.71
Table 10: Llama Mean Accepted Tokens (MAT) across datasets using HedgeSpec variants. Bold indicates highest performance per column.
D.6 VARIANTS OF HEDGING ALGORITHMIC CHOICE
In this section, we study how different hedging algorithmic choices affect the final outcome. The ablation considers factors: (i) using token acceptance-rate loss instead of expected-length-baasdd loss, and (ii) replacing the NormalHedge update rule with alternative algorithms, including Standard Hedge and AdaNormalHedde. HedgeSpec by default adopts the parameter- freee NormalHedge with expected-lengie -eaeeed--enntt---beedde based loss. The MAT results across datasets are reported in Table 10 and 11, showing consistent trends. We below show the workflow of NormalHedge Figure 7.
From the table, we see that the acceptance-rate-bunn based loss achieves comparable but slightly lower MAT, suggesting can be viable alternative for online learning in speculative decoding, though expected-length-baass
loss remains stronger overall. Switching Standard Hedge leads to larger drop performance. We attribute this to Standard Hedge's 's more conservative updating: tends to spread weight more broadly during the exploration phase, whereas NormalHedge shrinks weights more aggressively toward strong drafters. This conservatism could mitigated with careful parameter tuning, although such tuning would vary across scenarios and adds practical complexity. Finally, HdaNormalHedge also shows reduced MAT. This is somewhat surprising, as HdaNormalHedge enjoys first-order regret bounds and theoretically stronger. In our experiments, however, this advantage did not materialize.
D.7 ADDITIONAL RESULTS ON CUMULATIVE REGRET AND MAT VS. DRAFTER POOL TRENDS
In this section, we present additional results on cumulative regret for the Qwen-3-8B model, as well results on MAT with increasing number of drafters. The trends in Figure 8 and Figure 9 are consistent with those in Figure 5, demonstrating that HedgeSpec converges more quickly to near-zero regret and scales effectively the drafter pool grows.
24

---

Published as a conference paper at ICLR 2026
Datasets
Python
Math
Biology
Chemistry
MedQA
CNN.DM
SQL
HedgeSpec
6.32
7.27
5.66
6,61
6.08
5,10
7.52
HedgeSpec w/ Acc. rate loss
6.23
7.00
5.58
6,58
6.05
5.00
7.30
HedgeSpec w/ Standard Hedge
5.85
6.64
5.13
5.88
5.47
4.46
7.01
HedgeSpec w/ AdaNormalHeddge
6.26
7.12
5.57
6.39
5.94
4.99
7.40
Table Qwen-3-8B: MAT across datasets using HedgeSpec variants. Bold indicates highest per column.
5.5
HodgeSpec
UCBSpec
Spdc UCB5poc
Exp2Spicc
5
EAGLE
Exp3Spe:
Best Drafter
0.0
0.2
0.4
0.5
0.8
1.0
Time (iform leed Inforence a CCumulatiion Reghht
Number of Drofters MAI with increaning number chotters
Figure 8: Cumulative regret and MAT vs. number of drafters. HedgeSpec quickly settles with near-zero regret, and can scale up with larger drafter pool. Llama results show similar trend in highlighting our robustness.
D.8 THE GSMSK, HUMANEVAL, BIRD, MULTINEWS AND SPECBENCH RESULTS
In this section, present MAT and tokens-per-secone results the GSM8K, HumanFyal, Bird, MultiNews datasets as shown Table 12. These datasets are not included in the training data used construct the drafters. We find that without prior knowledge, HedgeSpec consistently outperforms all baselines across both domains. This demonstrates that HedgeSpec's orchestration of drafters can improve MAT and throughput without requiring prior knowledge, as long as some drafters excel in specific domains highlighting HedgeSpec's effectiveness in drafter selection.
In Table 13, we provide additional results from running the SpecBench experiments. We note, however, that the experimental setup differs slightly from our main evaluation. SpecBench contains substantial portion of tasks such as Natural Questions QA, translation, and RAG-style generation. Our drafter pool does not include experts specialized for these tasks tasks-although the online learner can still select the "best available" drafter for each query.
However, this actually reflects the real scnario, that you could have a bunch of expert drafters, but you cannot cover situation. In this sense, you usually have some additional "generalist" model other than the experts, like the EAGLE model, to handle the requests that outside the experts' expertise. With that being said, we added official EAGLE model in pool, and form up a experts + generalists pool for speculative decoding in this benchmark.
The results are shown below, we see that HedgeSpec still yeilds the best performance across different model. This experiments confirms HedgeSpec's effectiveness in selecting best drafters, regardless of OOD, for competing the best drafter in hindsight, leading to performance gain.
D.9 DISCUSSION OF HEDGESPEC ON BROADER SPECULATIVE DECODING SCHEMES
In this section, we extend our discussion on the integration HedgeSpec into broader speculative decoding schemes. We do believe HedgeSpec is integrable for different speculative decoding frameworks. At its core, HedgeSpec operates at an orthogonal layer: any speculative decoding algorithm could serve potential drafters in the pool. Theoretically, as long as we have verified trace (Theorem 3), and can use to get the token acceptance probability for EAL evalaution on other drafters, HedgeSpec select among them with provable no-regret guarantees.
In our main experiments, we build upon EAGLE framework, which is widely regarded as the most broadly deployed framework (Marques al., 2025; Tang al., 2025). Across benchmarks, EAGLE
25

---

Published as a conference paper at ICLR 2026
ACCaCTed Touen (MAD
HodgeSpec
UCBlppec
Hadgstpe:
Bed Dracter
Number Drabier
Number Draters
Figure 9: MAT vs. number of drafters on math workload on Llama and Qwen models. HedgeSpec scales effectively with larger drafter pools
Datasets
GSM8K
HumanEval
Bird
MultiNews
Method
MAT
Token/s
MAT
Token/s
MAT
Token/s
MAT
Token/s
Llama-3.1-8B
1.00
18.26
1.00
18.67
1.00
17.95
1.00
18.26
EAGLE
6.38
78.32
6.77
91.98
5.50
73.24
5.21
66.10
EXP3Spec
5.03
63.28
5,24
68.96
4.38
58.03
3.65
47.57
UCBSpec
5.05
66.11
5,46
72.05
4.67
62.60
3.58
46.04
HedgeSpec
6.77
84.32
7.54
97.21
6.67
83.87
5.53
65.20
Qwen-3-8B
1.00
14.82
1.00
14.71
1.00
14.91
1.00
14.89
EAGLE
5.05
60.59
4.84
58.81
4.08
47.91
4.00
46.81
EXP3Spec
4.90
53.57
4.15
48.62
3.73
43.67
3.21
36.26
UCBSpec
5.02
57.42
4.30
50.59
3.94
45.48
3.28
36.97
HedgeSpec
6.60
73.59
5.63
64.51
5.18
56.12
4.76
51.33
Qwen-3-32B
1,00
8.24
1.00
8.32
1.00
8.43
1.00
8.34
EAGLE
3.44
24.86
2.97
20.35
2.95
21.50
2.63
19.06
UCBSpec 5.00
37.50
3.97
29.66
3.85
27.56
3.48
25.09
EXP3Spec 4.86
36.38
3.88
28.62
3.78
27.36
3.34
23.46
HedgeSpec 6.57
42.37
5.17
33.82
4.92
34.72
4.84
33.25
Table 12: MAT (Mean Accepted Tokens) and Token/s (token generation rate) across GSM8K, HumanEval, Bird and MultiNews, which not part of the training datasets for the drafters. Bold indicates the best. Additional results regarding SpecBench shown in Table 13.Consistent with Table 3, HedgeSpec consistently outperforms all baselines across those two domains. Its shows that its orchestration of drafters improves MAT and throughput without prior knowledge, as long as there exists drafters excel in certain domain.
showS its SOTA performance over well-established speculative framework (i.e. Medusa, Rest, PLD). We believe that integrating HedgeSpec with EAGLE provides strong and representative demonstration of our method's effectiveness. Below, we further discuss how HedgeSpec can be integrated into these other frameworks as well.
Regarding PLD (Prompt-Lookup. -eckkding) (Saxena, 2023): PL.D utilizes the input prompt to guess the generation. In some tasks with input grounded generation, there are high chances of overlap between input and output. As a result, PLD uses only single surrogated drafter. Therefore, PLD not the best candidates for the multi-drafter framework.
Regarding Medusa (Cai al., 2024): Medusa is a well-known speculative decoding method. utilizes multiple Medusa heads predict the future tokens. These heads require training in order to align with the target task distribution. A potential integration with Medusa would follow a procedure similar EAGLE setup, that it obtains logits from the different set of expert Medusa heads and use them for full-information evaluation. This allows HedgeSpec to orchestrate the Medusa experts during speculative drafting.
Regarding REST (Retrieval- Based Speculative Decoding) (He et al., 2023): REST is another well- known speculative decoding framework. It treats different datastore the surrogate drafters, gen-
26

---

Published as a conference paper at ICLR 2026
Llama-3.1
Owen-3-8B
Qwen-3-328
Vanilla.
EXP3
UCB
Hodge
Vanilla
EXP3
UCB
Hodge
Vanilla
EXP3
UCB
Hodge
MAT
1,00
4.04
4,05
5.31
3.58
4.70
1.00
3.38
3.97
Token/s
18.24
52.19
52.68
60,80
14.32
41.02
41.23
80.70
8.12
24.86
24.24
27.73
Table 13: MAT and Token/s on SpecBench across different models (Llama-3.1, Qwen-3-8B, Qwen-3- 32B) under various speculative decoding strategies. Best results are bolded.
Model
HvmanEval
GSM8K
MT-Bench
Alpaca
MAT
Token/s
MAT
Token/s
MAT
Token/s
MAT
Token/s
Vicuna-7B-vlle
1.00
33.59
1.00
33.49
1.00
33.48
1.00
33.31
UCBSpec 1,45
44.60
1.38
42.33
1.10
37.86
1.38
43.85
Exp3Spec 1.43
43.55
1.36
41.99
1.11
38.40
1.44
46.63
HedgeSpec 1.88
55.27
1.62
46.58
1.28
41.42
1.69
52.87
Table 14: MAT and Token's across HumanEval, GSM8K, MT-Bench, and Alpaca for HedgeSpec comparing with other Bandit-based method on REST framework. Bold indicates the best.
erating future tokens by retrieving and composing relevant content from reservoir of existing knowledge. To fit into a practical multi-drafter selection framework, we can construct multiple such expert reservoirs, each specializing in a particular domain for token drafting. To integrate with HedgeSpec, we can calculate the token acceptance probability (EAL) from each drafter with the verified trace, and
select the best drafter fitting the requests.
We integrate HedgeSpec into the REST framework. Following the original REST setup, we use Vicuna-7B-vl.5 the target model. The expert drafters in this experiment include Python, Math, CNN-DM, MedQA, Alpaca, SQL., and Biology. Our goal remains the same: given pool of diverse drafters, determine how select the best drafter for each incoming task. We benchmark performance on HumanEval, GSM8K, MT-Bench, and Alpaca, with results shown in Table 14. Similar to findings under the EAGLE framework, HedgeSpec again achieves the best performance. This demonstrates the advantage of leveraging full-information feedback in drafter selection, allowing HedgeSpec to adapt to the best drafter in hindsight and ultimately improving overall efficiency.
E ETHICS STATEMENT
This paper presents work with the goal of advancing machine learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
F REPRODUCIBILITY STATEMENT
The models and datasets used in this paper fully open-sourced, with specifications provided in Sec- tions 4.1 and 4.2. Additional configuration details and implementation specifics are referenced the main text and included in Appendix B.5, D. and D.2. For theoretical contributions, complete proofs of the theorems are provided in Appendix A. Together, these materials ensure the reproducibility of results.
27


Reviews:
[{"review_id": "bnUeiE20bk", "reviewer": "Reviewer_G2tp", "summary": "The paper introduces speculative decoding with multiple specialized drafters. Contrary to previous bandit-based approaches where one picks single drafter and get information from them, author developed counter-factual estimator which can utilize mutliple drafter's accepntance rate, thereby converting bandit problem to online learning scenario, which proves to be more effective in EAGLE-3 based experiments.", "questions": "Questions \n\n* Can authors provide more experimental details about fine-tuning the drafters in 4.2? \n\n* Can you test the HedgeSPEC on batched inference scneario and prove the thorughput improvement?\n\n* While author cited the algorithm of the paper, it would be good to clearly state the overall algorithm (with base algorithm) is actually used in the experiment for clearer presentation.\n\n* (Minor) It seems like bandit-based approaches with multi-drafter scenario is first proposed by MetaSD [3], which is not properly reflected in the current manuscript.\n\n[1] (Li et al.) EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test\n\n[2] (Cai et al.) Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads\n\n[3] (Kim et al.) A Unified Framework for Speculative Decoding with Multiple Drafters as a Bandit", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Converting bandit problem to the online learning setting with theoretical guarantee (one-step estimator in Theorem 3) seems novel and interesting.  \n* Experiment is conducted on one of the SOTA method (EAGLE-3 [1]). \n* Proper ablations are conducted with sound presentations.", "weaknesses": "* **Drafter overhead** : While authors show utilizing multiple drafters in every SD step takes little overhead, this could be only hold for EAGLE-3 like model (where drafter size is way smaller than the target model). While EAGLE-3 shows SOTA performance, other method utilize relatively larger drafter (Medusa [2] for instance). Moreover, in real serving scenario, there might be SD framework with larger drafter. This could increase the cost of the drafter overhead linearly or reduce the merits of the online algorithm performance by increased drafter overhead. Discussion reagrding this scenario would be beneficial.\n\n* **Hyper-parameter for online algorithms** : The algorithm leverages (dealyed) Hedge algorithm which is standard algorithm in online learning. However, the regret bound is proved with the learning rate $\\eta$ which depends on the known horizon $T$ or current round $t$ (for anytime regret) but it might not be optimal for parcitcal performance. It would be good that author can clarify about the base algorithm they used. \n\n* **Non-stationary scenario** : In real-world scenario, a single query can be non-stationary especially for long context or contained in multiple scenario. In this case, Hedge algorithm might not be optimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VjJfFSGWGo", "reviewer": "Reviewer_dMBY", "summary": "This paper argues that multi-drafter selection for speculative decoding is not a bandit problem: by leveraging the single verified trajectory produced during standard speculative decoding, one can compute full-information feedback for all candidate drafters at no extra target-model calls. \n\nBuilding on this, the authors introduce HedgeSpec, which applies experts/hedging algorithms (e.g., Hedge/NormalHedge) with no-regret guarantees under two objectives—ETAP (expected token acceptance probability) and EAL (expected acceptance length). The paper formalizes ETAP/EAL, derives an unbiased off-policy estimator for EAL, and handles delayed/censored feedback. Empirically,  Llama-8B and Qwen 8B targets and pools of domain drafters are evaluated.", "questions": "1. Theory vs. Kim et al. (bandit). Under what logging or feedback constraints does your full-information assumption fail so that the bandit view is preferable? Please summarize regret/variance trade-offs and practical implications.\n\n2. Specialized drafter pools. Following Yi et al. (multilingual/specialized drafters), how does HedgeSpec perform when the pool contains varaints of EAGLE specialists per domain/language? How quickly is mis-specialization corrected under shift?\n\n3. Estimator robustness. Do you report empirical variance/CI for EAL across domains and K? Any stabilization (clipping, shrinkage, delayed-feedback smoothing)? \n\n\n4. Throughput & concurrency. With N drafters, what are tokens/s under realistic concurrent loads, and how do you cap parallel drafter evaluations to avoid HBM pressure/KV thrash? Please report GPU util, queueing delays, and prefill-verify overlap.\n\n5. Roofline & memory. What are per-drafter HBM and KV-cache footprints at target context lengths, and where do the kernels sit on a roofline plot for (i) multi-drafter prefills and (ii) target verify?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Uses the verified trajectory to evaluate all drafters, enabling no-regret experts methods without extra target calls.\n\n- Clear metrics  and formal treatment of delayed/censored feedback.\n\n- MAT and tokens/s improve versus EAGLE and bandit baselines across domains, but not domain specialized recent specdec are considered.\n\n- Scales with drafter pools for regret and MAT vs N are reported.", "weaknesses": "1. Related-work positioning is incomplete. The paper should explicitly acknowledge that the one other same online drafter-selection problem was formulated as a bandit [A], and compare assumptions/guarantees head-to-head (feedback availability, regret constants, robustness to partial/censored logs). It should also cite that motivates domain-specialized drafter pools [B].\n\n[A] A Unified Framework for Speculative Decoding with Multiple Drafters as a Bandit, Kim et al.\n\n[B] Towards Fast Multilingual LLM Inference: Speculative Decoding with Specialized Drafters, Yi et al.\n\n2. Estimator quality not fully characterized. Unbiasedness is shown, but variance under realistic  D (long contexts, domain shift) is not analyzed; large variance could slow adaptation or misrank experts early. \n\n3. Systems analysis is latency-centric. Deployment-relevant throughput (under concurrency), HBM/KV-cache footprints, and a roofline view (operational intensity vs. bandwidth ceilings for drafter prefill vs. target verify) are missing. Current overhead tables don’t fully address multi-tenant contention. \n\n\n4. Limited breadth of targets/drafters. Results center on 8B-class targets and EAGLE-style domain specialized drafters; broader speculative schemes (REST/Medusa/Spectr) and larger targets would strengthen generality. \n\n\n5. The method optimizes EAL/ETAP but reports MAT/tokens-per-second; a direct analysis of objective mismatch is absent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vOej68uJqA", "reviewer": "Reviewer_quC5", "summary": "This paper addresses adaptive drafter selection in speculative decoding for large language models. Instead of relying on a fixed small drafter, the authors propose HedgeSpec, an online learning framework that dynamically selects among multiple drafters during inference.\n\nThe key insight is that speculative decoding inherently provides full-information feedback—the target model’s verified trajectory allows computation of all drafters’ performance without additional target calls. This reframes the problem from a bandit to a full-information setting, enabling no-regret guarantees via standard Hedge algorithms.", "questions": "See weakness", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a clean and insightful reframing of drafter selection as a full-information online learning problem. This perspective simplifies speculative decoding analysis and allows provable no-regret guarantees without additional target calls。\n\n2. The proposed HedgeSpec framework is efficiently realized and demonstrates consistent throughput and acceptance gains across multiple domains and model families (LLaMA-3.1-8B and Qwen-3-8B).", "weaknesses": "- Although the evaluation phase introduces minimal overhead per drafter, it assumes access to sufficient computational resources for parallel evaluation. On resource-constrained systems, this could become a bottleneck. Since the results are only reported on 8 A100 GPUs, it would be helpful to include experiments on different hardware configurations and a breakdown of evaluation time and memory usage as the number of drafters N increases.\n    \n- The theoretical analysis relies on the assumption that feedback for all drafters can be computed without bias from the verified trajectory. Any deviation from this assumption may affect empirical optimality, and the error may accumulate as decoding process proceeds. An ablation across different numerical precisions would strengthen this claim.\n    \n- The evaluation scope is somewhat limited. (1) The method is described as training-free (Line 1043), but the draft models are pre-trained on specific datasets; and (2) the experiments primarily reuse these datasets. Beyond GSM8K and HumanEval, a broader evaluation on benchmarks such as SpecBench would be valuable.\n    \n- The experiments use Llama-3.1-8B-Instruct and Qwen-3-8B as target models. Demonstrating results on larger models would make the findings more convincing.\n    \n- Including experiments with batched prompts would provide insight into the method’s effectiveness in realistic, production-like settings.\n    \n- It would be interesting to discuss or demonstrate how the proposed approach integrates with other speculative decoding methods such as PLD, Medusa, and SAM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TVNzUR8axY", "reviewer": "Reviewer_PCBy", "summary": "This paper considers draft model selection in speculative decoding. It designs an online learning algorithm, HedgeSpec, which chooses the draft model in an online manner with full-information feedback, i.e., all the draft models can be evaluated without consuming much additional time. Theoretical guarantees for the token acceptance probability and expected acceptance length are provided, compared to the best fixed draft model in hindsight. Experiments are carried out with Llama-3.1-8B and Qwen-3-8B as the target models, and Eagle 3 series as the draft models.", "questions": "1. The proof of Theorem 4 invokes Theorem 1 from Joulani et al. (2013), which requires the delays to be independent of the forecaster’s prediction. However, this assumption seems violated in the setup of this paper, as the delays depend on the generated tokens of the chosen drafter model. Could the authors clarify this point?    \n2. Could the authors elaborate on Line 52: “Hedge or NormHedge to identify the best drafter _exponentially_ faster in N than bandit-based approaches”? It would be helpful to specify in what sense the improvement is exponential.\n3. According to Figure 2, does this imply that the statistics at one position are reused multiple times? Can the authors further clarify on how these statistics are shared or updated across positions?\n4. As claimed in Appendix B.5, evaluating more draft models slightly increases latency in the inference system. How does this latency scale with the number of draft models N? If the increase becomes non-negligible for large N, would a semi-bandit feedback mode—evaluating only a subset of the draft models per step (between bandit and full-information mode)—be a more practical strategy?\n5. If one relaxes the acceptance criterion on Line 96, does the proposed method still work? How to adapt the method to this scenario? \n\nPlease also refer to the weaknesses above.\n\nI favor the delicate way to evaluate all draft models and would appreciate it if the authors can resolve my concerns.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of using the verified trajectory to evaluate all the draft models is novel. It circumvents the bandit feedback setup in the literature and provides an alternative full-information with delayed feedback approach.\n2. It provides some theoretical guarantees on the performance of HedgeSpec for both acceptance probability and expected acceptance length objectives.\n3. Empirical experiments with Eagle 3 are conducted to demonstrate the effectiveness of the proposed approach.", "weaknesses": "1. **More clarity:** Some definitions of the notations are unclear. For example, do the two bounds in Theorem 4 hold at the population level or the prompt level? Are the probabilities conditional on $x_{<t}$ or on the entire input prompt? Is $T_{\\text{token}}$ fixed, or is it a stopping time? If it is fixed, what happens when the generated sequence terminates before $T_{\\text{token}}$? If it is a stopping time, how does the expected result look?\n    \n2. **Lack of interpretation:** More interpretation of the regret results would be helpful. In particular, for the result on accepted length (Line 299), while _AcceptLength_ is positively correlated with the quality of the draft model, its practical meaning is not well elaborated. For instance, the algorithm initiates speculative decoding only on a subsequence of the token sequence, and this subsequence is random.\n    \n3. **Limited draft methods:** The experiments are conducted exclusively with the EAGLE-3 series, providing limited evidence for the claim in Line 66 that “HedgeSpec could work with any given set of drafters on any speculative decoding method.”\n    \n4. **Mismatch between theory and experiments:** The theoretical results are derived under an adversarial setting, whereas the empirical studies are conducted under an i.i.d. assumption (Line 318). While parameter tuning in experiments is understandable, the shift from adversarial to i.i.d. settings may be too large. Moreover, the motivation for adopting the i.i.d. assumption is unclear, since tokens in autoregressive generation depend on all previous tokens.\n\n**Minors (mostly about writing)**:\n- I suggest that the authors carefully review the notations throughout the manuscript to ensure consistency and correctness, which would significantly improve readability. A few examples I noticed are listed below:\n    \n    1. Inconsistent notation: Accept_Length, $T_{Token}$.\n        \n    2. Line 199: an extra “=” in $q_i$.\n        \n    3. Line 209: the conditional probability expression appears problematic.\n        \n    4. The use of $\\gamma_{i,j}$.\n        \n    5. Line 760: $h$ should be $t$.\n        \n    \n- For self-consistency, it would be better to include the algorithms used in the appendix—for example, Algorithms 1 and 2 from Joulani et al.—as the _BASE_ algorithm in Theorem 4 does not appear within this manuscript.\n    \n- Since the paper claims that drafter selection can be performed in a full-information setting, it would be useful to explicitly discuss the corresponding losses and gains from both theoretical and empirical perspectives. There is likely a trade-off between faster learning and computational cost, as evaluating more draft models inevitably increases time and resource consumption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)