Faster Cascades via Speculative Decoding
Authors: Harikrishna Narasimhan
Wittawat Jitkrittum
Ankit Singh Rawat
Seungyeon Kim
Neha Gupta
Aditya Krishna Menon
Sanjiv Kumar
Venue: ICLR 2025 Oral (2025)
Keywords: Cascades
Speculative Decoding
Speculative execution
LLM
Inference
Adaptive Inference
Primary area: generative models
TL;DR: Faster language model cascades through the use of speculative execution

Abstract: Cascades and speculative decoding are two common approaches to improving language models' inference efficiency.  Both approaches interleave two models, but via fundamentally distinct mechanisms: deferral rule that invokes the larger model only for “hard” inputs, while  speculative decoding uses speculative execution to primarily invoke the larger model in parallel scoring mode. These mechanisms offer different benefits: empirically, cascades offer compelling cost-quality trade-offs, often even outperforming the large model; speculative cascades offer impressive speed-ups, while guaranteeing quality-neutrality. In this paper, we leverage the best of both these approaches by designing new speculative cascading techniques that implement their deferral rule through speculative execution. We characterize the optimal deferral rule for our speculative cascades, and employ a plug-in approximation to the optimal rule.  Experiments with Gemma and T5 models on a range of language benchmarks show that our approach yields better cost quality trade-offs than cascading and speculative decoding baselines.

Published as a conference paper at ICLR 2025
FASTER CASCADES VIA SPECULATIVE DECODING
Harikrishna Narasimhan', Wittawat Jitkrittum', Ankit Singh Rawat' Seungyeon Kim², Neha Gupta*t, Aditya Krishna Menon', Sanjiv Kumar 'Google Research, "Google DeepMind, Mistral AI (hnarasimhan, wittawat, ankitsrawat, adityakmenon, samjivkHagoggle.com
ABSTRACT
Cascades and speculative decoding are two common approaches to improving language models' inference efficiency. Both approaches interleave two models of different sizes, but via fundamentally distinct mechanisms: cascades employ a deferral rule that invokes the larger model only for "hard" inputs, while specula- tive decoding uses speculative execution to primarily invoke the larger model in parallel scoring mode, These mechanisms offer different benefits: cascades offer compelling cost-quality trade-offs, often even outperforming the large model; spec- ulative cascades offer impressive speed-ups. while guarantceing quality- neutray-. In this paper, we leverage the best of both these approaches by designing new spec- ulative cascading techniques that implement their deferral rule through speculative execution. We characterize the optimal deferral rule for our speculative cascades, and employ a plug-in approximation to the optimal rule. Experiments with Gemma and T5 models range of language benchmarks show that our approach yields better cost-quality trade-offs than cascading and speculative decoding baselines.
INTRODUCTION
Large language models (LLMs) have yielded significant advances in quality on a range of natural language processing tasks (Radford et al., 2018; Raffel et al., 2020; Brown al., 2020; Black et al., 2022; Chowdhery et al., 2022; Anil & et al., 2023; Touvron et al., 2023; Team et al., 2023; et al., 2024b;a), at the cost of an increase in inference latency. This has sparked a growing body of literature on reducing LLMs' inference costs without (overly) compromising on quality (Elbayad et al., 2020; Pope et al., 2022; Schuster et al., 2022; Leviathan al., 2023; Chen et al., 2023a; Sheng et al., 2023; Sun et al., 2024). One such line of work involves constructing a family of models of various sizes (e.g.. small and large model), and suitably orchestrating amongst them to make a prediction. Two canonical instantiations of this strategy are model cascading (Wang et al., 2020; Mamou et al., 2022; Varshney& & Baral, 2022; Khalili et al., 2022; Dohan et al., 2022; Chen et al., 2023b; Gupta et al., 2024; Ding et al., 2024) and speculative decoding (Stern et al., 2018; Chen et al., 2023a; Leviathan et al., 2023; Sun et al., 2024; Li et al., 2024a; Xia et al., 2024).
While similar in spirit, cascades and speculative decoding are fundamentally different in details. Cascades employ a deferral rule to identify "hard" inputs, and only invoke larger models on such inputs. For example, in two-model cascade, one first invokes the smaller model, and uses its associated probability of the generated output to decide whether to defer to the larger model. By contrast, speculative decoding uses small model to draft a block of tokens via standard auto- regressive decoding, which are then verified in parallel by a large model. One then accepts all drafted tokens until the first "implausible" one, which is rolled back based the larger LM's prediction.
Owing to their different mechanisms, both methods have complementary strengths. Cascades seek to output distributions that have the best quality for given cost budget, sometimes even yielding better quality than the individual models they are constructed with (Jitkrittum et al., 2023; Kim et al., 2023) ($3). By contrast, speculative decoding theoretically guaranteed to match the output distribution (or a close approximation thereof (Tran-Thiee 2023)), and is practically observed to provide impressive speed-ups (Stern al., 2018; Chen et al., 2023a; Leviathan et al., 2023; Sun et al., 2024). Given their complementary nature, natural question arises: can we leverage the best of both techniques?
Work done while working at Google.

---

Published as a conference paper at ICLR 2025
In this paper, we do so by designing new techniques for two-model cascades that im- plement their deferral rule in a speculative manner: we have the smaller model gener- ate drafts auto-regressively, and the larger model execute in parallel on the drafts to decide whether or not to defer on them. We show that this speculative cascading ap- proach yields better cost-quality trade-offs than both standard cascades and speculative decoding. In detail, we make the following contributions:
Verts und rolluck
Veriffer
Large
P
Target distmottion
Sma Janz regreskke
Defernine jpacsidd bg)
Drafter
Figure I:
Speculative cascade inference between a small and large LM via a deferral rule.
We introduce a general recipe for speculative execution, where we seek to mimic general target distribution that interleaves the drafter's s and verifier's distributions. Lossy speculative sampling (Tran-Thien, 2023) is a special case of this recipe for a particular target distribution ($4.1).
(ii) We show how common cascading rules, such Chow's rule (Chow, 1970) and confidence- difference thresholding (Jitkrittum et al., 2023), can be implemented speculatively by plugging in their target distribution into our framework. We refer to these as speculative cascades ($4.2).
(ii) We characterize the theoretically optimal deferral rule for a speculative cascade, and design speculative cascading technique that implements a plug-in estimate to the optimal rule (54.3, Lemma 4, Table 1). We also present token-specific variants of our deferral rules ($5).
(iv) Through experiments with Gemma (Team et al., 2024) and T5 models (Raffel et al., 2020) on a range benchmark language tasks including summarization, translation, reasoning, coding and QA, we show that speculative cascades are able to provide better cost-quality trade-offs than their sequential cascade and speculative decoding counterparts ($6).
Overall, we aim to develop principled approach to trade-off quality and inference costs by inter- leaving two models of different sizes, with promising empirical results. We hope to inspire future research adapting the proposed ideas with ingredients underpinning the state-of state-of-the-art the-art in speculative decoding (Cai et al., 2024; Li et al., 2024a;b; Chen et al., 2024).
A TALE OF TWO EFFICIENT LM INFERENCE STRATEGIES
Let V denote a finite vocabulary of tokens, with V denoting the set of all finite-length sequences generated by this vocabulary. Let Ay denote the set of all probability distributions over tokens in V. Given an arbitrary length sequence x = Z 22 ... IL. 4 and index S L,, we denote Te: 21 I- A language model (LM) is probability distribution over V*. Let IP denote the data generating probability distribution over V This could be, for example, distribution over prompt-response pairs that the LM may encounter during deployment, or a distribution of sequences used to pre-train the LM. We will measure the quality of an LM based on how closely mimics IP.
Suppose we are provided two LMs g and p, where is the larger (more expensive) model. Our goal is to design an inference strategy that selectively invokes q and p to trade-off between quality and latency (which may be approximated by the fraction of times that p is invoked). We will denote by q(ra) (<<)) the probability g associates to token I, E V given prefix I'e E pr-l, 1 and by p(z, (xe)) the same distribution from model p. Whenever it is clear from context, we will hide the conditioning on prefix Tee, and use the shorthand (+(-) for q(- ((-(xee)) and pt (-) for (((
Cascades are an effective strategy to trade-off cost and quality by having the smaller model q handle the "easy" samples, and the larger model p handle the "hard" ones (Gupta et al., 2024; Yue et al., 2024). A common cascading approach is confidence thresholding or Chow's rule (Chow, 1970; Jitkrittum et al., 2023), where we first run g on the input, and defer to p when g's confidence for its generated response is sufficiently low. This strategy is typically implemented at the sequence-level, where for a given prefix Zem we invoke g. evaluate its maximum conditional probability over all responses, and check whether falls below a threshold a E [0, 1]:
max
( ... .m Izm)
a.
If this holds, we defer to p generate response; otherwise, generate response with q. One may tune a to achieve a desired cost- quality trade-off. The literature also offers variants of Chow's rule that use more nuanced aggregation of pok-token uncertainties (Gupta et al., 2024).
2

---

Published as a conference paper at ICLR 2025
Table 1: Target distributions associated with different inference algorithms, where a is free parameter and [ 2 - a depends on o, and p. The last column indicates whether the execution is sequential (Algorithm 2), via an oracle (Algorithm 3), or speculative (Algorithm 5) (Leviathan et al., 2023). See (6) for details on 8. The third row presents variant of the BiLD algorithm of Kim et al. (2023), where D(g,p) p) is a measure of discrepancy between q and pt the original algorithm differs in the use of deterministic speculative decoding procedure with dynamic draft window (see BB)
Inference strategy
Deferral decision 6(g.p)
Target distribution mTu)
Execution
SpecDecoding (Leviathan al., 2023)
p(u)
Speculative
Lossy SpecDecoding (Tran: Thien, 2023)
max mmix (g(w), plo) mia)
Speculative
BiLD" variant (Kim et al., 2023)
1 (D(g.)) > aa
(1 5-5)- + 6
Speculative
TokenCascade [Chou] (Clow, 1970)
1( nuxx g(v) < 11-a)
(1 5) -(())
Sequential
Oracle [Diff] (Jitkrittum al, 2023)
1 maxx g(v) mmxx
(1 9()))
Oracle
SpecCascade [Chow]
1( max, g(v) 11-1- a)
(1 (-))) +6
Speculative
SpecCascade [Diff]
1( max, g(v) maxx, p(u) a)
(1- -)))))
Speculative
SpecCascade [OPT]
1( max, q(v) < max, p(y) Div(p.o)) ol)
(1 6) gg()) 4 p(M)
Speculative
Speculative decoding is an alternate strategy that applies token-level interleaving between q and p. seeking to provably match the larger model quality reduced inference cost (Stern et al., 2018; Leviathan et al., 2023). Given a prefix Zzt, we draft y draft tokens Tt..... ..t..-1 via auto-regressive sampling from g, and verify if these tokens can be accepted by running p in parallel on the y prefixes Eet.-.., ; We then rollback to the first rejected token & + j (where j" E (0, 1,..,,,7 7 replace Fz4z" with new token, and repeat the process with prefix #<0+5"+1-
1},,
During the verification stage, a draft token Tı+j generated by is accepted with probability
min 1, Pe+z((e=3) ) and rejected otherwise, recalling the shorthand (+5(.)) = ((1[=<05)) and (11(((115)) Pt+55) (.) = p(-(=<004)) A rejected token is then replaced by a new token sampled from a modi- fied distribution norm (max {0, Pttz(-) ()++(()) where norm(-) denotes normalization to sum to 1. This sampling process is provably equivalent to sampling 7 tokens auto-regressively from p for prefix <<t (Leviathan et al., 2023). We summarize this speculative sampling procedure in Algorithm 1. Each invocation of this algorithm generates at most 7 + next tokens (and at least one) for a given prefix Fec. One may run this algorithm multiple times generate a complete output sequence.
In practice, one may employ a lossy variant (Tran-Thies) Thien, 2023) of the above sampling that allows some deviation from verifier's distribution p. In this case, draft token Tz+j is accepted with
probability min (1, (1-(1-0) (((1)) where E [0, 1) is strictness parameter, with higher values indicating greater deviation from p. A rejected token may then be replaced by a token sampled from the residual distribution norm (max (o, 1/2 P++s(+) - (++)(+))) where , 2 - a a parameter that depends on a, q and p. A common heuristic is to simply set 3 = 1 (Zhou et al., 2024).
CASCADES MEET SPECULATIVE DECODING
Both cascades and speculative decoding interleave models of different sizes to reduce inference cost, but fundamentally differ in the mechanisms they use. As a step towards comparing the strengths and weaknesses of these approaches, we first describe how one may design a token-level cascade.
3.1 WARM-UP: TOKEN-LEVEI, CASCADES
It is straightforward to extend the sequence-level Chow's rule from $2 to form token-level cascade between y and p. For a prefix Iet, we first compute the smaller model's distribution q:(-), and check whether max (=(1) is below a pre-chosen threshold. If so, we evaluate pe(.), and sample Ie ~ (.): otherwise, we sample x, ~ (.).)
More generally, we may design token-level deferral rule 1t-1 {0, 1} that takes the prefix Tet as input and outputs a binary decision, with r( = 1 indicating that we defer to p (i.e., draw sample from p rather than q). For example, token-level Chow's rule can be written as:
Thooo (x)))) = max.ev ((()) 1 o,
(2)
3

---

Published as a conference paper at ICLR 2025
where is a threshold parameter, the higher the value, the lower is the frequency of deferral to p. One may also use other confidence measures than the maximum probability, such as the entropy of the small model'ss probability distribution. We elaborate in $D.1 that the choice of confidence measure would depend on the evaluation metric of interest; Equation (2) is typically prescribed when the cascade's quality is evaluated in terms of its accuracy against the data generating distribution on individual tokens, whereas entropy is prescribed when the metric of interest is the cross-entropy loss.
3.2 OPTIMAL TOKEN-LEVEL CASCADE DEFERRAL
While Chow's ' rule (2) is easy to implement, it can be sub-optimal if the smaller model's max-token probability is not reflective of which of the two models are better equipped predict the next token for a given prefix (Jitkrittum et al., 2023). Given this, it is natural to ask what the optimal deferral rule r for token-cascade looks like, and whether we can reasonably approximate this rule.
For this, we must first specify an objective to minimize at each step t. Following the prior cascade literature (Jitkrittum et al., 2023; Gupta et al., 2024), reasonable objective to minimize is the expected loss from the deferral rule against the data generating distribution P. with an added cost for deferring the larger model. We state this below for fixed prefix xict, using before the short-hand (-) for (())) and Pe(.) for pl p())
Loo xx)) = BewPe (nce) [L r(æ=1)) - L(V, gt) + +((x))) (e(u, pr) + +a]].
(3)
for a cost penalty a 2 0 and loss function E: V x Dy + R4.. - Common choices for e include the 0-1 loss ( (v, 1 (v A arg max, ge (<())) and the log loss (or(U, qt) log (q: (v)).
Lemma 1 (Optimal deferral for token-level cascades (Jitkrittum al., 2023)). The minimizer (3) is of the form:
(I<t) 1
(ewP() [e(e, > Ex ((u)))) + a.
Intuitively, compare the expected loss from q with the expected cost of invoking p, and decide to defer when the latter is smaller. We note here that this optimization problem is set up for a fixed prefix Fec. One may also consider the coupled optimization problem across all positions.
Plug-in estimator for (4). The optimal rule in (4) requires computing expectations over the data generating distribution P- which is not available during inference time. A common approach in the cascades literature is to replace the expected losses with the models' confidence estimates (Jitkrittum et al., 2023). For example, when L fy-1, it may be reasonable use max, (v) as an estimate of the expected 0-1 loss En,-P((laccc) [ 9:)] and 1 - max, (v()) as an estimate of E (1x (x): The extent to which these estimates are accurate depend on how well and p are calibrated (Guo et al., 2017). The resulting plug-in estimator for (4) thresholds the difference of
confidence estimates from both distributions:
Abier (x)
max, Zt (V) max, (v)
Similarly, when é flog. we may use the entropy - E. (v) log(q(()) from ge as an estimate of its expected log-loss, and similarly for ps (see $ D).
Remark (Diff rule is not realizable with token-level cascade). We cannot directly employ Pdits in a token-level cascade, as it needs the large model to be invoked at every step t. However, it serves as an oracle that allows to analyze the head-room available to improve upon Chow's rule.
3.3 CONTRASTING TOKEN-LEVEL CASCADE AND SPECULATIVE DECODING TRADE-OFFS
Token-level cascades and speculative decoding differ in the distribution over tokens they seek to mimic. Speculative decoding seeks to mimic the large model's output distribution, and is usually used when one wants to match the quality of the large model. On the other hand, token-level cascades seek to output distributions that closely approximate the label distribution and potentially offer good cost-quality trade-offs, sometimes yielding better quality than
the large model.
Cascades are useful when the draft model fares better than the verifier on some inputs, and one may want to retain the drafter's predictions even when it disagrees with the verifier. Even in cases where
4

---

Published as a conference paper at ICLR 2025
WMT:: Small Large
XSum: Base Large
13
Cascade (Chom]
12
Cascade [Chom]
Oracie
11
Orade
0.0 0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
41 #large-moodll al deferrals/ / #tokknns generated
#large-madel deterrals / #ttkanns generated
Figure 2: Plots of quality as a function of the number of deferrals to the larger model divided by the total number of generated tokens for cascades constructed from T5 models (under temperature sampling with T = 1). The left-most point represents the small model and the right-most represents the large model. We compare token-level cascades constructed with Chow's 's rule (Chow) and an oracle deferral rule (Diff). While speculative decoding will match the quality of the large model (see dashed horizontal line), the oracle deferral rule yields significantly better quality on range of deferral rates.
both the drafter and verifier fare poorly on some inputs (e.g., due to label noise), one may want to ignore the disagreement between the drafter and verifier to avoid triggering unnecessary roll-backs.
As a concrete example, we consider token- cascades of T5 models (Raffel et al., 2020) of two different sizes finetuned on WMT EN - DE translation Bojar et al. (2014) and extreme summarization (XSum) task (Narayan et al., 2018). We construct these cascades using Chow's 's rule in (2), and the Diff rule in (5), which noted in Remark 1, serves an oracle. In Figure 2, we plot quality as a function of fraction of samples deferred to the large model (number of deferrals divided vary the cost parameter a. Recall that speculative decoding is guaranteed to match the quality of the large model. In contrast, the Diff rule based cascades yield a wide range of cost-quality trade-offs, often outperforming the large model. Even Chow's rule, which is sub-optimal for cascading (Jitkrittum et al., 2023), outperforms the large model in small region. As noted by Kim al. (2023), this may be attributed the ensembling effect in a cascade.
by number of generated tokens),
However, compared to speculative decoding, token-level cascades may require a significantly larger number of deferrals to the large model to achieve the same quality. This is because token-level cascades are executed sequentially: whenever q defers, we execute once to generate one next token for the prefix accumulated so far, and the control transfers back to q. contrast, speculative decoding runs p in scoring mode to verify y draft tokens from in parallel. Moreover, the stochastic verification algorithm in speculative decoding often results in fewer tokens from q getting rejected compared to the deterministic deferral rules used in cascade. These observations motivate natural question: given their complementary strengths, how can we leverage the best of both these techniques?
4 SPECULATIVE CASCADES: LEVERAGING THE BEST OF BOTH WORLDS
In addressing the above question, we present our main contribution: speculative cascades, a principled approach to combining the better trade-offs token-level cascades offer with the faster execution of speculative decoding. Unlike token-level cascades, where the large model is called only when the small model defers, speculative cascades invoke the large model in parallel scoring mode after every y draft tokens. Consequently, they have the added benefit of being able to implement deferral rules that are not realizable with sequential cascade, and can thus potentially achieve lower latencies.
4.1 SPECULATIVE DECODING WITH GENERAL TARGET DISTRIBUTIONS
We begin by considering generic version of speculative decoding that seeks to mimic general target distribution derived from the drafter's and verifier's distributions. the proposed sampling procedure outlined in Algorithm 4, we sample tokens regressssrysylly before from the drafter's distribution. During the verification step, however, we do not compare the drafter's token probabilities against the verifier's distribution. Instead, we use user-specified target distribution π = T(q. p) E Ay derived from the drafter's ss and verifier's distributions at position t, for some function T(: that is inexpensive to compute. We accept draft token 2; when (xx;) I (2x)) and reject it otherwise with probability
1
x(2.) Upon rejection, we re-sample from the residual distribution norm (0, π (-) (()))).
((x)))
This general procedure not only encompasses standard speculative decoding (Leviathan et al., 2023) for T(y. p) p. but also includes lossy speculative decoding (Tran-Thies 2023) as a special case:
5

---

Published as a conference paper at ICLR 2025
Lemma 2. Algorithm 4 reduces to the lossy speculative sampling procedure in (Tran-Thier) 2023)
with parameters a and 3 when T(q, p) max(min(q(e), [-a), p(2) }.
4.2 FROM SEQUENTIAL TO SPECULATIVE CASCADES
Equipped with Algorithm 4, we now propose new cascading techniques that implement their deferral rule in speculative manner. Recall from $3.1 that token-level cascade of two models and is defined by deferral rule r : p
{0, 1). For prefix Eet, the next-token distribution at position t modeled by this cascade can be written as:
<(u)= = (1 - -((=))) q:(x) + r((z)) Pe(v).
In fact, for all the deferral rules described in $2, the resulting distribution can be described by target distribution function T, of the form:
Ts(q, ())) = (1 8(q. p)) ((v) 5(9.)) p(v),
(6)
for some function Av x Dv + {0, 1 that maps distributions (g.p) to a binary decision. For example, to implement Chow's 's rule, we may choose 8(g. p) 1( max, q(v) -x)... ot).
Our proposal is to then invoke the speculative sampling procedure in Algorithm 4 with To as the target distribution function. We outline this generic speculative cascading approach in Algorithm 5, and contrast it with the sequential execution of a deferral rule in Algorithm 2.
Interestingly, as noted below, the Diff rule, which was not realizable with a token-level cascade, can be efficiently implemented with speculative cascaded using ö(g, p) = 1( max, g(v) < max, p(v)- a). See Table 1 for summary of different deferral rules and corresponding target distributions.
token-level cascade, the large model's ss distribution cannot be used at the time the deferral decision is made (see Remark 1), as this would defeat the purpose of the cascade. With a speculative cascade, however, we can employ rules such Diff that depend on both q and p. This is because run the large model in parallel on drafts generated by the small model allowing us to compute both p(-) and q(-) on every prefix.
Remark (Diff rule is realizable with a speculative cascade).
So far we have considered deferral rules designed for use with (sequential) token-level cascades. In what follows, we derive the optimal deferral rule r for a speculative cascade, where we sample speculatively from a target distribution π (1 r(=<()) qe r(r<t) pt using qi the drafter.
4.3 OPTIMAL SPECULATIVE CASCADE DEFERRAL
We seek a deferral rule r pt-1 + f0, 1} that trades-off between quality and inference cost. As with $2, we measure quality in terms of the loss incurred against the data generating distribution. The inference cost, on the other hand, crucially depends on how frequently draft token is rejected in the verification phase, triggering a rollback. To this end, seek minimize the expected loss from the deferral rule subject to constraint on the resulting rejection rate. More specifically, (i) we show that the rejection rate can be computed using a simple closed-form expression (Lemma 3); (ii) we formulate a constrained optimization objective (7) and the corresponding the Lagrangian (8): (iii) we derive the optimal deferral rule that minimizes the Lagrangian (Lemma 4), approximate it with plug-in in rule (10), and provide regret bound guarantce for the approximation (Lemma 5).
Lemma 3. For a given prefix Tet,, and target distribution = (1 r(x=))) q: + +r(cct) . Pts the probability of a token drawn from draft distribution qı being rejected is equal to: r(z2)) Dr)( where Div(p.g) = Evev max p(u) - q(v}} is the TV distance between p and q.
TTv)= qe(v), and therefore there is no rejection or roll-back; when r ((re)) = 1, the rejection rate equals Drv qt), per Leviathan et al. (2023).
Intuitively, whenever r(zc))=
For fixed prefix F<t we formulate the goal of finding a solution to:
min Ep-F(-lrce) (1 - r(x=))) ((v.qt) + r(r2)) E(v, pt). s.t. r(zz)) Drv (pc. q:) S B, (7)
for some budget B > O. Equivalently, one may minimize an unconstrained objective similar to (3), for suitable cost parameter (see SD.5):
Tspoc (r; (et) (xwP(()ee) [(1 r(x<))) ((u. ge) + +((ee)) (l(v,p)) + a Drv(pt, Dry(ps:())), (8)

---

Published as a conference paper at ICLR 2025
Algorithm 1 SpecDecode
Algorithm 4 GenSpecSample
Input: Models q. p. Prefix zet. Block size y p) =
Input: Models q. P. Target distr. T, Prefix tee, Block size Y
// Sample 7 tokens regessegryyy from for j = 0 to 7- do
Output:
GenSpecSample(g, 2er, ?)
4++(()) = -111=<)))); Eets 9(+(()
end for
Algorithm 2 TokenCascade
// Run in parallel to score 7 draft tokens
Input: Models q. p. Deferral logic 6, Prefix tect (+(+) = g(|[2=e))
(++s() = p(-[reets); Vj h1 ?0 *etj = ((
if 6(q,, 0)
then Sample xi (.)
// Find the earliest rejected draft token
((=)(1+4)] a Ber mmnn (1, **t5(40+)) ) Vj E (r - 1}; a,
else
px(-)= = p(-ke))) Sample xy pr(-)
j = min [n] = 0}
end
// Sample new token from residual distribution pron()
Output: s6
Imrx {0, Te-s (+t5-()))) if < T+))(-)
Algorithm 3 OracleCascaca
else
Input: Models q. p. Deferral logic 5, Prefix Tee (:(-) = q( [r=); pe(-) = p(11x<c))
Sample D14s" ~ Pies
Output: 21,5.,,
if dgg. Pt)
then
Sample Æi ~ (.)
else
Algorithm 5 SpecCascade
Sample (-)-)
Input: Models g. Deferral logic 6, Prefix æes, Block size Ts(q. (1 5(g. p)) q 5(q.p) P) Output: GenSpecsample P. To:7
end
Output: 2t
Contrasting (8) with the deferral risk in (3) for a token-level cascade, the difference is that the cost of deferring to the larger model is no longer a constant, but depends on the similarity between and Pt. as measured by the total variation (TV) distance between them. Analgous to Lemma 1, we next derive the optimal deferral rule for (8), and then construct a feasible estimator for it.
Lemma (Optimal deferral for speculative cascades). The minimizer of (8) is of the form:
(((x)) =1 > Ex-P())xe)) [e(2, q.) > Eu-p(1 [L(v, Pc) + a- Drv(Pts q:).
When Pt and q: are similar, the rejection rate for qe is low, and hence the deferral decision will depend largely on which of the two models yields a lower expected loss. When pt and qe are very different, the optimal decision is to defer to only when it yields a substantially lower loss than dt-
Plug-in estimator for (9). The optimal rule requires estimating expectations with respect the data generating distribution P(-(x=)). We employ similar plug-in estimators as the ones used with token- level cascades ($3.2). When = La., we replace the expected 0-1 loss with (one minus) the maximum probability from the model, giving us:
fopr(x<<))
max, (v) < max. (t) - a Drv gc).
(10)
The efficacy of the plug-in estimator depends on how closely the individual models approximate the data generating distribution PF-()2)) this is formalized by the following regret bound:
Lemma (Regret bound for foer). Suppose = 60.1. Denote Pv(v) P(ulæz)). Then for fixed æzti
I Luspcc (Forr; xe)) min [rp < max P. (v) (v)) NEV
max (P((v) pc(1))-
-EV
One can now run the speculative cascading procedure in Algorithm using (10) as the deferral rule; the corresponding 8(.) is listed in Table 1, See $D.3 for similar derivation for & = 4oo--
5 BEYOND CASCADED DEFERRAL: TOKEN-SPECIFC INTERLEAVINGS
The deferral rules we have seen so far in (5) and (10) decide between the drafter's distribution (.) and the verifier's distribution pc(.) by comparing their maximum token probabilities. A downside to
7

---

Published as a conference paper at ICLR 2025
this cascaded form of deferral is that the specific draft token sampled 2, qe(-) may not be the same as the token that maximizes maximize qc(.). Thus, even when 2s is of poor quality, we may end up accepting it because 9: happens to be more peaked than Pt-
Token-specific interleaving. To alleviate the above problem, we propose the use of token-specific deferral rules r: V x V {0, 1 that use both the prefix z-<t and candidate token to provide a binary decision r( u) E 1. with O indicating that the token is of acceptable quality. We may then construct target distribution of the following form:
Tooee (v) = qe())
- r v)) + pc()) -7,
(11)
where 7= = T( o').q(((")) ( is a normalizing term chosen to ensure that Evey Tioken (v) = 1. This target distribution closely mimics (-) on tokens that the deferral rule r deems to be of acceptable quality, and defers to pe(.) otherwise. One can modify the generic speculative sampling algorithm in Algorithm 4 to use Trokan as the target distribution, as shown in Algorithm 6 in $E
To design the deferral rule r, we propose heuristic variant of the Diff rule in equation 4 (in $E, we discuss deriving a similar variant of the OPT rule in equation 9). Specifically, we compare the probability that the draft token is the incorrect next token to << according to the data- data-generating distribution P with the expected 0-1 loss that we would incur if we were to defer to the verifier pt:
r(I<,, v)
+
1 P(ela<)) > Egre(((lees) [oo .p)] + a,
(12)
for a cost parameter a. The following are some simple plug-in approximations to (12), where we approximate P(ulare)) with either ge (v) or (v)), and the expected 0-1 loss using max, Pe(v/):
Too (Te)) v)
(((v) < max. pc(t') - a
(13)
Toxxnnv (Fet, v) = 1
Pr(u) < max,' Pe(v') - a P((v) < maxy Pe(v') (1 -1-00 a).
(14)
(Tozenvs(#<11) v) =1
(15)
where (15) uses a multiplicative plug-in approximation.
The resulting target distributions have an intuitive form. For example, with (13):
TokkenV1 (v) = (Q(()) 1(v Ta) + pc()) EV ((()),
(16)
where To {v E V: (v) 2 max,' Pa("") a} is the set of tokens deemed important. For these tokens, TookenVi approximates (+):;; for the rest, it is a re-scaled version of p.)))
Contrasting with lossy speculative sampling. Recall that lossy speculative sampling (Tran-Thien, 2023) also seeks to mimic a token-specific interleaving of q: and pt. given by Tlossy (v) = max(min (v)) p.(e)) }, 88 }, for trade-off parameters a, 3 (Lemma 2). However, in some settings, this choice of target distribution may severely limit the range of cost-quality trade-offs that can be achieved by varying a and 3. For example, note that Lossy (v) = 0 whenever Pt (v) = O, making the trade-off parameters a and (3 irrelevant for such tokens. This can be particularly problematic when sampling with a small temperature or when applying top-/' sampling, where and p. may not share the same support. In contrast, our proposed approach to token-specific deferral enables a wider range of trade-offs under both temperature and top- sampling, by computing the deferral rule r in (11) on unscaled versions of g: and pt. while interleaving between scaled versions of yr and pr (see &C).
In fact, in the extreme case of greedy decoding (sampling with temperature T = 0), TLossy simply degenerates to Pt- For this special case, Leviathan et al. (2023) propose an alternate lossy version of speculative decoding with deterministic rejection criterion similar to (15). Interestingly, our proposed TokenV3 approach reduces to this variant when T + 0 (see $C.4). Thus TokenV3 can be seen generic deferral rule that is applicable to both greedy and non-greedy decoding.
6 EXPERIMENTAL RESULTS
We compare our speculative cascading techniques with both sequential cascades and standard speculative decoding on range of language benchmarks, including translation, reasoning, coding, QA, etc. We evaluate speculative cascades constructed from both the T5 v1.1 family of encoder- decoder models (Raffel al., 2020), and Gemma v2 decoder-only models (Team al., 2024). !
'Iluustrative colab notebook with Gemma models available at: http:://githho..mm//gggeeee-- cascades...
google- hataeeeeseerrrrrreeeeeteeeeereee
8

---

Published as a conference paper at ICLR 2025
Table 2: Reduction in latency (T = 1, 5) when matching the quality of the large model (cols
2-7), and the best quality metric without exceeding the latency of the large model (cols 8-13). Quality measured in terms of BLEU for WMT and ROUGE-2 for XSum and CNNDM. Rows 1-4 are the baselines; Rows 5-6 5-6 contain the proposed method with old deferral rules ($3); Rows 7-8 are with new deferral rules ($4). See FF2-F3 for results with varying temperatures and top-P sampling.
Latency, when matching large model's quality
Best quality without exceeding lagg model's lasency
Small - Large
Small - XI.
Small - Large
Small + XI.
Method
WMT
XSum
CNNDM
WMT
XSum
CNNDM
WMT
XSum
CNNDM
WMT
XSum
CNNDM
SegCascade [Chow]
1.55x
0.84x
0,98x
2,46x
0.93x
0.94x
16.56
12.97
9.91
16.29
16.40
11.18
TokenCascade [thow]
1.0%%
1.40x
D.2X
1.51x
16.52
13.30
17.09
11.44
Spechecode [Lossy]
1.61
1.10%
1.57x
2.17%
1.28x
2,07x
17.26
13.90
16.94
17.36
11.53
BILD"
1.34x
1.04x
1.38x
1.85x
1.28x
1.84x
16.49
13.81
10.14
15.90
17.35
11.35
SpecCascade [Chow]
1.4xx
1,01X
141x
2,01X
1.28x
197x
17.76
13.82
10.28
16.35
17.36
11.39
SpecCascade (oirr]
1.79x
1.17x
x.75x
2,44%
1.30x
2.15x
18.04
14.00
10.64
18.07
17.37
11.67
SpecCascade [OPT]
1.95x
1.17x
1,0xx
2,61x
1.34%
x
18.33
14,10
10.86
18.09
17.48
11.85
SpecCascade [Token]
1.85x
2,50x
1.40x
1
22.50
15.85
12.63
Cascades versus SpecDecode evaluation. Our evaluation protocol is markedly different from the standard evaluation of speculative decoding algorithms,, where the goal is to speed up inference with a large model while preserving output distribution. In contrast, our focus is on trading-off quality for lower inference costs by interleaving two models of different sizes. We also do not claim to develop a new sfate-of-the-art method for fast LM inference. Furthermore, the speculative cascades we design build on the original speculative decoding algorithm Leviathan et al. (2023). While one could potentially also adapt our proposal to other recent variants of speculative decoding (Cai et al., 2024; Li et al., 2024a), these involve wholly orthogonal suite of techniques to what we propose (such as architectural changes, allowing for multiple drafts, distillation, and so on; see BB).
Proposed methods and baselines. We evaluate proposed speculative cascades with four deferral rules: (i) Chow in (2), (ii Diff in (5), (iii) OPT in (10), and (iv) the Token-specific rule in (15). Of these, (i) and (ii) are existing deferral rules, while (iii) and (iv) are new rules we propose. We also present results for the V1 and V2 variants of the token-specific rules in $F.7.
We compare these with the following cascading and speculative decoding baselines:
ii Sequence-level cascade (Jitkrittum et al., 2023; Gupta et al., 2024) based on sequence-level Chow's rule in (1) (SeqCascade [Chow]).
(ii) Token-level cascade outlined in Algorithm 2, with token-level Chow's rule in (2) used for deferral (Chow, 1970; Gupta et al., 2022) (TokenCascade [Chow]).
(iii) Lossy speculative decoding described in $2, with both 3 1 (Leviathan et al., 2023; Zhou et al., 2024) (SpecDecode [Lossy]) and tuned using the procedure in Tran-Thien (2023) (Lossy*).
(iv) Big- Little Decoder approach (Kim et al., 2023), with both the original deterministic rejection rule (BiLD), and the stochastic rejection sampling variant of their method described in $B (BiLD").
Fine-tuned T5 cascades. Our experiments T5 models are based on the setup in Zhou et al. (2024); see $F.1 for details. We use T5-small (77M) as the small model, and either T5-large (800M) or T5-XL (3B) as the large model. In each case, we supervised fine-tune these models on three tasks: WMT EN translation (Bojar et al., 2014), CNN/DM summarization (Hermann et al., 2015), and XSum abstractive summarization (Narayan et al., 2018). We use temperatures T 0, 0.1, 0.5, 1.0, and block sizes = 3. 5, 7 (full results in $F). Following the protocol in Leviathan et al. (2023); Zhou et al. (2024), to measure latency, we evaluate the wall-clock decoding time with batch size
In Table 2, we report for the each method, (i) the reduction in latency from T5 cascades when matching the quality of the large model, and (ii) the best quality it can deliver without exceeding the latency of the large model. SeqCascade and TokenCascade are often seen to fare poorly on both quality and latency metrics, with the exception of WMT, where SeqCascade yields non-trivial speed-ups. SpecCascade [Token] often yields the highest speed-up and the best quality metrics, with OPT coming in second. The reason the Token-specific rule fares better than OPT and Diff is because the latter compute their deferral decisions based on which of qe (-) and p. (-) is more peaked; this can be a disadvantage when the sampled token is not close to the distribution mode, which is likely to happen when applying temperature sampling with high temperature.
9

---

Published as a conference paper at ICLR 2025
28 238
CRAON Siplot:
GSMBE 2278
23
Eruct Mauchh
20
Rougee
SpecDecate (aayy
Spec Decode (Lona)]
SpecDecode [taey]
SpecCincade (OPT]
Spec [OPT)
SpecCascade (3PI) SpecCascade Tokent
SpecCascade Thken!
SrecCascade [hiken]
0.4
0:30
5.05
0:30
0:15
0.26
2:00
0.00
0.05
3:00
0.5
2.23
0.25
0.00
0.22
3.04
0.56
2.08
2.10
0.32
Rejesioraaz
Rejection raze
VAbO sishot: 20 - 278 (T-1)
NDFP 20 (78 (T=1)
Natura CA I-anct: 270 IT
0.35
6.5
0.50
2.30
11 3.28
0.25
3.26
SpedDecode [Lasy] SpecCascade SpecCascade (oean)
SpecDecade (Lassy]
0.20
SpecDecode [Leay] SpecCascade 10PP! SpecCascade [Town))
SpecCaseade [OPT)
3.24
SpecCascade [Baken)
6.35
6.30
0.55
0.35
0.15
20
25
0.00 0.05
6.15
6.55
0.0
0.2 8.
64
Figure 3: Plots of quality vs. rejection rate for methods that interleave Gemma 2B with Gemma 27B (y 1). We use instruction cenneiio--uuedd models; for MBPP we report additional results with pre-trained models. See $F.7 for remaining plots, comparison to (13-14) and results on 2B + 9B cascades.
We present plots of quality vs. latency for the different methods in Figure 4 in $F. In each case, we vary the trade-off parameter and plot the quality metric as function of the relative latency to the large model. While methods that use speculative execution are considerably faster than sequential cascades (TokenCascade [Chow]), the latter offer better quality in the low-latency regimes. This is because unlike speculative approaches, which always call the large model after every steps, sequential cascades invoke the large model only when the small model defers. In $F.5-F6, we present additional comparisons to SpecDecode [Lossy"], and the original BiLD algorithm (Kim et al., 2023).
We also report results with varying temperatures in $F.2, and with top- sampling in SF.3. As the temperature or P becomes smaller, SpecDecode [Lossy] yields comparable quality as our methods, but is severely limited in the range of cost-quality trade-offs it offers (see discussion in $5). In contrast, SpecCascade [Token] offers wider range of trade-off off points even for low temperature or P values, and is able to match the quality of the larger model at lower latencies (see Table 5).
Few-shot Gemma cascades. To evaluate the Gemma model cascades, we use few-shot prompting with 8 language benchmarks: WMT, CNN/DM, GSM8K, MBPP, SQuAD 2.0, WebQuestions, NaturalQA and TriviaQA; many of these feature in the SpecBench suite (Xia et al., 2024). Figure 3 presents plots of quality vs. rejection rate with a 2B drafter and 27B verifier for 7 = 1. For brevity, we only compare the methods that fare the best in the previous experiments. With the exception of TriviaQA, SpecCascade [Token] is able to both match the 27B's s quality at a lower rejection rate and yield the best overall quality, often better than 27B. Since all three methods use the exact same implementation but with different rejection criteria, we directly compare their rejection rates.
Interestingly, the OPT rule is not effective as it with the T5 models. We attribute this to the differences in distributions between the two setups. With T5, the maximum token probability served as good indicator of token accuracy for both q and p. With Gemma models, however, we expect the large model to have closer alignment with the data generating distribution (due to it being several billion parameters apart from the smaller model), and hence using the large model probabilities to measure confidence for both the small and large model (15) yields better trade-offs than comparing the modes from the two model distributions. More generally, we expect SpecCascade yield significant gains over SpecDecode when there exists a slice of data where the small model performs comparable to or better than the large model. The larger this slice, the larger the improvement.
Conclusions. We have proposed new speculative cascading techniques that use a combination of auto-regressive drafting and parallel verification to implement their deferral rule, and shown that they yield better cost-quality trade-offs than standard sequential cascades and speculative decoding. A limitation of our approach is that while it offers lower latency via parallel execution, it also incurs higher total compute cost compared to sequential cascades. In the future, we wish to replace our plug-in in estimators with a router model (Gupta et al., 2024) trained on ground- -ruth samples to approximate the optimal rule, and to extend our proposal to more than two models.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. We thank Ananda Theertha Suresh and Ziteng Sun for insightful discussions and for help implementing speculative cascades with Gemma models.
REFERENCES
Rohan Anil and et al. PaLM 2 technical report, 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint arXiv:2 (00.27008.7772 2021.
Jonathan Berant, Andrew Chou, Roy Frostig, and Perey Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference empirical methods in natural language processing, pp. 1533-1544, 2013.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX- 20B: An open-source autoregressive language model. In Angela Fan, Suzana Ilie, Thomas Wolf, and Matthias Gallé (eds.), Proceedings of BigScience Episode #5 Workshop on Challenges & Perspectives in Creating Large Language Models, pp. 95-136, virtual+Dublin, May 2022. Association for Computational Linguistics. doi: 10. 10.18653//222222bbssiieeee //aclanthology. , 2022. wttp:///...igsccienceeeeeeee bigscience-1.9.
URL https:
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop statistical machine translation, pp. 12-58, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerberttVosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023a.
Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance, 2023b.
Zhuoming Chen, Avner May, Ruslan Svirschevski, Yu-Hsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable and robust speculative decoding. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openrevieee net. forum?id-rkaLoyGD.
Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Jie Huang, and Kevin Chen-Chuan Chang. Cascade speculative drafting for even faster LLM inference. arXiv preprint arXiv:2312.11462. 2023c.
C Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16(1):41-46, 1970.
11

---

Published as a conference paper at ICLR 2025
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayant Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier- Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Pa Scaling language modeling with pathways, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.144 14168, 2021.
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevier wtaeeenrvveeeeeeeeeeeeeeee
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-dickstein, Kevin Murphy, and Charles Sutton. Language model cascades, 2022. URL httep::/aaxxiv.org/bbb///// 10342
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In International Conference on Learning Representations, 2020. URL httts:////ppeeeeeiiew..n forum?id= SonummiddSggaaa Siggdddd-gK
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint 2rXiv:2404.167100 2024.
DeepSeek-Al et al. Deepseek-v2: A strong, economical, and efficient mixtute-of-experts language model, 2024a. URL /tttps:////aaeee wwtp://.wwg/bb///s...oom 04434.
Grattafiori et al. The Llama 3 herd of models, 2024b. URL
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 1321-1330. JMLR. .rg, 2017
Neha Gupta, Jamie Smith, Ben Adlam, and Zelda E Mariet. Ensembles of classifiers: a bias- perspective. Transactions of Machine Learning Research, 2022. URL https:://openreview.nn Berrmmidd11OQFVnee
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit. Singh Rawat, Aditya Krishna Menon, and Sanjiy Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations, 2024. URL https:://pperreviee. net met/forunt Kd=ggABSC
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh. Hasan Gene, Kurt Keutzer, Amir Gholami, and Sophia Shao. Speed: Speculative pipelined execution for efficient decoding, arXiv preprint arXiv:2310. 12072, 2023.
Wittawat Jitkrittum, Neha Gupta, Aditya K Menon, Harikrishna Narasimhan, Ankit Rawat, and Sanjiv Kumar. When does confidence- based cascade deferral suffice? Advances in Neural Information Processing Systems, 36, 2023.
12

---

Published as a conference paper at ICLR 2025
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601- 1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/ v1/P17-1147. URL htt.thtt/t//caaateett/e www.hhh/ooooooo//111/o//1 wwwp/o17-11//////
Leila Khalili, Yao You, and John Bohannon. Babybear: Cheap inference triage for expensive language models, 2022. URL
Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Thirty-sevente Conference Neural Information Processing Systems, 2023.
Tachyeon Kim, Ananda Theertha Suresh, Kishore Papineni, Michael Riley, Sanjiv Kumar, and Adrian Benton. Towards fast inference: Exploring and improving blockwise parallel drafts. arXiv preprint arXiv:2404.0922 09221, 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association Computational Linguistics, 7:452-466, 2019. doi: 10.1162/tacl a 00276. URL https://aclanthon
wrg//19- 1026.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 19274-19286. PMLR, 23-29 Jul 2023. URL
mlr wrwss/ wwt.2////eeiiaaaanneaa... html.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. In Empirical Methods in Natural Language Processing, 2024b.
Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding via double early exiting. arXiv preprint arXiv:2404.18911
2024.
Jonathan Mamou, Oren Pereg, Moshe Wasserblat, and Roy Schwartz. TangoBERT: Reducing
inference cost by using cascaded architecture, 2022. URL httt:://axxv.orgeee. httpg//aab//22.02//4...
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang. Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. pp. 932-949, 2024.
Giovanni Monea, Armand Joulin, and Edouard Grave. Pass: Parallel speculative sampling. arXiv preprint arXiv:2311.1358/ 2023.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, 2018
Reiner Pope. Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving lan- guage understanding by generative pre- traitraining. https://cdn. pomnni..om/rssearcccoooeee Language-unsupervisele unsuperviseed//anguuee htw.aauuuaneeeeeettannnnnnnnnnne pdf, 2018.
13

---

Published as a conference paper at ICLR 2025
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140067, 2020. URL http://jmls org/ wwtpa.....pppp///aaaaaa.aa papprs/vss/////////////////////p 28--74.hhe
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. (0.118533/1/D16-- 1 1264. URL https::////ccllanhooloooo org/ D16- 1264.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), ), Advances in Neural Information Processing Systems, 2022. URL https::/oppenreview.. net www.nnruttttaaeeeeee http://frumiiiduucccom 1c4LLC4
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, and Ce Zhang. FlexGen:
generative inference of large language models with single GPU. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31094-31116, PMLR, 23-29 Jul 2023. URL https://proceedngg.ttt. press/ v202/sheng23a. html.
Benjamin Spector and Chris Re. Accelerating LLM inference with staged speculative decoding. arXiv preprint Stti:22308.06622 2023.
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autore- gressive models. CoRR, 0tss/1811.03111 2018. URL http://arxiv. wtwpaaawwwwoiooaaaaeeeeeeerreeee hwwgg//b//aaa 1811.03115,
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024.
Gemini Team, Rohan Anil, and et al. Gemini: A family of highly capable multimodal models, 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at practical size. arXiv preprint 2408::2408.001 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Vivien Tran-Thien. An optimal lossy variant of speculative decoding, 2023. URL https: //vivienooo.giieee....e.e github. io... Unsupervised Thoughts (Blog). URL: https://github. com/ sivienooo//eetorred dddoddne...
Neeraj Varshney and Chitta Baral. Model cascading: Towards jointly improving efficiency and accuracy of nlp systems. arXiv preprint arXiv:2210.05522, 2022.
Siqi Wang, Hailong Yang, Xuezhu Wang, Tongxuan Liu, Pengbo Wang, Xuning Liang, Kejie Ma, Tianyu Feng, Xin You, Yongjun Bao, et al. Minions: Accelerating large language model inference with adaptive and collective speculative decoding. arXiv preprint 1022 2024.
Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M Kitani, Yair Alon, and Elad Ehan. Wisdom of committees: An overlooked approach to faster and more accurate models. arXiv preprint arXiv:2012.01988. 2020.
Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang, Yonggi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: comprehensive survey of speculative decoding, 2024.
14

---

Published as a conference paper at ICLR 2025
Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In The Tweifth International Conference Learning Representations, 2024. URL https://openreview..et// wttp://wwwwne//o/uumeeme 6okaSfANzh.
Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via speeulpeee decoding. arXiv preprint arXiv:2309.08816, 2023.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, San- jiv Kumar, Jean-François Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. In The Twelfth International Conference Learning Representations, 2024. URL https:/ http:::///ppneevieeee net www.nn/ooouumiii/sooooaaaaaa
15

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A Proofs
17
A.1 Proof of Lemma
17
A.2 Proof of Lemma 2
17
A.3 Proof of Lemma
18
Proof Lemma
A.5 Proof Lemma
B Further related work
21
C Contrasting Speculative Cascades and Lossy Speculative Sampling under Different Sampling Schemes
22
C.I Speculative cascades under temperature sampling and top-P sampling
22
C.2 Contrasting with lossy speculative sampling under temperature sampling
22
C.3 Contrasting with lossy speculative sampling under top- P sampling.
23
C.4 Lossy speculative greedy decoding variant by Leviathan et al. (2023)
24
D Optimal Deferral: Additional Discussion
25
D.1 Derivation of Chow's rule
25
D.2 Optimal sequential deferral when = flog
25
D.3 Optimal speculative deferral when = flog
25
D.4 Optimal speculative deferral for greedy decoding
ge..r...........................
26
D.5 Equivalence between (7) and (8)
27
E
Token-specific Speculative Cascade
28
F
Additional Experimental Details
29
F11
Experimental setup and hyper-parameters.
29
F2
Comparisons under varying temperatures and greedy decoding
32
E3
Comparisons under top-P sampling
33
E4
Comparisons under different block sizes
33
E5
Big Little Decoder (BiLD) variants
.
variants........................
33
Lossy speculative decoding variants .e.aanns........................
................................
36
F7
Token-specific deferral rule variants
37
G Limitations
39
16

---

Published as a conference paper at ICLR 2025
A PROOFS
A.1 PROOF OF LEMMA 1
Proof. Expanding the loss in (3), we have:
(r;
=(1-r(xz() (xeeP(E)))) [f(z,, q:]l + r(t<)) (Bger(()es)) [((Te, + a) =r(xee) (Bxee(()Ee)) (()) + P( (+)) ((( [f( 9:)]
This objective is minimized by a deferral rule r: - {0. 1) that minimizes, for each prefix I<t. the term within the parenthesis. Therefore the minimizer (*<x) whenever the term within the parenthesis is negative:
E (((ae)))))) [((T,, Pt)] + a - ((--F(()eee) (M qt) < 0,
and r* (x=2) 0 otherwise. Re-arranging the terms completes the proof.
A.2 PROOF OF LEMMA 2
Proof. The proof follows traight-forwardly from the results in (Tran-Thien, 2023). Recall from $2 that the lossy speculative decoding procedure of (Tran-Thiere Thinn,,, 2023) accepts a draft token x with probability:
<xx) min { p(x)
(17)
(1 - ax) ((x))
and replaces a rejected draft token with token sampled from the residual distribution:
Pres (x) = norm (max {o, p(()) -
-1()) alo)))
(18)
for parameters E 10, and B 2 1
We need to show that running Algorithm with the target distribution:
म (x) = max { min { ((0)) p(x)
p(x)
B
results in the same acceptance probability (17) and residual distribution (18).
The acceptance probability for draft token when running Algorithm 4 on is given by:
(x) min (1, (④x)
9(xx)
The corresponding residual distribution is given by: Plee (x) norm (max (0, T(x) - q(x)))
We consider three possible cases:
Case (i) ((x) > - p(() 2 1 p(x). In this case, π(अ) 1 T-a p(x). As result:
p(x)
n² (x) min
1, a) g(x) x(x);
max 0, i-a p(z) ((xx) )
pT. norm
0, p(x) ((1)) 1)
1
norm
max
Dres Pres(x)). (x))
17

---

Published as a conference paper at ICLR 2025
Case (ii): - p(x))= 2 3/ p(x))= > g(z). In this case, TTx) p(x)) As result:
K (x) = min { 1, B ((x)
p(x)
min 1, p(z)
x(x);;
a) .g(x)
1 p(x) ((10) })
Pres (x) norm
max 0, 3
Pree Prea(x)) (z).
Case (iii): I-a p(x)= 2 g(x))= 2 p(x)). In this case, x(x) ((x). As result:
R² (x) = 1 min { p(x)
(1 (1-a) g(xz) f k<z);
max { 0, B p(x) ((m)))
Pres (1) - 0 - norm
Pres Pres( (3).
three cases, the acceptance probabilities and residual distributions are identical.
A.3 PROOF OF LEMMA
Proof. Under target distribution ne, the probability of draft token drawn from qa being is rejected given by (Leviathan et al., 2023):
- (1- min { ((()) ))
(((1)
rejection probability
(v))
rEV
Σ
min {ee(u),
Σ (v) Σ min (
BEV
max (0, xx( qc()))
VEY
Expanding Te, the rejection probability becomes:
rejection probability
Σ
max {0, (1-r(c ((0) pr(v) qc(1))
V
When r( = 1, we have:
rejection probability
min (.())--((u)) = Drv(Pr, = r((xe)) Drv(pr: 9.).
NEV
When r(x)))= 0, we have:
rejection probability = 0 r(ret) Div(pt.
as desired.
A.4 PROOF OF LEMMA 4
Proof.
Expanding the deferral risk in (8), we have:
(ipoo(r) = r(xex) (Ex,)P(()ee) [( pb)] + a Drv(pe, q:) - En [( 4:)l)
+ (mn,)((l (1 4:)]
This objective is minimized by a deferral rule r pt- + {0, 1} that minimizes, for each prefix T<t- the term within the parenthesis. Therefore the minimizer r" (x whenever the term within the parenthesis is negative:
Ex [Z p.)] Div(pr. Emj-F(( [x q:) < 0,
and r" (xxce) otherwise. Re-arranging the terms completes the proof.
18

---

Published as a conference paper at ICLR 2025
A.5 PROOF OF LEMMA 5
For a fixed prefix Tet we can write the deferral risk in (8) as:
Lspee (r: = r(x<r) (Exee)())zed) [((Te,
+ a Drv(pe, qt) - Ex ((-leee) [f 9:)l) C,
where is term independent of the deferral rule r. Let p - {0, 1) denote the optimal
deferral rule that minimizes Lupec for any prefix Zet- We then have:
Taspee (FoPr; ==e)) - Tuppoc (rre))
(forr( -r"(xxr)) (Ez,)P())e)) [6( Pt)] + a Drv q:) (e [ q:)l)
Adding and subtracting m q (v) max.e Pc(v) to the term within the second parenthesis. we get:
Lapcc (Popri xice) - Lspec (x=))
((rer(( r" (x==))
max a Drv(pt, max (1)
wee
+ (forr( r" (xce)) (BenP(/Re)) [((x),P)) - (. [e(xt;
max (:(v) + max De (0) vEV
VEV
(foer((te))r r- (xxe))
(maxx((0) a Drv(pt. max (v)
wev
VEV
+ (forT(E<<)) - r- (x<c))
(BenP(Ike) [((xe, - 11 (naxp:(v)
+ (foFr((Ees) - r" (xx=t)
(1- - a (v) - E BReNP(-lee) [((xt;
$ (forr(E22) -r (z<1))
max (()) a Drv(pt.g) max (c(v)
NEV
+ ((pp((12)) - r" (x<r))
Bx [[(x),))]] - + max (v)
+ +foor(I22)) - -r -x)(Z=))-+
- ()) BxeeP((Izee) [e(I), VEV
(Forr(E<<) 1" (2xx))
(() + DTv(pt. qt) max (v)
NEV
lerm
+ ExpP((Iee) (((x))
max ma + V
max de (v) Exp [e(I,, 9:)
V
terma
term
(19)
where we have used the fact that (T)r) - (x=)) $1.
We bound each term separately. For the first term, consider two cases: (i) max (e(v) + a.
Drv q:) max Pe (v) S 0 and (ii) max (:(w) + a DTv (pt, 4t) max.e (c(r) >
When (i) holds, fort(x = 1; so irrespective of whether (*(2)) is 0 or 1,
term; < max (v) + Drv(p,, (:) max ge (v) S 0
V
eEV
When (ii) holds, Port (x)) = 0; so irrespective of whether r" (x<c) is or 1,
term <
max maxpp()) + a Drv(pt, 96) max maxxe(v) < 0.
vEY
19

---

Published as a conference paper at ICLR 2025
Thus have:
term] S O.
(20)
We next move to the second term. Since l E0-1, we have:
terma
[e(I);
max (v
NEV
It + arg max max pr(t)) "EV
max (1(
"V
way pr(u)
Σ
P(anftre)) .1
Wi = arg =rg pa
vEV
Let E arg va (v)).. Then:
term2 = |pr(t")
- S max (p(()) - P(o]oz2)) NEV
(21)
Similarly, show that:
termy
S max (+()) P(ul)re))) NEV
(22)
Substituting (20)-222) in (19) completes the proof.
20

---

Published as a conference paper at ICLR 2025
Inference stralogy
Deferal decision (g.pp)
Taget distribution n(w)
Exceution
SpecDecoding Levinthan ct (2023)
p(a)
Speculative
Lossy SpeeDeeoding (Tran- Thinn, 2023)
max(min(p(u)) gial
slul
Speculative
BiLD* (Kim et al., 2023)
1 ((..p) p) > a)
(1-6) g(a) + +
Speculative
Cascade [Chow] (Chow, 1970)
maxegv ((e) < <1
(1 (1-6)-00 (1-6)-g(m)-a + 4-p(()
Soquential
Cascade [ChowLog]
(n)ropy(y) a.))
(1 [1-48 (1-6)-g(u]- +6-
Soquential
Oracle [oiff] (litkritum et al., 2023)
1 maXuev g(e) maxGev)())--))
(1 - (1-)))(()] +d-(())
Oracle
Oracle [oiffLog]
1 entropy(e) < catropyig)
(1 11- (1-6)-9(m) +.
Oracle
SpecCascade [Chow]
1 q(e) <1
(1 (1-4)-g(e) +.6 +-6-(()
Speculative
SpecCascade [ChowLog]
1 entropy(y) >
(1 (1-6)-g(u] (o +.6p(u)
Speculative
SpecCascade [piffes]
1 maxugv g(e) maxuev
(1-6)- +6
Speculative
SpecCuscade [oiffLogl
1 entropy(p) < entropy(g)
(1 (1-6)ggu]- +6
Speculative
SpecCascade [oPTox]
1 muxuev glo) m(x p(v) Drv(p.
(1 - giu] + +
Speculative
SpecCascade [OPTLog]
1 entropy(s) < entnopy(g) (pv(2.1))
(1 g(a] + p(p())
Speculative
Table 3: Target distributions associated with different inference algorithms, where a is free parameter and / 2 1 - a is parameter dependent on q. P and a. The last column indicates whether the execution is sequential (Algorithm 2), via an oracle (Algorithm 3), or speculative (Algorithm 5) (Leviathan et al., 2023). The third row presents variant of the BiL.D algorithm of Kim et al. (2023), where (gg. p) a measure of discrepancy between g and p; the original algorithm differs from (Leviathan et al., 2023) in the use of deterministic speculative decoding procedure with dynamic draft window (see $B).
B FURTHER RELATED WORK
Several works have studied improving the drafting process in speculative decoding; these include having the drafter and verifier share their backbone (Stern et al., 2018; Kim et al., 2024; Cai et al., 2024; Monea et al., 2023; Hooper et al., 2023; Zhang et al., 2023: Elhoushi et al., 2024; Liu et al., 2024), using multiple small draft models (Chen et al., 2023c; Wang et al., 2024), using tree-structured draft batches (Spector & Re, 2023; Miao et al., 2024), distilling the drafter with the verifier (Zhou et al., 2024), and leveraging multiple sampled drafts (Sun et al., 2024; Chen et al., 2024).
The work that is most closely related to our specific proposal is the Big Little Decoder (BiLD) (Kim et al., 2023), which can be seen as another lossy variant of speculative decoding (Leviathan et al., 2023; Tran-Thien, 2023; Zhou et al., 2024). BiLD has two phases: a fallback phase, during which the drafter & is run auto-regressively until its maximum predicted probability is sufficiently low; and a rollbuck phase, during which the verifier p is run in parallel back to the point where D(g.p) p) > for a metric D that measures discrepancy and threshold a. The fallback phase implements Chow's deferral rule in (2), and allows for the draft window size to vary dynamically based on an estimate of how likely the draft tokens will be accepted: the rollback phase can be seen as deterministic variant of the rejection sampling algorithm of Leviathan al. (2023).
the prefixes generated by g and rolls
An advantage of BiLD over the rejection sampling algorithm in (Leviathan et al., 2023) is the use of Chow's rule to vary the draft window size. However, the final target distribution seeks to mimic, TRED Aannng. p) (v) 1(D(4,)) p) S a) (V) + I(D(g, p) > a:) p(v), is an approximation to y; specifically, the target distribution æ = TRLD((E p) is chosen to satisfy D(T, S a. Hence, cases where q deviates substantially from p, BiLD would choose p as the target distribution, even when q offers better quality on a prefix (where quality can be measured using a suitable loss function). In contrast, proposed approach in $4 uses speculative decoding approximate target distributions that seek to optimally cascade between g and p. In our experiments, we compare the efficacy of using BBiLD as the target distribution with the target distributions we propose in this paper (see Table 3).
21

---

Published as a conference paper at ICLR 2025
C CONTRASTING SPECULATIVE CASCADES AND LOSSY SPECULATIVE SAMPLING UNDER DIFFERENT SAMPLING SCHEMES
We contrast how speculative cascades and lossy speculative sampling behave under temperature sampling, top-P sampling and greedy decoding.
C.I SPECULATIVE CASCADES UNDER TEMPERATURE SAMPLING AND TOP-P SAMPLING
When implementing speculative cascades with temperature sampling and top-P sampling, pute the deferral rule on the original distributions and q, but use the deferral decisions to interleave the temperature-See -ecrradurr-seleedeee (or top-P truncated) versions of p and q.
For the cascaded deferral rules in Table 1, with the exception of OPT, we construct the target distribution in (6) as follows:
#1(1) = (1 8(ge, pt)) (v): + 8(4. S(p)
(23)
where S By + Dy denotes transformation of the distribution such temperature scaling or top-P truncation, and 5: Ay Ay {0, 1} denotes the deferral rule. One may run Algorithm with H. as the target distribution, and S(g)) and S(p+) as the drafter and verifier distributions.
In the case of the OPT rule, we would formulate the constrained problem (7) to use the TV distance between the distributions S(g:) and S(p+) to measure the rejection rate. The optimal deferral rule in Lemma would now use PTv(S(p))) S(q:)) instead of (Tv qr). To construct a plug-in estimator to this optimal rule, we still prescribe using the unscaled probabilities dr and pt to estimate the expected loss, giving us, for e fo-1:
8(p 9:) =1
max < maxp: - a Div (iy())p))
For the token-specific deferral rules in $5, compute the target distribution in (16) as follows:
Trokan (v) (w) (1 - r(re,, v)) + S(pe)(v) .7,
where the deferral rule r(ree, v²) is computed on unscaled distributions @e and Pt. and v) = Evey r( r') S(q:) (v') is normalizing term. For example, for the TokenV3 deferral rule in (15), we compute the target distribution as:
क (v) S(a)(v) E + S(p.)
Σ S(g)(t))
TTT.
where To = fu E pe(u) Z max, pe(v") (unscaled) distribution Pt-
((-()))(-000) (1-a)) is the set of top-ranked tokens by the original
C.2 CONTRASTING WITH LOSSY SPECULATIVE SAMPLING UNDER TEMPERATURE SAMPLING
When implementing lossy speculative decoding under temperature sampling, following Leviathan et al. (2023); Zhou et al. (2024); Tran-Thier (2023), we compute the acceptance criterion and the residual distribution using temperature-scaled drafter and verifier distributions. Specifically, accept a draft token v with probability:
(())(v)
min 1, (1 (1-a) (w)))())
Upon rejection, we replace the token by new token sampled from a residual distribution again constructed from temperature-scaled distributions:
(o. S(p.)() -(p))))s())))) -
norm
max
where 3 2 1 - a is parameter that depends on q and pt, and is such that Evey TLoasy (v) 1.
The resulting target distribution that the method samples from takes the form:
min S(g:) (v). ((p:)(v)
((p)
3 Loeay (v) = max
{ B
22

---

Published as a conference paper at ICLR 2025
This choice of target distribution may severely limit the range of cost-quality trade-offs that can be achieved by varying a and 3. For example, observe that
(o)))
TTossy (v)
and so the the trade-off off parameters a and / are not effective on tokens for which S(p:)(v) = 0.
This problem is exacerbated when sampling with temperature (i.e., greedy decoding) where Losey (v) becomes identical to ps. making a and | irrelevant.
Lemma 6. When sampling with temperature 0 (i.e., greedy decoding), TLossy (v) So(pc).
Proof. Applying temperature sampling with temperature to distribution p is equivalent to sampling from a transformed distribution (p))), where (p) assigns probability of to arg max, p(v). When (p)))) we have that:
Lossy (v) max (min (Su()))()) 0.
Since En Tlossy (v) = 1, it turns out that: TLossy (v) whenever So(p:) = 1. Thus: Tosss (2) (Pt)..
In contrast, because both our cascaded deferral rule and the token-specific deferral rules work with unscaled distributions, they provide meaningful trade-off' under temperature sampling, including when sampling with temperature O.
C.3 CONTRASTING WITH LOSSY SPECULATIVE SAMPLING UNDER TOP-P SAMPLING
When implementing lossy speculative sampling under top-P sampling, we accept a draft token with probability:
(p(p)(v) min (1 (p())(t))
where (p(p) truncates the distribution p to only retain the top-P fraction of tokens (i.c. smallest subset of tokens whose cumulative probability exceeds P).
Notice that as P gets smaller, (p))) assigns zero probabilities to a majority of tokens. As result for most draft token candidates the above criterion evaluates to 0, and the trade-off parameter has no effect on those tokens. Hence as > O, the trade-off parameter a becomes vacuous, and thus lossy speculative decoding fails to offer meaningful trade-offs. In fact, mirroring Lemma 6, lossy speculative decoding becomes identical to standard loss-less speculative decoding when P - 0.
speculative cascade does not suffer from the same issue it the trade-off parameter as a scaling parameter the acceptance criterion, but to construct a new target distribution that is amenable a higher acceptance rate even under top-!' sampling. For example, with the TokenV3 deferral rule in (15), a draft token v is accepted with probability:
{ (p(m(())) S (p(a
min
where me is target distribution defined using the trade-off parameter a that interleaves between Pt and as follows:
Te (v) Sr(()))) - 1 Ta) + Sr(p))())
Σ Sr(g)(v), WE..
E where Ta ( (v()) max (c (1-a)) is the set of top ranked tokens by the original untruncated distribution pe(.). Since the top-ranked tokens Ta are computed using the untruncated distribution, varying the trade-off parameter a still produces meaningful cost-quality trade-offs when speculatively sampling from (v).
23

---

Published as a conference paper at ICLR 2025
Table 4: Acceptance criterion for different speculative inference strategies under non-greedy and greedy decoding. Rows 2 and 3 indicates that under 0 temperature, speculative cascade with the TokenV3 deferral rule has an identical acceptance criterion as SpecDecode [Lsssy Greedyl; see Lemma 7.
Method
Ref.
Acceptance Criterion
7 0
T=
SpecDecade [Lossy]
Leviathan et al. (2023)
min 1. (I o) MoMoy
Siacel)
1( arg max, p(v()) p(u) > - a) p(v))
SpecDecode [Lossy, Greedy]
Leviathan et al. (2023)
Spactascade
[TokkeVV]]
This paper
min
Bireell Stotey
where r (16)
p(e) 2 (1 - a)) max, p(i!)
C.4 LOsSY SPECULATIVE GREEDY DECODING VARIANT BY LEVIATHAN ET AL. (2023)
For the special case of greedy decoding, Leviathan et al. (2023) propose an alternate lossy variant of speculative decoding (Appendix A.5 in their paper), where a draft token is accepted deterministically whenever (v) 2 (1 o:) max,' Pe()";: when the token is rejected, it is replaced with a new token sampled from (+).). We will refer to this variant as SpecDecode [Lossy, Greedy].
We now show that the proposed speculative cascades with the TokenV3 deferral rule (15) is identical to SpecDecode [Lossy, Greedy] when sampling with temperature 0.
Lemma 7. For any fixed trade-off parameter E [0, 1J. SpecCascade [TokenVa] is identical to SpecDecode [Lossy, Greedy] when sampling with temperature 0.
Proof. Let (p) denote temperature-scaled one-hot version of distribution p which places all its mass the mode of p. Let pa = (pt) and So())) With the TokenV3 rule, the acceptance criterion is computed against the target distribution in (16) with trade-off parameter a.
me (v) (v) 1(v Ta) + (e())
Σ ae(v')
e'ETa
where To = (u V: pe(u)= 2 max Pe()") ((-)))(11)) a}} is the set of top ranked tokens by the original (unscaled) distribution (r(.).
Under greedy decoding, the draft token is given by v² arg max, (v)). We consider two cases: (i) v" E To and (ii) & T.
In the first case, we have Te(x") = (.()). As a result, a draft token e is accepted with probability:
min {1, (o(ū(()))) min {1, G((v") (o()())))
11
((())))
In the second case, it is clear that the draft token v" is not the maximizer of pe(.). Furthermore, (v²) = pr (v*). ). As a result, the draft token v² is rejected since the acceptance probability for the token becomes:
min { (o(()(e))) min { "o (o(a)()))) (
Pe(w)) min { ((())) }
0.
(e((>)
It is then replaced with token sampled from: norm (max {0, (-()()) So (qr (-)))) norm (max (0, a.(-))) norm (p-()) =p+()) which would produce the token maximizing p-(.)-
In both cases, the sampling procedure is identical to that of SpecDecode [Lossy, Greedy].
Table 4 summarizes the acceptance criteria for different speculative inference strategies under temperature sampling and what they reduce to under greedy decoding.
24

---

Published as a conference paper at ICLR 2025
D OPTIMAL DEFERRAL: ADDITIONAL DISCUSSION
We provide additional discussion for the deferral rules derived in $3 and $4.
D.1 DERIVATION OF CHOW'S RULE
We show below that Chow's rule is a plug-in estimator to the optimal solution to the following objective
(rj((r = ((.ee) [1 (1-r(z=)) f(x,, 9r) r(xe))) .a a].
(24)
where the deferral rule is penalized with a constant penalty or [0, 1] for choosing defer the large model.
Following the steps as Lemma 1, it is easy show:
Lemma 8. The minimizer of (24) is of the form:
r (x(x)) =1 = Bappe() [R( q:) > a.
(25)
L = f.1, one may employ a plug-in estimator to (25) by replacing the expected 0-1 loss on with 1 m (v),, giving us fahow (x)e) in (2). If f llog. one may replace the expected log loss with the entropy of qe, giving us:
fcoowLog (F<e) =1 = entropy (((opy(((1<<x1) > a.
(26)
where entropy(g) = - -rev ((v) log(a(u)).
D.2 OPTIMAL SEQUENTIAL DEFERRAL WHEN =
flog
Recall that the optimal deferral rule for sequential cascade in Lemma takes the form:
r'(x<))= =
(n [6(2e, qr)] En,op()ee) [f( p:)] a.
When l = Logg we may use the entropy Evey (v) (v)) from q. as an estimate of its expected log-loss, and similarly for pt. giving us the plug-in estimator:
fDirtLog( =
Σvev (e(v) log(Io())) < Evev log(pi(u)) - a. (27)
D.3
OPTIMAL SPECULATIVE DEFERRAL WHEN & = llog
Recall that the optimal deferral rule for speculative cascade in Lemma takes the form:
(((x)) =
Ex,-P((lees) [e( q0] Eg,-P((eed) [(( pc)] + a Div gc).
When ( == loog may construct plug-in estimator for the above rule by replacing the expected log loss with the entropy from the distribution:
fOPTLOg (x<))) =
Ever log(g((v)) < EvEr Pi(v) log(p< Div(PL.
(28)
Lemma (Regret bound for fapFLog). Suppose bgo Suppose for a fixed xzt. [log( $ Bg and [og(p:(v)) < By: Yo V. for some By: Bp 0. Then:
Lapoc (Topri x2))) min Lnpoo(13x12 S E.E (P(v) + By P(o()zz))
Proof. The proof follows similar steps to that for Lemma 5, except in bounding the resulting term and termy for the log loss. this case,
term2 E, P((eea) [log(p((x))))
Σp((v) P((v) lo)(p(()))
NEV
25

---

Published as a conference paper at ICLR 2025
Σ
P(vlac)) log(p(()))
Ep(()) log(pi(v))
NEV
NEV
$ Σ .EV
P(vlx=)) - Σ p((w)
log(pe(v))
NEV
S Bp Σ
P(ul - Σ Pe(u)
"EV
NEV
Similarly,
P(ul0<1) - Eq( log(q:(v))
terma
VEV
DEV
S Bg
Σ
P(ulxc))
Σ (e(v)
VEV
Plugging these bounds into the equivalent of (19) in 5 for the log-loss completes the proof.
D.4 OPTIMAL SPECULATIVE DEFERRAL FOR GREEDY DECODING
When applying speculative cascades with greedy decoding, we shall see that both the optimal deferral rule OPT (10) is equivalent to the Diff deferral rule (5).
As detailed in $C.1, when implementing a speculative cascade with temperature-seate scaled distributions Le (v) xx qe (v) and (v) x (v) respectively, for temperature parameter > 0, the Diff and OPT deferral rules are computed as:
(x) =1
max, (v) maxy Pc(v) a,
and
Fa =
max, (e(v) < max, ( Drv Ør).
One may run Algorithm 5 with either Toser or rOPr as the deferral rule, and the temperature-scaled ğ as the drafter distribution and ir as the verifier distribution.
Lemma 10. When T - O, running Algorithm 5 with Fort the deferral rule and q and ii as the drafter and verifier distributions, is equivalent to running it with iostt as the deferral rule and d and the drafter and verifier distributions.
Proof. When T + O, note that & and Pe are one-hot encodings of arg max (((v) and arg max (r(v) respectively. As a result,
Drv(ār, = arg max (()) + arg max po(1)) .EV
DEY
When running Algorithm with PDPt as the deferral rule, we will accept a draft token with probability:
(OPT(V) min { (1 (.))( (c()) + (oPT(), (r(() G((x)
where
8oPT(q.P)
max g(v) < max p(u) - - WEV
(gargmax max q(e) # arg max p(())
arg
wEV
vEY
vEV
When running Algorithm 5 with Ppier as the deferral rule, we will accept a draft token with probability:
KDiE (v) = min {1 (1 épia (qt, pc)) q((v) + Spaa Pt) p:(u)
(x)
26

---

Published as a conference paper at ICLR 2025
where
8 (g.p)
m(max(()))
maxp(c) maxp(())aa) -aa -a)
We consider two cases:
(i) If arg ma (c(v) arg ma Pc()), then q: Pt. and irrespective of the outcome of sorr (t) or dpie (4t. Pt), we have that (orr(e) = KDaf (v). Furthermore, the token gets accepted in both cases.
(ii) If arg max (v))) o arg vax Pe(e), then
(OPT(U) = 14 Pi)
max
(u)
2
max a Spser(q.P)
KDiff (u).
VEV
NEV
In this case, when the draft token gets rejected, with deferral rule torr, we will sample new token from the residual distribution:
porr(1) m0n 6or pc)) (v) (0rr(a: pc) fe(v) (v))
= 6oPT(gr.PV (P) min{0, (x(v) [(())
When a token gets rejected with deferral rule SDIIS we will sample a new token from the residual distribution:
(V) x (DII Pt) min(0., (v))).
Since arg max,ev (v) * arg m (v()), Por & (opT(gt-) Pt) min (0. pc(v) {e)}} dDan (tt. min p.(u) qv( ( & (D)n Thus both the acceptance probability and the residual distributions for foer the the one we would have used had we run Algorithm with ADits as the deferral rule.
D.5 EQUIVALENCE BETWEEN (7) AND (8)
Since the prefix Zet is fixed (7), the constrained optimization we seek solve is of essentially of the following form:
min (1 11-77 r) co rr s.t. r. B, rE(0.))
for some coefficients cp. C1, 02 0. Since is a binary variable, we may formulate an equivalent unconstrained problem with the same minimizer:
min (1 -1-r) to r...r C2 rE(0,1)
where choose 0 when c2 S B and choose optimization problem is of the form in (8).
- - -c) otherwise. This unconstrained
27

---

Published as a conference paper at ICLR 2025
E TOKEN-SPECIHIC SPECULATIVE CASCADE
We provide a modification of Algorithm to accommodate the token-specific deferral rules in $5.
Algorithm TokenSpecCascade
Input: Models 4, p. Token-specific deferral rule r. Prefix dit,, Block size 7 (4. ()) = ( (1 - r(ace, v)) + p(v) Ewev r(rei, v') g(0')
Output:
GenSpecSample((f p. Troken (2))7)
Optimal token-specific deferral. Similar to $4.3, we may consider deriving the optimal token- specific deferral rule. We start by formulating similar optimization objective $4.3. For fixed prefix Fec. this would look like:
min Eu-F(Hlre)) [(V, Troken))
(29)
s.t. Div(Troken q:) B,
where Trokon (v) = (1 (1-r((tts)) q:(1) + 7 (v()) the target distribution resulting from the choice of r, 11 r(E v') (r") is normalization term, and B > 0 is a budget parameter.
However, unlike $4.3, the above constrained optimization problem does not directly lend itself to simple closed-form solution. In some highly simplistic special cases, may be able to derive trivial solution. For example, suppose = fo.1, and the mode of dr coincides with that of P( (rre)), ie., arg max q: (v) arg ma P(u]=<);; then the optimal token-specife rule is given by r(zee, v) = 0, for all V.
Under more realistic cases, may not be able to derive solution simple the OPT rule in (10). Therefore, in our experiments, we employ the three heuristic rules in equations 13-15, which are motivated by the form of the simpler Diff fff rule in (5).
28

---

Published as a conference paper at ICLR 2025
WMT: Small - Large (T=0.5)
WMT: Small - Large (T=1)
SpecDecade [Lossy]
22
SpecCascade [OPT]
SpecCascade [Token]
20
SpecCascade [Diff]
Spectascade [Chow]
18
TokenCascade (Chow)
BILD
16
14
0.50
0.55
0.60
0.65
0.70
0.4
0.5
0.6
0.7
0.8
Relative latency
Relative latency
CNNDM:
Large (T=0.5)
CNNDM: Small Large (T=1)
15.5
SpecDecodee [Lossy]
15.0
SpecCascade [OPT]
SpecCascade [Teken]]
14.5
Fougee
31
SpecCascade [Diff]
14.0
SpecCascade [Chow]
TelenCascade [Chow]
13.5
20
BILD
13.0
12.5
8
0.45
0.60
0.70
0.5
0.7
Relative latency
Relative latency
XSum: Small Large (T=0.5)
XSum: Small Large (T-1)
SpecDecode [Lossy]
SpecCascade [OPT]
14
SpecCascade [Taken]
ROUGE-2
ROUGE-2
SpecCascade [Diff]
16
12
SpecCascade [Chow]
TokenCascade [Chow)
14
1.0
Relative latency
Relative latency
Figure 4: Plots of quality vs. latency for T5 models with temperatures T = 0.5 and T = 1, and block size y Each method interleaves T5-small with T5-large. We include speculative cascades with the Chow, Diff, OPT and TokenV3 (referred to as Token) deferral rules, and compare it with three baselines: SpecDecode [Lossy]. TokenCascade [Chow] and BiLD*. The z-axis tracks the latency relative to that of calling the large model on all inputs. The horizontal dotted line denotes the quality of the large model.
F ADDITIONAL EXPERIMENTAL DETAILS
We provide additional details about our experimental setup and additional experimental results.
F.1 EXPERIMENTAL SETUP AND HYPER-PARAMETER
We first elaborate on our experimental setup and the hyper-parametern used.
T5 datasets. For the WMT English to German translation task (Bojar et al., 2014), we use validation sample of size 3,000 provided with the dataset. We set the maximum input length to 80 and the maximum output length to 80. For the Extreme Summarization (XSum) task (Narayan et al., 2018), use validation sample of size 11,305, and set the maximum input length to 1,024 and the maximum output length to 64. For the CNN/Daily Mail summarization task (Hermann et al., validation sample of size 13368, and set the maximum input length to 2,048 and the maximum output length to 128. Following (Zhou et al., 2024), we use ROUGE-2 as the evaluation metric for the summarization tasks.
2015),
We note that Kim et al. (2023) report ROUGE-L metrics for CNN/DM, which generally tend to evaluate to higher values than ROUGE-2. Furthermore, most of their experimental results with
29

---

Published as a conference paper at ICLR 2025
WNT. Small - Large (T=0)
XSum: Small - Largs -T
27.50
27.23
14
27.00
20
75
ROUGE
28 50
26.25
36
SpecDscods (Laay, Cready)) SpecCasceds (Taker))
SpecCarcads (oin)
26.00
13
SpocCanaads [Choo]
25.75
14
Tekrrascase (Chow)
25 50
0.425
0.455
3,500
0.525
5,550
0.575
0.600
0.00
0.70
0.75
0.90
latency
CANDM: Small
15.0
ROUSE-2
14.0
SpecDecade [Lossy, Groedyl SpecCassace [bean) SpscCescade [ain]
SpucCrscude (Chaw)
CannCa (Chow)
BLD
0.43
0.45
0.50
0.55
0.00
0.45
0.73
lalency
Figure 5: Plots of quality vs. latency for T5 models with greedy decoding (T= 0) with block size = 5. Each method interleaves T5-small with TS-large. The z-axis tracks the latency relative to that of calling the large model all inputs. The horizontal dotted line denotes the quality of the large model. SpecDecode [Lossy, Greedy] is the greedy version of lossy speculative decoding proposed in Leviathan al. (2023). SpecCascade [Token] uses the TokenV3 deferral rule in (15), As noted in $C.4, when T + 0, SpecDecode [TokenV3] is identical to SpecDecode [Lossy, Greedy].
greedy decoding (T 0), and hence, the ROUGE-L evaluation metrics they report in their paper tend to be higher for the same T5 models when compared to our numbers for ROUGE-2 with temperature sampling.
Gemma datasets. In addition to the WMT EN EN translation and the CNN/DM summarization datasets, we the GSM8K (Cobbe al., 2021) math reasoning dataset, the MBPP (Austin et al., 2021) Python programming dataset, and four question-answering datasets: Natural Questions (Kwiatkowski et al., 2019), TriviaQA (Joshi al., 2017). WebQuestions (Berant al., 2013) and the Stanford Question-Answering Dataset (SQuAD) 2.0 (Rajpurkar et al., 2016). In each case, we sample 1,000 prompts for evaluation. We employ few-shot inference, and set the maximum output length to 80 for WMT, to 128 for CNN/DM, to 320 for GSM8K and MBPP, and to for all the question-answering datasets.
Models. We construct cascades from T5 v1.1 family of encoder-decode models (Raffel et al., 2020), of different sizes T5-small (77M), T5-base (250M), T5-large (800M) and T5-XL (3B). 2 We follow the protocol in (Zhou et al., 2024): we initialize with the public checkpoints, pre-train them further for 100K steps, and supervise finetune the pre-trained models on three respective tasks. We finetune them for maximum of 250K steps on WMT, maximum of 100K steps XSum and maximum of 200K steps on CNNDM.
We construct the Gemma cascades from instruction-tunee decoder-only v2 models. For MBPP alone we additionally experiment with pre-trained models. We use 2B drafter, and either a 9B verifier or 27B verifier (Team et al., 2024).
Run-time evaluation. For each dataset, we evaluate the quality metrics on the entire validation set. For the run-time analysis in the T5 experiments, we adopt the protocol followed in Leviathan al. (2023); Zhou et al. (2024). We randomly sample 500 examples from the validation set, and calculate the wall-clock time taken for decoding with batch size of 1. We repeat this for three trials and report the average running time. All methods are run on the same TPUv4 device.
Thhe pre-trained checkpoints use available here.
30

---

Published as a conference paper at ICLR 2025
WMT Small IT=1)
WHT Small (T=0.5)
WNI Small Large (T-03)
SpeeDeecde [oweyl
SpecCascade (OPT)
26.5
27.5
(Tsken)
25.5
SpacDecode (Losay) SpecCascada
SoerDecadde [unvyy] SpecCassade
24.5
28.0
24.0
SpenCasrade
0:45 0.50 0.55 0.60 865 0.70 0.75 0.80
0.50
0.51
0.60
0.55
5.70
0.50
0:55
2.60
0.85
0.70
Reative Istency
Reativee iatency
Relative latency
XSum: Small Large. (T=1.0)
XSum: Small - Large IT-055
XSum: Small - Large (T=0.11
39
14
10
ROUCE-2
ROUGE
27
10
SpecDerodd [Loay]
SpecCaccade [OPT]
14
SpecCascade [OPT]
SpecDacode [Lowy]
15
SpeeCarcade [OPT)
SpenDecode [Lossy]
SpecCascade [Token]
SperCasiade [Tokoo]
SpocCascade [Taxen]
0.4
0.5 0.50 0.65 0.70 0.75 0.80 0.30
0.55
0.00
0.70 6.07 0.90
Relative Istency
istency
Relative Istency
CNNOM: Small - Large (T=1)
CNNDDW:: Small Large (T-0.5)
CNNDR: Small Large (T-01)
SpecDecode [Lasy]
12
SpecCaszade [OPT]
Speciascade [Tokur]
nOUGE-2
11
ROUGE?
24.5
ROUGE-2
15.0
24.0
14..5
13.5
Specitecode (Losag]
14.0
SpecDecade [Lnay] Spectascade [OPT]
13.0
SpenCrocade [OPT!
SpecCascade [Token]
Spectascode ['bien]
3.40
Relative lafency
Relative latendy
Relative latency
Figure 6: Plots of quality vs. latency for T5 models with varying temperatures. Each method interleaves T5-small with Ta-large. The r-axis tracks the latency relative to that of calling the large model on all inputs. The horizontal dotted line denotes the quality of the large model. As the temperature decreases, SpecDecode [Lossy] produces fewer unique trade-off points as its acceptance criterion is less susceptible to changes in the trade-off parameter a. In contrast, SpecCascade [Token], which uses the TokenV3 deferral rule, offers wider range of trade-off points; here the trade-off parameter is not used to construct a traded-off target distribution, and does not feature as a scaling term in the acceptance criterion.
T5 hyper-parameters For the T5 experiments, unless otherwise specified. we set the block-size y to 5 for all methods that use speculative execution. For the token-level cascades, we allow the small model to predict for maximum of 10 tokens (similar to (Kim et al., 2023)), before invoking the large model. This needed, as otherwise, the small model would predict long sequence, and when it eventually defers to the large model, the large model is bottle-necked by the pre-filling of the long prefix accumulated by the small model. We vary the trade-off parameter to vary the latency and plot quality as function of latency.
Gemma inference. When implementing speculative cascades and speculative decoding with Gemma models, we use block-size 7 = 1. In this case, for each prefix det, we have the drafter generate one draft token F, for the next step. We then invoke the verifier with the same prefix, and either accept the draft token or reject and replace it with the verifier's 's prediction. We repeat this process to generate the entire response.
BiLD baseline. For the BiLD method, we adopt the same discrepancy metric D as (Kim et al., 2023) for greedy decoding:
D(g.p) = -log (Irg
max (amaxxx()))) ) NEV
31

---

Published as a conference paper at ICLR 2025
WMT: Small (P=0.0)
WHT Small IP=0.5)
WMT Small (P-0.1)
25
27.0
24
26.5
27.0
23
26.5
22
25.5
SpecDacodk [Loas]
SpecDecode (Losoy)
Spechacode
21
SpecCas
25.0
SpenCascade
0.45
0.50
0.55
0.60
0.85
0.70
0.75
0.50
0.55
0.60
0.55
5.70
0.50
0.55
0.60
2.65
0.70
Reative intency
Relatvve latency
Relative latency
CNNDM: Small + Large [P=0.8]
CNNDM: Small - Large (P-0.5)
CHNDR: Small - Large IP-0.1)
15.5
ROUGE
ROUGE-2 2
15.0
13.5
SpecDecade [Lossy]
SpecDecode [Loey]]
SpecDeende [Lssy] Spoctascado [Tbion]
SpecCascade [Toxen)
13.0
SpocCascade (Tblen)
13.5
0.40 9:15
8.60 8.85 0.70 0.75
0.40
0:45 0.50 0.55
0.70 0.75
0.10
D.5
0.70 0.75 Relative latency
0.55 0.80
Relative latency
Relative Istency
Figure 7: Plots of quality vs, latency for T5 models under Top-P sampling with varying values of P. Each method interleaves T5-small with T5-large. The z-axis tracks the latency relative to that of calling the large model on all inputs. The horizontal dotted line denotes the quality of the large model. As P becomes smaller, SpecDecode [Lossy] able to produce fewer unique trade-off points as the scaling by the trade-off parameter a is less effective in its acceptance criterion ($C.3). SpecCascade [Token], which uses the TokenV3 deferral rule in (15), does not suffer from the same issue; here the trade-off parameter does not feature as a scaling term in the acceptance criterion.
and pick the value of the threshold a on this metric from the range [0, 10). For temperature sampling with non-zero temperature, we use the following natural analogue to the above D:
D(g.P) p) Eu-g (ogg--
Eg(v) log(p(v)).
In $F.5, we present comparisons between different implementations of this method.
Lossy speculative decoding. See $F.6 for details.
F.2 COMPARISONS UNDER VARYING TEMPERATURES AND GREEDY DECODING
In Figures 5 and 6, we provide additional plots of quality vs. latency for different inference strategies under greedy decoding (T' 0) and temperature sampling (T = 0.1, 0.5, 1.0) respectively.
As expected, see from the greedy decoding results in Figure 5 that all methods yield better quality metrics compared to their performance under temperature sampling. In this case, the proposed the lossy variant of speculative decoding proposed by
SpecCascade [TokenV3] is cquivalent Leviathan et al. (2023) for greedy decoding, which we refer to as SpecDecode [Lossy, Greedy]. See $C.4 for the discussion of this equivalence. Similarly, as noted $D.4, with greedy decoding, the OPT deferral rule coincides with the Diff deferral rule.
In the temperature sampling results in Figure 6, we compare the lossy speculative sampling proposed by Leviathan et al. (2023) for temperature sampling (SpecDecode [Lossy]) with two of our proposed methods: speculative cascades with the OPT and TokenV3 (referred to as Token) deferral rules. We can that SpecCascade [Lsssy] generates fewer unique trade-off points temperature T gets smaller. As noted in SC.11 this is because its acceptance criterion is less susceptible to changes in trade-off parameter a when S(pt) is peaked. In contrast, our proposed SpecCascade [Token] approach yields a wider range of orf points; here the trade-off off parameter is not used to construct a traded-off target distribution, and does not feature as a scaling term in the acceptance criterion.
The reason the Token-specific rule fares better than OPT is because the latter computes its deferral decisions based which of (+) and pc(-) is more peaked; this can be disadvantage when the
32

---

Published as a conference paper at ICLR 2025
Table Reduction in latency (T' - 5) when matching the quality of the large model, and the best quality metric without exceeding the latency of the large model. We use T5-small and T5-re as the small and large models respectively. Quality is measured in terms of the BLEU for WMT and ROUGE-2 for CNNDM. We apply Top-P sampling with varying values of P. The proposed SpecCascade [Token] method uses the TokenV3 deferral rule in (15).
Latency; when marching large model's quality
Bed quility withour exceeding large model liteay
p- 0.5
P-088
0.5
P- 0.8
CNNDM
CNNDM
CNNDM
SpecDucode [Losa]]
1.55%
1.48X
27.35
15.72
1361
Spectascade [Token]
1.74%
1.61%
1.9x
27.49
15.79
27.30
15.46
25.64
sampled token is not be close the distribution mode, which is likely to happen with higher temperatures. With lower temperatures, however, the sampled token is likely to be close the distribution mode, and as a result, the advantage that the Token-specific rule has over OPT diminishes.
F.3 COMPARISONS UNDER TOP- SAMPLING
We compare speculative cascades with the TokenV3 deferral rule (SpecCascade [Token]) with SpecDecode [Lossy] under top- sampling. As discussed in $C.3, when implementing SpecCascade [Token], the TokenV3 deferral rule in (15) is applied to the original drafter and verifier distributions; we use the deferral decision to then interleave top-P truncated versions of these distributions. In Table 5, report the results of our evaluation for varying P. while fixing the temperature T and 7=
is for the smallest value of / that our proposal offers the largest gains in speed-up over SpecDecode [Lossy]. Unsurprisingly, the smaller the value of P. the better are the quality metrics, with both methods being almost quality neutral for P 0.1.
As we elaborate in SC.3, as P becomes smaller, the trade-off parameter a in lossy speculative sampling becomes less effective, with the method becoming identical standard loss-less speculative decoding when P - 0. In contrast, our proposed SpecCascade [Token] approach does not suffer from the same issue as it uses the trade-off parameter not as scaling parameter the acceptance criterion, but to construct new target distribution that is amenable to higher acceptance rate even under top-/' sampling. Therefore we are able to tune a to get a wider range of operating points and match the quality of the larger model at a lower latency.
For example, in Figure 7, lossy speculative decoding with P 0.1 is able to offer only three unique trade-off points (despite sweeping through fine-grained grid from 10-6 to 1); in contrast SpecCascade [Token] is able to offer a wider range of trade-off points.
F.4 COMPARISONS UNDER DIFFERENT BLOCK SIZES 7
In Figure 8, present latency-quality trade-off plots for T5 cascades under different block sizes y In each case, we find that the proposed speculative cascading techniques outperform lossy speculative decoding across different latency values. Furthermore, higher values of y are seen yield wider range of trade-offs, with lower quality operating points shifting to the left, and better quality operating points shifting to the right. For example, with XSum, SpecDecode [Lossy] with ? matches the small model's quality at 0.64 relative latency, and matches the large model's quality at 0.85 relative latency; with 7 7, it matches the small model's quality at an even lower latency, but practically provides no speed-up when matching the larger model's quality. The reason larger block size can hurt speed-up at the higher quality regime is because can result in frequent rollbacks, thus defeating the purpose of using speculative execution.
F.5 BIG LITTLE DECODER (BILD) VARIANTS
In our experiments in the main text ($6). we compared against a version of the Big Little Decoder method (Kim et al., 2023) that applied Algorithm 4 to the target distribution TELD the authors seek to mimic ($B). We now show that this version performs similarly to the original BiLD algorithm in (Kim et al., 2023).
33

---

Published as a conference paper at ICLR 2025
Small 1y
WMT: Small (y=5)
WMT: Small (y-
22
[awsy]
(Lowa]]
[Lova]]
SpecCastade
SpecCasiade [OPT)
SpecCascade (OFTI
20
Spectascadd [Tbkkn]
SalcCasdadd [Taian]
SpacCascade [Toke]]
20
BLEU
1.6
16
14
0.55
0.00
0.85
0.20
0.75
0.80
0.45
0.50 0.55 3.40 0.88 0.70 0.75
0.00
0.4
0.5
0.6
0.7
0.8
0.9
Relative Istency
Relative latency
Relative latency
XSum: Small - Large (y 3)
XSum: Small - Large (ym5)
XSum: Small Large
[ym 7)
26
16
SpocD [Lasay]
SpecCiasaate [OP1)
14
14
SpecCascode [Token]
ROUCE-
ROUGE
20000
12
10
SpecCarcade [OPT]
SpecDaccde [Lowy]
10
SpecCarcade (OPT)
SpacDecode [Lowsy]
SpecCazcade [Taxon)
SpecCascade [Tkken]
0.70
0.75
0.80
5.85
0.7
0.8
0.4
0.5
0.7
0.8
3.0
Relative Iatency
Relative latency
lataavee latency
CNNDM: Small Large [y=3) 3)
CNNDM Small - Large (y 5)
CNNOM: Small - Large (y- 7]
12
12
nouCi-33
11
ROUGE-
ROUGE-2
20
Spac Decode (Losay]
Spec [Lowy]]
SpacDleodde [Loss)] SpeeCareade [OPTI
SpecCas [0PT)
SaecCae [OPT]
SpocCascade: [oken]
SparCarcade [Tkan)]
Sp SpecCascase [Tken)
Relative
Reliative latency
Relative latency
Figure 8: Plots of quality vs. latency for T5 models with with varying block sizes y. Each method interleaves T5-small with T5-large. The æ-axis tracks the latency relative to that of calling the large model on all inputs. The horizontal dotted line denotes the quality of the large model.
A key difference to the original algorithm in (Kim et al., 2023) is the use of the fallback phase, where the drafter is run until its maximum predicted probability max,ey ((v) < 1-ay, for a threshold af E [0, 1] (or until maximum block size of 10 is reached), and the use of a deterministic rollback policy where the verifier rejects a draft token whenever D(q.p) p) > a. our implementation, we adopt the speculative sampling algorithm from (Leviathan et al., 2023): we do not have the fallback policy and replace the determinisic rollback policy with the rejection sampling in Algorithm 4.
Figure 9 (top) compares the original version of BiLD with the version use in $6. We interleave
between a T5-small and T5-large model on WMT, using greedy decoding (T = 0) for inference. As prescribed by the authors (Kim et al., 2023), we use the following discrepancy metric for greedy decoding:
D(g.P) p) = log arg max q(v) NEV
We compare our implementation (BiLD"), where we set the block size to 5 (same as our proposed speculative cascading approaches), with the original BiLD for different choices of maximum block size y and different fallback thresholds axy. For both methods, we vary the threshold on D(g.p) to vary the latency and plot the resulting BLEU score.
A higher fallback threshold a / results in larger draft generation windows; this gives an advantage in the low latency regime, where most of the draft tokens are accepted. As a result, BiLD [> = 10, a 0.9] yields the lowest latencies, but also yields lower quality. A low fallback threshold results in very small draft generation windows, and consequently, in higher latencies. This is why BiLD ln 5, 0.1] is the slowest but yields high quality metrics.
Our implementation BiLD is seen to perform comparable to the best parameter choices for the original BiLD algorithm in Figure 9.
34

---

Published as a conference paper at ICLR 2025
WMT: Small Large (T=0)
BILD (y= 5, 0.1)
BILD (y= 5, 0.3
y= 5.
26.5
BILD (y 5, 0.73
BILD (y- 5, 0.99
= a0
26.0
BILD 10,
BILD (y= 10, 0.5)
25.5
( 10, dy
0.40
0.45
0.50
0.55
0.60
0.65
Relative latency
Small Large (T=1)
XSum: Small Large
SpecDncode [Lossy]
17.0
13
Spec Cascade [Lossy-]
ROUGE
16.0
11
15.5
15.0
SpecDecode [Lossy] SpecCascade [Lossy-]
14.5
0.15
Fraction calls to larger model
Fraction of calls larger model
Figure 9: Top: Plots of quality vs. latency comparing BiLD' with the original BiLD algorithm Kim et al. (2023) with varying maximum draft window size y and fallback confidence threshold a.. Bottom: Comparison of lossy speculative decoding with [Lossy] and 3 tuned using the procedure in (Tran-Thien, 2023) [Lossy*].
Small
SpecDecode
16
SpecCascade [OPT]
21
Spectascate (ewe133)
20
14
SpenCaerate [1
SpenCascate Bbwer92)
ROUGE?
13
12
17
SpeeBecdde (L
(OFT)
16
SpeeCinace (Tabani)
15
SpecCascace (Takani))
3.45
0.35
3.55
0.75
0.85
0.6
Reletivv lntency
Relative Intency
CNNON:: Largu (1-11
BOLGE-2
SpecBacede exey))
SpecDweadee
SpecDaecade [bben))))
SpecCancade [baan))3)
SpecCascade [baerw13
9.45
0.65 0.20
Relative Intency
Figure 10: Plots of quality latency for T5 models with all three token-specific speculative cascade deferral rules in equations 13-15, Each method interleaves a T5 small and T5 large model. The r-axis tracks the latency relative to that of calling the large model all inputs. The horizontal dotted line denotes the quality of the large model.
It is worth noting that while we view TBLDD the target distribution that the algorithm in (Kim et al., 2023) seeks to mimic, the presence of the fallback phase could mean that on some inputs output response is generated without the verification (or rollback) phase being invoked. In such cases, the output response will come solely from drafter it turns out that contains tokens for which D(g,,p) Pt) a.
35

---

Published as a conference paper at ICLR 2025
WHT Schot: 28 2288 (T
CHNOW Safat 28 278 (T-
275
25.0
22.0
135
SpecDerode II [oos
Spec [Iassy]
SpocCascade [OPT]
SowcCancade VioenvVl
20
SpecCascade (OPT]
Spec Cascade [[ocanV))]
125
SpecCascade TixenV/2
Cascaat Toxwnv21
130
SperCascade [Tx
SpecCascade [oov
0.00
0.30
2:33
2.23
0.50
0.05
0.30
0:13
0.20
0.23
rate
Peject
GSVAK Buthot: 27A
SQUAD 14hat: 23
0.75
0.70
0.60
0.50
0.55
SoecDecede [iossy)
Gecode [Loss]]
0.5
SpecCascade [0PT]
0.40
SpocCascade [OPT]
0.45
Misee/2
SpecCascade [TkonV
SpecCascade [Txoo)))
0.46
0.00
0.02
0:04
6:28
0,08
0.10
0.22
0.50
0.05
0.10
0.25
2
0:23
0.2
0.32
0.30
0.25
2.2
[Lavs]]
SpecCascade [0PT)
SpocCascade (OPT)
SpecCascade [okerva]
Spectascade [oosava)]
SpecCascade [oserV2)
0.55
SpecCascade [oxan22)]
Spee Dancade [io
Spec Descate [boweeV3
0.00
5.23
0.25
20
0.25
0.50
0.05
0.30
0.25
0.20
0.50
0:40
Re action rake
Peject rate
H4PP Iuehan: 20 270 [M] F
Thvas L-stet: 28 270 IT
0.65
0.60
0.55
0.55
SpozDecode (Lasay]
0.45
SpocDeaode [Loaay]
SpecEancade [[P1]]
Spectiscade [BMT]
0.40
SpecCascade VhkeeV3]
SpecCascade [bcenV33
SpesCascade [akanV2]
0.60
SpecCascade [oxanV2)
Spectan [[seeV1]
Spectanzade [boweVV3]
0.39
0.00
0.04
0.36
0.36
0.22
0.000
0.225
5,556
3,075
6.100
6:225 0.153
0175 0200
Ricstion 1282
Reedd on Fule
Figure 11: Plots of quality vs. rejection rate for Gemma models with all three token-specific speculative cascade deferral rules in equations 13-15. Each method interleaves a Gemma 2B drafter with a Gemma 27B verifier. The horizontal dotted line denotes the quality of the large model. We include all three token-specific speculative cascade deferral rules in equations 13-15.
F.6 LOSSY SPECULATIVE DECODING VARIANTS
In our experiments in the main text ($6), we compared against the lossy speculative decoding (Tran- Thien, 2023; Zhou et al., 2024) described in $2, with the parameter 3 set to 1. We now present results for this method with 3 tuned according to the procedure in Tran-Thien (2023), and show that choosing 8 1 fares at least as well tuning B.
The goal in Tran-Thien (2023) to choose a and /3 so as to maximize the acceptance rate for the draft token, while ensuring that the KL divergence between the resulting target distribution and p is within an allowable limit R. The authors prescribe specifying R, and for each prefix, tuning a and 8 to solve the resulting constrained optimization problem. To be consistent with the rest of our experimental setup, we vary a to vary the draft acceptance rate (note that each choice of a corresponds particular KI. divergence to p). and tune B > 1 a to satisfy the following condition
36

---

Published as a conference paper at ICLR 2025
shht- 28 (T-
GSM6K Banoo: 28 +39
0.75
25.0
0.55
25.0
0.60
0.55
Specterode II osossy)
SpecDerode [Iassy]
15.
SpocCascade [OPT]
0.50
SpecCascade (OPT]
SowcCascade VooenVVl
Spectanc [[ocanV:)]
SpecCascade VixenV22
0.45
SpecCascade Toxeni22
130
SperCascade
0.40
SpecCascade
0.00
0.25
0.25
0.50
0.22
0.04
0.28
0.10
Rey
Paject
SQuAD 14bot: 40
Weehquectane 1-shet:
0.60
0.29
0.5
0.28
0.5
0.52
Spe:Decede [oows)]
0.26
Gecode [Lossy]
0.50
SperCascade [0PT]
0.25
SpocCascade [OPT]
0.24
SpecCascade TikonV
SpecCascade [bxooV1)
0.46
0.3
0.00
0.05
0.20
2:25
0.20
0.25
0.30
0.35
0.00
0.55
0.30
2.13
C.22
0.25
0.500
[Loway]
6.425
[Lavsy]
SpecCascade [OPT]
SpecCascade (OPT)
0.3
SpecCascade [okerva)
Spectascade [locarv3)
SpecCascade [oserV2)
SpecCascade [Toxanv2]
0.2
Cancade [howeV1)
Spec Dencade [TooeeV3)]
012
0.00
0.05
0.33
0.25
0.20
0.00
0.50
0.10 0.15
Re rake
Reject rate
Figure 12: Plots of quality vs. rejection rate with Gemma 2B - 9B speculative cascades. Each method interleaves Gemma 2B drafter with Gemma 9B verifier. The horizontal dotted line denotes the quality of the large model. We include all three token-specific speculative cascade deferral rules in equations 13-15.
outlined in Tran-Thien (2023):
Σ max 0, p(u)
p(v) 3
max 0, Σ VEV
{(v))
NEV
We pick 3 using a grid-search over 1000 values between a and 10.
Since this tuning procedure, in turn, can add to the method's latency, for fair comparison, we plot quality as a function of the fraction of calls to the large model (rejection rate), instead of relative latency. In Figure 9 (bottom), plot these trade-off curves for loss speculative decoding with 3 = 1 (Lossy) and for speculative decoding with 3 tuned using the above procedure (Lossy"). We compare performances WMT and XSum, and in each case, interleave T5-small model with T5-large model.
In both cases, setting 3 1 provides trade-offs comparable to better than using tuned value of B. The reason using a tuned value of 3 fares worse than setting 8 = 1 might be because we are measuring quality in terms of BLEU ROUGE-2, which is different from the KI. divergence p objective that the tuning procedure in Tran-Thien (2023) seeks to optimize.
F.7 TOKEN-SPECIFIC DEFERRAL RULE VARIANTS
In Figure 10, we present iatency-quality trade-off plots for cascades constructed from a T5-small and T5-large model. We include in these comparisons, all three token-specific deferral rules in
37

---

Published as a conference paper at ICLR 2025
(15). In Figure 11, present trade-off plots for cascades constructed from Gemma 2B and Gemma 27B models with all three token-speest roken-specific rules, and in Figure 12, we include similar plots for cascades constructed from Gemma 2B and Gemma 9B models. We note that the trends with the 2B - +9 are similar to those seen with the 2B + 27B cascades.
With the T5 models, the results mixed, with the V1 and V2 variants sometime surpassing the V3 variant (which is the variant we included in the main experiments results $6). Interestingly, with the Gemma models, the V3 variant is seen outperform the others for most rejection rates, with the exception of the 2B BB+ cascade on SQuAD 2.0, where the V2 variant is better.
The reason for the V3 variant outperforming the V1 and V2 variants the Gemma models could be due to the fact that it uses the larger model's is distribution Pr(-) to measure confidence for both the drafter and verifier tokens (see LHS and RHS in (13)). We expect this be particularly helpful when there is a larger gap in sizes between q and p. and the larger model's distribution is better aligned with the data-gent generaann distribution compared the smaller model. Furthermore, as per the discussion in $5, the multiplicative form of the rule (15) results in target distribution with an intuitive form: it seeks to mimic ge (-) on the top-a ranked tokens by p-(-) and uses a re-scaled version of pr(-) for the other tokens:
TToeenv (v) = qc(()= 1( E Ta) + pc(o) (((),,
O'ETa
where To = (u E V pe(v) 2 max Pe(')) (1)-)).
38

---

Published as a conference paper at ICLR 2025
G LIMITATIONS
One of the limitations of our proposal is the use of plug-in estimators to approximate the optimal rule (9). While these approximations calibrated. An alternative to the use of plug-in estimators is to use router model explicitly trained to mimic the optimal rule using a validation sample drawn from (Gupta al., 2024). Another limitation is that the optimization objectives we seek to minimize are local objectives that seek to make the best deferral decision the current position l. doing so, they ignore the downstream effects of choosing a particular model in the current step. Devising a global deferral objective that takes downstream errors into account would be an interesting direction for future work. More broadly, paper seeks to improve cost-quality trade-off's in LM inference. It is important that such improvements do not unfairly advantage one slice the data or a subset of the population, at the cost of others. Ensuring that the trade-off gains that approach offers is equitable across different slices of the data is another important direction for the future.
effective practice, they rely the individual models being
39


Reviews:
[{"review_id": "vK9JV4A2i3", "reviewer": "Reviewer_jr8P", "summary": "The paper introduced new speculative decoding variations by combining two techniques: speculative decoding and cascade language models. This is achieved by applying cascade rules on the token level. Experiments showed that the proposed method achieves faster decoding and performs better when the temperature is high.", "questions": "1. What is the ground truth (mentioned in lines 76, 156, 166, etc) in this paper? Why would you optimize toward the ground truth probabilities?\n2. Line 353, what is OPT?\n3. What hardware is used for the experiment? How are the models implemented? \n4. What is SpecDecode [Token] in Figure 3?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "- The proposed method is lightweight and does not require supervised fine-tuning. \n- The paper flows naturally and is easy to read and understand overall.", "weaknesses": "- The reasoning behind the designs of the loss functions in Equations (3) and (8) is unclear. \n- The experimental design seems unfair and the improvements are limited. SpecCascade [Token] uses top tokens, while the other methods are evaluated with vanilla sampling at a temperature of 1. Sampling methods like Top-K and Top-P can lead to better performance by avoiding out-of-distribution tokens. To ensure a fair comparison, baseline methods should also use Top-K or Top-P sampling. Given the experiment in Figure 5, which shows that SpecCascade does not improve over lossy SD when T=0, I doubt whether the proposed method offers meaningful enhancements other than inserting a logic removing out-of-distribution tokens. \n- The evaluations in Table 2 and Figure 2 use gamma=5, which is not optimal; speculative decoding typically requires hyperparameter tuning for the best results. \n- The sub-figures in Figure 3 (and other figures with sub-figures) are not properly aligned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uJoCB2MXKD", "reviewer": "Reviewer_ySgY", "summary": "This work investigates the integration of two front-ended inference strategies: model cascades and speculative decoding. It introduces a novel decoding strategy called speculative cascading. This method combines the strengths of both strategies to achieve a more favorable cost-quality trade-off during inference. Experiments with Gemma and T5 models across  various benchmarks (e.g., summarization, translation, coding, and reasoning) demonstrate the effectiveness of this method, which yields better cost-quality trade-offs than cascading and speculative decoding baselines.", "questions": "Most of my primary concerns are outlined in the weaknesses section above. Here is an additional minor concern:\n\n- In line 527, the authors state, “a lower rejection rate directly translates to a lower latency.” It would be helpful to provide a direct demonstration of this relationship, showing the correlation between latency and rejection rate for clarity.\n- As I am not a specialist in the theoretical domain, I cannot fully assess the accuracy of the theoretical analysis presented in this manuscript. My opinion on this aspect may change after reading insights from other reviewers or engaging in further discussions.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper addresses a promising and challenging research direction by combining model cascades with speculative decoding.  Recent studies have shown interest in integrating speculative decoding with advanced techniques, such as contrastive decoding [1], to accelerate inference while enhancing the generation quality of LLMs. Speculative cascading complements these efforts by exploring model cascades in speculative decoding. Through both empirical and theoretical analyses, this work innovatively integrates various deferral rules within a speculative execution framework, which is non-trivial and could provide valuable insights for the the academic community.\n2. The design of speculative cascading is thorough and well-motivated. The authors provide a clear exposition of the theoretical foundations for both model cascades and speculative decoding, which they summarize effectively in Table 1. Speculative cascading is systematically crafted to leverage the strengths of each of these techniques.\n3. The authors conduct extensive experiments with the Gemma and T5 models, carefully detailing experimental settings and effectively validating the efficacy of speculative cascading. The results demonstrate that speculative cascading achieves better cost-quality trade-offs compared to conventional cascades and speculative decoding methods.\n4. The manuscript is clearly written, with a well-structured narrative, compelling motivation, detailed analyses, and transparent demonstrations that enhance its readability and impact.", "weaknesses": "1. **Fairness of Comparison**: In Table 2, the authors report minimal latency when matching the quality of a large model, as well as the best quality metric achievable without exceeding the latency of LLMs for each method. However, it is unclear if these comparisons are entirely fair. For instance, it would be helpful to know if the results for BiLD were reported under similar configurations, ensuring a consistent basis for comparison.\n2. **Applicability of Speculative Cascading**: Figures 2 and 3 suggest that the quality of speculative cascading can be significantly affected by relative latency. The manuscript would benefit from a more detailed discussion on optimizing the cost-quality trade-off during inference. Specifically, guidance on configurations for maximizing speed-ups while maintaining quality or improving quality without exceeding the latency of the original LLMs, would enhance understanding and practical applicability.\n3. **Quality Improvement of Speculative Cascading**: As shown in Figures 2 and 3, the quality improvements of speculative cascading are relatively modest across several tasks, including WMT 5-shot, GSM8K 8-shot, WebQ 1-shot, and NaturalQA 1-shot with Gemma models. This limited improvement in quality may constrain the broader applicability of speculative cascading. Additional discussion on scenarios where speculative cascading performs optimally would provide valuable context for readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CUFGzlg5TH", "reviewer": "Reviewer_Wu5Z", "summary": "This paper studies way to interleave two adaptive computation methods: cascading and speculative decoding. The authors present a framework that combines both methods, and aims to enjoy the benefits of both approaches. They also derive theoretical guarantees as to the optimal deferral rule for their method. The key idea is replacing the target distribution in speculative decoding with a different distribution, which takes both the small and large distributions into account. Experimental results on various benchmarks show that the best two variants lead better speed-accuracy tradeoffs compared to either approaches.", "questions": "- The intuition behind 4.4 (the first paragraph in that section) is not entirely clear to me. Can you please explain in more detail the problem you are trying to address here?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- An interesting and intuitive approach: in cascading, the small model can sometimes outperform the larger one. In contrast, in speculative decoding, the model is guaranteed to match the large model quality, but is typically faster. By combining them, the authors allow for fast decoding, with potential improvement, which leads to overall higher speedup.\n- Some of the claims are clever. I particularly like the intuition that only considering the confidence of the small model is sub-optimal, and we should also take the large model's confidence into account, and how this could be implemented in speculative decoding (Remark 2).\n- The best proposed method (SpecCascade [Token]) seems to outperform all baselines quite consistently.", "weaknesses": "* I had some trouble following section 4.3. A roadmap/intuition would have been helpful. In particular, I did not fully understand the role of Lemma 3, and the overall takeaway from this section.\n\n- The experiments section was also a bit hard to follow. It starts with outlying the different deferral rules, and then presents the baselines. It seems both parts are somewhat overlapping. It would be helpful to merge them and discuss the link between the two, and particularly not have a paragraph separating the two.\n\n- It is not entirely clear from table 2 which lines are the current work and which are previous work. I think only the last two lines are new, but the line separating the different groups appears before the last four lines. Also, the name SpecDecode (which hints of the current work) also appears earlier (SpecDecode [Lossy]). I think this name stems from the new perspective presented in this work, but it is still quite confusing and makes evaluating the results challenging. I would suggest clearly separating existing and new methods both visually (within the table) and by name to avoid confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)