Speculative Speculative Decoding
Authors: Tanishq Kumar
Tri Dao
Avner May
Venue: ICLR 2026 Poster (2026)
Keywords: inference
large language models
speculative decoding
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce an asynchronous speculative decoding algorithm wherein the draft model continuously speculates on top of anticipated verification outcomes, thus hiding drafting latency entirely.

Abstract: Autoregressive decoding is bottlenecked by its *sequential* nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them *in parallel* with a single target model forward pass. However, speculative decoding itself relies on a *sequential* dependence between speculation and verification. We introduce *speculative speculative decoding* (SSD) to parallelize these operations. While a verification is ongoing, the draft model *predicts* likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is up to 2x faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.

Published as a conference paper at ICLR 2026
SPECULATIVE SPECULATIVE DECODING
Tanishq Kumar ++
Tri Dao
Avner May
'Stanford University
Princeton University
'Together Al
ABSTRACT
Autoregressive decoding is bottlenecked by its sequential nature. Speculative de- coding has become a standard way to accelerate inference by using a fast draft model predict upcoming tokens from a slower target model, and then verify- ing them in parallel with single target model forward pass. However, specu- lative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be re- turned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is SAGUARO, optimized SSD algorithm. Our implementation is up to 2x faster than optimized speculative decoding base- lines and up 5x faster than autoregressive decoding with open source inference engines.
INTRODUCTION
Modern AI demands fast inference. Yet standard language model decoding generates single tokens sequentially, failing to leverage the massive parallel computation available on modern hardware. Speculative decoding (Leviathan et al., 2023 Chen al. 2023) (SD) is a technique introduced to alleviate this problem. Instead of slow, autoregressive sampling from a desired "target model," it uses fast "draft model" to predict the next few tokens that would be generated by the target model and then "verifies" these tokens in one parallel forward pass of the target model. This verification is done according to an algorithm that guarantees the resulting tokens are drawn from the distribution of the target model. In each verification, the target model decides both how many speculated tokens to accept, and samples an additional bonus token that follows all of the accepted tokens. This exploits the key fact that verifying many tokens in parallel is much faster than generating them sequentially. Though speculative decoding is effective, it is itself limited by a sequential dependence: verification must complete before the next speculation can begin. We ask:
Can we eliminate the sequential dependence between drafting and verification?
We introduce speculative speculutive decoding (SSD), unifying framework for methods that aim to parallelize drafting and verification. While in SD the draft model waits for the verification to com- plete before beginning to speculate the next round, in SSD the draft model predicts what verification outcomes are most likely, and prepares (pre-speculates) for all of them in parallel to verification taking place. If any of these prepared outcomes occurs, the draft model can immediately send the pre-speculated tokens to the verifier, thereby avoiding all draft overhead. Like ordinary speculative decoding, SSD is lossless.
There are three main challenges in optimizing SSD algorithms. First, the draft model must accu- rately predict the outcome of verification: not just how many speculated tokens will be accepted, but also which bonus token will be sampled at the end for each of those cases. The number of pos- sible outcomes proportional the vocabulary size, often very large. Second. any SSD algorithm must have a fallback strategy to handle failed attempts at predicting verification outcomes correctly.
"Correspondence tani shg@stannord..e..

---

Published as a conference paper at ICLR 2026
Speculate
Draft
Target
Draft
seLang
Predict & Speculate
Ours
250
Verify
Target
Verify
aee Boce Po0
00e
160
1.7x
100
1.0x
Speculate
Draft
Successful Prediction!
AR SD
SSD
Figure (Left) Ordinary speculation requires the verifier to wait idly for the draft to speculate. (Center) In our algorithm, speculation happens distinct hardware, in parallel. while verification taking place, based on predicted verification outcomes. This allows the latency drafting to be hidden, but risks "cache miss" if the draft fails to predict the actual verification. (Right) End-to- end performance of SSD averaged over four datasets spanning math, code and chat. Llama-3.1- 70B, batch size 1, greedy decoding on TP=4 H100s. Llama-3-IB draft for SD and SSD, Includes autoregressive (AR) and vanilla speculative decoding (SD) baselines.
At large batch sizes and temperatures, this happens increasingly often and speculating in-tiie negates the benefits of asynchrony. Third, we identify a subtle trade-off between the acceptance rate of the speculator and how well it is able predict verification outcomes, which must be navigated carefully to maximize speedups.
In resolving these challenges, present SAGUARO, an optimized SSD algorithm. SAGUARO achieves decoding speedups of up to 2x compared to optimized speculative decoding baselines, and up to 5x that of standard autoregressive generation families (Figure It incorporates targeted optimizations for each challenge:
range of datasets and model
In Section we frame the problem of predicting verification outcomes in terms con- strained optimization, and introduce technique that the most likely draft logits to predict the bonus token, doing so with up to 90% accuracy.
Section we identify tension between accurately predicting verification outcomes and generating high-quality speculations, and develop sampling algorithm that allows balancing the two. Appendix(A.3.1 gives a construction where this new sampling scheme necessarily yields end-to-end speedups.
In Section we examine various strategies to handle failed predictions, demonstrating that the optimal fallback strategy varies with batch size. Adopting this, SAGUARO outper- forms SD 20% even at larger batch sizes, despite doing more compute per batch element by decoding many possible outcomes simultaneously.
BACKGROUND
We briefly review speculative decoding to position SSD relative prior work.
2.1 SPECULATIVE DECODING
Speculative decoding accelerates sampling from target distribution Ptarget by using cheaper draft distribution Piraft to generate candidate continuations, and then using the target distribu- tion to accept selectively. To verify, the target first computes probabilities for all the speculated tokens in parallel in a single forward pass. Drafted tokens are accepted sequentially, each with probability min(1, Ptarget (x)/Paratt (z)). Upon the first rejection, remaining draft tokens are dis- carded, and the last-token target logits are used to sample a "bonus token." " This done by sam- pling the bonus token from modified distribution that guarantees the resulting sequence is dis-
2

---

Published as a conference paper at ICLR 2026
tributed Pearget. This modified distribution is called the residual distribution, and takes the form r(.) xc max (Ptarget (+) Paraft (-).0) In the case where all tokens are accepted, the residual distri- bution is simply the target distribution.
The expected number of generated tokens per round, and thus the overall efficiency of speculative decoding, is governed by the acceptance rate: the probability of accepting given token in the speculation, conditioned on accepting all prior tokens. In (Leviathan et al. 2023). it is shown that the acceptance rate can be written in terms of how well the draft distribution approximates the target distribution in the following way.
Theorem 1. (Leviathan el al. 2023
a Σ min (Ptargea)))))) Paraft (x)) = 2 3llptarget
Pdraft I/
We define a speculation at round T, denoted s" == (s) k) as the sequence of K tokens proposed autoregressively by the draft model. The length K of the speculated sequence is called the speculative lookahead.
Definition 2.
Definition 3. We define verification outcome at round T, denoted oJ = (k, E VT, where (oT-11 .....
.......-- -1 are the accepted draft tokens from round T 1,, and t is the bonus token sampled from the residual distribution.
2.2 RELATED WORK
Parallel Speculative Decoding. AMUSD (MeDanel) 2025) and PEARL (Liu et al., 2025) pro-
pose speculating the next round during ongoing verification, but only prepare for the verification outcome in which all tokens are accepted, which is one special case of many possible outcomes. SwiftSpec (Zhang et al. 2025) and SpecBranch (Shen et al. 2025) prepare larger cache consisting of a token tree branching off of the speculation being verified (thus enabling larger speedups), but both fallback strategies that do not work at large batch sizes. SpecBranch (Shen et al. (025) has similar motivation to SSD: it constitutes (approximately) a special case of the SSD framework where only single branching point allowed, where the fallback speculator is equal the regular speculator, and where the hyperparameters (branching point, number branches, speculation length) are dynamically chosen to maximize speedups. SwiftSpec (Zhang et al.) (2025) considers the special case of greedy sampling, and adopts a fallback strategy of just-in-time speculation that struggles at higher temperatures and batch sizes when cache-misses become inevitable.
Tree-Based Speculative Decoding, Numerous speculative decoding methods have been proposed that increase the expected number of accepted tokens by allowing the draft model speculate a free of tokens instead of sequence, thereby giving the verifier several token options at each posi- tion (Miao et al., 2024, Li al., 2024b) Chen et al., 2024, [Svirschevkii et al., 2024). Our method differs in important ways: First, existing tree-based methods are still sequential: speculate, then verify. Second, based methods introduce a large amount of verifier compute-whie is quite expensive due to the size of the target model-heecass, the entire tree must be verified in the target model forward pass. Our method, on the other hand, scales up the speculation compute by pre- speculating for many verification outcomes in parallel, but does not introduce any additional verifier compute. Lastly, it is important to note that our method can be combined with these tree-based approaches for further gains (see Appendix E for discussion).
Improved Draft Architectures. There have been many advancements in draft model architectures that can improve the acceptance rates and/or speeds of the draft model and which can be used in SSD for even better performance. For example, EAGLE (Li et al.) 2024a b) 20256) allows the draft model to take as input the powerful representations from the target model for the current prefix, while GliDe (Du et al. (204)) and LongSpee (Yang et al., (025) allow the draft model to perform cross-attention on the KV cache of the target model. Alternate model architectures. like diffusion
LLMs (Nie et al. 2025 Sahoo et al. 22024]] Li et al. 2025a Christopher et al.) 2025| Sammagh al.) D 2003) Wang et al. 2024), can also be used increase the speed with which the draft model can produce the speculated token sequence. We include more technical details on how SSD can fruitfully be combined with these improved draft
2025 Chen et al. 2026 or SSMs (Gu et al. 2022) Gu & Dao
model architectures (e.g., EAGLE-3; Li et al. (2225)) in Appendix
3

---

Published as a conference paper at ICLR 2026
3 THE SPECULATIVE SPECULATIVE DECODING FRAMEWORK
We introduce speculative speculative decoding. a framework to reason about asynchronous variants of speculative decoding (Section (.1), and present results comparing the expected speed of SSD to baselines (Section 3.2) Definitions introduced here will be important in Section
3.1 ALGORITHM
We present the SSD framework in Algorithm T on separate hardware. While the verifier is verifying the drafted tokens from round T. the speculator begins speculating round T +
The speculator and verifier processes in parallel
It does so by predicting likely verification outcomes, and then preparing speculations for each of these outcomes in parallel, and storing these in "speculation cache" (defined formally below). Then, when it receives the actual verification outcome, it checks whether this was one of the out- comes in the cache that had prepared for. so, it immediately returns Otherwise it defers to fallback speculation strategy.
For intuition on the S&D algorithm and why lossless, consider the similarities with specula- tive execution (González & González (1997)) for CPU optimization. Like speculative execution- which uses idle compute to pre-execute conditional paths in case they are later needed-SSI SSD pre- speculates
many possible verification outcomes that, when occurs, the corresponding token sequence is immediately available. That sequence is then verified exactly as in SD. If the real- ized outcome was not pre-speculated, SSD falls back to synchronous speculator, which is lossless since it reduces to ordinary SD. The following definitions will be important in understanding the SSD framework.
Definition 4. We define a speculation cache S as dictionary mapping a set of verification out- comes VT to their associated pre-computed speculations s" = (s) ..... ....,k) k)). We denote spec- ulated token sequence ST corresponding to an outcome oT E VT by ST (v")) as a cache lookup operation.
Definition 5. cache hit is when the outcome v" of verifying T- is contained in the speculation cache S A cache miss is when v² & S
Definition 6. Let Puit.p denote the probability of a cache hit on an iteration conditional on the previous iteration having been speculated by the primary speculator. Analogousty, Pla,b represents the probability of cache hit on an iteration conditional on speculating with the backup model. Finally, Phit denote the unconditional probability of a cache hit an iteration.
3.2 THEORETICAL RESULTS
We analyze the performance of SAGUARO relative to regular autoregressive decoding (Theorem and ordinary speculative decoding (Corollary 20).
Theorem 7. Let the primary and backup speculators take time T and Ta relative the verifier. Let Enit and Emiss denote the expected number of generated tokens from the primary and backup speculators, respectively. The expected speedup of Algorithm relative to autoregressive decoding is then:
Phit Ebit + (1 Phit) Emiss
speedupssD
Phit max(1, Tp) ) + (1 - Phit) (1 Tb)
The numerator corresponds to the expected number of tokens generated in each iteration of the algorithm, and the denominator corresponds to the expected latency of each iteration relative to autoregressive decoding. This implies two key corollaries.
Corollary 8. (Strictly Faster Than SD). Suppose run SD with given speculator M. Running SSD with primary and backup both set to M does no worse than SD (strictly better if Phit- TsD > 0). Corollary 9. (Speedup Sandwich) Suppose we choose a primary speculator for which drafting completes before verification (Tp < 1), and fast backup speculator (T; 0). Then if TsD; EsD
4

---

Published as a conference paper at ICLR 2026
Algorithm 1: The Speculative Speculative Decoding (SSD) Framework.
Function ma main (prompt, target, draft, backup spec):
asynchronously launch speculatoon (prompt, draft, backup spec)
generated tokens t- ver i fier (prompt, target)
return generated.fooens
Function verif verifier (prompt, target):
target.prefill (prompt)
WAIT TO RECEIVE tokens from speculator
generaled lokens
while True do
verify-outcont + target.verift (skec_tokens)
generated wookens.append(veriig
tokens)
SEND verify for-foooooootttoe to speculatoo
if token E veri fy cerrify_outoon ooutoome then
return generaled
end
WAIT TO RECEIVE spec_lokens from speculator
end
Function speculator (prompt, primary.sper backup_sped
spec_tokens draft.speculate (prompt) while True do
SEND spec.tokens verifier for verification
cache_keys + prredict.veiificatiio comes (skec_tokens) // Section
4.1
cache vals <- specul ate for outcomes (cache keys) // Section 4.2 WAIT TO RECEIVE verify outcome from veri fier if veri verify.outcome youtcome cache then
spec_tokens else
machelverify.outcme
spec.tokens + fallback.specultt...
verify_outooo outcome, primary-spec backup_spec
// Section 4.3
end end
represent the latency and expected number of generated tokens from draft model in SD, then the SSD speedup over SD can be bounded by:
(1 (TD)
Euit ESD
speedupssD
< (1+ TsD)
Eni
Phit
speedupsD
ESD
This equation reveals that the maximum speedup attainable by SSD is proportional to the latency reduction (1 +1+ TsD) from hiding draft latency, and the increase expected number generated tokens (Ena/ ESD) from increased drafting time. However, a low cache hit rate Pait reduces the effectiveness of this algorithm, as shown by the lower bound.
SAGUARO: AN OPTIMIZED SSD ALGORITHM
In this section, we present SAGUARO, our optimized instantiation the SSD framework. present the three optimizations introduce to attain them in Sections 4.1 (Geometric Fan-Out), 4.3 (SAGUARO fallback (.2 (SaguARO sampling), respectively, then combine and evaluate them end to end.
Setup. Unless otherwise specified, experiments are run with batch size and greedy decoding, target model on 4xH100. In SD, draft is colocated the same hardware. In SSD, it sits separate device, ie. 1xH100, since it speculates asynchronously during verification. We study two model families, I.ama-3 (main text) and Qwen-3 (Appendix F) across four datasets: Alpaca (Dubois al.
5

---

Published as a conference paper at ICLR 2026
2023
GSM8K (Cobbe et al. 2021). Further details Appendix B
UltraFeedback (Cui al. 2024), HumanEval (Chen et al.
2021
4.1
PREDICTING VERIFICATION OUTCOMES: BUILDING THE SAGUARO CACHE
Given a speculation s" that in the process of being verified, SAGUARO must build a cache ST for the most likely verification outcomes. The difficulty is that the space of possible verification out- comes is vast, of size approximately (K + 1)V. While the draft decodes all anticipated speculations once, parallel, using special attention mask (Appendix
Since there are O(KV) potential verification outcomes, it is impractical to pre-speculate for all of them in parallel. This motivates posing verification prediction as constrained optimization: given budget of B verification outcomes before the draft model becomes heavily compute bound and thus untenably slow during speculation, how should one select them to maximize the changes of cache hit?
4.1.1 ALGORITIM
Definition 10. We define the fan-out FE
= at position k to be the number of verification outcomes with k accepted tokens that are included in the specu- lation cache, given the previous iteration was speculated by the primary speculator. F2 is defined analogously for the backup speculator.
(W,"" ST
Spazelatian, pm Proporod
SAGUARO Verification Outcome Prediction Algorithm
Pessible Veriisation Ostoame
Given fan-out strategy {FP F (ours based on Theo- rem 12 SAGUARO takes the top-Fy tokens at each looka- head position & in the draft logits, and adds those to the speculation cache. These are the top- logits excluding the sampled token sent for verification at that position, which guaranteed not to be the bonus token.
Speculation, Being
3
4.1.2 THEORETICAL RESULTS
Schematic of speculation cache strategy. We allocate fan-out (bonus token guesses) over sequence length K + according to Theorem [2]
Figure 2:
We show how to optimize the choice of fan-out values F? and F to maximize the speedup under constraint on the size of the speculation cache.
The probability of a verification outcome (k, -) in a given iteration depends on whether the previous speculation was generated by the primary or backup speculator: Since the two have distinct acceptance rates, the probability a given number of tokens being accepted need not be the same between the two. Thus, if the default fan-out strategy when building the cache is to fan out uniformly at each sequence position, the cache hit rate on a given iteration depends on whether the previous iteration was spec- ulated by the primary backup speculator. These are the definitions of pF)t,p(F) and Puit,b(F), respectively. We see Figure [ that these functions (in fact their complement, the rejection rate) empirically follow power-law.
Definition 11.0 We say that a speculator has a r power-law cache hit rate the chance of a cache miss with fan-out F is equal to power-law of F with exponent for sequence drafted by this speculator. More specifically, this means - (F) = 1/FF VF E N. for some r >
We this definition to reason about how optimally select the fan-out values Fg and Fh under a computational constraint on the size of the cache. We consider the general case, when this assumption doesn't hold, in Appendix A
Theorem 12. (SAGUARO RO Cache Shape: Geometric Fan-Out) Consider a draft model with accep- tance rate op and power-law cache hit rate. Then the optimal choice of E& (FE) values for
This definition is closely related the notion of b power-law acceptance in Chen
2024

---

Published as a conference paper at ICLR 2026
k [0, K] under the constraint
Σ Fb B. follows capped geometric series:
Fi = Fo p f/(+r) vk < K, and
"p
FK Po Fo (/(1tt))
ap) (p)-1
where Fo can be chosen such that Σ FL B (closed form-equation in Appendix A).
This result cleanly reflects the intuition that the lengths of verified strings follow capped geometric distribution supported on the speculation lookahead. other words, is unlikely that j [0, K] tokens will be accepted by the verifier, we should not waste compute guessing and speculating on the bonus token at that position, and should lower Fg.
4.1.3 EMPIRICAL EVALUATION
We compare the end-to-end performance of using naive (uniform) fan-out strategy sequence length against the geometric fan-out strategy advanced by Theorem 12 In Figure [ we find improves cache hit rate (right) and end-to-end decoding speed, especially higher temperatures, where both SD and the uniform fan out begin to flounder.
Phit, (F))
Phit, (F)
Phie
Rejection rate
Rejection rate
Phit(F)
10-1
10-1
T-0 T-07, 18 dat
T-1 0, mmn
Te16. aut
2
16 32 64
128
2
16
32
64
128
2
16
32
64
128
F
F
F
Figure 3: Scaling of cache hit rate with fan out. The overall cache hit rate (F) (right) computed using Theorem 1. Rejection (cache miss) falls power law in draft-fan out, demonstrating that cache hit rates increase with cache size.
0.9
Decode Speed
250
(tokisec)
0.8
200
0.7
150
0.6
100
Temperature (Uniform)
Temperature
(Geometric)
Figure Advantage of geometric fan out strategy increases at higher temperatures, improving both speculation cache hit rate (right) and thus end-to-end speed (left). Results averaged over four datasets. At all temperatures, SSD with either fan out strategy outperforms ordinary speculative decoding.
4.2 BALANCING CACHE HIT AND ACCEPTANCE RATE WITH SAGUARO SAMPLING
The majority of the time, the bonus token sampled from the residual distribution (except when all tokens are accepted, in which case is sampled from the target directly). Recall the residual distribution is given by r(-) & max (Ptarget()) (-) Paraft (-), 0): it can be difficult to predict, especially sampling temperatures increase. We introduce novel sampling scheme that makes this residual easier to predict and therefore increases cache hit rates.
This exploits the fact that the residual distribution is function of the draft distribution (which includes the draft sampling scheme). We make the bonus token easier to predict by explicitly in- creasing the residual probability mass on the most likely draft tokens; and this is done by decreasing

---

Published as a conference paper at ICLR 2026
the corresponding probability mass when sampling from the draft. In biasing the draft distribution, however, we may decrease the acceptance rate by having moved the draft distribution farther from the target (see [T, inducing tradeoff between the acceptance rate and the cache hit rate, both of which contribute to end-to-end speed (see Figure | left). Appendix A.3.1 presents a construction (explicit Peraat).Prraft) where sweeping the SAGUARO coefficient C E [0.1 finds C* necessarily gives speedups.
4.2.1 ALGORITHM
Definition 13. We define sampling scheme function a: RV AV-1 from model logits Adraft E ERV to a probability distribution p E V
To understand how to design a sampling scheme that increases the residual probability mass on the most likely draft tokens, first note that the probability of token É in the residual distribution is proportional to max(Prarget (t) Paratt (t), 0). Thus, decreasing Päraft (t) increases the probability of in the residual. This exactly what cache aware sampling scheme does.
Definition 14. Given draft logits E RVV, define the SAGUARO sampling scheme dF,((2) for fan-out F and downweighting constant C' E [0, as
C- C-exp 22)
ift t(p
oFF((2)
exp
otherwise,
In practice, C' is hyperparameter found empirically.
4.2.2 THEORETICAL RESULTS
Theorem 15. For fan-out and primary speculator logits 2, the cache hit rate Phit of the SAGUARO sampling scheme increases monotonically as C - 0.
The new sampling hyperparameter C' allows a trade-ofi off between cache hit rate and acceptance rate. We plot this tradeoff Figure 5 (left). Figure 5 (right) presents an illustration of how SaGUARO sampling allows control of the residual distribution by manipulating the draft distri- bution during speculation. The intuition here that SAGUARO sampling deliberately suppresses the draft probabilities
the F cached tokens. By downweighting Paraft (f) on this set, the residual max (Parget (t) Pdraft (t), 0) is pushed to concentrate on those same tokens, increasing the chance that the bonus token lands inside the cache by construction.
4.2.3 EMPIRICAL EVALUATION
We show in Figure how there is a trade-off between acceptance rate and cache hit rate which we can navigate by using the SAGUARO sampling scheme with different values of C. Lower values of C' lead to higher cache hit rates and lower acceptance rates they bias the draft distribution away from the target control the residual distribution.
4.3 HANDLING CACHE MISSES WITH SAGUARO FALLBACK
We now discuss how we optimize the handling of cache misses in SAGUARO, and propose an optimal strategy for picking the backup speculator based on the batch size.
4.3.1 ALGORITHM
We design SAGUARO's cache miss strategy based on the observation that cache misses occur almost certainly at large batch sizes, and that when this happens, in SSD the whole batch must wait for the backup speculator to complete before being verified. We propose the SAGUARO fallback strategy: set the backup speculator to be equal to the primary speculator at low batch size. then switch to latenat speculator (Oliaro et al. 2024; Liu et al.) 2024)] Xu et al., (225) for larger batch sizes b > b², where the critical batch size b* is derived in the following section.
8

---

Published as a conference paper at ICLR 2026
Cache Hit Rate Acceptance Rate
Draft Distribution (Beffor Saguare Sampling!
Residual Disttr iuution
No
0.75
0.70
14h1
Fors1
14h
-3.5
Draft Distribution (Atter Saguare Sampling)
Residuat Distribution
0.30
bakky
All Neddial Naw
0.5
0.50
0.53
2.73
0.00
Vecal, Sorted Bank
Vercal, by Druft Bank
Cache
Figure 5: We introduce SAGUARO sampling, a novel sampling scheme designed specifically for SSD. (Left) It interpolates between high cache hit rate and high speculative acceptance rate. (Right) Illustrative schematic for how SAGUARO sampling increases residual probability mass on the top draft tokens, encouraging the sampled bonus token to lie in the speculation cache by construction.
4.3.2 THEORETICAL RESULTS
We prove that SAGUARO's backup speculator strategy is optimal, under the conditions that SSD uses high-quality (slow) speculator (primary), and lower-quality (fast) speculator (backup). We begin with a corollary to Theorem 7 which accounts for the impact of batch size on the expected speedup attained by SSD. This result assumes that in SSD) the whole batch waits for the backup speculator to complete before verifying the batch.
Corollary 16. At batch size the expected speedup from SSD is equal
Phit Enit (1 Phit) Emuis
speedup
which approaches
Phi mmxxmx Tp) + phit) ) (1 Th)" Phi Enit + Phit) Finiss
as b xc.
+ Tb
In our implementation, the backup strategy is to return random tokens and the primary strategy to do just-in-time speculation. As the batch size increases, the entire batch stalls on the latency of the backup speculator as cache misses happen more frequently. This forces the choice of a low latency speculator at larger batch sizes, as the following theorem makes precise.
The optimal cache miss strategy given two speculators of varying speed (primary and backup) is to set the backup equal to the primary for batch sizes < b", and not otherwise. We solve for b" in Appendix
Theorem 17.
4.3.3 EMPIRICAL EVALUATION
We show in Figure that as the batch size increases, using a fast backup speculator that returns random tokens outperforms using slow but more accurate neural speculator just-in-time, We also see that we can reduce latency by scaling draft-time compute. Dedicating more draft devices leads larger speculation caches and thus higher cache hit rates.
END-TO-END EVALUATION
In Figure 7 compare the end-to-end performance of SAGUARO key baselines, including own implementation of ordinary speculative decoding, as well as the strongest open source inference engines (vLLM/SGLang) We also include EAGLE-3 when supported by open source engines. We attain almost 5x speedup on average compared to our autoregressive baseline, and are to twice as fast as the optimized speculative decoding baselines, establishing new state of the art.
Note that speculators, like those based n-grams (Liu et 2024 Oliaro et al. (022))
casily improve upon this random token strategy by using extremely fast non-neural
FEven the strongest public EAGLE-3 implementations have suboptimal performance. We find SGLang
the strongest baseline for both SD/EAGLE. so include just those here for clarity.

---

Published as a conference paper at ICLR 2026
250
Decoding Speed
per Sequence
Fast Backup
tookens/sec/batet
Neural Backup
200
Sync Baseline
tok/s/s04
160
140
150
120
Batch Size
Draft GPUs
Figure 6: (Left) Optimal backup speculator (fast neural) depends batch size, Theorem 17 predicts. temperature O. (Right) We forecast how cache hit rate/decoding speed improves with draft compute (fan out) temperature 1.0 when cache hit rate is 0.6 with one draft device.
Though SSD is designed to trade off compute for lower latency, it nonetheless pushes the Pareto frontier across both lateney and throughput, with biggest gains lower batch sizes where the draft model is not yet compute-bow bound. This means that SSD does not just improve the best possible decoding speed, but is also more compute efficient per device.
Llama3 End to End Speedups
Throughput-Latency Pareto Frontier
ber
be16
$D
5.5x
Throughput (tok/s/GPU)
300
5.2x
4.7x
400
250
3.9x
4.1x
b=8
300
150
200
b=d
b=4
50
100
b=1
b=2
HumanEvaiUllrraF
Alpaca
GSMBK
Average
140
160
180
200
220
240
260
Latency (tok's per sequence)
55D
SGLang
(EAGLE-3)
Figure 7: (Left) End-to-end decoding speed comparison of SSD compared to SD and standard au- toregressive decoding on 1ama--31-Instree 70B across four datasets. (Right) SSD expands the throughput-latency Pareto frontier.
CONCLUSION AND LIMITATIONS
Speculative decoding trades compute for latency, but drafting and verification remain synchronous. We introduce speculative speculative decoding (SSD), which parallelizes even this sequential de- pendence. We derive performance bounds, study each component the design space, and distill the results into SAGUARO, which achieves to 2x speedup over optimized speculative decoding and up to over autoregressive generation across datasets and model families.
Like all speculative decoding methods, SSD targets latency, not throughput; throughput-boumg workloads like large-scale RL offline data generation may benefit. Though SSD uses ad- ditional hardware, it outperforms SD even after adjusting for extra hardware.
Our work suggests number of natural next steps. First, SSD is compatible with popular SD vari- ations like EAGLE and token-tree speculation (Appendix E), and study of the combined design space would be valuable. Second, SSD admits a simple way to further decrease latency: scaling the number of draft devices and thus speculation cache size. What are the tradeoff's here? Third, extending SSD to cluster-scale inference via drafter-verifier disaggregation (with similar motiva- tions in prefill-decode disaggregation (Zhong et al.) 2024)) may make sense because drafting gets more quickly compute bound in SSD than in SD or autoregressive decoding. Studying this would illuminate the gains to be reaped during large-scale inference in production settings.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We gratefully acknowledge the support of the Schmidt Sciences AI2050 fellowship, the Google ML and Systems Junior Faculty Awards, and the Google Research Scholar program. We thank Together Al for supporting this project with compute resources. Tanishq Kumar thanks Tatsunori Hashimoto and Luke Bailey for helpful comments and discussion.
REFERENCES
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. In Findings of the Association for Computational Linguisties: EMNLP, 2020. URL https:
7 /arxiv.orq/ http//bb////0.....
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri
Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 10777, 2024. URL
wrtpa///.rrrra/////oooooo..sss http://as/22411110 10774
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023. URL
org. abs /2lss/233222 2302. 01318
Jian Chen, Yesheng Liang, and Zhijian Liu. Dflash: Block diffusion for flash speculative decoding, abs /2 602. 06036
2026. UtL.sttp::/aaxxv
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss,, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. htttps: http://ogg//bbbb/a/gobbbe 03377
Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable and robust speculative decoding. In Advances in Neural Informa- tion Processing Systems, 2024.
Jacob K. Christopher, Brian R. Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, and Ferdinando Fioretto. Speculative diffusion decoding: Accelcrating language generation through diffusion. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 May 4, 2025, pp. 12042-1 2059. Association for Computational Linguistics, 2025. doi: 10. 100.65///1/22255.AAALLLLLLL. maacl-loma 601
Language Technologies, NAACL 2025
URL petttps://do:.. .ta.oo..org01..111... wtt.g///....1//11//2112
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxis: orr/ aa/aabba/2111 14168
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie. Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. |RIhttts:: wtw.xaivvvrrrr abs/233 O. 01 01377
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Process- ing Systems (NeurIPS), 2022. URL httph:tt//// mtetps://arxiv..rrttr
11

---

Published as a conference paper at ICLR 2026
Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, and Yang You. Glide with a cape: A low-hassle method to accelerate speculative decoding, 2024. URL https:h/aaxiiii
2402. 02082
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023. URL htattpp://axxiv.or///aaat
14387
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm infer- ence using lookahead decoding. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024. URL https: //tp://.a.iiv.vi... org wbwpp//aaa//42..... 2202
José González and Antonio González. Speculative execution via address prediction and data prefetching. In Steven Wallach and Hans P. Zima (eds.), Proceedings of the 1 11th interna- fional conference Supercomputing, ICS 1997, Vienna, Austria, July 7-11, 1997, pp. 196-203. ACM, 1997. doi: 10.1 10.11455263580226663111 URL /RL/:ttpss///0...rr//0.....
263631
Albert Gu and Tri Dao. Mamba: Linear-Tima Sequence Modeling with Selective State Spaces. arXiv, 2023. doi: 10448550/aXxx..311.0077
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learning Representations (ICLR), 2022.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL httpthttt///aat
org/ abs
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (SOSP), 2023. doi: 10.48550/a2Xiv.230999 .0000000010.... URL htttts/////axi
/org/ brg/abs/2333 2309. 06180
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding, 2023. URL http:/tpp..//rriv.orr/aaa///////
17192
Guanghao Li, Zhihui Fu, Min Fang, Qibin Zhao, Ming Tang, Chun Yuan, and Jun Wang. Diffuspec: Unlocking diffusion language models for speculative decoding. CoRR, abs/2510.02355 2025a. doi: (0.48550/AXXI22550002555 02358
htttp::/ 42g/10..4500/axi....... 2510.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: speculative sampling re- quires rethinking feature uncertainty. In Forty-first International Conference on Machine Learn- ing, ICML 2024, Vienna, Austria, July 21-27. 2024, 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, 2024b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference accel- eration of large language models via training-time test, 2025b. URL https::/arxiv..or
abs/ 0aes/2503.0m 01840
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. CoRR, abs/2401. 173777 2024.
Tianyu Liu, Yun Li. Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, and Xiao Sun. Pearl: Parallel speculative decoding with adaptive draft length, 2025. URL https: // /aaaiiiiiv.. www.///.oo.oo abs/ 2408. 11850
12

---

Published as a conference paper at ICLR 2026
Bradley McDanel. AMUSD: asynchronous multi- speculative decoding for LLM accelera- tion. In IEEE International Symposium on Circuits and Systems, ISCAS 2025, London, United Kingdom, May 25-28, 2025, 2025.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang. Xinhao Cheng, Zeyu Wang. Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jía. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM Interna- tional Conference on Architectural Support Programming Languages and Operating Systems, Volume 3, ASPLOS 2024, La Jolla, CA, USA, 27 April 2024- 1 May 2024, 2024.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In Proceedings of the Thirty-Ninth Conference on Neural Information Processing Systems (NeurIPS 2025), 2025. URL
htttps://axxis wwtpaarwwww....aaaeaaeeraeeee/ae abs 25.2.02222200 09992
Gabriele Oliaro, Zhihao Jia, Daniel F. Campos, and Aurick Qiao. Suffixecoding: Extreme speeu- lative decoding for emerging ai applications. CoRR, abs/2411.0es 1.04975, 2024.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander M Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion lan- guage models. In The eiihty- Annual Conference on Neural Information Processing Sys- tems, 2024. URL https: / cotnee/fooummideee Aaw.rrrrnnnn.--AAAAAAAAAAAA
Mohammad Samragh, Arnav Kundu, David Harrison, Kumari Nishu, Devang Naik, Minsik Cho, and Mehrdad Farajtabar. Your LLM knows the future: Uncovering its multi-token prediction potential. CoRR, abs/2507. 2025. doi: 1048550ARXXV:250.111555 URL https: 7/doi. org/100 488444855505/1 2507 11851
Jay Shah, Ganesh Bikshandi. Ying Zhang. Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. In NeurlPS 2024 (Spotlight Poster), 2024. URL poster version.
wwwp://..wr/bb//////o/// 08608
Spotlight
Yuhao Shen, Junyi Shen, Quan Kong, Tianyu Liu, Yao Lu, and Cong Wang. Speculative decoding via hybrid drafting and rollback-awar branch parallelism. arXiv preprint arXiv:2506.01979,
2025.
Ruslan Svirschevski, Avner May. Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec: Massively parallel speculative decoding for interactive LLM inference consumer devices. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Junxiong Wang, Daniele Paliotta, Avner May. Alexander M. Rush, and Tri Dao. The mamba in the Ilama: Distilling and accelerating hybrid models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances Neural Information Processing Systems 38: Annual Conference on Neural formation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: /papt:://ppperssee .ips cc/paper wiless/ppper/2224//44 72393306
Conference..cceee Ab Cotteaaaeeeetttrccrccotoeeeeeeee httel
Hao Xu, Jiacheng Liu, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Infini-gram mini:
Exact n-gram search at the internet scale with fm-index. CoRR, abs/2506. 12229, 2025.
Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, and Bo An.
Longspec: Long-context lossless speculative decoding with efficient drafting and verification, 2025. BRLLttts:
Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for Ilm inference serving. arXiv preprint arXiv:2501.01005, 2025.
URL |tl.tt /tapps:://axxt wttpoag//../..o.o 01005
13

---

Published as a conference paper at ICLR 2026
Gyeong-In In Yu, Joo Seong Jeong, Geon- Woo Kim, Soojeong Kim, and Byung- Gon Chun. Orca: A distributed serving system for Transformer- Based generative models. 16th USENIX Symposium Operating Systems Design and Implementatiin (OSDI '22, pp- 521-538 Carlsbad, CA, jul con http://ooeeeeeeeeeeeeeeeeeeeeeee
2022. USENIX Association. URI. htt.hhtss://wwww www.wwww.nii....mm x. o
presentation/y/
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. Advances Neural Information Processing Systems (NeurIPS), 2020.
URL wtt.htttps///axxi..ooo org. abs/2003/30 14062
Zhenyu Zhang. Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng. Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: heavy- hitter oracle for efficient generative inference of large language models, 2023. URL https:
/ltgg//bbb//.......// 2306 14048
Ziyi Zhang, Ziheng Jiang, Chengquan Jiang. Menghan Yu. Size Zheng, Haibin Lin, Henry Hoff- mann, and Xin Liu. Swiftspec: Ultra-low latency Ilm decoding by scaling asynchronous specula- tive decoding, 2025. URL https::tt.............. / htt.axiivvorrr abs 2506. 1130911399
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024. USENIX Association, 2024.
14

---

Published as a conference paper at ICLR 2026
A THEORETICAL RESULTS
Unless otherwise stated, assume throughout that T 1, that max 1) 1. This is because SSD has the primary speculator (draft with custom attention mask) overlapped with verification by construction, that it finishes before verification is complete. The speculation lookahead chosen to make this true in our implementation. We also assume that different sequences a batch are unrelated, that the probability of cache hit is iid Phit-
A.1 THEOREM PROOF: MODELING THE SPEEDUP FROM SSD
We copy Theorem for reference:
Theorem 18. Let the primary and backup speculators take time Tp and T3 relative to the verifier. Let the expected number of generated tokens from the primary speculator be Enit and from the backup speculator be Emiss- The expected speedup of Algorithm 7 relative autoregressive decoding then:
Phit Ehi + - (Phtt Finiss
speedup
Phit max(1, TT) + (1 Phn) Tb)
The global cache hit rate Pait can be expressed in terms of the cache hit rates Phit.p and Phit.b- cor- responding to whether the last round's 's speculator the primary or the backup, respectively (assuming |Pitt.p Puit,bl < 1):
Phit.b
Phit
1 Phit,b - Phit.p
Proof. We will prove this theorem in two parts:
Part I: We will first prove this speedup equation, assuming we know the overall cache hit rate Pai (conditioned choice of lookahead, cache topology, sampling algorithm, etc.).
We will then prove the functional form of Phit, as a function of the cache hit rates Phit,p and Puit, of the primary and backup speculators, respectively.
Part 2:
A.1.1 PART
In each iteration of SSD, the expected number of generated tokens is Ebit if there is cache hit, and Emiss there not. Similarly, the latency max(1, Tp) if there is cache hit, and 1 T, there not this is because backup speculation, which takes time T,, begins only after verification (which we assume takes unit of time) completes.
Therefore, it is clear that the expected number of generated tokens and the expected latency (relative to autoregressive decoding) in one iteration of SSD are:
E # Generated tokens]
Phit Ebit + (1 Phit) Emiss
E Latenccy
=
Phit max(1, Tp) + (1 - Phi)) (1 Tp)
Thus, the expected speedup is:
E Generated tokens]
speedup
E|Latency]
Phit Enit + (1 - Phit) Emiss
Phi m,, + (1 -hitt (1 + Tb)
This concludes part of the proof.
15

---

Published as a conference paper at ICLR 2026
A.1.2 PART
We will now prove that the overall (unconditional) cache hit rate Phit is equal
Phit.b
Phit + Phit,b Phit.p
where phit.p and Phit.b are the cache hit rates conditioned on the prior iteration being speculated by the primary and backup speculators, respectively.
The challenge in deriving a closed-form solution for the overall cache hit rate Phit is that Phit at iteration I" of the SSD algorithm depends on whether there was a cache hit in the previous round (in which case the primary speculator was used) or not (in which case, the backup speculator was used).
In order to deal with this recursive property of Phin, we first write Phit a recursive equation, which considers the iteration number the algorithm. We consider both the base case (1 0), where for now will assume we always use the non-neural spec, along with all rounds thereafter:
Phic (0)
Phit,p:
because we use the primary speculator
0,
Phit - 1) Phit.p + (1 (1-Phic (T - 1)) Phit,
We rearrange and unroll this recurrence:
Phin (T)
=
Phit (T - 1). (Phit.p - Phir,b)
+ Phit,b
Phit (T -
1)) 1). Phitt.b letting r Phit,p - Phit,b
(phic(T -2) .r +
Phit,b
Phit (T 2) Phatb Phit,
(Piic(T' 3) r + +Pit..) x² + Phit,b T + Phit,
Phit (T - -3) n Phith 12 Phit, T Phit,b
Pai (0) r + Phit,b
pl
1=0
T
Pait (0) - Phit.b
Using the stated assumption that |r| = |ait. converges and the second term converges to --r
Phit,bl
< 1, we can see that the first term above
as expected.
Puast PPait-PPha. Pha.p
This concludes the proof.
A.1.3 PROVING THE TWO COROLLARIES
We now prove the two corollaries of Theorem copied below:
Corollary 19. (Strictly Faster Than SD). Suppose
SD with given speculator M. Running SSD with primary and backup both set to M does no worse than SD (strictly better if Phit. TsD > 0).
Proof. Let Enit Emiss ESD. and Tn To TSD. Then
Phi Ehit (1 Phit.) Emuiss
specdup
Phit max [1, Ip)) + (1 Phit) (1 + Tb)
Phit EsD (1 Phit) ESD
Phit max(1, (1 Phit) TSD)
ESD
Phit max (1, + Phit) . (1 TSD)
16

---

Published as a conference paper at ICLR 2026
Recall that the speedup from SD is EsD/(1 Tsp). This final term is strictly greater than the SD speedup if Phit and TSD > O, because in this case max(1, TsD) < ++ TSD. If Phit = 0 or TSD = 0, then the SSD speed is equal to the SD speed.
n
Corollary 20. (Speedup sandwich) Suppose we choose a primary speculator for which drafting completes before verification (Tp < 1), and a fast backup speculator (T, = 0). Then if TsD, Esn represent the latency and expected number of generated tokens from a draft model in SD, then the SSD speedup over SD can be bounded by:
+ (1+Tsn)
Emit ESD
speedupssD speedupen
- (1+ (1+Tsn)
Enit
Phit
ESD
Proof. By assumption, Tp < 1 and Ta = 0. So the SSD speedup is:
Ebit + (1 - Phit) Emiss
speedupssD
Phit max Tp) (1 phit) + Th)
Phit Ena + (1 - Pant) Emiss
Recall the SD speedup equation is:
ESD T
speedupsp
So,
speedupssD
Phit
Epit + (1 Phit) Emiss
speedupsD
ESD((1 +TDD)
Because Phit S we get the upper bound (assuming Enit 2 Emisss):
speedupssD speedupsD
(1 + TsD) ESD
Fait
Because Emiss
(bonus token is always generated), get the lower-bound:
speedupssD speedupsD
Phit
Ema + (1 - Pait) Emiss
ESD/
TSD)
2 1 + TSD
Ema
EsD Phit-
A.2 THEOREM PROOF: OPTIMIZING SAGUARO CACHE TOPOLOGY
To prove Theorem 12 will first prove Theorem 21 which is general version of the theorem. Then, Theorem 12 will be an easy corollary of this more general theorem.
Theorem 21. The choice of F (and equivalently. FR)) values that maximizes the speedup of SAGUARO under the constraint Ex F < B (where K is the speculative lookahead), is:
K
Σ
EP
B,
k=0
Ophit.
Op aF
(FR)
ap
(Fo)
Vk K, and Ophi.p
OF
0p 0ppit.p (Fx)
Kiall
(1 - (1-ap)
0p
(Fu).
OF
OF
17

---

Published as a conference paper at ICLR 2026
Proof. To understand how to optimize cache topology, first rewrite the speedup equations from Section 3.2 now making explicit how the speedup depends on the Fg and Fg values:
speedup ( (F
Phit (ER),1FF) Ehic
(1 (1-Pist (FE),.(E)
Emiss:
where
Phit,b((F())
Phit (FP): (FF)((F))
1+ Phin, - Phit,p ((
We can express phit.p more precisely in terms of:
The acceptance rate ap of the primary speculator, and
The functions phatpp Fi)) and Phit.p (F)): These functions describe the probability of cache hit conditioned on (1) the last round's speculator was the primary one, (2) ls tokens were accepted, (3) whether
K.all
the tokens were accepted (Phtt, K.all) or not (phit.p). and (4)
Fg verification outcomes were prepared for in case tokens were accepted. Note that this can be estimated empirically by comparing the draft and target probability distributions for a set of sequences in a calibration set.
We can do the same for bait.b, the cache hit rate when the last round's speculator was the backup.
K-1
K.all Phtt.p(FF)) ag PPiup
(Fk) +
Σ a0(1 ap) Phs,,(Fk), and
These equations are direct result of the fact that the chance of accepting exactly k tokens is a (1- ap) for & < K, and of for ls K, when the acceptance rate is ap.
Given these equations, we can now optimize the cache topology (i.c., the choice of Fk values) to maximize speedup. We notice that the speedup is monotonically increasing Pait. and that Phit is monotonically increasing in both Phit.p and Phit,b (casy to see by taking first derivatives of Phits and seeing they are always positive). Thus, we must simply maximize Phit.p ((F and (f under the constraints that Σ
=k=o FR <
B and Ex F
<
B. We do this below.
A.2.1 MAXIMIZING Pait,p Patt,p((FT)) ((FP)) AND PRitD P ((F()
As discussed, we want to maximize the probability of cache hit under the budget constraint Dk= Fx < B. We do this for phit,p and the proof identical for Phit.
K-1
max Phit, (K, max af Puit.p FxSB
K K.all
(Fx)
Σ
a ap) pritt, (F+)
Σx FASD
Σ.
We will solve this maximization problem with Lagrange multipliers.
K-1
K Fi B)
K K.all A) ap Phitt, (FK) +
Σ d
ap) phit..(Ph) +
(
k=D
Now, will take the derivative with respect all the variables, and set it to zero
18

---

Published as a conference paper at ICLR 2026
0C
8
a
ap)
OFF Phit.p (Fi) =
for k < K
OFF
0C
k 0 K,lli
(Fk) + =
0FK
p
OFk
Phit,p
OC
Σ Fx B k=0
0X
We notice that for all k, ac
ar, are equal to one another (all equal to -)). Thus, we see that:
0 (Fa)
0
(1 ap) OFk
ap(1 - ap)
Phit,p
(Fi) ...
OFK
ap
k-1(1 (1 ap)
OF
Phit,p
K-1 K-1 (FK-1) = Op OF
K
Phit.p
K,all
(Fx)
0
- i
Phit.p
(Fk)
" 0F
Pit.p (Fo)
vk < K, and
0F
0
0F%
P
hpt.p
Kllll
(Fk) = (1- ap)a
k
OF
Phit.p (F))
This gives the desired result.
We now use this result to prove Theorem 12 which is special case of the above theorem in the case where the speculators have r power-law cache hit rates.
Theorem 22. (SAGUARO Cache Topology) The optimal choice of PE (equivalenty, Fb) values for k E [0, K] for SAGUARO under the constraint Ex= Fh S B, and under the assumption that the speculator has acceptance rale ap and r power-law cache hit rate, follows a geometric series (for k < K):
F& Fo dp 2/((+r) vk < K, and
FR = Fo Po.05 Fo.ap k/(1+r). (1 ap) -11(11)))))))))))
where Fo can be chosen as follows so that Exn FP B.
B
Fo = k/(1+r) a1,)) (1 (1- ap k((1+r) / ap
1/(1+r)
Op
Proof. Now, substituting Phit,p (F) = 1 - F (and thus 2 DF. Phit,p (F) = (F)
F-r-1), 1) get:
F-
a kr. F- wk K, and
F -- = (1 - ap) ap K r For bーr-1
FR
Fo Fo vk < K, and
FK = Fo ((14r)/(+r))
K/ k/(++) (1 - a 1/((1r)) (1/4))1 d
To solve for the exact sequence of F& fan-out values, we plug the above values into the budget equation:
K-1
B = Fo dp
k/(1tr) (1 ap) 11(1+r) +
Σ k=
Fo a
k/(tt)
19

---

Published as a conference paper at ICLR 2026
Solving for Fu gives:
B
Fo k/(1+r)
(1- a0-
E Up
K-1 k/(++r)
We can simplify this further using the equation for the sum of the geometric series:
1 1- ck
Σ p (01-)
Σ
T-c for c oz 2/(1+))
Plugging this in gives the desired result.
B
Fo
k/(+t)) (1-ap) op) -100-(((+r) (1- ap k/(ttr)// (1- 2/(1+r)
ap
A.3 THEOREM 15 PROOF: OPTIMIZING SAGUARO SAMPLING ALGORITHM
Theorem 15. For fan-out F, and draft logits 2, the cache hit rate Phit of the SAGUARO sampling algorithm increases as C 0.
Proof. Fix fan-out F and draft logits z. Let S == (z) denote the F cached tokens (where S depends only on z, not on C). Write w; == (xp()+) and define
Σ WP
= Σ wt.
Z(C) = CA B.
agS
Then the SAGUARO sampling scheme induces the draft distribution
Cw.
te S
Z(C) W1
Pc(1)
$ S.
Z(C))
Fort S,
d
W/ B
dc. Pc(t)
(CA + B)² 2 0.
and for t & S,
W,A Po(1) (CA B)² < 0.
Thus increasing C' increases (t) on cached tokens and decreases (t) off the cache. Let g(() == Prarget (1) and define the (unnormalized) residual mass
(t) max(g(t) (t), 0).
Since (t( a nonincreasing function of (t).). the above monotonicity gives that for t e S. ut()) nonincreasing in C; for t S, uc(t) is nondecreasing in C. Then define
in(C) = Σ (t)(), out(C) = Σ (t).).
FES
EES
Then iC(C') nonincreasing C' and out(C') is nondecreasing C.
For any verification case in which the bonus token is sampled from the residual distribution (i.e., when & < K in SD/SSD), the probability that the sampled bonus token lies the cache
iC(C)
Phit = in(C) - oCt
with the convention that iC( + out(C) = 0 (zero residual mass), then Pai (C) = Then since the map b) a/(a b) is nondecreasing in and nonincreasing in for a, b > 0, and since in(C)) (C)) decreases while (C' increases with C,, it follows that Phie (C') is nonincreasing in C, concluding the proof.
20

---

Published as a conference paper at ICLR 2026
A.3.1 CONSTRUCTIONS SAGUARO SAMPLING CAN GIVE SPEEDUPS
Construction Fix the target distribution
q = (0.49, 0.49, 0.01, 0.01),
over vocabulary of V 4 tokens, and fix arbitrary E E (0, 0.01). Define two draft distributions
p(s) (0.5 - E, 0.5 E, e, e),
(e()) = (0.48 + E, 0.48 + E, 0.02 - E, 0.02 - =).
Theorem 23. Construction demonstrates a draft and target distrbution (q. p) where SAGUARO sampling necessarily yields end-to-end speedups by transforming + p' using some SAGUARO coefficient C e 1].
Proof. We proceed in three parts. First, we show p' leaves acceptance rate unchanged. Then, we show p' induces a strictly higher cache hit rate by construction. These two are enough to show (g.p') must result end-to-speedups. We conclude by showing there exists (by construction) SAGUARO co-efficient such that (pp) =p' This suffices to conclude that SAGUARO sampling results in end-to-end speedups here.
Part 1. Acceptance rate is unchanged. By construction, all coordinate-wise differences have magnitude 0.01 - 8:
00.49 - (0.5 =)l = 0.01 - |0.01 - =| = 0.01 E,
and similarly |0.49 - (0.48 + +) 00.01 - (0.02 - a)1 = 0.01 - 6. Hence
- (1=p(=)]1 = 4(0.01 =) = 1|g - p'))
Therefore, by Theorem | the speculative acceptance rates are identical:
a(p(s)) =
= 11- 4(0.01 - =) = 0.98 + 2s.
Part 2. Cache hit rate strictly increases. Take fan-out F = 2, so the cache is the top-2 draft tokens S = {1, 2} for both p(=) and (s)). Let (t) max (((())))))))))))) p(t), 0) and (t( x (t()) be the residual distribution. Since p(s) = p(s)2
= q2 and p()) = p(E) < Q3 = q4, we have (1) "p(e) (2) 0 and (3) = up(e) (4) = 0.01 - s. so Tp(e) supported {3, 4] and
pms((((E) = Σ (t) = O.
(E
Conversely, p(()) = p(())2 < <n = 22 while p'())3 = p' p(()) > 93 94. so the residual concen- trates on {1,2) and
Pm)(l'(=))= = Σ (t) 1. 1E(1.)
Part 3. There exists an explicit (e) [0, with SAGUARO sampling. Apply SAGUARO sampling with cached set S = (1,2) and coefficient C [];
(C)())(t) te [1,2} (p())(t)
(t x
te (3,4).
By symmetry, (1( = (2) and (3) = (4), and the odds ratio is
Pc (1
C 0.5 -
Pe (3)
Choosing
C*((E) = (0.48 + +))/ - -)
(0.488+c)
<0.5-s)/= (0.5 - =00.02 -) E (0,
makes (e) = ('() exactly by construction. This suffices conclude the proof.
21

---

Published as a conference paper at ICLR 2026
A.4 COROLLARY [6 THEOREM 17 PROOFS: OPTIMIZING SAGUARO FALLBACK STRATEGY
We first prove corollary 16 and then Theorem both copied below for reference:
Corollary 24. At batch size the expected speedup from SSD equal to:
Phit Bhit + (1 Phit) Emiss
speedup
which approaches
Phit max [1 Tp) + (1 - Phin) (1 + Tb)"
Phit Enit + (1 Phit) Emiss
as b 20
+ Tb
Proof. For each element of the batch, it gets cache hit it will generate Ehit tokens, and otherwise Emnss- The latency an element of the batch, however, depends on whether any element of the batch had cache miss. If the latency for the entire batch is 1 Tg. Otherwise, if every element of the cache had a hit, the latency is max(1, (1, T,). Thus, we can see that:
E H Generated tokens]
=
Phit Ebit + - (h-Phit) Emiss
E L'[Lateny]]
=
b max T2) + (1 - phan) (1 + TT)
Thus, the expected speedup is:
E # Generated tokens]
speedup
El Latencyy
Phit Euit + (1 - Phit) Fimiss
=
Phit max(1, T,) (1 phat) + Tb)
Noticing that plat - as grows concludes proof.
Theorem 25. The optimal cache miss strategy, conditioned only being able choose between high-quality slow speculator (primary), and a lower-quality speculator with negligible latency (backup), to use the primary speculator for batch sizes b < b", and the backup speculator other- wise. The value of b* is given by:
log 1 Tp
((1+)
Exit
6"
log(Phit
Tp Phit
Enit + Tp (1 Phit) Fmiss
Proof. Using the slow primary speculator as the backup speculator gives expected speedup:
Phit Enit + (1 - Phit) Fhit
speedupalow backup
phi .max Tp) (1 p (1 Tp)
Enit.
Pla (1 plan) (1 (p)
Enit
1 Tp Tp
Using the fast backup speculator (with T₁ as the backup speculator gives expected speedup:
Phit Enit + (1 Phit) Emiss
speeduppast.Jacka
Phi max 1, Tp) + (1 - phin) (1 +1 Tb)
Phit: Euit + Phit) Emiss
Ph. max(1, Tp) + phrt)
Phit Fea + (1 Phat) Emniss--
=
22

---

Published as a conference paper at ICLR 2026
We set these equations equal to each other and solve for b, which gives the equation in the Theorem statement.
Fiit
Puit Euit + (1 - Phit)) ) Emisss
1 1 Tp Tp phit
Ehit
1+ Tp Tp Phit
Phit
Emit + (1 - Phit) Emiss
1
Enit
plat
1 Tp
Tp
Phit Enit + (1 - Phit) Emiss
((++7) Tp
Enit
log
log(Phit)
Tp Phit Eni + Tp (1 Phit)
Emiss
Now we must simply show that for batch sizes b < b" it is better to use the slow backup speculator (a.k.a,, the primary speculator), and for 2 b" it is better to use the fast backup speculator.
We do this by seeing that the speedupstow backup monotonically decreasing the batch size (negative derivative with respect b), whereas the speedupfastt does not depend on b (obvious from equation). This shows that there is value b* that makes these two speedups equal, then for b < b* the slow backup option gives larger speedup, whereas for b 2 b" the fast backup option gives larger speedup.
@speedupstow backup
0 db
Enit
db
Tr Tp Pris
Enit
Tp Piu lolo
(1 Tр
Tp phn
which is clearly negative because log(phit) and everything else is positive. This concludes the proof.
B IMPLEMENTATION DETAILS
B.1 SYSTEMS DESIGN
Overall Design. We implement SAGUARO in custom inference engine written in PyTorch. Our implementation incorporates PagedAttention (Kwon et al., 2023), continuous batching YYu et al. 2022). tensor parallelism, BF16 mixed precision, torch compilation, and CUDAGraphs.
The architecture is built around a target model split across devices, and small draft on a separate device. The two communicate through NCCL once per speculation round, wherein the target sends only the results of the previous round (number of accepted tokens, bonus token sampled), and the draft model uses this key lookup into its pre-prepared speculation cache, and immediately returns the tokens and logits corresponding to the pre- rre-prepared speculation on a cache hit. and either random outputs just-in-time speculation in the case of cache miss. The key point that the target model never sees the draft-side speculation cache: it is sent the speculated tokens for its verification outcome immediately after verification. particular, no KV cache ever transferred between the devices.
Implementation Details. The engine orchestrates from a coordinator process on the main GPU. A scheduler paired with block manager handles prefill/decode scheduling and page-table bookkeep- ing. ModelRunner on each target GPU prepares attention metadata and executes forward passes. In async mode, the draft model runs in separate process its own GPU. Target and draft com- municate once per iteration via NCCL with fused payloads: the target sends cache keys (sequence ID, accepted-prefft length, recovery token), current sequence lengths, draft block tables for KV ad- dressing, and per-row temperatures; the draft returns a cache-hit bitmap. K speculative tokens per sequence, and K K-step logits for acceptance.
While the target host maintains page-table bookkeeping for both models, the draft KV cache ten- resides the draft device. The scheduler ensures the target has sufficient pages for K 1
23

---

Published as a conference paper at ICLR 2026
Custom SSD Attention Mask, = K= ==4,7=2
You
Figure 8: Custom attention mask for multi-query decoding of all BF(K + verification branches in parallel. This mask is for B = 1, K 3, uniform fan-out F at depth 2. Black indicates tokens that can be attended to. The left block shows attention to the verified prefix: diagonal bands show each branch attending only within its forking path.
multi-query decoding steps, preempting sequences when lookahead reservations cannot be satisfied. After verification, both page tables are reconciled: completed pages are finalized (hashed for prefix caching), and any pages allocated beyond the accepted suffix are deallocated. This rollback require- ment-undoing allocations for rejected tokens that wrote into pre-allocated Iages-necessitates host-side post-processing step after each verification.
Design Decisions and Performance Engineering. During draft speculation, F(K 1) branches of each sequence are decoded in parallel using a custom sparse attention mask that al- lows each branch attend to the verified trunk and its own forking path. We use FlashAttention kernels (Shah et al. (2024) where possible, falling back to FlashInfer (Ye et 20,55 for multi-query decoding paths requiring custom masks. An example mask shown Figure 8
Materializing these masks depend on prefix length, B, K, F, and step i-is substantial overhead. The sparse, non-coalesced memory access patterns custom-mask attention kernels dominate critical path, limiting how many steps we can profitably draft. Thus, most end-to-end speedup comes from hiding draft latency rather than increasing lookahead. Since accepted branches land in fragmented KV cache locations, we perform an extend (prefill) operation of the previous speculation into KV cache before each round async decoding. This corresponds to the "Glue & Recurrence" column in Figure 8 enabling all F forked branches attend to the same prefix.
B.2 EXPERIMENTAL DESIGN
Datasets. We take 128 prompts from each dataset and sample 512 decoding tokens for each. We use vanilla sampling (not top-p or top-k). We measure decoding throughput, excluding prefill. All experiments are done on single node of NVIDIA H100s, though we only use TP=4 (AR, SD) or 5 (SSD) GPUs at once.
Baselines. We compare to the two most widely used open source inference engines: vLLM and SGLang, both with autoregressive and standalone speculative decoding. Though SSD outperforms methods like EAGLEE3,, it can be combined with these methods see Appendix E., The implemen- tation of both autoregressive and vanilla speculative decoding our inference engine comparable to that in both vLLM/SGLang, highlighting that we have strong, tuned baselines.
B.3 NUMERICAL RESULTS
C SSD OVERHEAD
Let B be the target batch size, K the speculation lookahead, and F' the (uniform) fan out factor for the draft model guessing verification bonus tokens.
24

---

Published as a conference paper at ICLR 2026
Model
Dataset
AR tok/s
SD tok/s
SSD tok/s
SSDISD
SSD/AR
Llama-3, I-Istrucc
70B/IB
HumanEval
54.7
176
283
1.60x
5.17x
Ultrafedback
54.7
138
215
1.55x
3,93x
Alpaca
54.7
145
224
1,55x
4.10x
GSM8k
54.7
301
1.60x
5.50x
Average
54.7
161.8
255.8
1,58
x.68
Qwen-3 32B/0.6B
HumanEval
88.8
146
222
1.52
2.50x
Ultrafeedback
88.8
122
174
1,43x
1.96x
Alpaca 88.8
127
185
1,47x
2.08
GSM8K 88.8
152
234
1.54x
2.64x
Average 88.8
136.8
203.8
1.49x
2.29
Compute. Let è be the compute required for draft forward pass, normalized by units of target FLOPs. Since the draft decodes BK(K + F) tokens per round SSD but BK tokens per round in SD, we incur factor of 6(K + 1)F more FLOPs on the draft relative to ordinary speculative decoding. Recall this is because each of the draft steps decodes B(K + 1)F tokens parallel using custom attention mask.
In much the same way that SD uses more FLOPs to achieve lower latency (see (Leviathan et al, 2023), Section 3.4), SSD applies the same philosophy latency than even SD itself. the SD setting, tokens speculated by the draft that were rejected by the target constitute wasted compute. In the SSD setting (in addition the above), we have that entire chains decoded pre-emptively in parallel for anticipated verification outcomes constitute wasted compute. Thus, SSD introduces new tradeoffs between compute and latency that were not possible before.
more FLOPs achieve lower
Memory. The draft model must build up a speculation cache as it speculates asynchronously. This has possible verification outcomes as keys and the corresponding tokens/logits as values. Concretely, this means storing tensor of BK K 1)F tokens that are decoded, the form of length- K speculations stored for B(K + F) possible verification outcomes. For each of these tokens, we also store logits of size V, so the speculation cache stores overall O (BFK(K + 1)(V + 1)) bits, ending up around hundreds of megabytes in practice. Given this cache refreshed every speculation round, this ends up small enough to be non-issue in practice, as the HBM on modern GPUs much larger.
Communication. The draft and target model synchronize once per speculation round. The target model sends the outcome of the previous round of speculation (the number of accepted tokens and recovery token for each sequence, O O(B) bits of information). The draft sends back the newly speculated tokens ("cache hits") well their logits (which the target will need for verification). This is O(BVV) bits of information. All communications are device to device over NCCL via NVLink, which fast enough that communications are bottleneck in practice.
D EXTENDED RELATED WORK
SSD complementary to other lines of work that accelerate LLM inference through orthogonal means: hardware- algorithms like FlashAttention (Dao et al. 2022). KV-cache management
and paging (Kwon et al.] 2023 ZZhan et al.) 2023; Hooper et al. 2024). sparse or approximate
attention (Beltagy al.] 2020; Zaheer al. 2020), and multi-token tree-based draft methods
(Cai et al. 2024] Fu et al. 2024).. SSD targets different bottleneck-thh sequential dependence
between drafting and verification- and can principle be combined with any of these.
E COMBINING SSD WITH SD VARIANTS
EAGLE-3. EAGLE-3 Li et al. 2025b) improves the acceptance rate a by training the draft to condition on target activations extracted during the previous round's verification. When target acti- vations are unavailable -pecuulau,,,on, the draft conditions on its own activations as a surrogate, with acceptance degrading slowly as self-generated activations used.
25

---

Published as a conference paper at ICLR 2026
This is entirely compatible with SSD, with nuance: since the SSD draft begins speculating round T+11 pre-emptively, it does not yet have target activations from round T (unlike standard EAGLE-3, which waits for them). With lookahead K, the draft thus conditions tokens, lowering acceptance on the latter K. This can he mitigated by training the EAGLE-3 draft preserve acceptance over longer periods of self-conditioning. This is depicted in Figure
up to 2K self-generated
EAGLE3
Targer activation
Ver fied prefix
Round T-1 Speculation,
Round T
toxen
SSD-EAGLES
Drafted token
Round Speculation concurrently being verified
Round Speculation loking place
takens
profx
Figure 9: Comparison of activation conditioning in EAGLE-3 vs. SSD-EAGLE-3, In standard EAGLE-3 (bottom), the round T' draft conditions almost entirely on target activations extracted from the already-verified round 7--1. In SSD-EAGLE-3 (top), round T speculation begins before round T-1 verification completes, so the draft must substitute its own activations (light nodes) for the not-yet-available target activations. Conditioning more draft activations can degrade quality of EAGLE speculations if the draft model is not trained to use draft self-conditioning effectively.
In our implementation, the target model sends the number of accepted tokens per sequence well as the target activations at each relevant token position to the draft over NCCL for the draft model condition when speculating.
Token-Tree Methods.
Token-tree SD methods (Miao et al. 2024 Chen et al. 2024
Li et al. 2024b
2025b) can be combined naturally with SSD. SSD drafts linear chain for many possible verification outcomes. To combine SSD with these token-tree approaches, one would simply pre-speculate (and verify) token tree for each verification outcome instead of token chain. practice, this requires more elaborate attention mask than that presented in Figure
F ADDITIONAL. EXPERIMENTS
Here, we reproduce similar trends from the main text on the Qwen3 model family, demonstrating that our results and algorithms are model and dataset agnostic.
26

---

Published as a conference paper at ICLR 2026
Rejection rate
Rojection
Pre(F)
1000
out
T-1.2 6.40
T-d. cmh
T-d.3. 1.20..1.mmm
10-2
Figure 10: Rejection rate and cache hit rate scaling for Qwen-3 model family. Like the Llama-3 model family, we see it is an approximate power law in the fan-out, that cache hit rates increase steadily as we grow F, the size of our cache.
Qwen3 End to End Speedups
250
2.6x
2.5x
2.3x
200
2.1x
2.0x
tokis
150
HumanEvalUltraaF
Alpaca
GSMBk
Average
AR (Ours)
SD (Ours)
SSD (Ours)
SGLang (SD)
SGLang (EAGLE-3)
Figure 11: End-to-end decoding speed comparison of SSD compared to SD and standard autore- gressive decoding on Qwen-3 32B across four datasets.
0.9
Decode Speed
Cache Rate
0.8
150
100
Temperature
Temperature
SD
SSD (Uniform)
SSD (Geometric)
Figure 12: End-to-end speed for Qwen-3 model family compared synchronous baselines.
27

---

Published as a conference paper at ICLR 2026
200
Fast Backup
Decoding Speed per Sequence Mkens/sec/bate/
Neural Backup
175
Sync Baseline
150
125
100
1
2
4
16
Batch Size
Figure 13: Batch size scaling of Qwen-3 models compared to synchronous baselines.
28


Reviews:
[{"review_id": "ngmxXBOhde", "reviewer": "Reviewer_kXox", "summary": "This paper proposes to parallelize the draft and verifier models in speculative decoding by running them on separate hardware devices. The proposed approach demonstrates superior performance compared to traditional speculative decoding in the reported experiments.", "questions": "Could you improve the paper’s structure to make the algorithm easier for readers to follow?\nAdditionally, could you add more experiments to address the weaknesses noted in points 2 and 3, particularly by including comparisons with other speculative decoding methods and providing more detailed experimental setups?\nFinally, could you clarify how your method preserves the original distribution, both conceptually and mathematically?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1.\tThe proposed method outperforms standard speculative decoding (SD) in the reported experiments.\n2.\tIntroducing parallelism between the draft and verifier models represents an interesting and promising direction for advancing speculative decoding.", "weaknesses": "1.\tThe algorithmic description is difficult to follow in the current paper organization. It is unclear how the proposed method preserves the original distribution, and while the paper provides extensive theoretical analysis, it offers limited explanation and intuition about the algorithm itself, particularly in Section 3.\n2.\tThe paper lacks baseline comparisons against other speculative decoding (SD) approaches, making it hard to assess the relative performance and contribution of the proposed method.\n3.\tThe paper does not include a dedicated experimental section—experiments appear only as a subsection—and there is insufficient detail regarding the setup, such as which models were evaluated and under what conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jJGdasIu4h", "reviewer": "Reviewer_h3mq", "summary": "This paper introduces Speculative Speculative Decoding (SSD), a framework that parallelizes the drafting and verification stages of speculative decoding by having the draft model predict likely verification outcomes in advance and pre-speculate for them. The authors propose SAGUARO, an optimized SSD algorithm that addresses three key challenges: accurately predicting verification outcomes, balancing cache hit rate with token acceptance rate, and handling cache misses efficiently. SAGUARO achieves up to 2x speedup over standard speculative decoding and up to 5x over autoregressive decoding on open-source inference engines.", "questions": "1. I would like the authors to provide the wall time for their method. For example, while the target model is performing Verify, how many tokens can the draft model generate in that same amount of time?\n\n2. I would like to know the number of accepted tokens for your method. Compared to EAGLE3, does your method accept more or fewer tokens per round?\n\nI will adjust my score based on the author's rebuttal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a principled and theoretically grounded approach to overlapping drafting and verification, effectively eliminating drafting latency through pre-speculation.\n\n2. SAGUARO introduces practical and novel techniques—geometric fan-out allocation, cache-aware sampling, and adaptive fallback strategies—that collectively address core challenges in SSD.\n\n3. The evaluation is comprehensive, covering diverse models, datasets, temperatures, and batch sizes, and shows significant end-to-end speedups while maintaining compatibility with existing acceleration methods.", "weaknesses": "1. The authors do not compare their method with existing SOTA methods (e.g., EAGLE2/3). They argue that their method is orthogonal to EAGLE and can be combined with it. However, the paper does not report the performance of their method when combined with EAGLE3. I would like to know what performance improvement could be achieved if they were combined.\n\n2. The authors do not compare their method with existing Token Tree SPD methods, which I believe is a missing baseline. The proposed method (SSD) has the draft model send only a single speculative sequence to the verifier, and while the verifier validates that sequence, the draft model computes other possible branches in parallel. In contrast, Token Tree SPD methods have the draft model generate an entire \"tree\" structure at once, providing multiple candidate tokens at each position, and the verifier must validate all branches and nodes of this tree in a single forward pass. I consider these to be two similar solution approaches, and therefore the authors should compare these two types of methods. For example, they could replace the Token Tree SPD component in EAGLE3 with their SSD method and report whether the performance increases or decreases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TCjWacTlW6", "reviewer": "Reviewer_Jb3k", "summary": "This paper introduces Speculative Speculative Decoding (SSD) — a new framework that parallelizes both speculation and verification in a distributed manner. While typical SD alternates executions between drafter and verifier, SSD removes this dependence and performs drafting and verification concurrently and asynchronously. In SSD, while verification for round T is ongoing, the draft model predicts the likely verification outcomes and pre-speculates corresponding continuations.", "questions": "- While the deployment of LLMs typically runs on $2^n$ GPUs, SSD actually requires $2^{n} + 1$ GPUs. How can users deploy the SSD for serving efficiently? e.g. if using homogeneous GPUs, using a Hopper GPU for only the small draft model can be an overkill as much of the compute and memory are not saturated. If using heterogeneous GPUs (i.e. running the speculators on another type of GPU), how much impact will the communication between speculators and verifiers be?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- This paper has novelty in terms of serving the draft and target models asynchronously.\n- This paper has clear derivations and proofs (Theorem 7, 12, 17), which make the framework analytically sound.\n- The author has conducted extensive benchmarks on multiple model families, detailed ablations (fan-out, sampling, fallback), and clear visualizations.", "weaknesses": "- The flow of the paper can be improved, e.g. Section 4 shows the results first and then explains the algorithms\n- Can give a better flowchart of the workflow in Figure 1, e.g. label the step index\n- This method requires an independent GPU for the speculator, this can increase the cost of serving and system complexity (e.g. heterogeneous GPUs)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C8S09m39JD", "reviewer": "Reviewer_QxMH", "summary": "This paper introduces ​Speculative Speculative Decoding (SSD)​, a novel framework that parallelizes the drafting and verification phases in speculative decoding. By pre-speculating likely verification outcomes during ongoing target model verification, SSD eliminates the sequential dependency between drafting and verification, achieving significant latency reductions.", "questions": "1 How does SAGUARO’s memory footprint scale with batch size and fan-out? Could cache eviction policies improve efficiency?\n\n2 Could the pre-speculation idea be combined with approximate drafting (e.g., EAGLE-3) for further gains?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1 The idea of speculating on speculationis both creative and impactful. By leveraging idle draft-model compute during verification, SSD addresses a fundamental bottleneck in existing speculative decoding methods. The theoretical framework (Algorithm 1) is elegant and generalizable.\n\n2 The paper provides strong theoretical analysis.\n\n3 Comprehensive Empirical Validation.", "weaknesses": "1 While SSD reduces latency, the parallel speculation across multiple branches increases draft-model FLOPs. The paper should quantify this overhead more explicitly.\n\n2 Although comparisons to standard SD are thorough, benchmarking against recent parallel SD methods (e.g., SwiftSpec) or tree-based decoding (e.g., SpecInfer) would strengthen the claims. Does SSD outperform these methods in high-batch or high-temperature regimes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)