Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling
Authors: Yongchang Hao
Lili Mou
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
Speculative sampling
Auto-regressive generation
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose Cactus, a speculative sampling method that guarantees controlled divergence from the verifier distribution while increasing throughputs.

Abstract: Speculative sampling (SpS) has been successful in accelerating the decoding throughput of auto-regressive large language models by leveraging smaller draft models. SpS strictly enforces the generated distribution to match that of the verifier LLM. This is unnecessarily restrictive as slight variations of the verifier's distribution, such as sampling with top-$k$ or temperature, would also be acceptable. Typical acceptance sampling (TAS) alleviates this issue by accepting more tokens using entropy-based heuristics. However, this approach distorts the verifier distribution, potentially degrading output quality when the verifier encodes critical information. In this work, we formalize the speculative sampling algorithm through the lens of constrained optimization. Based on this formulation, we propose Cactus (**c**onstrained **ac**cep**t**ance spec**u**lative **s**ampling), a method that guarantees controlled divergence from the verifier distribution and increasing acceptance rates. Empirical results across a wide range of benchmarks confirm the effectiveness of our approach.

Published as a conference paper at ICLR 2026
CACTUS: ACCELERATING AUTO-REGRESSIVE DE- CODING WITH CONSTRAINED ACCEPTANCE SPECULA-
TIVE SAMPLING
Yongehang Hao Lili Mou
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta
Canada CIFAR AI Chair
Longchal@ualbeere
doublepower honpllppoerr.mueggmeee com com
ABSTRACT
Speculative sampling (SpS) has been successful in accelerating the decoding throughput of auto-regressive large language models by leveraging smaller draft models. SpS strictly enforces the generated distribution to match that the verifier LLM. This is unnecessarily restrictive as slight variations of the verifier's distribution, such as sampling with top-k or temperature, would also be acceptable. Typical acceptance sampling (TAS) alleviates this issue by accepting more tokens using entropy-based heuristics. However, this approach distorts the verifier distribution, potentially degrading output quality when the verifier encodes critical information. In this work, we formalize the speculative sampling algorithm through the lens of constrained optimization. Based on this formulation. we propose Cactus (constrained acceptance speculative sampling), a method that guarantees controlled divergence from the verifier distribution and increasing acceptance rates. Empirical results across a wide range of benchmarks confirm the effectiveness our approach. The code is publicly available
INTRODUCTION
Auto-regressive large language models (LLMs) have driven remarkable advances in machine learning and artificial intelligence (Vaswani et al. 2017 Brown et al.) (020) Kaplan et al.) 2020). yet their growing size comes with steep computational costs: generating each token requires memory-bound forward pass through hundreds of billions of parameters, which bottlenecks LLM
throughput (Yuan al. 2024). Speculative sampling (SpS) addresses this by first using a smaller draft model to propose certain number of candidate tokens autoregressively. then verifying the candidate tokens in parallel with the large-scale verifier LLM (Stern et al., 2018 Xia et al.) 2023] Leviathan et al. 2023 Chen et al.) 2023). Since SpS can emit multiple tokens per large-model invocation, it substantially speeds up auto-regressive generation by alleviating the memory-bound issue.
Despite its success, SpS enforces strict distributional equivalence with the verifier, causing correct but lower-probability tokens be rejected. In real-world applications, exact adherence to the original distribution is generally not required
(Holtzman et al., 2020 Meister et al., 2020). Typical acceptance sampling (TAS; Cai et al. 2024) mitigates this issue by accepting proposals based entropy-driven heuristics (Hewitt et al. 2022
Meister et al. (023). However, we show in this paper that TAS improves acceptance rates at the cost of distorting the verifier's output distribution and risking semantic drift when the verifier encodes critical information.
In this work, we reformulate speculative sampling as a constrained optimization problem, explicitly trading off acceptance rate against divergence from the verifier's distribution. Guided by this theory, we introduce Cactus (constrained acceptance speculative sampling), simple yet principled modification that enforces hard constraint on distributional divergence while enabling higher acceptance rates.
htttp::///h wwtp:://o.oooaaaoaaaaaa httere////tiiiicco///aaaaaa////r

---

Published as a conference paper at ICLR 2026
Algorithm 1 Generalized formulation of speculative sampling
Require: sampling steps draft model p, acceptance rate o, and recover distribution
1: f 1, -- [BOS]
2: while not end do
3: D Drafting m tokens
4: for < 0, m. do
5: Fiti ~ p(-l(eeet
> a cetti is concatenation of Z<t and ( C++i
6:
u4 ~ 10 end for
(O, 1) is the uniform distribution between [0, 1]
7:
8:
c o min : uj
D c the length of accepted draft tokens >> Tite is sampled from the recover distribution
9:
Zitc g(-1xx<tt)
t ++++
11: end while
We conducted experiments on a wide range of benchmarks with multiple state-of-the-art large language models. Results show that Cactus consistently improves generation throughput compared with the lossless SpS. In addition, Cactus preserves the generation quality and diversity of the verifier model, due to its explicit divergence constraint.
APPROACH
We first provide a generalized formulation for speculative sampling. This enables a theoretical analysis of speculative sampling under constrained optimization framework. Based on this analysis, propose algorithm, Cactus, which provably approximates the verifier distribution q while achieving higher acceptance rates.
2.1 GENERALIZATION SPECULATIVE SAMPLING
Speculative sampling. The vanilla speculative sampling (SpS; Chen et al. 2023) uses draft model p()(x<t) that has a significantly smaller memory footprint than the verifier model q( (())==))- At time step t, SpS repeatedly samples E N4 tokens Ia...., 31+m-1 from in an auto- regressive manner. Then the verifier evaluates the probabilities of the tokens and calculates the acceptance rafe ((0+45 ((0<++)))
[0, m). If any token Te+s is rejected, then tokens 2i+j+1,... (ttmm are also discarded. As backup, SpS resamples Tt+- using the recover distribution (((4+)/x<<())) x (g(- (o(-(=<14))- (0f-<44))((((2eeeee.ee p(-1(x<c+)))+++ where (+)+ denotes max(0,-) -). The final accepted tokens are Tt. ..,, Zt+jj. By this draft-and-verily scheme, SpS accelerates regr-segeese decoding by avoiding the need load the large verifier model q from memory at every step. This approach has been shown effective in practice (Zhou et al.) 2024 Hu
min 0(===+=((==c===//p((((z+_)+ [x=<)+)) for all
ct al. 2025).
We formalize the draft-and-verify scheme Algorithm | vanilla SpS algorithm (Chen al. 2023) produces any target distribution with an optimal acceptance rate.
Under this setting, we can show that the
Observation 1. Consider any desired target distribution h and draft model Algorithm 7 produces the target distribution h exactly if the acceptance rate and recover distribution are defined as
((z1(0<)) min h1 (()) p( cc)) 1
and g(x.(aee)) h(()((er)) - p( ( (())) Ex'=pp1 o(x (
respectively. addition, this acceptance rate o is optimal for achieving the highest acceptance rate.
Proof. See Appendix A.1
2

---

Published as a conference paper at ICLR 2026
2.2 APPROXIMATING SPS CONSTRAINED OPTIMIZATION
Observation | provides a foundation to produce an arbitrary target distribution with the optimal design under the draft-and-veriff scheme. Our insight is that, instead of producing fixed verifier distribution q as the target distribution h like SpS. we may utilize this observation to dynamically select distribution h close q while yielding higher acceptance rates based on function Φ. This can be formulated as constrained optimization problem.
For each step 4, assume the drafted token has index n. Let h E IRV be the parameters to be optimized. The ideal h is given by h())æx))) hī where ht is the solution of the following problem:
max h
min minnha//p(laaaa
(3)
s.t. IV--1 Dy(hila((arre))) S 8.
(4)
(5)
Here, the hyper-parameter 2 0 controls the closeness to the verifier model and D, is any f- divergence metric used measure the distance between q and h. Once the optimal found, can then derive the corresponding ø and by Observation
The above formulation falls into the framework of constrained convex optimization, which we show has the following solution.
Theorem 2. The optimal solution of h in objective is
=i=n otherwise,
h4
(6)
1]. The function f is the one used in the T-g(a (()
where 7 is any root of the equation
1-7
= g(nlx<())
g(m)eze)) + +(1 g(nl2==e)) ((=(((<))))
(7)
over the interval [g(n| +oo), clamped into definition of f-divergence.
Proof. See Appendix A.2
Theorem 2 theoretically characterizes the trade-off between closeness the verifier model q and the acceptance rate induced by é. In particular, the theorem suggests that the drafted token now has at least the same higher chance of being accepted (since 7" 2 9n)). For other non-sampled tokens, their probabilities are scaled down proportionally so that h remains valid distribution.
worth noting that, since the solved in Equation (6) depends on the sampled token n, the solution is different for different sampled tokens. As result, the effective distribution of the overall algorithm halg might have a divergence other than 8 from the target distribution q. To this end, we provide the following theorem to guarantee the controlled divergence of the effective distribution.
Theorem 3. Let on and 9u denote the functions that follow the solution in Theorem when the sampled token n. The distribution of the overall algorithm given by
halg
Σ p(n(ace) (n)en +1- (n))g.., IElIVII
(8)
where en is a one-hot vector with only non-zero element at index n. In addition,
(hate (q(()o=))) S min{T(o), Dy (x -
(9)
for any 2 O. Here, the function T: : [0, +00) - [0, + is continuous and non-decreasing in with value of 0 at 8 0.
Proof. See Appendix A.3
3

---

Published as a conference paper at ICLR 2026
In essence, despite the h in Equation is solved specifically for the sampled token the divergence between the overall distribution and the target distribution is still implicitly controlled. In particular, for any target divergence 0 S Oalg < +oo imposed the overall algorithm, we can always find a proper 2 such that (h (l S (()) S dagg- While I does not admit closed- form expression, 5 itself is hyper-parameter. In practice, quality-trooghput trade-off. This confirms the soundness of our framework.
can tune ő to achieve the desired
In fact, our framework also offers a novel theoretical interpretation of typical acceptance sampling.
Proposition 4. Typical accepiance sampling (TAS: Cai et al. 2024 implicitly solves a variant of the optimization problem in objective (3), where the f-divergence is substituted with the cross-entropy (h. ())=x))....
Proof. See Appendix A.4
The suboptimality of TAS arises from the nature of cross-entropy. Specifically, the cross-entropy can be decomposed as
Hh, gl ((())))))) = DKL (blllg((( + ((h).
(10)
Mode capturing
Certainty
Here, the KL divergence encourages to focus on the mode of q (since h the first argument), while the entropy term encourages h to be deterministic. However, the summation allows h collapse into a deterministic distribution at the expense of increasing divergence, therefore failing to capture the full shape of q. In fact, TAS always yields h with entropy 0 while increasing the divergence by least H (g). As a result, the produced distribution may diverge significantly from the verifier model, especially when q carries high entropy and thus rich information.
2.3 CACTUS: CONSTRAINED ACCEPTANCE SPECULATIVE SAMPLING
Based on our analysis above, we propose to use only the KL divergence as the measure of "distance". Specifically, this corresponds to the function f(t) log t. Combined with our Theorem 2 y" is the root of
((nlaze) ) + (1-g(n|(<<()) ((7) =
7
1-7
(11)
7 ) +(1 (1-7) log
1-7
log
((((pare))
(((((())))))
(12)
=6.
(13)
However, since t is transcendental function involving terms like æ log it cannot be solved in closed form. We therefore approximate / by its second-order Taylor series expanded at Yo
(7) (70) + ('00) 70) + 4² (r 20
(14)
2
This approximation is justified by noting that 8 is typically small and 7 remains close to g(n|æ<)).
Corollary 5 (Cactus's solution). Let the f-divergence in objective 3 be the KI. divergence. The solution to Equation (4) is given by
7"
fi=n,
h((l = I-q(n|aes)
- g(() otherwise,
(15)
where 7- min (9(n|0<1) + /20g(n) x())
Proof. See Appendix A.5
n
4

---

Published as a conference paper at ICLR 2026
In other words, Cactus modifies the distribution of the verifier model by increasing the probability of the candidate token n by a small "bonus" determined jointly by g(n x<ee) and 6. We further show that Cactus's 's solution is more conservative than the exact solution when the verifier is less confident, ensuring that it strictly satisfies the divergence constraint such cases.
Corollary 6. When the exact solution " is not greater than 0.5 (i.e. the token is not likely to be accepted), our approximation always satisfies the divergence constraint:
DE. (k S 6,
(16)
where (1(12 x2e)) is given by the approximated solution in Equation
15
Proof. See Appendix|A. A.6
It is easy to see that the bonus probability attains its maximum when g x<t))
0.5. In
practice, LLMs generally have more than 100K tokens (Dubey et al.) 2024 Qwen Team 2024).
so a probability around 0.5 indicates strong model confidence in the token. However, SpS could still reject the token solely because the draft model is overconfident (i.e., p(n|ar<e) is large). Cactus increases the acceptance likelihood in such scenarios by modifying the verifier distribution accordingly.
Compared with TAS's criterion function, Cactus only requires reading the probability at token rather than accessing the full vocabulary. This allows Cactus to further reduce memory access overhead, especially in surge-vocabular settings. More importantly, Cactus 's divergence is tightly controlled with minimal entropy change, whereas TAS yields only low-entropy solutions.
3 EXPERIMENTS
3.1 SETTINGS
Datasets. We evaluated Cactus on three popular benchmark datasets for large language models: (I) The GSM8K (Cobbe al.] 2021) test set contains 1.3K high-quality grade school math word problems, designed to assess a model's ability to apply mathematics to real-world scenarios. Following common practice in LM-Eval (Gao al. 2024), used 5-shot examples for each test instance. The final accuracy score is averaged over all samples. (2) The IFEval (Zhou et al.
2023) benchmark measures instruction-following ability. It consists of 500 "verifiable instructions" whose outputs can be heuristically evaluated. For example, a prompt might be: "Write a blog post with 400 or more words about the benefits of sleeping in hammock," which can be automatically checked by counting the number of words. (3) The GPQA (Rein et al. (202)) diamond benchmark includes approximately 200 challenging science questions authored by domain experts, designed to test models' scientific knowledge. For instance, sample question is: "The angular size of the event horizon of a supermassive black hole the centre of galaxy parsecs is measured to be 0 10-17 degrees. Find the order of magnitude of the entropy of the black hole." " Following common practice (Gao et al. 2024), we format the prompts to include four multiple-choice options. Models are then evaluated by generating chain-of-thought (Wei et al. 2022 followed by the final answer.
distance of d 10
Evaluation metries. For all three tasks, the results are extracted from the generated text by regex matching with the corresponding format. These results are then compared with the gold labels using strict-match accuracy (i.e.. if the strings are identical and otherwise). Final scores are obtained by averaging the accuracies over all samples. Following previous work (Dubey et al. (024), the regex for GSM8K and GPQA is the "flexible-extraet" pattern, which selects the first number the generated sentence regardless of whether the model adheres to the few-shot examples. For IFEval, we use the woouunt--eee-sstiit---eeeeee-eee model to strictly follow all the instructions.
regex as defined in Qwen Teaml (2024), which requires the
In addition to task scores, we report the average acceptance length (AL) for all runs. Specifically, ALm refers to the expected number of accepted tokens among drafted tokens. A higher ALm. generally indicates faster generation. However, a method may artificially inflate AL by accepting low-quality draft tokens that lead to longer, less efficient chains of thought. Although AL remains
5

---

Published as a conference paper at ICLR 2026
The results on three benchmarks: GSM8K, IFEval, and GPQA. We report the "strict- match" accuracy as the score with the standard regex pattern for each task. ALm indicates the number of accepted tokens when the draft length m. Rej denotes the total number of rejected tokens throughout generation in relative scale, where we use the SpS runs as the reference (labeled as "Ref").
Table 1:
(a) The results of Qwen 3 8B verifier and Qwen 3 0.6B drafter.
GSM8K
IFEval
GPQA
Name
Scoret
AL
Rej+
Score
AL
Rej
Score"
AL
Rej*
Verifier
84.31
84.66
41.07
SpS
83.78
4.49
Ref
84.66
2.59
Ref
40.91
3.70
Ref
TAS
86.58
5.49
-29%
85.40
3.28
-27%
41.41
-42%
10
Cactus 0.75
85.97
5.65
-34%
85.03
3,40
-31%%
41.42
5.33
-47%
Cactus 1.0
86.35
5.72
-37%
84.10
3.44
-32%
39.39
5.44
-48%
SpS
84.46
5,44
Ref
84.10
2.74
Ref
42.93
4.23
Ref
TAS
85.51
7.23
-35%
84.10
3.77
-29%
38.89
6.68
-46%
20
Cactus 0.75
86.66
7.50
-37%
85.95
3.76
-30%
40.01
6.89
-47%
Cactus 1.0
86.43
7.61
-39%
84.84
4.05
-33%
39.90
7.05
-49%
(b) The results of Qwen 3 14B as verifier and Qwen 0.6B as drafter.
GSM8K
IFEval
GPQA
m
Name
Score
AL
Rej
Score"
ALL.
Rej
Score
AL
Rej*
Verifier
91.71+0.52
85.09
40.07
SpS
91.12
4,27
Ref
85.03
2.19
Ref
39.39
3.37
Ref
TAS
92.65
5.24
-30%
86.14
3.00
-25%
38.89
4.99
-46%
10
Cactus 0.75
92.12
5.35
-31%
86.87
3.04
-29%
44.95
5.14
-50%
Cactus 1.0
93.10
5,44
-32%
85.96
3.03
-30%
43.43
5.16
-51%
SpS TAS
91.89
5.11
Ref
84.47
2.27
Ref
40.91
3.84
Ref
92.87
6.78
-32%
85.03
3.49
-27%
40.40
6.41
46%
20
Cactus 0.75
92.15
7.15
-36%
86.69
3.45
-30%
45.46
6.46
-46%
Cactus 92.87
7.00
-34%
86.32
3.60
-30%
45.46
6.74
-50%
high in this case, the behavior lead to lower overall throughput due to unnecessarily lengthy outputs. To present a more complete picture of generation efficiency, we also measure the number of rejected tokens during generation, which reflects both the acceptance rate and the total length of generation.
Implementation details. We used the Qwen 3 series as our main testbed for two reasons: (1) the models come variety of sizes, ranging from 0.6B to 14B parameters, enabling wide range of choices of model pairs; (2) the models are trained to generate with internalized chain-of-thought
reasoning (Wei et al. 2022), which makes them a natural use case for speculative sampling given the longer generation lengths (Yang et 2025bb. For all experiments,
used the recommended generation parameters (Yang et al. 2025a), where top-p set to 0.95, top-k is set to 20, and temperature is 0.6.
3.2
MAIN RESULTS
As shown in Table speculative sampling (SpS) serves strong baseline that closely preserves the output distribution of the verifier model. Across all three benchmarks (GSMSK, IFEval, and GPQA), SpS maintains similar accuracies to the verifier (e.g., 84.46 vs. 84.31 on GSM8K with m = 20 Table Ia and 91.89 vs. 91.71 in Table lb]. This aligns with the theoretical claim that SpS is nearly lossless in generation quality. Additionally, the number of accepted tokens (ALm) ) for SpS reaches 5.44 on GSM8K and 4.23 on GPQA with m = 20, indicating that the verifier model invoked less frequently.

---

Published as a conference paper at ICLR 2026
Ya Fix
Yo Rio 20
323
0:68 145
Coctue 0.73
Coctus 1.0
0.68 88 GSMBK
0.68
0.08
GPOA
0.08
GSMBK
14B GPO4
AV
AccorsO (6)
Average Length
Length
Average
Length
(AL)
Figure Accuracy-aceeptanee across benchmarks and model settings. The a-axis shows the average accepted length (AL), and the y-axis shows accuracy normalized by the standard deviation from the verifier.
Typical acceptance sampling (TAS) outperforms SpS in terms of acceptance rate, achieving more accepted tokens and lower rejection rates. For example, on GSM8K with m = 20, TAS improves ALm from 5.44 to 7.23 (Table Tap and reduces the rejection rate by 35%, which is consistent with our approximation analysis in Section 2.2] However, TAS often introduces distributional shifts that degrade performance. For instance. GPQA in Table Ia TAS yields lower accuracy than SpS (38.89 vs. 42.93), likely due to accepting plausible yet suboptimal tokens, especially when the verifier distribution contains fine-grained decision signals.
By contrast, our proposed method, Cactus, achieves the highest acceptance rates across all benchmarks while maintaining or improving accuracy. When ó 0.75, Cactus consistently surpasses both SpS and TAS in AL while achieving strong accuracy, e.g., score of 86.66 on GSM8K with m 20 (Table la and 45.46 on GPQA with m 20 (Table Tbp. notably outperforming all baselines. When d = 1.0, Cactus further increases ALm. to 7.61 on GSM8K with a score 86.43 (Table Tap. or to 7.00 with score of 92.87 using a larger verifier (Table (b). Notably, unlike TAS, Cactus does not degrade performance on challenging benchmarks such as GPQA. Instead, achieves both high acceptance rates and stable accuracy, validating its theoretical foundation in constrained optimization and demonstrating practical robustness across diverse tasks.
3.3 IN-DEPTH ANALYSES
Accuracy against acceptance rates. We visualize the accuracy-acceptance trade-off in Figure ! where accuracy is measured in standard deviations (r) from the verifier mean, and throughput is quantified by the average accepted length (AL). Each subplot corresponds to a specific benchmark and verifier-drafter pair. The dashed black line indicates the verifier performance, and the red dashed line marks the -o threshold, a commonly used indicator of statistically significant degradation.
As shown, TAS improves throughput over SpS but often suffers from accuracy drops, notably falling below the - 1a threshold on GPQA with the 8B verifier (m = 20). In contrast, Cactus consistently preserves accuracy (remaining within above the verifier confidence range) and frequently exceeds it, such as on GSM8K and IFEval with both 8B and 14B verifiers. This demonstrates that Cactus effectively improves decoding efficiency without compromising (and sometimes even enhancing) generation quality.
It is also worth noting that the improvements from Cactus are stable across tasks with different characteristics. For instance, on challenging benchmarks like GPQA, where other methods either exhibit significant degradation (e.g., TAS) or achieve limited throughput gains (e.g.. SpS), Cactus substantially increases AL while maintaining accuracy above baseline. This highlights the strength of our constrained acceptance framework balancing aggressive token acceptance with distributional fidelity.
The importance of divergence control. Our Cactus dynamically manipulates the target distribu- tion to increase the chance of accepting the sampled tokens. Since this inevitably pushes the target distribution h from the verifier q to be more similar to the draft model p. it resembles the notion argue that Cactus is superior
of mixing the distributions g and p by interpolation. However, to simple interpolation, given that it uses a principled approach which comes with a divergence guarantee. We empirically justify this argument by the following experiment.

---

Published as a conference paper at ICLR 2026
GSMBK
IFEvl
GPQA
85
80-
B2-022
Ston
Score
CY075
Ied
30-
73
Cartue dln
Cacta
25
keepp
Coctue plog
(Eokoy
30
70
80
90
40
6U
8U
40
50
60
70
01
90
100
Accestance Rate IAR.
Acceptance Rate (AR, %)
Acceptance Rate (AR, %6
Figure 2: Task score vs. acceptance rate for the 0.6B+14B Qwen combination without top-p/top-k sampling arguments. Solid and dashed lines are 2egree-2 polynomial fits.
Verifier
Sp5
TAS
Coctus
Spoodup us Verfeer (1 la fasten
1.4
1.2
1.5
0.8
05
0.6
0.6
5p5
Cacus 0.75
Cartos 1.0
5p5
1A5
Cactus 0.75
Cactus
Sp5
Cactos 0.75
Cactus
Sp5
Cactos 0.75
Cactus 1.0
Figure 3: Wall-time normalized throughput (y-axis) across different model sizes and draft lengths. The wall time of single verifier model is always normalized 1.
Here, we produce data points by running grid search on å for Cactus and interpolation rate for interpolation, respectively. As shown, Cactus consistently outperforms interpolation the similar acceptance rate. For example, at similar acceptance rate of approximately 90%, Cactus achieves score above 86 (8 1e4) on GSM8K, whereas interpolation only reaches a score below 72 (a = 0.9). Notably, even at 96.3% acceptance rate, Cactus maintains a higher score (above 80), further confirming the necessity of divergence control in our method.
Throughput comparison. In Section 3.2, we used ALm and Rej proxies for throughput, as they are invariant to hardware and system conditions. Here, we additionally report wall-time results, all measured on A100 40GB GPUs with identical CPU and memory configurations. We used VLLM (K et al. 2023) with its default compilation settings to ensure realistic inference conditions. The results on GPQA are shown in Figure
Across all settings, Cactus remains competitive or superior to all baselines. In particular, Cactus 0.75 and 1.0 yield significant improvements in the 0.6B+14B setting, where Cactus 1.0 achieves nearly 1.9x speedup over the verifier alone with m 10, while also maintaining the highest score on GPQA (see Tb). In contrast, TAS slightly underperforms Cactus nearly settings. Notably, as discussed in Section 2.2 and verified in Table [b] TAS lacks explicit divergence control. These results highlight the benefit Cactus's 's constrained acceptance strategy, which more effectively balances fidelity and efficiency than existing baselines.
Evaluating on more model series. To assess the generality of our method, we go beyond Qwen and evaluate three additional model series: Gemma (2B 9B, Gemma Team (2024)), DeepSeek R1 (1.5B + 7B, (uoo et al. (2025), and LLaMA (1B 8B, Dubey et al. (2024)). Each model pair represents a distinct series developed by different teams with varying training methodologies. Tollowing/Bachmannn Bachmann et al. (2025). we additionally evaluate top-k decoding a naive lossy baseline, where draft tokens are accepted if they fall within the top-5 candidates according to the verifier. All drafter-verifee pairs follow the same speculative decoding setup, and accuracy is measured with standard task-specific metrics. We also include SpS and TAS baselines under equivalent configurations to ensure fair comparison. The results presented in Figure
8

---

Published as a conference paper at ICLR 2026
Verfter
Top
3p5
TAS
Coctus
Germaa 20 +
R1 1:50 +0
Unma 13 80
80
45
25
50
565
TAS
Cartas 0.75
Caruaa 1.0
5p5
1AG
Casta 0.79
Caruo
Spp
745
Cacua 0.79
Cacuus 1.0
Figure 4: Evaluating on GSM8K with three model pairs.
Table 2: The results of Qwen 3 32B as verifier and Qwen 3 1.7B as drafter on three benchmarks: GSM8K, IFEval, and GPQA. We report the "strict-match" accuracy and the acceptance length (AL).
GSM8K
IFEval
GPQA
Name
Scoret
AL
Score
AL
Score"
AL
SpS TAS
95.30
5.03
83,36
2.61
40.40
3.73
10
94.10
7.02
83.73
4.16
40.40
6.12
Ours (6 1)
94.40
7.13
85.21
4.47
41.92
6.36
Top-k decoding consistently underperforms the verifier model, reaffirming the importance of using principled verifier-guided sampling like Cactus. Across all settings, Cactus delivers strong and consistent performance. For DeepSeek RI and Gemma, Cactus notably outperforms TAS. While SpS and TAS perform well on LLaMA, Cactus matches their accuracy and retains its robustness across models. These results support the conclusion that Cactus generalizes well across diverse architectures and remains competitive superior regardless of the underlying model series.
Scaling to larger models. To evaluate the scalability of our method under more inntere conditions, we conduct exnductexperrmentt on larger model pair: Qwen 1.7B (drafter) and 32B (verifier). This setting involves significantly higher parameter counts than the reported 14B maximum in the main table, serving to verify performance where memory bottlenecks are typically more prominent. We maintain the standard speculative decoding setup with draft length of m 10 and report both accuracy and acceptance length (AL).
As shown in Table 2 Cactus demonstrates superior efficiency (achieving the longest acceptance lengths) across all three benchmarks. and SpS on IFEval and GPQA, while achieving comparable result GSM8K. These findings confirm that the effectiveness of Cactus naturally extends to larger models, delivering consistent improvements in acceptance rates while maintaining accuracy.
terms of task performance, it notably surpasses TAS
4 RELATED WORK
The draft-and-verify scheme. The most related line of work is the draft-and-verifyy scheme to accelerate auto-regresssiv decoding. The foundation of this scheme lies in the acceptance algorithms (i.e., designing the acceptance rate and recovery probability functions in Section 2). This includes vanilla speculative sampling (Chen al., 2023 Leviathan al. 2023) and typical acceptance sampling (Hewitt et al. 2022,MMese et al. 2023] Cai et al.) 2024). Cactus belongs to this category, as shares the same framework but utilizes a different acceptance strategy. For this reason, we extensively compare it against both methods. In addition to acceptance algorithms, building specialized models for this scheme has been shown be effective (Kim al., 2023) Liu et al., 2024 Liao et al., 2025). For instance, Cai et al. (2024) fine-tune multiple heads for generating subsequent tokens; Li et al. (2024b 2025) propose EAGLE, which introduces an additional head for draft token generation; Bachmann et al. (2025) propose Judge Decoding, training a binary classifier to augment the acceptance rate function. However, these methods require substantial training resources, whereas Cactus is training free acceptance rule. We also expect that Cactus can be directly applied to any method that utilizes either SpS or TAS as the underlying principle. Another generalization of speculative sampling involves using multiple draft tokens verifiers Yang al.

---

Published as conference paper at ICLR 2026
2024
Chen al. 2024 Jeon al. 2024 For example. Miao et al. (2024) propose SpecInfer with tree- beeebbaed draft generation; TreeBoN (Qiu et al. 2025 integrates speculative sampling into best- of-N tree-search decoding. We leave the exploration of more integrated versions of multi-drafter or multi-verifier Cactus to future work.
Low-complexity attention for Transformers. Transformer models generate sequences in auto-regressive manner (Vaswani et al.) 2017). Since each token attends to all previous ones, generation time grows quadratically with sequence length (Wang al.; 2020). To address this, previous work has proposed low-complexit attention variants (Child et al. [2199] Zaheer et al., 2020 Tsai al., 2019 Kitaev al., 2020 Choromanski et al. 2021). These methods modify the Transformer architecture itself. Cactus can be combined with these methods since they also follow the auto-regressive paradigm. In addition architectural changes, decoding complexity can also be reduced by manipulating the KV cache (Zhang et al. [0033] Li et al.) 2024a Cai et al.) (025). For instance, SnapKV (Li et al. 2024a) evicts less relevant tokens from the prompt before generation; Hao et al. (2025) Wu et al. (2026b) compress context with more efficient representation. These techniques are orthogonal to speculative sampling methods like Cactus.
Minimizing overheads of Transformers. Without approximating the Transformer architecture, overheads can still be reduced to accelerate decoding. Flash Attention (Dao et al., 2022 Dao 2024), for example, uses tiling techniques to avoid memory-bound operations, and has
widespread
adoption (Wolf et al. 2020 Kwon et al., 2023). Memory-efficient attention (Rabe and Staats 2021)
reorders computation
maintain constant memory usage regardless of context length. Another
line of work applies quantization to model parameters (Lin et al.] 2024) [adri and Shaji, 2023 Liu
al., 2025). The benefits are threefold: (1) reduced memory footprint due to lower-precision data types; (2) alleviated memory bottlenecks during decoding; and (3) improved hardware efficiency
optimized kernels. All these methods can be seamlessly integrated into speculative sampling approaches, including Cactus.
5 CONCLUSION
In this paper, we propose a constrained optimization framework for analyzing and improving speculative sampling methods. Building upon this framework, we introduce Cactus, novel training- free speculative sampling method that increases acceptance rates while maintaining a provably controlled divergence from the large verifier model. Cactus uses only basic element-wise operations, making it highly practical and lightweight for real-time inference. We empirically evaluate our method on a variety of benchmarks and confirm its effectiveness. As LLMs continue to grow in size and cost, our method provides theoretically grounded yet practically efficient solution for scalable deployment.
ACKNOWLEDGMENTS
We thank the reviewers and chairs for their efforts. The research is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Amii Fellow Program, the Canada CIFAR AI Chair Program, donation from DeepMind, and the Digital Research Alliance of Canada (alliancecan.ca).
ETHICS STATEMENT
We certify that all authors of this project adhere to the ICLR Code of Ethics (https://icle cc ht//public/CodeoFEttin
Our research does not involve human subjects, practices related to dataset releases, potentially harmful content, potential conflicts of interest and sponsorship, iisseriiiaation/biaaffaaa concerns, privacy and security issues, legal compliance, or research integrity issues.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
All of our experiments use publicly accessible datasets and models. Specifically, the datasets we used can be found at the following links via HuggingFace:
GSM8K:
IFEval:
GPQA: http:://uuguuggioc..oowuussstttr
The models can be found at the following links:
Qwen 3 0.6B:
Qwen 3 1.7B: https//hwngginiiircccooovvvvv//r
Qwen 4B:
Qwen 3 8B: http://hwggiiggaaaaccc///vvee//a
Qwen 14B: http:s/hnggggnggioccccooo/x////2
Qwen 32B: http://lagggnnngiec.cco/vvvvvv/2
Gemma 2B: https//wuuginngghcccoooooo////mn
Gemma 9B: http://wwggiingaaccoooooo//eemgn
DeepSeek R1 1.5B: http://wwwgnnnoocccoooe/o//////p
DeepSeek R1 7B: http:///waeennpoe.oooooppppoppop Qw
LLaMA IB: http:s///uuggiggggc...c//tttttts
LLaMA 8B: http://mgggnnniccccoooccoooll//m
In addition, our code is publicly available at https:/shhhiicccmmAAAAAAAAAAAAA
REFERENCES
Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas K Kohler. Judge decoding:
Faster speculative sampling requires going beyond model alignment. In ICLR, 2025. URL
httts:://ppenrvvile wwtppperevlew..ee//oooooeee otgpeprrweeeeee.eeeeeeoooorrooor
Hicham Badri and Appu Shaji. Half-quadratic quantization of large machine learning models.
Online manuscript, 2023. URL https:: httpss///rropoxx wwr.ox...iitii withh...... 10 http//hglllooooo//oooooooooooooe
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Herbert-Voss, Gretchen Krueger, T. Henighan, R. Child, A. Ramesh. Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S. Gray, B. Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, I. Sutskever, and Dario Amodei. Language models are few-shot learners. In www.eeewiiss.ccpppeeeeeeeee wttppapee//222//aaa.....
NeurIPS, 2020. URL https::/prroeeedingg 1457c0d6bfeb496744 otg.a//oo.....1........222-----2
.hmll
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In ICML, 2024. URL
mlr wwwpppeessvveee /v35/////////// ca iai2.b.... html
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu. Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling. In COLM, 2025.
net.
[oormmidde sazum?id-ay> wttpiiddayyyggleplu 7gezU87
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, arXiv preprint 01318
00211:2302.01318. 2023. /

---

Published as a conference paper at ICLR 2026
Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Kevin Chang, and Jie Huang. Cas- cade speculative drafting for even faster LLM inference. In NewrIPS, pages 66226-86244 2024. URL https: https://proceeddggg wwwpp//www....o.o....oooooooooo. www.eeriss.p.ppppeeeeeeeeee cc/paper files/paperr2222/ html
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse Transformers. arXiv preprini arXiv: 10509, 2019. URL https:://axii...rg////////////
10509
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with Performers. In ICLR, 2021. URLLattps:://peeeeeeee
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021. welhttss://arkee org/
wwtg/as///////oo////////////aaer abs 22122.11.1111
Tri Dao. FlashAttentionn2:: Faster attention with better parallelism and work partitioning. In ICLR, 2024. URL [RLLLttts///openreett http://wwww....e..eeo...eeeeeenn 222mmmm222mmm2222299.9..
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory--fficient exact attention with I0-awareness. In NeurIPS, pages 16344-16359, 2022. URL
hepp://.pe................eepeee hash/ 2ataa//67722aa200000000200004440
html
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang. Archi Mitra, Archie Sravankumar, and et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. BRLhhttss//arxii org/ http/bb/////07 /bt.222..22723
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang. Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 2024. URL https: http::://eeeooooo...o tttp:/eedooo.../..oooeo www.:r///e.rrr..o.ooerr.o 1260B
Gemma Team, Gemma: Open models based on Gemini research and technology. arXiv preprint 0rX2v:2403.08295, 2024. URL htttpsstts/////iiii org.
wttp://wwwww...r/a////////////// http//bb//2/0 08295
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang. Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chenggi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li. Hao Zhang. Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen. Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, Tao Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li. Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang
12

---

Published as a conference paper at ICLR 2026
Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He. Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan. Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DeepSeek-RI: Incentivizing reasoning capability in LLMs via reinforcement learning. Nature, 645:633-638, 2025. URL httpps:/wwwl nat ure
Yongchang Hao, Mengyao Zhai, Hossein Hajimirsadeghi, Sepidehsadat Hosseini, and Frederick Tung. Radar: Fast long- long-context decoding for any Transformers. In ICLR, 2025. URL https:
John Hewitt, Christopher D. Manning, and Percy Liang. Truncation sampling language model desmoothing. In Findings of EMNLP, 2022. URL htttt:://allatttta.ataa/// wwtp:ooo...gg//22......
Findings-emn_p 249
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text de- generation. In ICLR, 2020. h htt.://wwww...eeeoooooooooo///aa
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. URL
httpps://operrett
net http:/oormmmnmneeeeeee
Yunhai Hu, Zining Liu, Zhenyuan Dong, Tianfan Peng, Bradley McDanel, and Sai Qian Zhang- Speculative decoding and beyond: An in-depth survey of techniques. In Findings of EMNLP, 2025. pttttps://acaanthon wwt.gyy5.rg222222.. ftnee2..fiiiiiin findings-emea
716
Wonseok Jeon, Mukul Gagrani, Raghavv Goel, Junyoung Park, Mingu Lec, and Christopher Lott. Recursive speculative decoding: Accelerating LLM inference via sampling without replacement. In ICLR Workshop Large Language Model (LLM) Agents, 2024. URL https://arx::
org/ /a//bb//22/ 200a/2402.111 14160
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL BRLLhttps: htttttttt//aaaittie http:s///aaxiiw.rg/bbbb.s
08361
Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. big little decoder. In NeurIPS, pages 39236-39256,
Speculative decoding with
2023.
URL https:
//fpcceeloggs.euurissccco//oosss fiapsserrrriieeeee htt.aaaaaa.aaaaaee222222222a.5.- Con.zeeett-Cooeeeeceee.c html
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient Transformers. In ICLR, 2020. htw.hh/pps//ggaarreeeeeeeee.o/ep
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with PagedAttention. In SOSP. 2023. URL https://dl. wal.llaccoorggg.. att.aaaa...oo.o/oo.././///////
00110000066633..1 3613165
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from Transformers via speculative decoding, In ICML, pages 19274-19288 2023. URL htttttts//soocedin.ii mlr.
html
Yuhong Li, Yingbing Huang. Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you httppss/pproccedigss...nnee neurips cc/paper htce/aaaer_illlee//pp//////////2 Baab418222263e077333322 oeeee
looking for before generation. In NewrlPS, 2024a.
URL
13

---

Published as a conference paper at ICLR 2026
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling
requires rethinking feature uncertainty. In ICML, 2024b. URL https://proceedingg
mlr. wwwpppresssv... Ftress/v235/112455 html
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of lan-
guage models with dynamic draft trees. In EMNLP, 2024c. URL htllttss///caatthooloo..
org/2024. emnlp-main 422
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference acceleration of large language models via training-time test. In NeurIPS, 2025. URL https:
/ //openreview httt:/orrmmiddeeeellllo.
Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. Reward-guided speculative decoding for efficient LLM reasoning. In ICML, 2025. URL
Ji Lin, Jiaming Tang. Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang. Chuang Gan, and Song Han. AWQ: Activation- aware weight quantization for on-device LLM compression and acceleration. In ML- Sys, 2024. URL htt.lt/pwspppooccd............ep
hash/ 2aaaa//2222aaaeeeeeeaaaaeeeeeee.
html
Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang- Kangaroo: Lossless self-speculative decoding via double early exiting. In NeurlPS, 2024. URL ottp:s:/pproceed_nssseeee..
cc ft/ppppee_flless/ppppppreeeeeeee hhmm
hash
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant: LLM quantization with learned rotations. In ICLR, 2025. URL hittps://gpenreeviews net/
Eorum22d==or05DE6F2
Clara Meister, Tim Vieira, and Ryan Cotterell. If beam search the answer, what was the question? In EMNLP, 2020. URL
emn maninnlllminnn- 170
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Locally typical sampling. TACL, 11:102-121, 2023.
___00336
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. SpecInfer: Accelerating generative large language model serving with ree-based speculative inference and verification. In ASPLOS, 2024. URL https://dl. acm. org httg:/oi/110 10. 1tepa//.111111//111 1ee.11111322660 3.e..365555 3651335
Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. Faster cascades via speculative decoding. In ICLR, 2025. URL TtLpbttps:// http::/:s////eeevvieeee net/ wttp://forrmmmm.oss200eoe
Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Huazheng Wang, Kaixuan Huang, Yue Wu, and Mengdi Wang. TreeBoN: Enhancing inference-time alignment with specula- tive tree-search and best-of-n sampling. In Findings of EMNLP, 2025. URL https://
wwl.antholoyy.org//000 htt.://.oorr022......ieeeeee.ee
Findings-emnlp 1140
Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024. URL https:
Markus N. Rabe and Charles Staats. Self-attention does not need on memory. arXiv preprint 212.. 05682
12.00221 2021. URI/hhttsss///rrivvoorg////
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Grogle-proof
Q&A benchmark. In COLM, 2024. URL https://openzevi net mettfoorrmidd Ti 7167584b98
14

---

Published as a conference paper at ICLR 2026
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep
autoregressive models. In NeurlPS, 2018. / hnw.::eeee..nnnn..............p. cc/
ppaer/201s/hashi htt.://2.....22255.55.522222222.
Vivien Tran-Thien. An optimal lossy variant of speculative decoding. Unsupervised
Thoughts (blog), 2023. URL -
a http://:o....o..ooooooooaaaaaaaa
.ttml html
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for Transformers's attention via the lens of kernel. In EMNLP-IJCNILP, pages
2019. URL https://
aclanthology. org/D19-1443
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. URL. https://procett (etbbee233547deee
ngs. neurips www.ees......aaaeeeeeeee wwwpeper.pppp..p.ppppp fi es wwt.esspp/eee/eeeeeeeeeeeeeeeeee paper/20177
fbd053cl 44a84
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.
2os/2006.cm 748
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In NeurlPS, 2022. URL URL/atpss:
ViQLMeSB
J
Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation. In ACL, pages 10817-10834, 2023. URL https:://cccatthhhhhht ogy.
bxg/2023.ac1-1oo
605
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick VOn Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural natural language processing.
In EMNLP, 2020. URL httpps://aclaattkooooo.ooo/2/222
emn p-lll-peemme p-lppo-eems6
Zijun Wu, Yongchang Hao, and Lili Mou. Ulta-low-ddimensionall prompt tuning via random projec- /tattps://opennevieee...
tion. In EACL, 2026a.
Zijun Wu, Yongchang Hao, and Lili Mou.
TokMem: Tokenized procodural memory for large
language models. In ICLR, 2026b. URL https://apennevveww.ntttt www.nnte/forummdtt
RW JEfPPPij
Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of EMNLP, 2023.
org/2223. findings-emnip 257
Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang. Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency large language model inference: A comprehensive survey of speculative decoding. In Findings of ACL, pages 7655-7671, 2024. URL https: //aclanthology ort. findings-acl. .56.456
An Yang. Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li. Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09333 2025a. URL https://eree wwwpxxvvoorgaaaeoom http://////25.... 2200.003888
15

---

Published as a conference paper at ICLR 2026
Sen Yang, Shujian Huang, Xinyu Dai, and Jiajun Chen. culti-candidate speculative decoding. arXiv preprint arXiv:2401.06706 2024. URL hattps://axxi.. org/ btrg/abb//a httpp//2411..660 06706
Wang Yang, Xiang Yue, Vipin Chaudhary, and Xiaotian Han. Speculative thinking: Enhancing small-model reasoning with large model guidance at inference time. In COLM, 2025b. URL
fttps://openrest
net forum? f 185ooHo
Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. LLM inference unveiled: Survey and roofline model insights, arXiv preprint arXiv:2402. 16363, 2024. hRtphttss///axxtt wwwpwii.org/a///////// httpaa///....eee 16363
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang. and Amr Ahmed. Big Bird: Transformers for longer sequences. In NeurlPS, pages 17283- 17297, 2020.
URL httpss//pproceedingg..eeter neurips. pttppp//e222///att/ 681288222224777211188111------13
html
Zhenyu Zhang. Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H20: Heavy-hitter oracle for efficient generative inference of large language models. In NeurIPS, fataap/pereLeeee/p/e////23233 hash/ http://eaaa1117777eee020023311a0
2023. URL
html
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follloiing evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. URL
07911
Zixuan Zhou, Xuefei Ning, Ke Hong. Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. survey on efficient inference for large language models. arXiv preprint arXiv:2404. 14294, 2024. UtLhttps::/ /ttp:::////xvvv..//. http://awww.....r////////0.. 14294
16

---

Published as a conference paper at ICLR 2026
A TECHNICAL PROOFS
A.1 PROOF OF OBSERVATION I
Observation 1. Consider any desired target distribution h and draft model p. Algorithm produces the target distribution h exactly if the acceptance rate and recover distribution are defined as
Φ(z((<<)) min
and g(ar| h( p( x (<))) 21)))
Ext~ol1 - 6x [x<t)]
respectively. In addition, this acceptance rate o is optimal for achieving the highest acceptance rate.
Proof. Let n be the selected token and æ be the context. Then at each step, the resulting distribution of the algorithm is:
Pr(n|x) Pr(n- p(-(z) and a S )(x(z))
Σ p(la)) Pr(n g(-1x) and o(t[z))
(17)
=p(n|2z))((1al (a))
E
[1 < [));
(18)
s-pp-110)
where the first term the right hand side indicates the sampled token is accepted. The second term means the originally sampled token is rejected, and the current token n comes from the recover have
distribution g. Since would like Pr(n|z> x)) h(n/z))
p(n|=))(n|1)) + g(n x
[1 - ((il))] = h(n(2))
(19)
i>p(-1an (m)
- [(|l) = h(nlæ)) - ((
(20)
R <(t]=)]
hence proving the expression for g. Here, Φ can be any function that maps to [0, 1] that makes g distribution. Since the expression g is self-normalizing, we only need to make sure that all g(l
values are non-negative. Specifically,
S g(ilaz)
(21)
= h(i|z)) - p(il=))(ilar) 2 0
(Image(ø) S [0, 1))
h(la))
(i|[z) S p(ilz)
(22)
Again, with Image(d) C [0, 1]. we have
<(i(z)) min
(23)
((il
which gives the optimal acceptance rate min
[((()) p((a))
1
A.2 PROOF OF THEBRRE 2
Before proceeding to the proof of the theorem, we first show the following technical lemma.
Lemma 7. Let f : R+ - be convex with f(1 = O. For any a E [0, and sub-distribution
(q(1))es over S with Q = Eies g(i) 0, the solution
Σ9(05 h1(2)
min (h(a))
(24)
g(i)
iES
Σ h(ii) a, h(i) 20
(25)
h² hh((()) (9(1)) for all i S.
17

---

Published as a conference paper at ICLR 2026
Proof.
Let X .. Define h(1) X²(2). Then, we have
= =a
(26)
satisfying the constraints. For any feasible h o h, define r(2) have
A(1 q(6)
By Jensen's inequality,
Σ (0000( 21 ( = ( (1
(27)
with equality iff r(t) = À for all i E S. Thus h is the unique minimizer.
We can now show the theorem below.
Theorem 2. The optimal solution of in objective 3 is
7
if= n, otherwise,
h4
(6)
coc)) a(1 (())))))
where 7" is any root of the equation
( ((nlpree)) + +(1 (iggnn/ree)))
1-7
5 = g(nlaett)
(1-
77)
over the interval ((n]x<<)) too), clamped into [g(n x<)), 1). The function f the one used in the definition of f-divergence.
Proof.
hn
max min
(28)
s.t. h E 1V-1
(29)
(1 (+)))(())=))))))))))) 6.
(30)
Here, AV denotes the probability simplex, and
h((1) (())
Dy(hlg)= Σ {1)f
(31)
iEV
is the f-divergence. The objective
Nn ((())
min
(32)
is maximized when ha_ is as large as possible. However, since min(-, 1} caps the value at 1, the p(n) maximum achievable is 1 (when h, p(n)). Thus, the problem reduces to maximizing h, under the constraints, as increasing hn directly improves the objective until h, 2 p(n). To maximize hns we allocate as much probability mass hn as allowed by the constraints. Let ? hn The
remaining mass 1 - must be distributed over i + n. By Lemma 7 the optimal allocation for #
is:
1-y 1 - q(n) ((1)
h(11)
(33)
1-7 where T-q(a) ensures Eitn h(1)
Substitute hi = 7 and h())
1-g(n) 1-2 (i)) into Dy
D, {()))
+ Σ q())
(1--2))
(34)
((())
ifn
18

---

Published as a conference paper at ICLR 2026
Simplify the second term using Σifn ((1) 1 g(n): ?
+ (1 {(n)) (1-9(2)
Dj = {())f
(35)
g(n))
The constraint Dj S becomes an equality optimality (since increasing 7 further would violate the constraint). Thus, ? solves:
q())f
( q(n))
7 + (1
q(n))
(1-97)) = 8.
(36)
Finally, since y" may exceed I (when set too large to attain), truncated into [q(n), 1] as proper probability value.
A.3 PROOF OF THEOREM 3
Theorem 3. Let on and 9n denote the functions that follow the solution in Theorem when the sampled token is n. The distribution of the overall algorithm is given by
halg
Σ
p(nlare)) (n)en) th (n))
(8)
=E|[VI]
where en is a one-hot vector with only non-zero element at index n. In addition,
Dy |la (x)))) s min (T)), ()(p( (
for any d 2 O. Here, the function I [0, +oo) 00, +oo is continuous and non-decreasing in with value of at 6 0.
Proof. We work single step and suppress the context Øer. Fix p and q on finite alphabet. For each drafted index n, let hn be any target with (h,,ll) < 5. The conditional output is
Tu = (n)er + (1
and the algorithm's step output is
halg
Σ p(n)ra- n
Let
Hs = (hmd): (b,llll 58 Vn, ((1) = 0= (1) = 0).
(37)
F(hh)))) == >=Dy(hal: 14).
(38)
Define
r(6) = sup F((ha)m) E [0) (h,)CHs
(39)
and note that I depends only on (p, q. and the budget By construction,
Dy (hanhll S T(8) for every feasible family (h,,) E Ha.
(40)
It is straightforward to show that D (hatglly) q) < (r(l))) given that the all-acceptance distribution simply p. Thus it remains to show that has the claimed shape: non-decreasing, r(0) = 0. and continuous in the extended-real sense.
19

---

Published as a conference paper at ICLR 2026
Basic properties of r. (i) [(0) 0. I then h for all n, hale q and thus [(0)= = Dy(l9)) = O.
(it) Monotonicity. If 62 2 81 then Hs, Hay so [()1) I T((2) by definition of the supremum.
(iii) Continuity. We show right- and left-contnnitty. On a finite alphabet, the set of probability distributions is compact (a simplex), and with support alignment the feasible set Ns is closed (as the preimage of [0, 6] under the continuous function max,, Dy(-Ilg)) and thus compact. The map (hn), hig is continuous (operations involved continuous their domains), hence F is continuous.
We first show the right-continutty Let 6% 4. 6. For each & pick (h," E Ha. with F((h))))) )n) 2 [( - E.. where E/ + O. Since the alphahet is finite, the feasible families live a finite product of simplices, which is compact; therefore, there exists subsequence (not relabeled) such that h/
h for each n. By continuity of Dy(-1l9). (y(h))) = lim (h (1g) S lim, 6% = 6, so (hr), Ho. Continuity of F gives
lim sup ((6)) S
lim (F((h( +EE) = F((H)))) & r(5).
k-+00
Monotonicity gives ((6)= lim infx+40 [(6x), hence lim I(6x) = [
We then show the left-continutty. Let 6 t 5 and fix E > Choose (ha), E Hss with F( 2 (()) 8. For 0 E 10 define hn.0 argument,
0)h; + Oq. By convexity of D (-1lg) in its first
(y(Mn,oll)) (1 (1)) + OD, (gllg) S (1 - -00 < 6
so (ln,0)n E H(1-0)5- By continuity of F, for sufficiently small 0 >> 0 we have
F((hn,))n) F((h)))) - 2 [(6) 25.
For all large & with 6g > (1 - 0)6, monotonicity gives
I(6x) 2 T((1 - 0)6) 2 F((hn,)))) 2 r(8) 22.
Thus lim x (()x) 2 (()), and since monotonicity gives lim sup; I( - I(6), we have limp--ooo ((6x) (0))
In conclusion, by definition of r, for every feasible family (hw.) E Hs. Dj (y (4) (((),
with I' non-decreasing, continuous on [0, oo), and I(0)= = 0. This proves the theorem.
A.4 PROOF OF PROPOSITION 4
Proposition 4. Typical acceptance sampling (TAS; Cai et al. 2024) implicitly solves a variant of the optimization problem in objective 33) where the Idiivergence is substituted with the cross-entropy
H (h, (())x(()).
Proof. Given the optimization problem:
min hn ((n))
max h
s.t. h E A 1--1
(41)
H ②) < +
(42)
where (g) is the entropy. The optimal solution concentrates mass on (n, m). Equation equivalent to
442 is
Ela(1 h()) log 1
> -6.
(43)
q(i)
20

---

Published as a conference paper at ICLR 2026
To maximize hn 7, must minimize the LHS of (43) Based Lemma 8 the resulting
distribution is always two-point distribution. Let m = arg max, (()). For fixed h 7, the optimal
allocation places all remaining mass
= n
1(1)
7 = m
(44)
otherwise
Substituting the optimal form into (42
1
? log
(1 7) log g(m)
[q) +
g(n)
1 log
1
? log g(n)
(()) 8 log
{(m)
((m)
H + log p(m)
7
(45)
log ((m) ((n)
Since is probability, its maximum is reached when
Y= 1 log g(m)
< H(q)+ + logg {(m)
q(n)
- ((n) 2 ( exp(-6),
(46)
which is the acceptance rate used in TAS.
It should be noted that theory here is used reveal the soundness of the TAS acceptance function, without aiming to replicate the exact TAS algorithm. However, based on our framework, can derive the exact TAS algorithm by adding an (h)) 0 constraint and an threshold the cross-entropy limit, which we omitted for simplicity.
In the proof above, we invoked the following technical lemma.
Lemma 8. For any [0, 1), the minimal value of Eitn h(ii) log (() is achieved when:
h(m) = 1- Mi1 Vi + n, m.
(47)
We provide the proof below.
Proof. Let h(t) ax(1 7) for i n, where E, az 1. Then:
1 q(i)
Σla()) - a4 7) log
(48)
inn
minimized when a; concentrates on m arg max gi). since log q(ं) is minimized at i = m.
A.5 PROOF OF THEOREM 5
Corollary (Cactus's solution). Let the f-divergence in objective 3 be the KL divergence. The solution to Equation (14) is given by
fi=n, otherwise,
h(ilæ<))
( x()) 1-q(n|aes)
1-
(15)
where 7" = min (g(n)zt<)) + + (1 (x))))))))))))))
21

---

8 < (1/2 1 9)² 2q 4) Published as a conference paper at ICLR 2026
Proof. We first compute the derivatives of & at Yo:
((7)) (7%) = 0,
(49)
1
and *7(7)
(50)
The unique root in [q(n) too) is then
28 (>0))
7o
((nlanee))) + V 22 (n)(=<)))
We clip this value to the interval (g(n/==)) 1] to ensure validity as a probability.
A.6 PROOF OF COROLLARY
Corollary 6. When the exact solution > is not greater than 0.5 (i.e., the token is not likely to be accepted), our approximation always satisfies the divergence constraint:
(xu(hll)) S 6,
(16)
where fu(1 (x2)) is given by the approximated solution Equation
15
Proof. L=t =- g(z x<ee) for brevity and define the quadratic-approximate root
3 : + g q).
(51)
Because (f()) log? - log have ('()) 0 for every 7 (4, 1); hence & is strictly increasing on [g. and the equation (+7) admits a unique root y (q, 1].
Taylor's theorem with the Lagrange remainder, expanded at
gives, for some & (q, 7),
(g)g)
(()(E) (7 -
Φ(7)
(7 4)²
(52)
2
6
T2(1)
For the Bernoulii KL,
1 1 2n
(7) 7(1 7)
(53)
72(1 7)2
Whenever 7 2. the factor 1 1-2 is non neganiee and therefore (E) S 0. It follows that
(7 2) 2g(1 - q)
*(^) < T2(r)
by (q. 2
Choose & such that T2(9) = 6, this yields the expression given above. 4 S 1 or equivalently
(54)
then the above inequality gives +04) < 6. Since strictly increasing, we obtain
(55)
This result ensures that our approximation never overestimates y when the verifier model is not confident about the current sampled token.
22

---

Published as a conference paper at ICLR 2026
Table 3: The results with Qwen 3 14B as verifier and Qwen 3 0.6B as drafter.
GSM8K
IFEval
GPQA
m
Name
Score
AL
Wall
Score
ALT.
Wall+
Score
AL
Wall-
SpS
91.12
4.27
1.00x
85.03
2.19
1.00x
39.39
3.37
1.00x
Mentored
91.66
4.51
1.20x
61.37
2.88
0,96x
40.91
4.31
0,93x
SpecCas
88.40
6.42
0.85x
69.50
5.02
0,54x
32.83
6.27
0.68x
10
TAS
92.65
5.24
0.86x
86.14
3.00
0.82x
38.89
4.99
0.72x
Cactus
93.10
5.44
0.87x
85.96
3.03
0.78x
43.43
5.16
0.70x
Cactus 10
92.72
5.73
0.83x
84.66
3.41
0.74x
39.40
5.71
0.69x
Table 4: Speedup comparison on Spec-Bench using Qwen 14B as the verifier and Qwen 0.6B as the drafter. We report the speedup ratio relative to standard autoregressive decoding. "Accepted" denotes the mean number of accepted tokens per step.
MT Bench
Trans.
Summ.
QA
Math
RAG
AL
Overall
SpS Cactus
2.01 x 2.09
1.40x
1.92x
1.85x
1.83x
1.86x
3.20
1.81
1)
1.40x
2.04x
1.95x
1.86x
1.92x
3.29
1.88x
B ADDITIONAL EXPERIMENTS
Mentored decoding. blog post proposed Mentored decoding Tran-Thice 2023). which uses binary search to generate a target distribution such that (gll)) S 8. Compared with Cactus, there are two major differences: (1) Mentored decoding allows sampled tokens be accepted even when the verifier has zero probability, violating the principle of adhering to the verifier's ' mode; (2) more importantly, the solution found via numerical optimization procedure, significantly slowing down the decoding speed and defeating the purpose of high-throughput decoding. We conduct additional experiments to compare Cactus and Mentored decoding (using = as recommended),
As shown in Table Mentored decoding has the least acceptance rate gain at the cost of increasing the per-step generation time. For example, on GSMSK, the overall wall time even longer than that of the naive SpS method by 20%. In addition, the performance degrades severely IFEval, echoing our analysis of the misplacement of the divergence arguments.
Speculative cascading. More recently, Narasimhan et al. (0025 proposed speculative cascading (SpecCas), which dynamically decides if the sampled token will be verified by the large model based on the difference between the two distributions. Essentially, it is mathematically equivalent to mixing the draft and verifier distributions as the target distribution at different steps. We therefore conduct experiments with SpecCas (the [OPT] variant and 0.1 for better quality).
The results in Table show that SpecCas significantly increases the acceptance rate and the decoding speed. However, its generation quality is not as good as that of other methods, even when we choose hyper-parameters to favor higher generation quality. with 5 10 for Cactus. With similar wall-time acceleration generation quality is considerably higher. We hypothesize that this is due to the lack of explicit divergence control in SpecCas, whereas the other methods (especially Cactus) guarantee controlled "distances." Given that the primary focus of this paper is to introduce a new, principled method, we leave deeper investigation of these methods future work.
the other hand, we also ran experiments GSM8K and GPQA, Cactus 's
Evaluations on Spec-Bench. To provide more comprehensive assessment of Cactus across diverse scenarios, we conduct evaluations on Spec-Bench (Xia et al., 2024), unified benchmark designed to test speculative decoding methods across multiple distinct domains, including multi- turn conversation (MT-Bench), translation (WMT), summarization (CNN/DM), question answering (natural questions), mathematical reasoning (GSM8K), and retrieval-augmente generation (RAG). This broad coverage ensures that the observed speedups not limited specific task types but consistent across varied real-world applications. We use the Qwen 14B model as the verifier and the 0.6B model the drafter, maintaining temperature of 0.6.
23

---

Published as a conference paper at ICLR 2026
Table 5: Ablation on GSM8K using Qwen 3 14B verifier with different drafter sizes (ò 1).
Draft Size
Score
AL
Rej
Verifier (Oracle)
91.71
0.6B
93.10
5.44
-32%
1.7B
92.50
6.78
-60%
4B
92.57
7.76
-76%
The results are summarized Table Cactus is tested without any hyper-parn paper-parameter tuning (6 1). However, it immediately yields acceleration over the SpS baseline. In addition, Cactus consistently outperforms SpS across different domains, achieving an overall speedup of 1.88x (+88% gain over autoregressive decoding). This significant reduction in compute cost is achieved without additional training. worth noting that these speeds are measured using the HuggingFace Transformers framework (Wolf al. 2020), which is less optimized for speculative sampling. We anticipate that the real-world performance gains would be even larger with a better implementation such as vLLM woon al. 2023), indicated by other experiments.
Impact of draft model size. We employ same-family models to ensure aligned tokenization,
consistent with standard practice (Leviathan et al., [023) Chen et al.) 2023). To investigate the impact of drafter capacity, we evaluate Cactus on GSM8K using a Qwen 3 14B verifier with varying drafter sizes (Ta
Increasing the drafter size to 4B significantly boosts the mean accepted length (AL) to 7.76 and reduces rejection rates by 76%, while maintaining high task accuracy. These results confirm that Cactus effectively scales with stronger drafters, translating improved draft quality into greater decoding efficiency.
C
CASE STUDY
In this section, we discuss whether the choice of affects qualitative measures such as reasoning ability. We gather the results of the first sample from GSM8K, where ō is set to different values when running Cactus with the Qwen 3 0.6B + 14B model pair.
From the case study in Table 6 (more divergence allowed). Consequently, the result is wrong due to the low-quality chain-of- thought. This confirms that the divergence control in Cactus helps maintain qualitative measures.
see that the reasoning is poor and lengthy when ó is large
D BROADER IMPACT AND FUTURE DIRECTIONS
Broader impact. By improving the inference efficiency of large language models without sacrificing output quality, our method reduces computational costs and energy consumption. This contributes to more sustainable deployment, broadens access to high-performance language models, and supports environmentally conscious machine learning practices. Additionally, Cactus can enable faster, lower-cost applications in education, healthcare, and low-resource settings.
Future directions. Our goal in this paper framework, not to exhaustively optimize every dimension of the system. Accordingly, we identify several extensions and leave them for future exploration by the community: (1) Model scale. We capped evaluation at 32B parameters to keep the methodology clear and costs tractable. Pushing to substantially larger backbones could reveal sealing behavior (e.g., effects on acceptance rates, latency, and robustness) and is best investigated in follow-on work, including studies of scaling laws and distributed inference. (2) Model training. We emphasize a training ffree method to highlight the mechanism itself. While targeted tuning (e.g., LoRA for the draft (Hu et al. 2022 Wu et al., 2026a), verifier calibration, joint sequence-lewe distillation (Wen et al. 2023) may further improve proposal quality and reduce disagreement error, such engineering is orthogonal to
introduce and analyze the draft-and-verify
contribution and thus deferred. (3) Memory usage. Draft-and-verif introduces extra memory for the draft model and caches. Techniques like quantization, weight sharing, cache reuse, selective offloading, and early-
24

---

Published as a conference paper at ICLR 2026
exit heuristics could lower this footprint, but thorough treatment would distract from the main result; we leave these optimizations to future work. (4) Leveraging ensemble effects. In our main experiments, we observe that Cactus often performs better than the verifier model. For example, Cactus surpasses the verifier's accuracy by standard deviations on both IFEval and GPQA. We hypothesize that this is because Cactus enables "healthy" ensemble effect by combining model distributions. Leveraging ensemble effects in speculative sampling could be explored in future work.
E
THE USE OF LARGE LANGUAGE MODELS
Throughout this paper (with this paragraph being an exception), we use large language models to help identify grammar errors. Specifically, we prompt ChatGPT "Revise grammar errors with minimal changes of the original text", followed by the latex source code of each paragraph. In addition, we use ChatGPT and DeepSeek RI triple-check all technical proofs. The code for plotting all the figures is initially generated by ChatGPT, which is further revised by the authors according to the authors' aesthetics. We certify that the originality and scientific contributions of our method do not come from any large language models.
25

---

Published as a conference paper at ICLR 2026
Table 6: Generations for different values of
Task
Question: Janets ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for \$2 per fresh duck egg. How much in dellars does she make every day at the farmers' market?
8 <think> let me try to figure out how much Janet makes each day at the farmers' market. Hmm, so the problem says she has ducks that lay 16 eggs per day. Then she eats three for breakfast and bakes muffins with four eggs each day. Then she sells the remaining eggs for $2 each. Alright, let's break this down step by step. 'n \nFirst eggs. Then she uses some for breakfast and baking. Let's see, for breakfast, she eats 3 eggs. Then for baking, she uses eggs. So total eggs used each day would be 3 + 4
total eggs per day: 16
eggs.
Therefore, the remaining eggs would be 16 - 7
eggs. Then
she sells those eggs at $2 each. So, how much does she make? let me check again. If she uses 3 eggs for breakfast and 4 for baking, that's s 7. 16 minus 7 is indeed 9. Then 9 times 2 is 18. Yeah, her daily earnings would be $18. Hmm, is there any possibility I made mistake here? Maybe I should re-read the problem to make sure didn't miss anything. An \nThe problem says she lays 16 eggs per day. Then she eats 3 for breakfast and bakes muffins each day with four. So the total used is 3+4=7. Subtract that from 16, which leaves 9. Multiply by $2 for selling, 9+2=18. Yep, that's correct. I think that's the answer. </think>\n 'nJanet ducks lay 16 eggs per day. She uses 3 eggs for breakfast and eggs for baking muffins,
9 eggs multiplied by $2, which is 18 dollars. An \nWait
that seems right.
totaling 7 eggs used. The remaining eggs are $16 7 9/8. She sells these at $2 per egg, earning $9 \ttimes 2 = 18\$. An An #*###*## 18
8 = 10c
<think>'nOkay so letV's try to figure out how much Janet makes every day at the farmers'' market. Let me start by breaking down the problem step by step. [lengthy COT omitted] so total eggs used is 4
16. Then, after eating for breakfast, leftover is 16 - 3 = 13, then she sells them at 2 each: 13 $26. in \nYes that seems to make sense. Therefore, the answer should be 26. But let me check if the question is asking "how much money does she make every day at the farmers'' market?" So yes, selling the remainder at $2 per egg. Therefore, answer is $26. An \nTherefore which in the previous problems they had 5. This is a different case. So I think that correct. An </think>\n \nAnswer : 26)n AnThe problem states that Janet ducks lay 16 eggs per day. She eats 3 eggs for breakfast and eggs to make muffins daily. After these expenses, the remaining eggs are $ 16 - 4 == 9, which are sold at $2 per egg. Therefore, the total money she earns at the farmers'' market is 9 Vtimes 2 - 18 $. \n AnHowever the phrase "bakes muffins for her friends every day with four" might imply that she uses 4 eggs per muffin, meaning she makes 4 muffins. If she uses 4 eggs each muffin, the total eggs consumed would be $ 4 Vtimms
the answer is 26,
16 $. Since she eats 3, the remaining eggs are $ 15 - 13 $, which are sold for $ 13 Atimes 2 = 26 $ Hence, the correct answer is +*26+* In In #*#* 26
26


Reviews:
[{"review_id": "rI6kDnBQOI", "reviewer": "Reviewer_sKn6", "summary": "This paper proposes CACTUS, a novel constrained acceptance speculative sampling algorithm that accelerates large language model decoding while maintaining output fidelity. The key idea is to cast speculative sampling as a constrained optimization problem that explicitly trades off divergence from the verifier distribution against acceptance rate. By leveraging an f-divergence–bounded objective, the authors derive a closed-form solution (with KL divergence) that ensures provably controlled deviation from the verifier while allowing more draft tokens to be accepted. Extensive experiments on multiple model families (Qwen, Gemma, R1, LLaMA) and benchmarks (GSM8K, IFEval, GPQA) show that CACTUS consistently improves throughput and acceptance length without hurting accuracy. Compared with typical acceptance sampling (TAS), it demonstrates both higher acceptance rates and more stable quality, validating its theoretical motivation.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written. The proposed approach is novel. The evaluation is comprehensive and analysis is in-depth.", "weaknesses": "1. The observed improvements of CACTUS appear to correlate strongly with the capability of the underlying LLMs. For instance, larger verifiers (e.g., 14B) consistently outperform smaller ones (e.g., 8B), and reasoning-tuned models achieve higher acceptance and throughput than non-reasoning counterparts. This raises the question of whether the proposed method inherently benefits from the stronger self-consistency and error-correction capacity of powerful models, rather than from the constrained optimization formulation itself. A more detailed analysis or discussion of this correlation would strengthen the empirical claims.\n2. The evaluation primarily reports acceptance length and rejection ratio as proxies for throughput, but omits key metrics such as average generation length or tokens per completion. Without these, it is difficult to fully interpret both the quality improvements and the real speedup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RBHNPORxt1", "reviewer": "Reviewer_6cgh", "summary": "This paper introduces Cactus, a novel speculative sampling method for accelerating auto-regressive decoding in large language models (LLMs). Building on prior work like vanilla speculative sampling (SpS) and typical acceptance sampling (TAS), the authors reformulate speculative sampling as a constrained optimization problem to trade off acceptance rates against divergence from the verifier model's distribution. They derive a closed-form approximation using KL divergence and provide theoretical guarantees on bounded divergence. Empirical evaluations on benchmarks like GSM8K, IFEval, and GPQA using models from Qwen, Gemma, R1, and LLaMA series show that Cactus achieves higher acceptance rates and throughput than baselines while preserving or improving output quality.", "questions": "1. How sensitive is Cactus to δ? Can the authors provide ablation experiments?  \n2. How to select the optimal δ? And how should δ be chosen for different tasks?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and highly readable, with the authors providing a detailed introduction to the proposed method, along with in-depth theoretical contributions and proofs.\n2. The \"training-free\" nature and minimal computational overhead of Cactus make it extremely easy to integrate into existing inference frameworks, endowing it with high practical value.\n3. The experiments cover a variety of benchmark datasets as well as multiple model families and sizes. The results consistently show that Cactus outperforms SpS and TAS in terms of acceptance length and rejection rate, while maintaining stable accuracy.", "weaknesses": "1. It only includes three types of datasets: mathematical reasoning, instruction following, and scientific question answering, and lacks experimental results on Spec-Bench [1]—a more commonly used benchmark for speculative decoding. Spec-Bench covers multiple tasks such as multi-turn dialogue, translation, mathematical reasoning and so on.  \n2. The experiments only cover small-to-medium-sized models (with a maximum of 14B parameters) and do not involve larger models (e.g., 70B parameters) where memory bottlenecks are more prominent.  \n3. The baselines are relatively outdated. Key baseline methods such as Medusa [2] and EAGLE-2/3 [3, 4] are not comprehensively compared in the main results.\n4. The code in the anonymous link (https://anonymous.4open.science/r/Cactus-2E4D/) is empty.\n\n[1] Xia H, Yang Z, Dong Q, et al. Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding [C]. Findings of the Association for Computational Linguistics ACL 2024. 2024: 7655-7671.\n\n[2] Cai T, Li Y, Geng Z, et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads [C]. International Conference on Machine Learning. PMLR, 2024: 5209-5235.\n\n[3] Li Y, Wei F, Zhang C, et al. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees [C]. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024: 7421-7432.\n\n[4] Li Y, Wei F, Zhang C, et al. Eagle-3: Scaling up inference acceleration of large language models via training-time test [J]. arXiv preprint arXiv:2503.01840, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H0Y9jGDsGl", "reviewer": "Reviewer_Qb8d", "summary": "The core idea—casting speculative decoding as a constrained optimization and applying a KL-divergence bound with a second-order Taylor approximation—appears as an incremental re-interpretation of existing acceptance strategies rather than a fundamentally new algorithmic principle. Theoretical development largely repackages known mechanisms, while empirical gains emphasize acceptance/throughput and depend heavily on the hyperparameter δ; accuracy robustness improvements are inconsistent across tasks. Overall, the contribution does not clear the bar for novelty and impact.\n\nThe manuscript exhibits noticeable typesetting glitches (e.g., unnecessary hyphenation and line breaks in the title/long tokens, inconsistent styles for algorithms/tables/captions, and minor inconsistencies in notation and reference formatting. These issues hurt readability and professional polish; a thorough pass against the official template is needed.", "questions": "The manuscript exhibits noticeable typesetting glitches (e.g., unnecessary hyphenation and line breaks in the title/long tokens, inconsistent styles for algorithms/tables/captions, and minor inconsistencies in notation and reference formatting. These issues hurt readability and professional polish; a thorough pass against the official template is needed.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The work introduced a principled, divergence-controlled acceptance rule (Cactus) that balances fidelity and efficienc.", "weaknesses": "The manuscript exhibits noticeable typesetting glitches (e.g., unnecessary hyphenation and line breaks in the title/long tokens, inconsistent styles for algorithms/tables/captions, and minor inconsistencies in notation and reference formatting. These issues hurt readability and professional polish; a thorough pass against the official template is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kr4iG7JQM0", "reviewer": "Reviewer_CzSq", "summary": "This paper proposes Cactus, a novel speculative sampling method that provides explicit control over divergence from the verifier’s distribution to improve the token acceptance rate. It formulates speculative sampling as a constrained optimization problem that maximizes the draft token acceptance probability while keeping the target distribution within a bounded divergence from the verifier. Empirical results across multiple models and datasets show that Cactus achieves higher acceptance rates, often with improved accuracy compared to baseline methods.", "questions": "Q. What are some possible explanations for the often better accuracies?\n\nQ. The anonymous link to code seems broken?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposes principled trade-off between efficiency and fidelity. The paper proposes viewing speculative sampling as choosing an appropriate distribution $h$ that achieves a desired trade-off between draft token acceptance rate and divergence from the verifier. This view allows methods including SpS and TAS to be viewed in a unified framework.\n\n2. Practical implementation. Based on the constrained optimization formulation, the paper derives a closed-form solution with cost comparable to SpS.\n\n3. Empirical evaluations. Experiments on multiple models and datasets show improved acceptance length compared to SpS and TAS.", "weaknesses": "1. Missing evaluations on drafter quality. Qwen 3 models are used as both the verifier and the drafter. Since these models belong to the same family, their output distributions are likely to be similar, making it unclear how the results would change with different levels of inherent divergence between the verifier and drafter. Also, how does performance vary when using drafters of different sizes?\n\n2. Missing evaluations on interactions with temperature. For the main experiments, one set values for top-p, top-k, and temperature (fairly low) is used. Further analysis on how different methods interact with sampling hyperparameters would be useful since the focus of the paper is on speculative $\\textit{sampling}$.\n\n3. Discussion on approximation validity. Cacus leverages second-order Taylor approximation assuming $\\delta$ remains small. It is unclear whether $\\delta$ values such as 0.75 justifies the approximation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TOYG2JFXp6", "reviewer": "Reviewer_PuX9", "summary": "This paper introduces Cactus, a constrained acceptance speculative sampling method aimed at accelerating auto-regressive decoding in large language models (LLMs) while preserving output distributional fidelity. The work frames speculative sampling as a constrained optimization problem balancing acceptance rate and divergence from a verifier model. Theoretical analysis yields a principled solution for constrained divergence (using KL divergence as a metric), and the authors propose a practical algorithm that maximizes accepted tokens per verification step. Empirical evaluation on multiple benchmarks (GSM8K, IFEval, and GPQA), using various pairs of drafting and verifying models, demonstrates improved throughput and competitive or superior accuracy versus classical speculative sampling and typical acceptance sampling approaches.", "questions": "1. The method currently relies on manual or grid-based tuning for the divergence parameter $\\delta$. Is there a theoretically principled or adaptive way to set or anneal $\\delta$ based on runtime feedback (e.g., based on target throughput or accuracy)? What would be the pitfalls in automating this choice?\n\n2. While Cactus aims to bound divergence and preserve verifier fidelity, do any empirical failure cases show semantic or correctness failures? For example, what types of errors arise in GSM8K or GPQA when $\\delta$ is set too high?\n\n3. How would the approach scale to extremely large verifiers (e.g., >100B parameters) with draft models that are structurally or statistically misaligned? Does the performance gain or accuracy gap increase as model size scales up?\n\n4. Related to Figure 1 and Table 1: In specific cases (e.g., GPQA with Cactus 1.0), Cactus surpasses the verifier's accuracy. Is this genuine improvement or due to variance/noise in evaluation or artifacts of metric normalization?\n\n5. Could the framework be generalized to allow for multi-token acceptance rules (e.g., chains or blocks of tokens) beyond per-token KL constraint, and what would the mathematical/algorithmic implications be?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Formalization and Theoretical Rigor: The paper provides a thorough formalization of speculative sampling as a constrained optimization problem, with mathematically sound derivations (see Equations in Section 2 and supporting proofs in Appendix A). The use of $f$-divergences, primarily KL, is well-motivated and leads to explicit solutions for the acceptance/recovery functions.\n\n2. Principled Design: Compared to prior heuristics like typical acceptance sampling (TAS), Cactus mathematically guarantees bounded divergence from the verifier distribution. The method is grounded in strong theoretical justifications (see Theorem 3, Corollaries 6 and 7, and explicit treatment of the divergence-acceptance trade-off).\n\n3. Empirical Breadth and Depth: The experiments include a wide variety of benchmarks, LLM architectures, and model sizes. Table 1 provides detailed results across tasks and models, showing Cactus consistently offers higher acceptance rates (e.g., $\\mathrm{AL}_{m}$ and Rej columns), often without sacrificing accuracy compared to SpS and TAS. Additional evaluations address wall-time throughput (Figure 3), cross-architecture robustness (Figure 4), and trade-offs with alternative approaches (interpolation and SpecCas).\n\n4. Clarity and Transparency: Mathematical notation is mostly clear, with comprehensive appendices containing proofs and derivations. The design of the method is explained with both equations and pseudocode (Algorithm 1), and decision points are clearly highlighted.\n\n5. Practical Impact: The method is training-free, requires only elementwise operations on output probabilities, and can be deployed to accelerate LLM inference in real-world conditions.\n6. Figure/Table Engagement: Key quantitative claims are backed up by explicit references to figures and tables. For example, Table 1a and 1b directly show improvement of Cactus in both acceptance rate and quality, while Figure 1 graphically summarizes the accuracy-acceptance tradeoff across diverse settings.", "weaknesses": "1. Empirical Baseline Coverage and Ablations: While the paper includes SpS, TAS, Top-k, Mentored decoding, and SpecCas as baselines, it does not systematically evaluate against very recent tree-based decoding advancements (such as recursive or blockwise candidate strategies). This limits direct benchmarking of Cactus's relative value in the face of alternative parallel or hierarchical sampling frameworks. The omission can be seen in the lack of explicit mention of, e.g., \"Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement\" or similar tree/tree-of-N approaches, which could address the same efficiency-quality tradeoff. Including these could strengthen the empirical section.\n\n2. Scope of Generalization Claims: Although Figure 4 and several model series are included, experiments mostly focus on models up to 14B parameters and a limited set of benchmarks. Claims about scalability to substantially larger backbones (hundreds of billions of parameters) or broader domains (e.g., multilingual or multimodal tasks) remain unproven, and the discussion in Appendix C defers these questions to future work.\n\n3. Selection and Tuning of $\\delta$ (Divergence Constraint): The hyperparameter $\\delta$ critically determines the quality-throughput tradeoff. While the mechanism for grid search and choice is described, no principled or automated selection strategy is suggested, nor is the sensitivity to $\\delta$ analyzed in detail beyond grid-search plots (see Figure 2). This may be a practical barrier for deployment, especially for practitioners unfamiliar with the KL/gamma tradeoff mechanics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)