CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs
Authors: Zhiyuan Ning
Jiawei Shao
Ruge Xu
Xinfei Guo
Jun Zhang
Chi Zhang
Xuelong Li
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Efficient Inference
Speculative Decoding
Primary area: deep_learning
TL;DR: In this paper, we propose a novel Cascade Adaptive Self-Speculative Decoding (CAS-Spec) algorithm which constructs speculative draft models by leveraging dynamically switchable inference acceleration (DSIA) strategies

Abstract: Speculative decoding has become a widely adopted as an effective technique for lossless inference acceleration when deploying large language models (LLMs).
While on-the-fly self-speculative methods offer seamless integration and broad utility, they often fall short of the speed gains achieved by methods relying on specialized training.
Cascading a hierarchy of draft models promises further acceleration and flexibility, but the high cost of training multiple models has limited its practical application.
In this paper, we propose a novel Cascade Adaptive Self-Speculative Decoding (CAS-Spec) method which constructs speculative draft models by leveraging dynamically switchable inference acceleration (DSIA) strategies, including layer sparsity and activation quantization.
Furthermore, traditional vertical and horizontal cascade algorithms are inefficient when applied to self-speculative decoding methods.
We introduce a Dynamic Tree Cascade (DyTC) algorithm that adaptively routes the multi-level draft models and assigns the draft lengths, based on the heuristics of acceptance rates and latency prediction.
Our CAS-Spec method achieves state-of-the-art acceleration compared to existing on-the-fly speculative decoding methods, with an average speedup from $1.1\times$ to $2.3\times$ over autoregressive decoding across various LLMs and datasets.
DyTC improves the average speedup by $47$\% and $48$\% over cascade-based baseline and tree-based baseline algorithms, respectively.
CAS-Spec can be easily integrated into most existing LLMs and holds promising potential for further acceleration as self-speculative decoding techniques continue to evolve.

CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs
Zhiyuan Ning Jiawei Shao" Ruge Xu²
Xinfei Guo
Jun Zhang Chi Zhang"! Xuelong Li TeleAI, 2 Shanghai Jiao Tong University. Hong Kong University of Science and Technology,
{telegraphpolehead,, saccheadd,chhhree schrodinger iereiingernneeiiiii xinfei..ii..iie xinfei.guo)@$sjte guo} edu. cn, Schoooiw..hangg1100hhhhttteeeee en, eejzhangoust. hk, xuelong 1 eeeieeeeee.... org
Abstract
Speculative decoding has become widely adopted an effective technique for lossless inference acceleration when deploying large language models (LLMs). While on-the-fly self-speculative methods offer seamless integration and broad utility, they often fall short of the speed gains achieved by methods relying on specialized training. Cascading a hierarchy of draft models promises further ac- celeration and flexibility, but the high cost of training multiple models has limited its practical application. Self-Speculative Decoding (CAS-Spee) -pec)) method which constructs speculative draft models by leveraging dynamically switchable inference acceleration (DSIA) strate- gies, including layer sparsity and activation quantization. Furthermore, traditional vertical and horizontal cascade algorithms are inefficient when applied to self- speculative decoding methods. We introduce Dynamic Tree Cascade (DyTC) algorithm that adaptively routes the multi-level draft models and assigns the draft lengths, based the heuristics of acceptance rates and latency prediction. Our CAS-Spec method achieves state-of-the-art acceleration compared to existing on-the-fly speculative decoding methods, with average speedup from 1.1x to 2.3x over autoregressive decoding across various LLMs and datasets. DyTC improves the average speedup by 47% and 48% over cascade- based baseline and tree-based baseline algorithms, respectively. CAS-Sper can be easily integrated into most existing LLMs and holds promising potential for further accelcration self-speculative decoding techniques continue to evolve.
this paper, we propose novel Cascade Adaptive
Introduction
Large Language Models (LLMs), such as Llama [1], Mixtral [2], and Qwen [3] are rapidly evolving and increasingly adopted in various applications. However, their autoregressive generation pro- cess, where tokens are produced sequentially, coupled with their massive parameter sizes, leads to substantial inference latency and high computational cost. To mitigate this bottleneck, speculative decoding [4-6] has emerged as highly effective and widely adopted approach, achieving significant speedup without compromising the quality of the generated text. It operates by utilizing a smaller, faster "draft" model to generate sequence of multiple future tokens concurrently, which are then verified by the larger, more accurate "target" model in parallel, thereby reducing the number of expensive forward passes.
Work done during internship TelcAT Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite its promise, standard speculative decoding introduces new burden: it requires training and maintenance of separate draft model. This not only demands additional training data and computation resources, but also requires careful tuning to ensure compatibility and efficiency aligned with the target model. To address these challenges, self-speculative decoding approaches, self- speculative decoding methods [7-9] have been proposed. These techniques cleverly derive draft predictions from the target model itself, typically by skipping certain blocks in the modules or leveraging model compression techniques, thus eliminating the external training burden. While self-speculation simplifies the deployment pipeline, it often provides limited acceleration. In contrast, cascade speculative decoding [10] introduces hierarchy of draft models, enabling a multi-stage approach with higher potential speedups and greater flexibility. Yet it requires multiple distinct draft models, making it largely impractical for widespread adoption in real-world systems.
In this paper, we aim to harness the potential of cascade speculation without the crippling overhead of training multiple draft models. We introduce Cascade Adaptive Self-Speculative Decoding (CAS-Speec, novel framework that brings the performance benefits of cascade speculative decoding without the overhead of training multiple draft models. CAS-Spec dynamically constructs hierarchy of speculative draft stages using the target model itself by leveraging Dynamically Switchable Inference Acceleration (DSIA) strategies. These include techniques such layer sparsity and activation quantization, enabling the creation of multiple draft models embedded within the target model's inference process. To coordinate this hierarchy at runtime,
further introduce Dynamic Tree Cascade (DyTC) algorithm to adaptively route the draft models, construct draft trees as well as controlling the draft lengths. It leverages heuristics based on token acceptance rates and latency prediction to maximize throughput.
Our contributions are summarized as follows:
We propose CAS-Spec, novel speculative decoding framework that creates multiple on-the-fly draft stages from a single target model. CAS-Spec achieves lossless inference acceleration without requiring additional draft model training.
We introduce Dynamic Tree Cascade (DyTC), an adaptive routing algorithm that dynamically manages the draft models and their lengths based on heuristics of acceptance rates and latency predictions. DyTC provides 47% and 48% improvement in average speedup over the cascade-basee and tree-based baseline, respectively.
We demonstrate through extensive evaluations that CAS-Spec achieves state-of-the-art (SOTA) acceleration among on-the-fly speculative decoding methods, delivering speedups ranging from 1.1x to 2.3x over autoregressive decoding across various LLMs and datasets.
This work presents a practical and efficient approach to significantly accelerate LLM inference, paving the way for wider deployment of powerful language models in latency-sensitive sensitive and resource- constrained scenarios.
2 Preliminary
Autoregressive decoding, where each token depends on the previously generated ones, requires sequential execution, limiting the inference speeds of L.MMs. Speculative decoding [4-6] offers a general framework to mitigate this issue by predicting multiple future tokens using faster draft model Ma and verifying them with the target model M, in parallel. To address practical challenges and unlock greater acceleration, various methods have been developed focusing on the nature and utilization of the draft model, leading to approaches like Self-Speculative Decoding and Cascade Speculative Decoding.
Self-Speculative Decoding. Self-speculative decoding (SSD) aims to eliminate the need for external draft models altogether, thereby reducing the overhead of training separate draft models. A common approach of SSD is to derive draft token predictions directly from the target model M₁ itself during the inference process. Several strategies have been explored, including layer sparsity [7, 9, 11], early-exiting [8. 12, 13], efficient attention [14, 15]. Jacobi decoding [16-18] and activation quantization [19]. Other methods such as EAGLE [20-22] and Medusa [23] reuse the target model's hidden states to generate draft tokens efficiently. By leveraging parts of the target model's own information or computation, SSD avoids the burden of separate draft model training and extra memory footprint for maintaining the kay-v (KV) cache of the draft LLM.

---

Cascade Speculative Decoding, Vanilla draft models are commonly autoregressive LLMs, which means they can also be accelerated by speculative decoding. Further acceleration is promised by employing multiple draft models, typically ordered by decreasing sizes and latency, e.g.. Applying this technique recursively leads to similar approaches known as vertical cascade [10], hierarchical speculative decoding [15] or multi-level speculative decod- ing [24]. In the direction of draft token generation, since rejecting a draft token means rejecting all the following draft tokens, the acceptance of early draft tokens is more important than the later ones. Cascade Speculative Drafting (CS- D [10] further proposes horizontal cascade which generates early draft tokens using a slightly slower draft model with a high acceptance rate, and subsequent draft tokens with progressively faster draft models. By leveraging vertical cascade and horizontal cascade, CS- CS-Drafting achieves further acceleration over the vanilla speculative decoding.
{Mdy Ma........MMMM
Torget Mocel
Valusa
Ru Valtasa
Veumm 120
5300
2.50
A,
1161
Eor
DWIFT Phrnt
SHIFT Caza Ports
6.5
65
644
044
860
503
04
67
0
SMIFT
A. (coecpted accuptance salo)
9.n (oxpected accestance rako)
(a) Spec-Bench Results
(b) vertical cascade Bound
(c) horizontal cascude Bound
Figure 1: (a) Comparison of on-the-fly SSD methods (Lookahead, SWIFT) and methods with statistical draft models (e.g. PLD) on Spec-Bench, tested on NVIDIA H100 GPU. (b) Theoretical effective bound of vertical cascade for a draft model MMa, to be beneficial in the cascade speculative decoding compared with vanilla speculative decoding of Ma, alone. The x-axis is the expected acceptance rate a(Me. and the y-axis is the cost coefficient ((M,,Ma,). The SWIFT data points are from the Spec-Bench results for Vicuna-7B-vl 3 model. (The acceptance rates of PLD are between 0.1 and 0.5 in this setting.) (c) Theoretical effective bound of horizontal cascade, similar to (b). In this case, we consider d1,2 which is commonly similar to 0r,d2 in practice.
3 Motivation and Analysis
Although self-speculative decoding methods like EAGLE [20-22], BiTA [18] and LayerSkip [8] have shown promising speedups, these methods still require training for several days on node of 8 common server GPUs like Nvidia A 1000. While the on-the-fly SSD methods like Lookahead [16]
and SWIFT [9] are training- free, they are superior in terms of speed, even falling short of minimal retrieval- drafting methods like prompt lookup (PLD) [25] on Spec-Benchi [26], as shown in Fig. 1a. Since retrieval-based methods like PLD utilize the repeating tokens in the generation process to produce draft tokens, they are inherently lightweight and universally applicable, which weakens the competitiveness of the current training-free SSD methods. To break this trade-off between speed and ease of use, using a combination of multiple training- free SSD methods stands out as a potential solution. Chen et al. [10] present provable improvements over the standard SSD methods by cascading multiple draft models horizontally and vertically. However, not all LLMs have a series of smaller draft models available like the FLAN-TS [27] family used in CS-Drafting Requiring multiple draft models is a significant limitation for the practical application of cascade speculative decoding (CSD). We note that the training- free SSD methods can be used to construct cascade of draft models for speculative decoding. Though it has been proven that a retrieval-bai based statistical model with negligible cost (e.g. PLD) can almost always gain further speedup against the standard speculative decoding [10], it is not guaranteed that such CSD can be faster than SD with the statistical model alone. It thus leads to the first research question to be addressed:
RQ I:
Can existing raining-free self- seff-speculative decoding methods be used to construct an effective cascade of draft modely above retrieval -based methods for on-the-fly speculative decoding?
Firstly, we establish theoretical effective bound for an intermediate draft model to bring a speedup in CSD. We adopt the same notations as in DS-Drafting for the sake of clarity:
3

---

Expected acceptance rate (MM,,M)) The probability that draft token from a draft model Ma verified as correct by the target model Mr
Cost coefficient e(M We define c(Mt,Ma, as the ratio of inference times, indicating how much faster the draft model Ma is compared to the target model M, for single forward step.
Expected walltime improvement factor (EWIF) T: It represents the predicted gain in overall execution speed (wall time) assuming that the acceptance of each token is an i...d. Bernoulli trial.
As proved by Chen et al. [10], the EWIF of speculative decoding is: TsD
+1 [ok+1]
1-oki1
(T-o)(ck+T))
where (z) + (z 1) 1 1-a of generating tokens p; and a = a(M,, The EWIF for a vertical cascade with two draft models, MMd (slower, higher quality) and Maz (faster, lower quality), is [10]:
(1-az) is the probability generating function of the probability
1 1- (a)
Tyc(M, May)) (1 a)) ned, + nkcaa))
(I)
where ((x) = (Ma, MMa,) (z), a MaM and cd2 Cd2 are c(M,, (Me,MMs respectively. Adapting from Chen et al. [10], Thm. 4.5, the EWIF of horizontal cascade for two draft models is:
Ra,
1 1-a dj ad,
Ra, adg (1-22
+ adi 1-0012
THC(M,
(2)
1 kd, Cd2 + Cdg
where ad, = Ma,)) as, =
cd, = olMoMMMs) cd2 = MdM)
Then we get theoretical bound by solving the inequalities Trc(Ma,.Ms) 2 TsD(Ma,) and
1-a8 1o
THo(Ma, Magz) 2 TSD(Ma,), where TsD(Ma,) (I-as,][a b1+ ties are presented in the Appendix B.
The solution of these inequali-
However, the solutions of the inequalities are highly dependent on the hyperparameters of the speculative decoding scheduling, such as kd,, kdg and n.
Thus, we should compare the EWIF of these methods with optimal hyperparameters to find tighter bound, i..
max maxIvC(Ma, 2 maxTTsD(Ma) u.k
max THOLMs 2 maxTsD(MM.) ka Kag
(3)
Kg
Kg
This inequality (Eq. 3) does not readily yield a closed-form expression for Cd., as the maximization over integer hyperparameters (ko, n, k kay, kaa)) typically requires numerical evaluation of given model parameters like the expected acceptance rates and cost coefficients. Therefore we conduct a numerical simulation to find the theoretical effective bound for Ma, to be benificial in the CSD. When Má2 is retrieval- based statistical model with negligible cost, we assume Cd2 0.01 and (MMMMMa = aMMs, MM.). The simulation results of the borderline of ca, and (MMe, are shown in Fig. 1b and Fig. lc.
With the theoretical effective bound established, we can analyze the performance of existing training- free SSD methods in the context of CSD. For instance, SWIFT [9] employs layer sparsity strategy, the distribution of its acceptance rates and cost coefficients on Spec-Bench are illustrated in Fig. lb and Fig. Ic. As observed, most of the data points of SWIFT lie above the theoretical effective bound, indicating that naive HC or VC cascade with SWIFT as the intermediate draft model does not guarantee speedup over using PL. alone for speculative decoding.
While it's feasible to cascade multiple training-free SSD methods for CSD, more effective cascade algorithms are necessary to fully leverage their potential, with tree-based structures being a promising direction. Since the EWIF of CSD depends on the scheduling algorithm of different draft models, is possible to achieve a better speedup by adaptively routing the draft models and assigning the draft lengths. This leads to our second research question:
RQ 2: Can we achieve further speedup by adaptively routing the draft models and assigning the draft lengths, with regards the characteristics of different DSIA strategies?
This question is explored in Section 4.2, where we introduce the Dynamic Tree Cascade (DyTC) algorithm for online scheduling of CAS-Spec.

---

4 Cascade Adaptive Self-Speculative Decoding
To address the outlined challenges and leverage the potential of training-free SSD methods, we propose Cascade Adaptive Self- Self-Speculative Decoding (CAS-Spec). CAS-Spec constructs a hierarchy of draft models using various inference acceleration strategies applied to the target model M,. then employs dynamic mechanism to route through this hierarchy and determine draft lengths.
4.1 Construct Draft Models with Dynamically Switchable Inference Acceleration Strategies
Definition 4.1. A Dynamically Switchable Inference Acceleration (DSIA) strategy is technique that modifies the inference process of model to accelerate the token generation, which can be dynamically switched on or off during inference. These strategies can often be parameterized (e.g.. by the degree of sparsity, quantization bit-width).
Each DSIA strategy, potentially with different parameter settings, can viewed as creating a distinct "virtual" draft model Ma, derived from M₁. Examples of DSIA strategies include:
Layer Sparsity: Skipping a subset of transformer layers or a subset of attention and FFN blocks in Me to generate draft tokens [7, 11].
Early-Exiting: Using predictions from intermediate layers of M, as draft tokens [8, 12, 13].
Activation Sparsity: Keeping only a subset of neurons (activations) in each layer to reduce compu- tation and memory movement of weights. It commonly requires batch size to be I or small. 3 Activation Quantization: Using lower precision (e.g.. INT4) for activations and (partial) KV cache of the model during draft generation, as explored by QSpec. It requires weight-only quantized target model for considerable speedup.
Efficient Attention: Using an efficient attention mechanism like StreamingLLM [30] for draft generation. This is explored the context of SSD by TriForce and MagieDec. Such methods are commonly more performant in long context generation.
To construct a hierarchy of draft models, there are three approaches in general:
Mixing-DSIA Cascade: Using orthogonal DSIA strategies create a series of draft models. For example, Ma, could be a layer-sparse model, while Ma₂ could have both layer sparsity and activation sparsity.
Using conflicted DSIA strategies to create series of draft models. For example, Ma, could be model with FP8 quantized SageAttention2 [31], and Ma, could be model with StreamingLLM attention.
Replacing-DSIA. Cascade:
Scaling-DSIA Cascade: Using the same DSIA strategy with different parameter settings (c.g.. different degrees of sparsity) to create series of draft models {Md, .Md..... MM. }.
In the spectrum of the trade-off between speed and accuracy, each intermediate draft model Ma, typically satisfies (MMe,Ma)) 2 a(Me,Ma... and c(M,,Ma,) < e(M The last model in the hicrarchy is expected to be the fastest and least accurate, such as an extremely fast, often non-neural retrieval-basee method.
Definition 4.2. In hierarchy fo draft models [Md, Mag..... }. the bottom draft model Má. a model that serves as the final stage in a cascade draft models, which means it cannot be further accelerated by speculative decoding.
Prompt Lookup Decoding (PLD) [25] is a prime example. Statistical n n-gram models or small, fixed draft heads like those in EAGLE [20-22] and Medusa [23] could also serve this role. However, since EAGLE and Medusa require the hidden states of the target model, their performance will be greatly affected when using the hidden states from DSIA draft models. For simplicity and considering its proven efficacy in CS-Drafting [10], we often consider PLD as a default Ma..
The Ma, are inherently training- free if the DSIA strategies are training-free. By choosing training- free bottom draft model and DSIA strategies, CAS-Spee can be implemented without training multiple draft models and thus more easily integrated into wide range of LLMs. Among the listed DSIA strategies, layer sparsity does not require special condition to gain speedup. We choose it for easy
'There are no existing works on SSD with activation sparsity yet. but it is promising direction with recent advances in this technique [28, 29]
5

---

comparison with other speculative decoding methods. CAS-Spec with other DSIA strategies like activation sparsity and quantization is discussed in the Appendix C.
4.2 Dynamic Tree Cascade (DyTC)
Tree attention allows for more flexible and efficient speculative decoding by enabling the parallel verification of different branches of draft tokens [23, 32J. Similarly, early tokens in draft token tree should also be prioritized for more accepted tokens.
Proposition 4.3. Tree Cascade (TC) assigns the different draft models in the draft token tree to maximize the expected acceptance rate of the early draft tokens.
Given a set of available DSIA strategies and a bottom draft model, there are many possible configura- tions for the cascade of draft models, forming set of candidate draft models.
In CAS-Spec systems, the challenge is to determine where to start generating tokens (in the draft token tree), which draft models to use, and when to switch between them. Since the dimensions of the search space are large, is impractical to use global optimization methods, which are commonly used in vanilla speculative decoding to find the optimal draft length. Instead, we propose to use a aeuristic-based approach to dynamically adapt the hyperparameters of the scheduling algorithm during inference. This dynamic adaptation is guided by heuristics based on continuously updated estimates of acceptance rates and latency predictions. Inspired by the idea of dynamic draft tree expansion [33], we propose to leverage online optimization to adapt the expansion of Tree Cascade: Proposition 4.4. Dynamic Tree Cascade (DyTC) is dynamic scheduling algorithm that adaptively selects the draft models and their configurations in a tree structure based the online acceptance rates and latency predictions.
Acceptance Rate for Draft Configurationss For each potential draft model configuration Ma, (including DSIA variants of M, and vertical cascade combinations), DyTC maintains an online esti- mate (MM, of its acceptance rate. This estimate is continuously updated using an Exponential Moving Average (EMA) mechanism:
Unew X Öprer + (1 - A) Exrecent
where Öprev the estimate from the previous step, &recent is the acceptance rate computed from a local history window of the most recent H/ generation steps (we use H 20 and À = 0.7 in our experiments), and À controls the balance between stability and responsiveness to changing generation contexts.
Critically, for computing orecent, we focus on the acceptance of the first draft token generated by each configuration, rather than the overall ratio of accepted to drafted tokens. Specifically, if the local history contains outcomes {01, 02, 0H) where o, E {0, 1) indicates whether the first drafted token was accepted, then: åreeent
The EMA-based update ensures that á adapts to the dynamic nature of text generation, where accep- tance probability can vary significantly across different tasks (e.g., translation vs. summarization) and even within single sequence. For cold starts initialization, brief calibration phase can be performed at the beginning of generation to gather initial acceptance statistics for each configuration, detailed in Appendix D.
Token-Level Information for Drafted Tokens. While the configuration selection in F FindBestConf IndBestConfigurat www.oniiirraaattotttteeeeeo (Algorithm 2) relies on the aggregate acceptance rate estimates described above, the calculation of accumulated acceptance rate IT'; aj for already but not-yet-verified tokens does incorporate token-level information. Specifically, we consider:
For neural draft models: the normalized probability (logit) of the drafted token, which is positively correlated with acceptance likelihood [33].
For non-neural drafts (e.g., PLD): longer length of the n-gram match indicating higher confidence.
This teken-level refinement allows DyTC to more accurately estimate the quality of different candidate branches in the draft tree when selecting the next leaf node to expand. However, for future tokens (those not yet generated), we cannot access such token-level information without actually running the draft model, which would be prohibitively expensive for all candidate configurations. Therefore, the configuration selection relies on the historical acceptance rate estimates.

---

Horizontal Cascade
Vertical Cascade Horizontal Cascade
Each circle represents token generated by:
Vertical Cascade
Dynamic Tree
Má;
Cascade (DyTC)
reeedoeedd Paralial
Step 1:
Step 2: 2-level VC
Ma;
3-level
Decoding
ventication
vet,,, PC(M,....))
VC Ma)
Ma, (y2 (Rotiom Draft Modal)
3)
2-level VC VCIMa,.
Stop
Step
Bottom Draft Model
3-level VC VOVM,,
Figure 2: Illustration a example of the Dynamic Tree Cascade (DyTC) algorithm when n
Hardware-Aware Latency Prediction (²): The cost coefficient c(M,, Ma.) depends on the specific DSIA strategy and the hardware platform. DyTC utilizes latency model predict these costs. We predicted the roofline latency the hardware platform with Bayesian lincar regression.
DyTC Algorithm. Firstly, we start by generating tokens at the leaf node where the accumulated acceptance rate 11/=1 dj is the highest. Here / is the path length from the root node to the leaf node and di is the estimated acceptance rate of the j-th node in the path. At each decoding step. DyTC evaluates different cascade configurations. "configuration" involves a selection of DSIA models, their draft lengths, and their arrangement. For simplicity, consider round of vertical cascade as a holistic step. For example, at the s-th step of generation, DyTC considers:
Generate with a single model Ma, with a draft length of k, (commonly small for finer control).
A vertical cascade:
(..,, )), with an expected draft length k, (the draft length cannot be strictly controlled since it depends on the acceptance of the low-level draft tokens).
End the generation of the draft tokens (if the accumulated acceptance rate I j==1 & is too low).
The decision-making process aims to maximize the overall EWIF, choosing proper configurations Ma, and hig. Firstly, start with Greedy Search that optimize the predicted local speedup
[I ayak, [1 di of the current step. However, the speedup of CSD may not obey the Greedy Choice Property, that is, choosing the locally optimal solution each step does not guarantee globally optimal EWIF. For instance, if there are two draft models: Ma, with d'd, = 0.9 and cd, = 0.4, and Ma, with agg = 0.8 and c²2 = 0.3, the local speedup when K; = is (MM,), = 0.8 Re 2.67 at the first step. The Greedy Search algorithm would select 0.4 2.25 and (MM,) = 0.3
f. a(1 a(1-af))
Md, every step, with suboptimal overall EWIF of 1.554. On the other hand, if we the horizontal cascade of Ma, and Md we can achieve an overall EWIF of 1.615.
Dynamic programming could be used to find the optimal global solution, but the computational overhead prohibitive for online scheduling since the search space grows exponentially with number of steps. Inspired by the concept of "admissible heuristic" from the A* algorithm[34], propose to adjust the local optimization target considering not only the estimated speedup brought by the current step, but also the estimated least futuree speedup to address this issue. We define the least future speedup as the EWIF of using the Bottom Draft model Ma, for the following draft step. So the subproblem of the s-th step is maximize the following objective function:
6(1-6*)) &.aadd
ToMa,, M.MM,,ka K.) II a
where (MM (.dd,,k)) k.) čk + cú.
(5)
1=1
Since 1 dj only depends on the chosen leaf node, we first find the best leaf node with the highest
accumulated acceptance rate. We stop the generation of the draft tokens if ad ed I dg < Emén
7

---

Algorithm 1: Dynamic Tree Cascade (DyTC) Draft Generation
Input Initial prefix zg. Maximum tree size Mtree_maz
Set of candidate draft model configurations S, Bottom draft model Ma., with dd Cd Minimum overall speedup threshold min. Maximum draft length per expansion step kmax
Top-K token selection K, Top-P tree probability threshold Ptree
Output: Generated draft token tree T,
Initialize T, with Noot representing the last bonus token Zig Dictionary tracking accumulated acceptance rate P aacel (accl[Noot! <- 1.0
3 Dictionary tracking active nodes Jactive[Nroon f- True // Mark Nroot as active leaf while T,.size() < -Mrrreemaz do
Nieaf +- org maXNeT, (Pacee Dactive [N] = True}
if Nicaf f is null then
// No more active leaves to expand (S", ,T) <- ( -iialleetoorriiuaaaiaaaaorrppppp Od, &d,, kmaa))
L break
S is null then
10
DuctivelNieaaf t False break
11
// No beneficial configuration, stop expansion
12
< GetSequenceTollode
13
Faiblings < +GetSiblingTokess (Nteaf)
14
if S is not Md, And Esiblings is not empty then
15
L x + concat siMMings)
// Tree-based sequence parallelism
y E x + G xx)²))
16
åg current estimate (S*) for i < to k* do
18
for j in arg po [i, do N Nyarent + wew.eiilingoooeeeeNeaa....
20
// returns Niaaf if j 0
21
N new add coadd__hhil(Npoorrr porents y [[, info from S*)
22
if = then
23
Nfirst + Nnew
24
Pace Pace(Nnew) Paec((paaeet) x åg-
25
Dactivel [Nnw]] + True
26
N Nparent + Nfirst
27
if T sr.size)) 2 Meree_ max then return T,
28
// Tree size limit reached
29
return T,
where lmin is a threshold for the minimum local speedup, or the total tree size exceeds the maximum size m. Then we get the best configuration Ma, and k, by solving the optimization problem in Eq.
Ma, K2 arg max Ts(Ma, ks), s.t. K2 [1, kmax]
(6)
Má,
Tree-based Parallel Draft Generation. Draft models constructed with DSIA strategies are them- selves an LLM variant, with memory-bounded inference process. Tree attention allows for not only parallel verification of multiple candidate draft paths, but also parallel generation draft paths. Following the idea of tree-based parallel decoding in SpecInfer [32], we can generate draft tokens for multiple sibling leaf nodes in parallel. Given the TOP-K selected siblings N ......., Nem..
(m = K - 1) of the selected leaf node Nieaf- select TOP-P sibling nodes based on the normalized probability of the drafted candidate tokens. In memory-bounded decoding process, a slightly larger sequence of input tokens each step is acceptable and does not significantly affect the overall latency of draft generation. Thus, we use the same draft length k, for all selected sibling nodes to simplify the implementation.
The algorithm summarized in Alg. 1. The detailed algorithms functions FindBestLeaffodee, Fi FindBestConfig, and GenerateDraftTook presented in Appendix D.

---

Table
Overall speedup compared Autoregressive Decoding on Spec-Bench. Models:
Vi cuna--B-vvi 7B.v1.3, 3, Aicuaa11B-11 3, and Aicuna-33B-a v1. 3. [36] Bold indicates the best perfor- mance among training-free methods. Underlined indicates the best overall performance, including
methods with training. CAS-Spec denotes CAS-Spec with Kangaroo and PLD
Model
Method
MT-Bench
Trans
Summary
QA
Math
RAG
Overall
Lade
1,386
1.172
1.173
1.253
1,567
1.078
1.274
PLD
1.563
1.046
2.276
1.109
1.603
1.642
1,539
7B
SWIFT
1.073
1.075
1.096
1.019
1.067
1.055
1.064
CAS-Spec
1.598
1.103
2.268
1.145
1.664
1.676
1.578
Kangaroo
1.698
1.307
1.548
1.409
1.658
1.581
1.534
CAS-Spec
1.727
1.312
2.327
1.407
1.701
1.695
1.696
Lade
1.281
1.065
1.132
1.128
1,480
1.068
1.196
PLD
1,418
1.020
2.104
1.035
1.577
1.673
1.458
13B
SWIFT
1.155
1.087
1.196
1.040
1.106
1.142
1.119
CAS-Spec
1,562
1.134
2.063
1.107
1,582
1.691
1.524
Kangaroo
1.652
1.244
1.483
1.340
1.652
1.508
1,482
CAS-Spect
1.732
1.251
2.337
1.401
1.719
1.689
1.673
Lade
1.295
1.085
1.159
1.165
1,535
1.114
1.229
PLD
1.431
1.047
1.891
1.061
1.523
1.396
1.385
33B
SWIFT
1.218
1.187
1.244
1.152
1,218
1.221
1.206
CAS-Spec
1.547
1.176
1.862
1.186
1.563
1.490
1.481
5
Experiments
We conduct comprehensive experiments to evaluate the effectiveness of our proposed Cascade Adaptive Self-Speculative Decoding (CAS-Spec) algorithm. We aim to answer the two research questions posed in Section 3.
5.1 Experimental Setup
We evaluate CAS-Spec on range of widely-used open-source LLMs, including Llama-2-1 [35] and Vicuna-vi. [36] family. These models represent different architectures and training objectives, allowing us to assess the generalizability of CAS- CAS-Spec. All experiments ensure lossless decoding, meaning that the output is identical to that standard autoregressive decoding. We choose Spec- Bench [26] and for evaluation. Spec-Bench is comprehensive benchmark including various tasks such as multi-turn conversations, mathematical reasoning, and summarization. For all datasets, we measure the generation speed for producing 1024 tokens. To demonstrate the versatility of CAS-Spec, we conduct experiments on eevver-grad GPU (NVIDIA HI00 80GB). The primary metric is Speedup, defined as the wall time of autoregressive decoding divided by the wall time of the speculative decoding method.
CAS-Spec Configuration. For CAS-Spec, we construct hierarchy of draft models using the following DSIA strategies, chosen for their training-free nature and effectiveness:
DSIA (Layer Sparsity): Skip every other Transformer layer Mr, following SWIFT [9].
DSIA (Early Exiting): Exit after subset of Transformer layers, then decode with trained adapter, following Kangaroo [13]. (Kangaroo is a non te--traaninn-frre method and only provides the adapter weights for 7B and 13B models.)
bottom draft model (Ma,,) ): Prompt Lookup Decoding (PLD) is used as the final, fastest stage, as it has negligible computational cost.
Our CAS-Spec implementation primarily uses three-level DSIA cascade: Ma, and Mag followed by Ma, (PLD). The DyTC algorithm dynamically selects between these options and adjusts draft lengths (himan set to 5, Imin set to 1.1). Detailed configurations and hyperparameters for CAS-Spea provided in Appendix E.

---

5.2
Main Results
Table summarizes the main speedup results. CAS-Spec consistently outperforms all baseline on- the-fly (training-free) speculative decoding methods across all tested models, and datasets. CAS-Spec achieves speedups ranging from 1.10x to 2.27x. This significantly surpasses individual training- free methods like PLD and SWIFT. Notably, CAS-Spec performance is competitive with, and in some cases exceeds, reported numbers for Kangaroo, despite CAS-Spec being entirely training-fiee frreen With trained methods like Kangaroo, which leverage small tuned head for early exiting, we can achieve more substantial gains over both PLD and Kangaroo. The detailed comparison between training-Foe free and not-training- free methods is provided in Appendix F.I. As shown in Figure for Vicuna- 7B 3, CAS- Spec achieves an average speedup of 47% % over CS- -S-Drafting [10](VC+HC) and 48% over tree algorithm in SWIFT [9].
5.3
Discussion
The experimental results robustly demonstrate that CAS- Spec achieves SOTA speedups among on-the-fly specula- tive decoding methods. Addressing RQI: Our findings confirm that training-fres self-speculative methods can be effectively layered to construct cascade that significantly outperforms a single, strong statistical draft model like PLD under proper cascade scheduling. Addressing RQ2: The ablation study on DyTC clearly shows its superiority static cascade scheduling. The ability dynamically route through the draft model hierarchy and assign draft lengths based on runtime heuristies (acceptance rates and latency predictions) is crucial for maximizing performance. This adaptability allows CAS-Spec to handle variations in generation difficulty and hardware characteristics more effectively, boosting
1.8
PLD
1.2
1.17
Speeciup Ratio
1.07
1:06
0.91
0.0
0.6
0.4
applications [37, 38].
0
HG
Tr
DyTa
Speculation Algorithms
The training-Foe free nature of CAS- CAS-Spec, combined with its high performance, makes a practical and attractive solu- tion for accelerating I.I I.LM inference diverse deployment scenarios. It can be readily integrated with existing LLMs without the need for costly retraining or maintaining sepa- rate draft model weights and KV caches (beyond the DSIA modifications to the target model's inference path).
Figure 3: Speedup of different meth- ods relative to baseline. AR (1.0) and PLD (1.54) reference lines are shown. The vertical line separates two groups of methods.
Conclusion
In this paper, we introduced Cascade Adaptive Self-Speculative Decoding (CAS-Spec), a novel algorithm for lossless LLM inference acceleration that eliminates the need for training separate draft models. CAS- constructs a hierarchy of speculative draft models by leveraging dynamically switchable inference acceleration (DSIA) strategies, applied to the target model. This approach offers significant flexibility and ease of integration.
core contribution of our work the Dynamic Tree Cascade (DyTC) method. DyTC adaptively routes generation through the multi-level draft models and assigns draft lengths based on online heuristics of acceptance rates and latency predictions. This dynamic scheduling allows CAS-Spec to optimize its performance continuously during inference.
Our experiments demonstrate that CAS-Spee achieves state-of-the-a acceleration among on-the-fly speculative decoding methods. The results validated our hypotheses that (1) training-free DSIA strategies form effective cascades over fast bottom draft models, and (2) dynamic scheduling via DyTC further enhances these gains.
CAS-Spec offers a compelling solution for practical LLM deployment by providing substantial speedups without the overheads associated with training and maintaining external draft models. Its adaptability and ease of integration make it valuable tool for wide range of LLMs.
10

---

References
[1] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL httpp:///axiii wwg/ /202//2302..0 13971.
[2] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[3]
Qwen, :, An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: /aaxiv..v..////////////////aa org/ wtwp///2........ 15115.
[4] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding, 2023.
[5] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving. Jean- -aptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. 2023.
[6] Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding:
Exploiting speculative execution for accelerating seq2seq generation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909-3925, 3925, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/0233 fnn.2...nnnnggg--em..222.. aclanthology. wrgg windinnnneemiii 257/
URL https://
[7] Jun Zhang, Jue Wang. Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mchrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11263-11222, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ acw.-lllllon..ooooo long.607. 607. URL /tttps://acaatttatat
v1/2024.acl-oon
org acl 1ong..on...60. 607/
[8] Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. LayerSkip: Enabling early exit inference and self-speculative decoding. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 12622-12642, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1220:5////112222aalllolooog 681/
URL https: / /aalaaaaaannooooyyyooo org/2024. 2024. acl- long.
[9] Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, and Wenjie Li. SWIFT: On-the-fly self- speculative decoding for LLM inference acceleration. In The Thirteenth International Con- ference on Learning Representations, 2025. URL https://openrrview net/ EKJhH5D5wA.
form??dd nit.rrru
[10] Ziyi Chen, Xiaocong Yang. Jiacheng Lin, Chenkai Sun, Kevin Chen-Chuan Chang, and Jie Huang. Cascade speculative drafting for even faster Ilm inference. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 66226-86242. Curran Associates, Inc., 2024. URL htwps:tttt/ttt /ttppee:///roooeeeiigg.ssss www.::ss.....rpre.eeeeeeeeeeeeee cc55083taefcaabb005dd0202222225 Porpe
wwtpccccpppeeeeeeeeeeeeeee faaper_fille wllllffllleeeeeeee Dtteee//apper/222///12.. Conference pdf.

---

[11]
consistency large Anonymous. CLasp: In-context layer skip for self-specullative decoding. In Submitted to ACL Rolling Review December 2024, 2025. URL http::://pperreviewwwi/e wwt.:mmmndooo t1BBPa0z8z. under review.
[12] Jiahao Liu, Qifan Wang, Jingang Wang, and Xunliang Cai. Speculative decoding via early- exiting for faster LLM inference with Thompson sampling control mechanism. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors. Findings of the Association for Computational Linguistics: ACL 2024, pages 3027-3043, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18633/1120244 / acl aclanthology htw.hhoooooyyogg/2222222 wwg//024........o.o findings acl. 179/.
URL https://
[13] Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Duyu Tang, Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding for accelerating LLMs via double early exiting. In The Thirty-eighth Annual Conference Neural Information Processing Systems, net/ 2024. URL https: http::::///p/eeeeeeeeeeee wwt./eeeuniiiieao 1rrrmmid1133o00mmpp
[14] Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency- throughput tradeoff for long context generation with speculative decoding. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrewe http/forummidCSSJJaaaa
[15] Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. In First Conference on Language Modeling, 2024. URL https: htttps://opnrreveww net/forummid HVK6n13197.
[16] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. In Proceedings of the 41 International Conference Machine Learning, ICMI/24. Jorg. 2024.
[17] Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: language models. In Proceedings of the 41 International Conference on Machine Learning. ICML'24. JMLR.org. 2024.
(18) Feng Lin, Hanling Yi, Yifan Yang. Hongbin Li, Xiaotian Yu, Guangming Lu, and Rong Xiao. Bita: Bi-directional tuning for lossless acceleration large language models. Expert Systems with Applications, 279:127305, 2025. ISSN 0957-4174. doi: https://doi.org/10.1011 j.eswa.2025. 127305. URL httph:t////w// http:::/ww..ccieecdddeetttt sciencedirect. wwtp://cce//ccccceieieeeeeeeeeee ctm////ceeeeee//ttttii///ppiiiii 50957417425009275
[19] Juntao Zhao, Wenhao Lu, Sheng Wang, Lingpeng Kong, and Chuan Wu. Qspec: Speculative decoding with complementary quantization schemes, 2025. URL 2410. 11305.
[20] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference Machine Learning, ICML'24. JMLR.org, 2024.
[21] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421-7432, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 1/2.6663/// aclanthology org/:///////4/20 ac..anttoloog..orr//22..... 0rg/2004..mm1pmmii..... emnlp-main.n... 422/.
URL https://
[22] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test, 2025. https: abs/2503. 01840.
org/
[23] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple Il inference acceleration framework with multiple decoding heads. In Proceedings of the stst International Conference on Machine Learning, ICMI.24. JML.. 2024.
12

---

[24] Evangelos Georganas, Dhiraj Kalamkar, Alexander Kozlov, and Alexander Heinecke. MI- specqd: Multi-level speculative decoding with quantized drafts, 2025. URL https:////axii trggabss25503 13565.
[25] Apoory Saxena. Prompt lookup decoding. GitHub repository, 2023. URL htRphhtps:////thuub Accessed: 2024-01-03.
com/apon wapporvmaangpproopp.t....mmmo
[26] Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehen- sive survey of speculative decoding. In Aun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024, pages 7655- 7671, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 118863//1/2222/ findinggs- acl.456. findings-acl.456. URL http:::://alaathlloggooo org/ 2024.findime findings--a...... 456..
[27] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetune language models, 2022. URL https: / arxiv org/ abs/2210 2210. 11416.
[28]
Yixin Song. Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large language model serving with consumer-grade gpu. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, SOSP '24, page 590-606, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400712517. doi: 10.1 00.114522644715533655666 URL https://doi 10.g//1...0 1145/3694111 366669999497...99444..9 3695964.
[29] James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Training-free activation sparsity in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https::///// /ttp://evee..eeeeeeeeeeee hwtprerrrriieenee/ffourmm net/forum? id=dGVZwyg5tV.
[30] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL //pen//v/eeeeeeeeeeeeee wwwpwwwwwwweeeeeeeeeeeeeeeeee fort//form?? iddimmddddGG75S
[31] Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageatten- tion2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. In International Conference Machine Learning (ICML), 2025.
[32] Xupeng Miao. Gabriele Oliaro, Zhihao Zhang. Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serving with tree- speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS '24 page 932-949, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703867. doi: 10. 10.1145732206663351133..
URL https: //op::::///....o/ org/ 11..1..115556226660 3651335.
[33] Yunfan Xiong, Ruoyu Zhang, Yanzeng Li, Tianhao Wu, and Lei Zou. Dyspec: Faster speculative decoding with dynamic token tree structure, 2024. URL https: http://axxxv....abbba. 11744.
[34] Peter Hart, Nils Nilsson, and Bertram Raphael. formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 100) 107, 1968. doi: 10.1119/tscc.1168.33000 300663. URL /tsttp:://ooi org/ /g//100.11 1109/tsscc.c... 1968. 300136.
[35] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The laaa 3 herd of models. arXiv preprint ar 207722240..21788 2024.
13

---

[36] Wii-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL
[37] Jiawei Shao and Xuelong Li. Ai flow at the network edge. IEEE Network, pages 1-1, 2025. doi: 20.10000/EETT.2225555515
[38] Hongjun An, Wenhan Hu, Sida Huang. Siqi Huang, Ruanjun Li, Yuanzhi Liang, Jiawei Shao, Yiliang Song, Zihan Wang, Cheng Yuan, Chi Zhang, Hongyuan Zhang, Wenhao Zhuang, and Xuelong Li. Ai flow: Perspectives, scenarios, and approaches, 2025. URL https: / //arxiv..o www./ogg/////a/eeee/ooooeee 2506. 12479.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We include a clear statement of the main claims made in the abstract and introduction. The analysis, experiment results and implications all support the claims made and our contributions. The scope of this paper is stated as well.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We comprehensively discuss the limitations of our work the paper in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
14

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
This paper includes only some computations, which are quiet simple in mathematics. We found that the provided equations and explanations may fit the readers well in understanding.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We have illustrate all the important experimental settings in Experiments. Other information, such as training and evaluation details, is included the Appendix Our results can be fully reproduced by following all the information.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
15

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will submit the code of the full project following the NeurIPS code and data submission guidelines. Meanwhile, the training dataset and evaluation benchmarks we used in this paper are all open-access, and related settings and processing codes are also included in the submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
coompolicee ooicyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. /t//cc/pp//o.... http//ullcggggudcee Seeiiidee/oooSubbn wwweeeSuumiiiiooniin wwwisssoiicccyiyi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified all the training and test details in the Appendix E, The related codes are also submitted.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We do conduct extensive experiments that support the main claims of the paper and report all the results in Section and Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We implement experiments on H100 80GB for different backbone LLMs. We have displayed the detailed information of setting in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://nneuruupeeeeee http://s....ppllleccceeecenaaaae
Answer: [Yes]
17

---

Justification:
We have reviewed the code to confirm with the NeurIPS Code Ethics..
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
There no direct societal impact from our work. Our contribution focuses solely on improving the inference efficiency of large language models through a decoding speedup technique. The content or accuracy of the generated responses- -including any potential misinformation or ethical concerns- is entirely determined by the underlying backbone model, which our method does not modify or influence.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
18

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Part of our training code and the used data and benchmark datasets are from open-access projects and works. We have cited the responding works and authors.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have included a tutorial of our released
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
19

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.MM policy (https: http://eeerrsppeeeee http://uurpss..c/ooeeeeeeeeeeeee for what should or should not be described.
20

---

A Limitations and Future Work
While CAS-Spes demonstrates promising performance, its effectiveness inherently tied to the quality and efficiency of the chosen DSIA strategies. The online estimation for DyTC, though lightweight, introduces some computational overhead and require short warm-up period for optimal performance. Moreover, since dynamic scheduling depends on the heuristics of the generation of certain token sequence, its improvement is lessened when the batch size is large. Other limitations include the incompatibility of CAS-Spec with current SOTA speculative decoding methods like EAGLE3 due to their reliance on hidden states of the verify model.
Future research could explore integrating more sophisticated or newly developed training-free SSD techniques as DSIA components within the CAS-Spec framework. Further refinement the DyTC algorithm, potentially incorporating more advanced online learning for scheduling, could yield additional improvements. Investigating the application of CAS-Spec even larger models or different modalities, and exploring hardware co-design to optimize DSIA strategy execution, are also promising avenues for future work. As the field of see--speculatie decoding continues evolve, CAS-Spea provides flexible and powerful framework to leverage these advancements.
B Theoretical Analysis
For certain hyperparameters (kg, n, k. ka k.2). we can derive the theoretical bounds for the cost coefficient of MMr such that the speedup of the cascade is greater than the speedup of standard speculative decoding with Maz alone.
For vertical cascade, the solution of the inequality (Ma 2 (TD(Mag) gives the bound:
Oue, (o(az))))
O'd, (Cay ko +1)
Cdi 1
(1 + nkcas)
1o+1
adi
a
d2
1 1-0xa)
where p(ad,) = (1 (1-as, (1+tka ea,)
For horizontal cascade, the solution of the inequality T THo(Ma, Maz)) 2 TSD(Mag) gives the bound:
Kay +1 ds
Kdz)
)
( (11 Ord ( kids + +1] (1 + Kids Gdz))
Odg (1 Od2s
Raj
Cd S
+ Otd
kid
1 add
1 1 Odds
kdz
a
C DSIA
Other than the layer sparsity, also consider the activation quantization and activation sparsity as two candidates Spec framework. For activation quantization, we refer to the implementation of QSpec. While for activation sparsity. we adopt the TEAL's method of this DSIA strategy.
Furthermore, since the activation quantization and activation sparsity are both orthogonal the layer sparsity, we can construct draft models with Mixing-DSIA Cascade. For instance, we can construct Ma, as the W4A4 quantized model, and Ma, as the W4A4 model with layer sparsity and activation sparsity.
However, using these DSIA strategies inherently requires a weight-only quantized draft model with batch size of 1. This configuration is suitable for edge inference, but not general enough for large-scale inference scenarios. Therefore, we do not include the experiments on these two DSIA strategies in our CAS- framework by now.
D Dynamic Tree Cascade Algorithm
Maintaining Estimates for Inactive Configurations. For draft configurations not currently selected in given step, their acceptance rate estimates are preserved and do not decay. When a previously unused configuration becomes active again (e-g., due to changing generation context), estimate updated based on actual verification outcomes. In cold-start scenarios for configurations that have
21

---

never been used, we initialize estimates using either: (1) brief offline profiling on representative tasks, or (2) heuristic priors based on the DSIA strategy's aggressiveness (c.g.. higher layer sparsity implies lower expected acceptance rate).
Note that for vertical cascade configurations like (C(Ma; Ma,,), maintain single acceptance rate estimate corresponding to the highest-level draft model Ma, since all draft tokens are ultimately verified by this model before proceeding MM2.
The detailed functions mentioned the Algorithm are as follows:
Algorithm 2: Dynamic Tree Cascade (DyTC) Functions
Function FindBest Conf ewwuutiiooorrteeee Srtrrbeestttm..guttrtioooorrrre (Sandidates dd, Edu, liman):
MMest choice <- null; _best_choiee choice + 0; max_ obj_vall val t-
foreach configuration S E Scandidates do 0g <- current estimate (s) êg f- current estimate 8(S) for k if Egkk Cd.. & then
kiman do
continue
float Eaccepted: if āg 1.0 then Eacrepted &- k
10
11
else
Facceette + ( (15))((1 aa)
13
float Trat t- (Eaccepted + a (c + Con)
14
if Teal -aa mmaa_ob) ____oj_jvo val then
15
mox mor__bjj_voe +- TToal S
Mbest_choice kbest_ cchoice t- k
17
if max obj_vall val < then
19
L return (null, 0)
20
return (Mbest, [bbestcchiices choicess Rhest__phoiee) choice)
E Experimental Details
In this section, we provide the details of our experimental setup, specifically for CAS-Spec configura- tion in the main experiment. Our draft models are constructed using mainly layer sparsity based SWIFT, which is training-free and on-the-fly DSIA strategy. The hierarchy is got by Scaling-DSIl Cascade, which tunes the layer sparsity to get different draft models. We also consider the Prompt Lookup Decoding (PLD) as the bottom draft model. The configuration of CAS-Spec is as follows: Ma. has around 0.4 layer sparsity, while Ma, has around 0.6 layer sparsity. Thus it leads to multiple candidate Ma, for each step:
basic model: Mai (TS = 0.4), (TS = 0.6), Ma, (PLD) 2-Level VC: VC (Ma,; Ma,), (C,,
3-Level VC: VC (Ma, VC(Ma,
However, due the insuffieient gap between the layer sparsity of Ma; and May. the 3-Level VC configuration turns out to be less efficient and hence rarely used in the DyTC algorithm. Therefore, we present the results of 2-Level VC in the main experiment.
F Additional Experimental Results
Figure shows the speedup of different methods on Vicuna-7B-vi.3 model. refers to the layer sparsity only drafting without tree attention. VC, HC, VC+HC are the framework of using the vertical
22

---

cascade and horizontal cascade with PLD, based on the implementation of CS-Drafting Tr is the standard SWIFT implementation with Tree Attention. Tr+VC follows the same implementation of CS-Drafting but with the tree attention. DyTC is the version with Dynamic Tree Cascade algorithm. Compared VC+HC configuration, DyTC achieves 73% improvement in average speedup. Compared to Tr (SWIFT), DyTC achieves 47% improvement in average speedup.
F1
Comparison with Trained Methods
Table 2: Comparison with trained methods on Vicuna-7B-vl. model.
Method
Training-Free
#Mean accepted tokens
Speedup
PLD
Yes
1.75
1.54x
SWIFT
Yes
3.01
1.06x
CAS- (SWIFT.PLD)
Yes
3.43
1.58x
Speculative Decoding (Vicuna 68m)
No
2.27
1.44x
Medusa
No
2.39
1.69x
EAGLE
No
3.57
2.05x
EAGLE2
No
4.36
2.21x
In comparison with trained speculative decoding methods, our CAS-Spec framework with SWIFT and PL.D achieves higher speedup than vanilla speculative decoding methods with Vicuna 68m, while being training-free. The gap between training-free and not-training free methods is significant, as trained methods like Medusa and EAGLE achieve higher acceptance rates and speedups, leveraging the full model's capabilities.
23


Reviews:
[{"review_id": "Beh58Htnfn", "reviewer": "Reviewer_5NsN", "summary": "This paper proposes \"CAS-spec\", a method that dynamically builds a cascade of (training-free) draft models (and determines their corresponding lookaheads) to maximize speedup. The types of draft models considered fall under the category of \"self speculative decoding\" where pruned or quantized versions of the target model are used as the speculator. The algorithm is dynamic because it maintains an online prediction of the acceptance rates and latencies of each of the draft models, and adjusts the cascade structure accordingly. CAS-spec attains meaningful speedups over the other training-free speculative decoding baselines considered in the paper (Lade, PLD, SWIFT).", "questions": "- How do you maintain estimates of acceptance rates for draft models not being used in the current cascade?\n- I'm confused by the interplay between the token tree being constructed, and the dynamic cascade algorithm. How are cascaded speculators combined with tree-based speculation/verification?\n- Do you actually use token trees in your experiments, or do you just use sequences?\n- What hierarchy of draft models do you use in your experiments? In the main paper it says the available draft models are PLD and 50% layer-sparsity, but in Appendix E it says PLD and 40% and 60% layer sparsity?\n- Could you compare with Eagle and other SOTA speculative decoding techniques, to clarify the gap between training-free and not-training-free methods?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The idea of building an optimal cascade automatically, without requiring an extensive grid search in an exponentially large search space, is quite nice.\n\nWeaknesses:\n- The paper suffers from a lot of \"mathiness\" and lack of clarity. The presentation is extremely dense and full of math, even though if I understand correctly actual implemented algorithm is quite simple: A cascade consisting of PLD -> layer-sparsity (50%) -> target model, where the algorithm uses an A*-style search algorithm to efficiently find the best hyperparameters (lookahead for each draft).\n- The proposed method need not only apply to training-free draft models. It would have been quite informative to see how this method can help accelerate speculative decoding with SOTA draft models (e.g., Eagle-3).", "was_revised": true, "final_justification": "I thanks the authors for their response.\n\nI remain concerned about:\n1. Clarity: I'm still not sure I understand the algorithm. Thus, perhaps it would be best for this paper to undergo serious revision and be resubmitted to a future conference.\n1. Impact: If this only applies to training-free draft models, that limits the impact of this work. A deeper investigation into how this method works with non-training-free draft models would help demonstrate broader impact.", "rebuttal": "We sincerely thank the reviewer for their thoughtful feedback and engagement with our work. Below, we address the key points raised:\n\n**[W1]. On Clarity and \"Mathiness\"**\n> The paper suffers from a lot of \"mathiness\" and lack of clarity. The presentation is extremely dense and full of math, even though if I understand correctly actual implemented algorithm is quite simple: A cascade consisting of PLD -> layer-sparsity (50%) -> target model, where the algorithm uses an A*-style search algorithm to efficiently find the best hyperparameters (lookahead for each draft).\n\nWe appreciate the feedback on the presentation and agree that the core contributions could be highlighted more clearly.\n\nThe mathematical analysis in Section 3 and 4 is intended to address a key research question (RQ1): can a cascade of training-free self-speculative decoding (SSD) methods provably outperform a single, strong statistical baseline like PLD? Our theoretical bounds (Fig. 1b, 1c) demonstrate that this is non-trivial and depends on a delicate balance of acceptance rates and costs, motivating the need for an adaptive algorithm.\n\nWe acknowledge that our description may have oversimplified the algorithm's implementation. The core novelty of our Dynamic Tree Cascade (DyTC) algorithm is not a fixed cascade or a simple A* search. Instead, at each step of generation, DyTC dynamically:\n- Selects a configuration from a set of candidates (e.g., LS(0.4) only, LS(0.6) only, VC(LS(0.4), PLD), etc.) based on online estimates of acceptance rate and latency.\n- Determines the optimal draft length (k*) for that chosen configuration.\n- Constructs a tree structure by generating multiple candidate sequences in parallel, as illustrated in Figure 2.\n\nThis step-wise optimization is what allows CAS-Spec to adapt to the changing difficulty of token generation throughout the sequence.\n\nIn the revision, we will significantly streamline the mathematical exposition in Section 4 and add a clearer, high-level overview of the DyTC algorithm's dynamic, step-wise decision-making process to better distinguish it from a static cascade.\n\n**[W2/Q5]. On Comparison with SOTA Trained Models like EAGLE**\n> The proposed method need not only apply to training-free draft models. It would have been quite informative to see how this method can help accelerate speculative decoding with SOTA draft models (e.g., Eagle-3).\n\nOur primary focus was to push the limits of what is achievable with on-the-fly, training-free methods, which offer maximum ease of deployment. While applying CAS-Spec to trained draft models is an interesting direction, there are two key challenges:\n\n1. Methods like EAGLE rely on specific hidden states from the full target model to generate drafts. Introducing an intermediate layer-sparsity draft model (like SWIFT) would alter these hidden states, likely disrupting EAGLE's prediction mechanism and reducing its effectiveness.\n2. Diminishing Returns: SOTA trained models like EAGLE already achieve very high acceptance rates. The potential speedup from adding another, less accurate draft model in a cascade is minimal and could be outweighed by the overhead of the additional stage. Our framework would likely learn to route directly to the EAGLE draft, bypassing the intermediate stages.\n\nHowever, other trained speculative decoding methods can be naturally integrated into our CAS-Spec framework. For instance, we experimented with a SOTA self-speculative decoding method based on early exit (Kangaroo) instead of SWIFT on Vicuna-13B:\n\n| Method | MT-Bench | Translation | Summarization | QA | Math Reasoning | RAG | Overall |\n|--------|----------|-------------|---------------|----|----------------|-----|---------|\n| **Kangaroo** | 1.65x | 1.24x | 1.48x | 1.34x | 1.66x | 1.51x | 1.48x |\n| **PLD** | 1.56x | 1.04x | 2.28x | 1.13x | 1.54x | 1.67x | 1.53x |\n| **CAS-Spec (Kangaroo,PLD)** | 1.72x | 1.23x | 2.35x | 1.40x | 1.72x | 1.69x | 1.67x |\n\n\nWe will add a discussion in the paper clarifying the scope (training-free methods) and elaborating on these technical reasons why cascading with models like EAGLE may not be straightforward or beneficial, thereby contextualizing CAS-Spec's SOTA position within the training-free paradigm.\n\n> Could you compare with Eagle and other SOTA speculative decoding techniques, to clarify the gap between training-free and not-training-free methods?\n\nWe will include a comparison with EAGLE and other SOTA speculative decoding techniques in the revised paper. Currently we've completed the comparison in the following table:\n\n| Method | Training-Free | #Mean accepted tokens | Speedup |\n|--------|---------------|-----------------------|---------|\n| **PLD** | Yes | 1.75 | 1.54x |\n| **SWIFT** | Yes | 3.01 | 1.06x |\n| **CAS-Spec (SWIFT,PLD)** | Yes | 3.43 | 1.58x |\n| **Speculative Decoding (Vicuna 68m)** | No | 2.27 | 1.44x |\n| **Medusa** | No | 2.39 | 1.69x |\n| **EAGLE** | No | 3.57 | 2.05x |\n| **EAGLE2** | No | 4.36 | 2.21x |\n\nIn comparison with trained speculative decoding methods, our CAS-Spec framework with SWIFT and PLD achieves higher speedup than vanilla speculative decoding methods with Vicuna 68m, while being training-free. The gap between training-free and not-training-free methods is significant, as trained methods like Medusa and EAGLE achieve higher acceptance rates and speedups, leveraging the full model's capabilities.\n\n**[Q1].On Estimating Acceptance Rates for Inactive Models**\n> How do you maintain estimates of acceptance rates for draft models not being used in the current cascade?\n\nThank you for this clarifying question. The DyTC algorithm maintains online estimates (â) for all candidate draft configurations, not just the one currently in use. The process is as follows:\n- All configurations are initialized with acceptance rates from prior heuristics, (or a brief warm-up phase in cold-start situations).\n- At each generation step, the FindBestConfigurationForStep function (Alg. 2) evaluates the potential speedup of all candidate configurations using their most recent â values.\n- The chosen configuration is used to generate tokens. Its acceptance rate â is then updated using a moving average based on the outcome of the verification step.\n\nThis ensures that if a previously sub-optimal configuration becomes more promising (e.g., due to a change in generation context), the algorithm can switch to it.\n\nWe will revise Section 4.2 to make this online estimation and update process for all candidate models more explicit.\n\n**[Q2, Q3]. On the Interplay Between Cascade and Token Trees**\n> I'm confused by the interplay between the token tree being constructed, and the dynamic cascade algorithm.\n> How are cascaded speculators combined with tree-based speculation/verification? Do you actually use token trees in your experiments?\nYes, our experiments use token trees, as depicted in Figure 2 and implemented in Algorithm 1. The interplay is as follows:\n- The \"Tree\" refers to the overall structure. From a single verified token, we can grow multiple parallel branches of draft tokens.\n- The \"Cascade\" refers to the method used to generate the tokens along a single branch.\n- At each expansion step, DyTC selects the most promising leaf node in the tree. It then chooses the optimal cascade configuration (e.g., a vertical cascade of LS(0.5) and PLD) to generate a new branch of draft tokens starting from that leaf. This allows for flexible and efficient exploration of the generation space.\nWe will add a paragraph in Section 4.2 to explicitly describe this relationship, clarifying that the cascade is the \"branch generator\" within the broader \"tree\" structure.\n\n**[Q4]. On Inconsistent Draft Model Descriptions**\n> What hierarchy of draft models do you use in your experiments? In the main paper it says the available draft models are PLD and 50% layer-sparsity, but in Appendix E it says PLD and 40% and 60% layer sparsity?\n\nThank you for catching this inconsistency. Our apologies for the confusion. The detailed configuration in Appendix E is outdated. The main paper's description of using 50% layer sparsity is correct.\n\nOne reason we changed the configuration was that the vertical cascade of PLD -> LS(0.4) -> LS(0.6) required more forward processes of LS models and fell short of the expected speedup. Hence this configuration was seldom selected in our scheduling algorithm, leading us to simplify the cascade to just PLD and LS(0.5) for clarity."}, {"review_id": "6l1dY3FV4L", "reviewer": "Reviewer_zLee", "summary": "This submission introduces Cascade Adaptive Self-Speculative Decoding, a methodology for LLM inference acceleration under speculative decoding settings. The proposed approach dynamically adapts the employed draft model through the training-free creation of a hierarchy of draft LLMs, extracted from the target model (e.g. through pruning, quantisation etc). Alongside, a routing mechanism (dynamic tree cascade) is proposed to guide the generation process, by adopting different draft models and drafting lengths by monitoring heuristics such as acceptance rate and inference latency at runtime. The proposed methodology is shown to improve inference efficiency compared to relevant approaches from the literature, across a variety of models and datasets.", "questions": "Please considered replying to the points raised in the comments section above.", "limitations": "Some limitations of the proposed approach are discussed in the appendix, along with proposed future work directions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- CAS-Spec addresses a very interesting and timely problem. The proposed  DyTC is a novel and potentially impactful direction.\n- Overall the manuscript is easy to follow and proposed contribution is clear.\n- The experimental results demonstrate that the proposed approach is effective, in terms of acceleration to AR decoding, and achieves on-par or better speed-up to other technics. \n\nComments:\n- Consider pointing more clearly to the connection between the adopted baselines and related work [10]. Since CS-Drafting is the most closely related approach from the literature, amore comprehensive comparison and discussion is required. Why is CS-Drafting not included as a baseline in Table 1 ?\n- It is unclear why the proposed DyTC only considers running heuristics such as the acceptance rate and inference time and no token-level information such as prediction confidence for the tokens in the sequence. As the authors rightfully pointed out, past performance is not a strong indication for future behavior of the model.\n- Further experimental analysis would benefit the reader understand the behavior of the proposed approach. Consider reporting statistics about the selected model sizes across different tasks/ stages of the generation process, potential comparison with an Oracle scheduling strategy, as well as qualitative examples of the DyTC behavior.", "was_revised": true, "final_justification": "Increasing my score for 3 to 4, following the authors' rebuttal that satisfactorily address my raised concerns.", "rebuttal": "We sincerely thank the reviewer for their insightful and constructive feedback. We are encouraged that they found CAS-Spec to be a \"novel and potentially impactful direction\". We will address the comments below and are confident that incorporating these suggestions will significantly strengthen our paper.\n\n**[Q1]. On the Comparison with CS-Drafting**\n\n> Consider pointing more clearly to the connection between the adopted baselines and related work [10]. Since CS-Drafting is the most closely related approach from the literature, a more comprehensive comparison and discussion is required. Why is CS-Drafting not included as a baseline in Table 1?\n\nThis is a very important point, and we appreciate the opportunity to clarify our relationship with CS-Drafting.\n\n* CS-Drafting is a key inspiration for our work. However, it operates on a pre-existing family of smaller models (e.g., the FLAN-T5 series). In contrast, our CAS-Spec framework is designed for more common scenario where it starts with a single large target model and dynamically creates draft models on-the-fly.\n\n* While we cannot compare with the specific *implementation* of CS-Drafting, we absolutely compare against its core *principles*. Our ablation study in **Figure 3** directly evaluates the static vertical (`VC`), horizontal (`HC`), and combined (`VC+HC`) cascade strategies proposed by CS-Drafting, using our on-the-fly draft models. The results clearly show that our dynamic **DyTC** algorithm provides a 73% speedup improvement over horizontal cascade baseline(`HC`) and 48% speedup improvement over the best static cascade baseline (`VC+HC`). This highlights the importance of draft tree construction and dynamic scheduling in achieving significant performance gains.\n\nFor a more detailed comparison, we will add the cascade strategies from CS-Drafting as baselines in our experiments. For example, the speedup of CS-Drafting with SWIFT and PLD of Vicuna-7B on Spec-Bench is as follows:\n| Method | MT-Bench | Translation | Summarization | QA | Math Reasoning | RAG | Overall |\n|--------|----------|-------------|---------------|----|----------------|-----|---------|\n| **CS-Drafting (SWIFT,PLD)** | 1.01x | 0.95x | 1.21x | 0.98x | 1.05x | 1.08x | 1.07x |\n| **CAS-Spec (SWIFT,PLD)** | 1.59x | 1.05x | 2.13x | 1.11x | 1.61x | 1.65x | 1.58x |\n\nWe will revise Section 5 to explicitly state this distinction. We will clarify that while CS-Drafting is not in Table 1 due to the lack of model families, the core static cascade strategies from CS-Drafting are used as crucial baselines in our ablation study (Figure 3) to isolate and demonstrate the significant benefits of our dynamic DyTC algorithm.\n\n**[Q2]. On the Choice of Heuristics for DyTC**\n\n> It is unclear why the proposed DyTC only considers running heuristics such as the acceptance rate and inference time and no token-level information such as prediction confidence for the tokens in the sequence. As the authors rightfully pointed out, past performance is not a strong indication for future behavior of the model.\n\nThis is an excellent point regarding the potential for more fine-grained control.\n\nIn our current implementation, when selecting the best configuration for a given step, DyTC primarily relies on the online estimates of acceptance rates mainly based on past performance. For selecting a configuration of draft models, we have to predict how different configurations will perform **in the future**. In `FindBestConfigurationForStep`, we cannot know the token-level information like logits for future tokens. And running each of them to get the token-level information would be too expensive.\n\nOn the other hand, we actually considered token-level information in the estimation of acceptance of already **drafted** tokens in order to calculate the *accumulated acceptance rate* $\\prod_{j=1}^{l_s} \\hat{\\alpha_j}$. We didn't explicitly mention this in the algorithm description, because CAS-Spec framework is designed to be applicable to various draft models, and the token-level information can vary (e.g. logits for different draft models and number of matched tokens for n-gram PLD). We acknowledge that this could be made clearer in the paper if we explicitly explain how token-level information is used in the acceptance rate estimation of drafted tokens.\n\nWe will revise the paper to clarify this point, explaining the difference between estimating acceptance rates for future tokens (which relies on past performance) and past tokens that are drafted but not yet verified (considering token-level information for the drafted tokens). We will also discuss the potential for future work to incorporate more sophisticated token-level heuristics, such as prediction confidence, to further improve the dynamic scheduling process.\n\n**[Q3]. On Further Experimental Analysis**\n\n> Further experimental analysis would benefit the reader understand the behavior of the proposed approach. Consider reporting statistics about the selected model sizes across different tasks/ stages of the generation process, potential comparison with an Oracle scheduling strategy, as well as qualitative examples of the DyTC behavior.\n\nThese are fantastic suggestions that would greatly enhance the paper's clarity. We agree that showing how DyTC adapts is crucial.\n\nWe will perform additional experiments and add a new section to the Appendix with the following analyses:\n1. We will include a visualization showing the distribution of draft configurations selected by DyTC for different tasks from Spec-Bench (e.g., summarization vs. coding). This will demonstrate how the algorithm automatically adapts its strategy, for instance by favoring faster, more repetitive models for one task and more accurate, conservative models for another.\n2. We will provide a concrete generation example, illustrating how DyTC switches between draft models in response to the changing generation context in the Appendix.\n3. An oracle scheduler is an excellent benchmark for the upper bound. However, the search space of permutations of all possible configurations on a sequence of tokens is exponential, requiring an impractical amount of experimentation. While implementing a perfect oracle is non-trivial (as it requires knowing the future), we will discuss the concept and its implications in our analysis, framing it as the theoretical goal that DyTC aims to approximate.\n\nFinally, we will update Table 1 to bold the best-performing method in each category as per convention.\n\nWe are confident that these additions will provide a much deeper understanding of DyTC's adaptive behavior and significantly strengthen our contribution. We thank the reviewer again for their thoughtful and valuable suggestions."}, {"review_id": "yZOF871wNk", "reviewer": "Reviewer_8EAL", "summary": "This paper introduces CAS-Spec, a new method to speed up LLM without needing extra training. Current methods either require costly separate \"draft\" models or offer limited speed gains. CAS-Spec tackles this by creating a hierarchy of \"virtual\" draft models from the main LLM on the fly, using techniques like layer sparsity and activation quantization. It then uses a Dynamic Tree Cascade algorithm to intelligently select the best draft model and sequence length during token generation, based on predicted acceptance rates and hardware efficiency. Experiments show CAS-Spec achieves 1.6x to 2.1x faster inference compared to other training-free methods, making LLMs respond much quicker.", "questions": "- What are some detailed features used when prediction latency in Section 4.2?\n- How does the method estimates the acceptance rate using the history information?\n- How does the method sensitive to the hyperparameters such as `k_max`, `t_min`, and the sliding window size for estimating the acceptance rate?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n- The paper addresses the central topic of LLM inference by accelerating speculative decoding. The method has great capacity yet introduces no extra training cost. This makes feasible the cascaded method that used to be too costly.\n- The approach is novel. The approach uses different kinds of ways to create a hierarchy of virtual draft models from the target LLM during runtime and the algorithm is adaptive.\n- The experiments are comprehensive and the ablation study is convincing.\nOverall, I do not have too many concerns about the paper.\n\nCons:\n- How does the method sensitive to the hyperparameters such as `k_max`, `t_min`, and the sliding window size for estimating the acceptance rate?\n- How about the overhead and complexity of the DyTC algorithm?\n- I would expect more details of the latency prediction model.", "was_revised": true, "final_justification": "The authors has addressed my concern. I maintained my score.", "rebuttal": "We are deeply grateful to the reviewer for their positive and encouraging assessment of our work. We are particularly pleased that they recognized the novelty and potential of our training-free approach, and found our experiments to be comprehensive and convincing. We appreciate the thoughtful questions, which will help us further improve the clarity and completeness of our paper.\n\nWe address each of your points below.\n\n**[W1/Q3]. On Hyperparameter Sensitivity**\n\n> How does the method sensitive to the hyperparameters such as k_max, t_min, and the sliding window size for estimating the acceptance rate?\n\nThis is an excellent question regarding the practical robustness of our method. We designed the hyperparameters to be intuitive and generally stable.\n\n* `k_max` (Maximum Draft Length): This parameter serves as a safeguard to prevent the algorithm from generating overly long (and likely incorrect) draft sequences, especially early on when acceptance rate estimates are noisy. In practice, the objective function in Eq. 4, which balances draft length against acceptance rate, naturally penalizes long drafts if the acceptance rate (`â`) is not near-perfect. We found that performance is not highly sensitive to this value as long as it is reasonably set (e.g., 4-10), since `t_min` will also prevent excessive exploration of long drafts. We used `k_max=5` in our experiments as a conservative choice.\n\n| `k_max` | 1 | 2 | 3 | 4 | 5 | 6 | 8 | 10 | 12 | 15 |\n|---------|---|---|---|---|---|---|---|----|----|----|\n| Speedup | 1.22x | 1.35x | 1.49x | 1.56x | 1.58x | 1.57x | 1.58x | 1.57x | 1.56x | 1.54x |\n\n* `t_min` (Minimum Speedup Threshold): This is a pruning hyperparameter to prevent the algorithm from exploring branches that are unlikely to yield any speedup. We set it to 1.1 to ensure that any chosen configuration is predicted to be at least 10% faster than autoregressive generation. The algorithm's performance is robust to small variations here; the main effect of increasing `t_min` is slightly more conservative generation, while decreasing it may lead to more speculative (and potentially wasteful) attempts.\n\n| `t_min` | 1.0 | 1.05 | 1.1 | 1.15 | 1.2 | 1.25 | 1.3 |\n|---------|-----|------|-----|------|-----|------|-----|\n| Speedup | 1.54x | 1.56x | 1.58x | 1.58x | 1.56x | 1.53x | 1.54x |\n\n\n* Sliding Window Size: This parameter controls the trade-off between stability and responsiveness in our online estimation. A small window makes the estimate more responsive to recent changes but also more susceptible to noise. A large window provides a more stable estimate but adapts more slowly. We used a window of the last 20-30 generation steps, a standard choice for moving averages that provided robust performance across all models and datasets, indicating it does not require per-task tuning. We will present the figure for the sliding window size versus local acceptance rate in the Appendix.\n\nTo provide a more quantitative answer, we will add a new **Hyperparameter Analysis** section in the Appendix. This section will include an ablation study showing the impact on overall speedup when varying `k_max` and `t_min`, demonstrating the method's robustness.\n\n**[W2]. On the Overhead and Complexity of the DyTC Algorithm**\n\n> How about the overhead and complexity of the DyTC algorithm?\n\nThis is a crucial practical concern. The DyTC scheduling logic is designed to be extremely lightweight.\n\nThe core of the scheduling decision involves iterating through a small, fixed set of candidate draft configurations (in our experiments, fewer than 10 candidates) and, for each, evaluating the simple closed-form objective function (Eq. 4) for `k` from 1 to `k_max`. This amounts to a few dozen floating-point operations.\n\nWe measured the wall-clock time of the DyTC scheduling logic and found it to be less than 0.1 ms on average. This is negligible compared to the latency of even a single transformer layer's forward pass. Therefore, the overhead of the DyTC algorithm only takes up about 0.5% to 2% of the total computation time on different GPUs.\n\nWe will add a subsection in the Appendix titled \"Algorithm Overhead\" that formally discusses this complexity and provides these quantitative timing results to clearly demonstrate its negligible impact.\n\n**[W3/Q1/Q2]. On the Latency Prediction Model and Acceptance Rate Estimation**\n\n> I would expect more details of the latency prediction model. What are some detailed features used when prediction latency? How does the method estimates the acceptance rate using the history information?\n\nThank you for requesting more detail on these key components.\n\nAs stated in Section 4.2, we use a simple Bayesian linear regression model for its efficiency and effectiveness. The features used to predict the latency of a draft generation step are:\n1. The chosen DSIA strategy (e.g., Layer Sparsity at 0.4, Layer Sparsity at 0.6), treated as a one-hot encoded categorical feature.\n2. The proposed draft length (`k`).\n3. The context length (number of tokens already generated in the current sequence).\n\nThis offline-calibrated model accurately predicts the hardware-specific cost of executing different draft configurations.\nWe will further discuss the accuracy of latency predictions for different configurations of draft models in the Appendix.\n\nWe use an online weighted moving average to estimate the acceptance rate for each candidate draft configuration. The process is as follows:\n1. We maintain a local history (a queue of fixed size, 20 in the experiments) of the outcomes for each draft configuration that has been recently used.\n2. An outcome is recorded as a tuple: `(accepted_tokens, drafted_tokens)`.\n3. The estimated acceptance rate `â` for a configuration is simply `λ * â_original + (1 - λ) * â_history`, for that specific configuration. This provides a continuously updated, adaptive estimate of its performance in the recent generation context. (We set `λ` to 0.7 in our experiments to keep the estimate responsive to recent changes while still incorporating historical performance.)\n\nWe will expand Section 4.2 to include these specific details about the features for the latency model and the weighted moving average mechanism for the acceptance rate, ensuring the implementation is fully transparent.\n\nWe are confident that incorporating these clarifications and additional analyses will address all the reviewer's concerns and further strengthen the paper. We thank the reviewer again for their constructive feedback and positive evaluation."}, {"review_id": "PPOP0j5H2g", "reviewer": "Reviewer_7MsQ", "summary": "This paper proposes Cascade Adaptive Self-Speculative Decoding (CAS-Spec), a novel method for accelerating LLM inference without additional training. It constructs speculative draft models using dynamically switchable acceleration techniques such as layer sparsity and activation quantization. To further enhance efficiency, the authors introduce the Dynamic Tree Cascade (DyTC) algorithm, which adaptively routes draft models based on acceptance rates and latency estimates. CAS-Spec delivers incremental improvements over existing training-free baseline methods.", "questions": "1. Could the authors provide more details on the computational complexity or runtime overhead associated with applying CAS-Spec?\n\n2. Additionally, in high-throughput scenarios (e.g., with batch sizes greater than 1), does CAS-Spec maintain its speedup advantage compared to baseline methods?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. CAS-Spec addresses training-free speculative decoding, offering practical value in scenarios where additional training is impractical or unavailable.\n\n2. The method dynamically integrates existing training-free acceleration techniques to build a more efficient draft model.\n\n3. This paper targets a practical deployment scenario where multiple draft models are not always available for a given LLM.\n\n\n**Weakness**\n\nThe main concern is the relatively modest speedup achieved by CAS-Spec. As shown in Table 1, for 7B-scale LLMs, CAS-Spec outperforms the second-best method (PLD) by only 0.037 on average. The improvements for 13B and 33B models are similarly limited. Given the added complexity of dynamic switching and the DyTC algorithm, the marginal gains makes the overall contribution of CAS-Spec appear somewhat incremental.\n\nA minor issue: Section 3 states that self-speculative decoding methods require substantial training time (e.g., several days or weeks). This appears to overstate the cost—such methods typically use a few billion tokens for training and have relatively few trainable parameters. Training for weeks seems unrealistic. While training-free approaches are valuable, training-based methods can also be efficient and high-performing, and should be evaluated fairly.", "was_revised": true, "final_justification": "In light of the additional experiments provided by the authors, I will retain my present assessment.", "rebuttal": "We sincerely thank the reviewer for their thoughtful evaluation and for recognizing the practical value and novelty of CAS-Spec. We are encouraged by the positive assessment of our work's relevance to real-world deployment scenarios. Your insightful comments have helped us identify areas where we can provide greater clarity.\n\nWe address your concerns and questions below.\n\n**[W1]. On the \"Modest\" Speedup and Incremental Contribution**\n> The main concern is the relatively modest speedup achieved by CAS-Spec. As shown in Table 1, for 7B-scale LLMs, CAS-Spec outperforms the second-best method (PLD) by only 0.037 on average. The improvements for 13B and 33B models are similarly limited. Given the added complexity of dynamic switching and the DyTC algorithm, the marginal gains makes the overall contribution of CAS-Spec appear somewhat incremental.\n\nWe appreciate the reviewer’s sharp analysis of the results in Table 1. While the overall average speedup over the second-best method (PLD) may appear modest, it's more difficult get further speedup over PLD than it is over SWIFT. This is limited by the inefficiency of current training-free self-speculative decoding methods like SWIFT when it comes to small models.\nWith trained methods like Kangaroo, which leverage a small tuned head for early exiting, we can achieve more substantial gains over both PLD and Kangaroo:\n\n| Method | MT-Bench | Translation | Summarization | QA | Math Reasoning | RAG | Overall |\n|--------|----------|-------------|---------------|----|----------------|-----|---------|\n| **Kangaroo** | 1.65x | 1.24x | 1.48x | 1.34x | 1.66x | 1.51x | 1.48x |\n| **PLD** | 1.56x | 1.04x | 2.28x | 1.13x | 1.54x | 1.67x | 1.53x |\n| **CAS-Spec (Kangaroo,PLD)** | 1.72x | 1.23x | 2.35x | 1.40x | 1.72x | 1.69x | 1.67x |\n\nAlso, more powerful training-free self-speculative decoding methods is emerging recently like CLaSp in ACL 2025, which further achieves around 10% to 20% speedup over SWIFT. Incorporating these methods into our CAS-Spec framework will likely yield even greater speedups.\n\nMeanwhile, ablation study in Figure 3 shows that a static cascade (`VC+HC`, inspired by CS-Drafting) achieves only a 1.07x speedup. Our dynamic DyTC algorithm elevates this to 1.57x, a 48% improvement over the best static baseline. This demonstrates that the core algorithmic contribution is not the cascade itself, but the dynamic intelligence to manage it, which yields substantial, not marginal, gains.\n\nIn our revision, we will add a paragraph to the results analysis (Section 5.2) to highlight this nuanced interpretation, emphasizing the adaptability of CAS-Spec and contextualizing its performance against the most relevant baselines for each task type.\n\n**[W2]. On the Stated Training Cost of Other Methods**\n> A minor issue: Section 3 states that self-speculative decoding methods require substantial training time (e.g., several days or weeks). This appears to overstate the cost—such methods typically use a few billion tokens for training and have relatively few trainable parameters. Training for weeks seems unrealistic. While training-free approaches are valuable, training-based methods can also be efficient and high-performing, and should be evaluated fairly.\n\nThank you for this fair point. We agree that our language may have been slightly imprecise. While \"weeks\" might be an overstatement for some well-optimized setups, the training for methods like EAGLE or Medusa still represents a significant barrier.\n\nOur intention was to contrast this with our truly \"on-the-fly\" approach, which has zero pre-computation, data, or training-specific engineering requirements.\n\nMoreover, our approach can also be seamlessly integrated with trained self-speculative decoding methods like Kangaroo, as shown in the table above.\n\nWe will revise the text in Section 3 to be more precise, changing \"weeks\" to \"several days\" and explicitly mentioning the required hardware and data resources for different speculative decoding methods to provide a fairer and more accurate comparison.\n\n**[W3/Q1/Q2]. On Computational Overhead and Batch Size Performance**\n\n> Could the authors provide more details on the computational complexity or runtime overhead associated with applying CAS-Spec?\n\nThis is a crucial practical concern. The DyTC scheduling logic is designed to be extremely lightweight.\n\nThe core of the scheduling decision involves iterating through a small, fixed set of candidate draft configurations (in our experiments, fewer than 10 candidates) and, for each, evaluating the simple closed-form objective function (Eq. 4) for `k` from 1 to `k_max`.\n\nWe measured the wall-clock time of the DyTC scheduling logic and found it to be less than 0.1 ms on average. This is negligible compared to the latency of even a single transformer layer's forward pass. Therefore, the overhead of the DyTC algorithm only takes up about 0.5% to 2% of the total computation time on different GPUs.\n\nWe will add a subsection in the Appendix titled \"Algorithm Overhead\" that formally discusses this complexity and provides these quantitative timing results to clearly demonstrate its negligible impact.\n\n> In high-throughput scenarios (e.g., with batch sizes greater than 1), does CAS-Spec maintain its speedup advantage compared to baseline methods?\n\nThis is an excellent question. As noted in Appendix A, the speedup from speculative decoding techniques, including ours, is most pronounced in latency-sensitive, small-batch scenarios (like real-time chatbots), where inference is often memory-bandwidth bound. As batch size increases, inference becomes more compute-bound, and the relative benefit of reducing forward passes diminishes for all speculative methods. This is a common weakness of speculative decoding techniques, not just CAS-Spec.\n\nCurrently, self-speculative decoding methods like SWIFT and Kangaroo did not provide the implementation for batch sizes greater than 1, so we did not test CAS-Spec with larger batch sizes.  However, literature like SmartSpec show that decreasing the draft length will boost the performance in high-throughput scenarios. We expect that CAS-Spec will also slightly reduce the performance degradation by dynamically estimating the latency and hence limiting the number of forward passes.\n\nWe will add a subsection to the Appendix detailing the negligible computational overhead with quantitative measurements. We will also expand the discussion in Appendix A to clarify the expected performance characteristics under varying batch sizes.\n\nWe are confident that these revisions will address your concerns and further strengthen the paper. We thank you again for your constructive feedback and supportive rating."}]

Author rebuttal:

Meta-review: This paper proposes a novel Cascade Adaptive Self-Speculative Decoding algorithm that constructs speculative draft models using dynamically switchable inference acceleration strategies. Three reviewers posted positive comments while one reviewer posted relatively negative comments initially. CAS-Spec demonstrates significant innovation by enabling dynamic, step-wise optimization of the cascade structure during token generation, which differs from static cascade strategies and allows for adaptation to changing generation contexts.It achieves state-of-the-art acceleration compared to existing on-the-fly speculative decoding methods on both edge and server platforms, with a 48% improvement over static cascade baselines shown in ablation studies. 

Therefore, the AC recommends to accept this paper. AC also suggests authors revise the manuscript carefully in the final version. Especially, The description of the algorithm needs to be clearer and more understandable. A deeper investigation into how this method works with non-training-free models is also required.
Decision: Accept (poster)