Title: CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Efficient Inference

Speculative Decoding
Abstract: Speculative decoding has become a widely adopted as an effective technique for lossless inference acceleration when deploying large language models (LLMs).
While on-the-fly self-speculative methods offer seamless integration and broad utility, they often fall short of the speed gains achieved by methods relying on specialized training.
Cascading a hierarchy of draft models promises further acceleration and flexibility, but the high cost of training multiple models has limited its practical application.
In this paper, we propose a novel Cascade Adaptive Self-Speculative Decoding (CAS-Spec) method which constructs speculative draft models by leveraging dynamically switchable inference acceleration (DSIA) strategies, including layer sparsity and activation quantization.
Furthermore, traditional vertical and horizontal cascade algorithms are inefficient when applied to self-speculative decoding methods.
We introduce a Dynamic Tree Cascade (DyTC) algorithm that adaptively routes the multi-level draft models and assigns the draft lengths, based on the heuristics of acceptance rates and latency prediction.
Our CAS-Spec method achieves state-of-the-art acceleration compared to existing on-the-fly speculative decoding methods, with an average speedup from $1.1\times$ to $2.3\times$ over autoregressive decoding across various LLMs and datasets.
DyTC improves the average speedup by $47$\% and $48$\% over cascade-based baseline and tree-based baseline algorithms, respectively.
CAS-Spec can be easily integrated into most existing LLMs and holds promising potential for further acceleration as self-speculative decoding techniques continue to evolve.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Cascade Adaptive Self-Speculative Decoding (CAS-Spec), a novel method for accelerating LLM inference without additional training. It constructs speculative draft models using dynamically switchable acceleration techniques such as layer sparsity and activation quantization. To further enhance efficiency, the authors introduce the Dynamic Tree Cascade (DyTC) algorithm, which adaptively routes draft models based on acceptance rates and latency estimates. CAS-Spec delivers incremental improvements over existing training-free baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. CAS-Spec addresses training-free speculative decoding, offering practical value in scenarios where additional training is impractical or unavailable.\n\n2. The method dynamically integrates existing training-free acceleration techniques to build a more efficient draft model.\n\n3. This paper targets a practical deployment scenario where multiple draft models are not always available for a given LLM.\n\n\n**Weakness**\n\nThe main concern is the relatively modest speedup achieved by CAS-Spec. As shown in Table 1, for 7B-scale LLMs, CAS-Spec outperforms the second-best method (PLD) by only 0.037 on average. The improvements for 13B and 33B models are similarly limited. Given the added complexity of dynamic switching and the DyTC algorithm, the marginal gains makes the overall contribution of CAS-Spec appear somewhat incremental.\n\nA minor issue: Section 3 states that self-speculative decoding methods require substantial training time (e.g., several days or weeks). This appears to overstate the cost—such methods typically use a few billion tokens for training and have relatively few trainable parameters. Training for weeks seems unrealistic. While training-free approaches are valuable, training-based methods can also be efficient and high-performing, and should be evaluated fairly.\n\nQUESTIONS:\n1. Could the authors provide more details on the computational complexity or runtime overhead associated with applying CAS-Spec?\n\n2. Additionally, in high-throughput scenarios (e.g., with batch sizes greater than 1), does CAS-Spec maintain its speedup advantage compared to baseline methods?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CAS-Spec, a new method to speed up LLM without needing extra training. Current methods either require costly separate \"draft\" models or offer limited speed gains. CAS-Spec tackles this by creating a hierarchy of \"virtual\" draft models from the main LLM on the fly, using techniques like layer sparsity and activation quantization. It then uses a Dynamic Tree Cascade algorithm to intelligently select the best draft model and sequence length during token generation, based on predicted acceptance rates and hardware efficiency. Experiments show CAS-Spec achieves 1.6x to 2.1x faster inference compared to other training-free methods, making LLMs respond much quicker.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n- The paper addresses the central topic of LLM inference by accelerating speculative decoding. The method has great capacity yet introduces no extra training cost. This makes feasible the cascaded method that used to be too costly.\n- The approach is novel. The approach uses different kinds of ways to create a hierarchy of virtual draft models from the target LLM during runtime and the algorithm is adaptive.\n- The experiments are comprehensive and the ablation study is convincing.\nOverall, I do not have too many concerns about the paper.\n\nCons:\n- How does the method sensitive to the hyperparameters such as `k_max`, `t_min`, and the sliding window size for estimating the acceptance rate?\n- How about the overhead and complexity of the DyTC algorithm?\n- I would expect more details of the latency prediction model.\n\nQUESTIONS:\n- What are some detailed features used when prediction latency in Section 4.2?\n- How does the method estimates the acceptance rate using the history information?\n- How does the method sensitive to the hyperparameters such as `k_max`, `t_min`, and the sliding window size for estimating the acceptance rate?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission introduces Cascade Adaptive Self-Speculative Decoding, a methodology for LLM inference acceleration under speculative decoding settings. The proposed approach dynamically adapts the employed draft model through the training-free creation of a hierarchy of draft LLMs, extracted from the target model (e.g. through pruning, quantisation etc). Alongside, a routing mechanism (dynamic tree cascade) is proposed to guide the generation process, by adopting different draft models and drafting lengths by monitoring heuristics such as acceptance rate and inference latency at runtime. The proposed methodology is shown to improve inference efficiency compared to relevant approaches from the literature, across a variety of models and datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- CAS-Spec addresses a very interesting and timely problem. The proposed  DyTC is a novel and potentially impactful direction.\n- Overall the manuscript is easy to follow and proposed contribution is clear.\n- The experimental results demonstrate that the proposed approach is effective, in terms of acceleration to AR decoding, and achieves on-par or better speed-up to other technics. \n\nComments:\n- Consider pointing more clearly to the connection between the adopted baselines and related work [10]. Since CS-Drafting is the most closely related approach from the literature, amore comprehensive comparison and discussion is required. Why is CS-Drafting not included as a baseline in Table 1 ?\n- It is unclear why the proposed DyTC only considers running heuristics such as the acceptance rate and inference time and no token-level information such as prediction confidence for the tokens in the sequence. As the authors rightfully pointed out, past performance is not a strong indication for future behavior of the model.\n- Further experimental analysis would benefit the reader understand the behavior of the proposed approach. Consider reporting statistics about the selected model sizes across different tasks/ stages of the generation process, potential comparison with an Oracle scheduling strategy, as well as qualitative examples of the DyTC behavior.\n\nQUESTIONS:\nPlease considered replying to the points raised in the comments section above.\n\nLIMITATIONS:\nSome limitations of the proposed approach are discussed in the appendix, along with proposed future work directions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes \"CAS-spec\", a method that dynamically builds a cascade of (training-free) draft models (and determines their corresponding lookaheads) to maximize speedup. The types of draft models considered fall under the category of \"self speculative decoding\" where pruned or quantized versions of the target model are used as the speculator. The algorithm is dynamic because it maintains an online prediction of the acceptance rates and latencies of each of the draft models, and adjusts the cascade structure accordingly. CAS-spec attains meaningful speedups over the other training-free speculative decoding baselines considered in the paper (Lade, PLD, SWIFT).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The idea of building an optimal cascade automatically, without requiring an extensive grid search in an exponentially large search space, is quite nice.\n\nWeaknesses:\n- The paper suffers from a lot of \"mathiness\" and lack of clarity. The presentation is extremely dense and full of math, even though if I understand correctly actual implemented algorithm is quite simple: A cascade consisting of PLD -> layer-sparsity (50%) -> target model, where the algorithm uses an A*-style search algorithm to efficiently find the best hyperparameters (lookahead for each draft).\n- The proposed method need not only apply to training-free draft models. It would have been quite informative to see how this method can help accelerate speculative decoding with SOTA draft models (e.g., Eagle-3).\n\nQUESTIONS:\n- How do you maintain estimates of acceptance rates for draft models not being used in the current cascade?\n- I'm confused by the interplay between the token tree being constructed, and the dynamic cascade algorithm. How are cascaded speculators combined with tree-based speculation/verification?\n- Do you actually use token trees in your experiments, or do you just use sequences?\n- What hierarchy of draft models do you use in your experiments? In the main paper it says the available draft models are PLD and 50% layer-sparsity, but in Appendix E it says PLD and 40% and 60% layer sparsity?\n- Could you compare with Eagle and other SOTA speculative decoding techniques, to clarify the gap between training-free and not-training-free methods?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}