Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models
Authors: Yinrong Hong
Zhiquan Tan
Kai Hu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Speculative Decoding
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a new dynamic tree speculative decoding method that leverage the inference cost and achieves improvements against baselines.

Abstract: Large Language Models (LLMs) face significant inference latency challenges stemming from their autoregressive design and large size. To address this, speculative decoding emerges as a solution, enabling the simultaneous generation and validation of multiple tokens. While recent approaches like EAGLE-2 and EAGLE-3 improve speculative decoding using dynamic tree structures, they often neglect the impact of crucial system variables such as GPU devices and batch sizes.

Therefore, we introduce a new dynamic tree decoding approach called CAST that takes into account inference costs, including factors such as GPU configurations and batch sizes, to dynamically refine the tree structure. Through comprehensive experimentation across six diverse tasks and utilizing six distinct LLMs, our methodology demonstrates remarkable results, achieving speeds up to 5.2 times faster than conventional decoding methods. Moreover, it generally outperforms existing state-of-the-art techniques from $5\%$ to $20\%$. The code is available at \url{https://github.com/EAGLE-Research/sglang-eagle4}.

Published as a conference paper at ICLR 2026
INFERENCE-COST-WWANN DYNAMIC TREE CONSTRUC- TION FOR EFFICIENT INFERENCE IN LARGE LANGUAGE MODELS
Yinrong Tan"; Kai Hul!
I: Beihang University E Fund Management Co., Ltd.
ABSTRACT
Large Language Models (LLMs) face significant inference latency challenges stem- ming from their autoregressive design and large size. To address this, speculative decoding emerges solution, enabling the simultaneous generation and valida- tion of multiple tokens. While recent approaches like EAGLE--2 and EAGLE-3 improve speculative decoding using dynamic tree structures, they often neglect the impact of crucial system variables such as GPU devices and batch sizes.
Therefore, we introduce new dynamic tree decoding approach called CAST that takes into account inference costs, including factors such as GPU configurations and batch sizes, to dynamically refine the tree structure. Through comprehensive experimentation across six diverse tasks and utilizing six distinct LLMs, our methodology demonstrates remarkable results, achieving speeds up to 5.2 times faster than conventional decoding methods. Moreover, it generally outperforms existing state-of-the-an the-art techniques from 5% 20%. The code is available at attpaa/s/ggghhbbccc///AAddeeeeee ata.ee/aaaacccclaggaaaaaeeaeee
INTRODUCTION
Large Language Models (LLMs) have showcased remarkable capabilities (OpenAl 2023 Touvron
al. 2023) and are extensively applied across various domains. Nevertheless, the vast scale their parameters, often exceeding hundreds of billions, presents notable challenges. This is particularly evident during the autoregressive text generation process, where generating each token involves referencing previous tokens, resulting in considerable latency. In real applications like chatbots, producing hundreds to thousands of tokens can render LLM inference slow and resource- inteurieei This shows the need for better inference speedup methods.
To address this challenge, speculative decoding techniques (Leviathan et al. 2023 Chen et al. 2023) have emerged. These methods swiftly generate initial tokens and validate them concurrently, thereby diminishing inference latency by generating multiple tokens traditional speculative decoding adopts chain-structured draft, recent progressions have introduced tree-structured drafts to boost efficiency. For example, EAGLE (Li et al. 2024b) utilizes static draft tree structure, incorporating a fixed number of candidates at each stage. However, this fixed approach overlooks the context-specific nature of token acceptance rates, contradicting the fundamental premise of speculative sampling that simpler tokens can be predicted by smaller models. Subsequently. EAGLE-2 (Li et al. 2024c) and EAGLE-3 (Li et al. 2025) leverage dynamic trees to further enhance performance.
single forward pass. While
While EAGLE-2 and EAGLE-3 have begun to harness the potential of dynamic tree structures, they fall short in adapting the structure based on crucial factors like GPU devices and batch sizes. Our proposed unified approach tackles this limitation by modeling the effects of variables such as device type and batch sizes as costs. The motivation behind our approach is that higher number of tokens does not always equate to better performance. Taking into account the inference cost, there exists
'These authors contributed equally.
Corresponning author.

---

Published as a conference paper at ICLR 2026
critical value beyond which adding more tokens becomes inefficient, slowing down the overall process. Drawing on these insights, we introduce a novel conscionss strategy that dynamically determines the tree's depth, token count per layer, and the number of tokens to be validated by the target model. Integrating this inference cost-aware dynamic tree construction method with the cutting-edge technique EAGLE-22 EAGLE-3 yields advanced method: Cost-Aware Speculative Tree (CAST), This method adjusts the draft tree structure dynamically by balancing the trade-off' between accepted token numbers and inference cost, resulting in accelerated speedups.
Our comprehensive evaluations span six distinct tasks: multi-turn conversation, code generation, mathematical reasoning, instruction following, summarization, and question answering. The datasets utilized encompass MT-bench (Zheng et al., 2023), HumanEval (Chen et al. (02T). GSM8K (Cobbe)
al. 202T), Alpaca (Taori et al. 2023) ) CNN/Daily Mail (Nallapati al. 2016), and Natural
Questions (wwitkkwwski et al. 2019)). We benchmark our method against ofate-of-the-art speculative
decoding techniques: standard speculative decoding (Joao Gante 2023 Leviathan et al. [023]
Chen al. 2023), Medusa (Cai et al.
2024). PLD (Saxenal 2023 Lookahead (Fu et al. 2023).
EAGLE (Li et al. 2024b), EAGLE-2 (Li et al. 2024c). and EAGLE-3 (Li al. 2025). Experiments
conducted across various LLM series with different batch sizes, including Vicuna, LLaMA3,
Qwen2, and distilled DeepSeek-R.. R11. Our method consistently surpasses all baseline approaches, achieving speedups of up 5.2x and typically delivering speed enhancements ranging from 5% to 20% compared to the previous state-of-the-art method.
In summary, our paper offers the following contributions:
We propose a new dynamic-tree-based speculative decoding method CAST based on the trade-off between the number of tokens to verified and the inference cost.
The proposed method generalizes previous state-of-the-art methods EAGLE-2 and EAGLE- 3 and also systematically considers the impact of batching and GPU, which is less discussed in the literature.
We conduct extensive experiments among 6 tasks and 6 models. The proposed method usually achieves 20% speedup than the previous SOTA method and up to 5.2x speedup than the vanilla autoregressive method.
2
RELATED WORKS
Speculative Decoding The goal of speculative decoding is to accelerate LLM inference without losing output quality. Its core idea is to separate proposal from verification: a lightweight draft model suggests tokens, and the base LLM validates them. This shifts much of the workload the draft model while preserving consistency, reducing latency compared with conventional step-by-step decoding.
Early work focused on greedy decoding. Stern et al. (2018) introduced blockwise decoding and Sun al. (2021) proposed instantancous methods, both allowing multiple tokens per step. Later, speculative sampling (Leviathan et al. 22233 Chen et al. 2023) extended the idea to non-greedy
settings, establishing broad applicability.
Subsequent methods improved draft efficiency and base-model alignment. SpecInfer (Miao et al. 2023) used traft-model ensembles and tree-mask attention. Medusa (Cai al. (024) leveraged MLPs on internal states to predict multiple tokens. EAGLE (Li et al. (024b) expanded free proposals for higher acceptance. Draft-and- Verify frameworks (Zhang et al. [023] Hooper et al., 2023 Yang et al.
2023 Monea et al.] 2023 Li et al. 2024a Yi et al. [2224] Liu et al. 20224) Sun al. 2024 Elhoushi
ct al.
2024| Svirschevski et al. 2024) introduced early exits and partial model reuse, partitioning the
into fast generators and verifiers.
More recently, dynamic draft trees emerged. GLIDE and CAPE (Du et al. 2024) added fallback branches for uncertain cases but limited expansion. EAGLE-2 (Li et al.
2024c) removed such
constraints for fully adaptive growth, while EAGLE-3 (Li et al. 2025) further relaxed training restrictions, yielding more effective speculative decoding.
2

---

Published as a conference paper at ICLR 2026
Batching Method A complementary line of work studies how batching be combined with speculative decoding to better leverage GPUs. Existing methods mainly target the conventional Chain-based paradigm, while tree-structured batching remains largely unexplored.
Su et al. 2023) first analyzed how batch size affects chain-style decoding, revealing trade-offs between improved parallelism and synchronization overhead. Building this, Qian al. (2024) proposed a strategy that parallelizes not only across batches but also along the draft-token axis, enabling finer GPU utilization and higher throughput. Most recently, Wu al. (2025) introduced specialized techniques that further boost batched speculative decoding, demonstrating that careful batching design can accelerate inference scale.
PRELIMINARY
In this section, we will briefly recap some of the needed knowledge and notions in LLM inference. Let #111 (I1, ,2, denote the language sequence. We will consider two autoregressive models as follows:
Target Model: Pr( F1:)), the high-quality, accurate model whose predictions we aim to approximate efficiently, which is usually a large model and has a bigger inference cost.
Draft Model: PD( æ1a). a lightweight, fast model used to propose candidate tokens.
The objective is to sample from Pr more efficiently using PD without compromising the quality of the output distribution.
3.1 SPECULATIVE DECODING
The motivation of speculative decoding (Leviathan al. 2023 Chen al. 2023) is that some tokens may be "easy" to predict and can use a smaller model to generate to make inference more efficient, and also the initial model is used to verify the correctness of the predictions.
Given context Vis, the draft model will first generate a sequence of d tokens autoregressive: 4.41 ~v Pp(- 21:1). le+2 Pp(. T1.s, 2 -itd Pp(- Zi:t, f2+1=4d-1). Let #++1:+d denote the predicted draft sequence, the tokens are verified sequentially and once a token is accepted by the target model, we can drop the hat symbol.
Starting from i = 1. each token Ze+t is verified by the target model as follows:
Calculate the draft probability: qi PD( T1sti-1).
Calculate the target probability: p. := (r #1sti-1).
uniform random number vı Uniform(O, 1) is drawn. The token is accepted if: u, S min (1, Otherwise, we reject the remaining tokens and fall back to sampling from a residual distribution: It-i Pr(- -1=+1-1). where Pr = norm max(0. 4)). )))).
can be shown that the above procedures can ensure the overall output sequence sampled from the target model distribution Pr (Leviathan et al. (023).
3.2 EAGLE
The previously discussed speculative decoding method predicts the tokens in an autoregressive chain and verifies them sequentially. It has the disadvantage of once rejecting token, all its subsequent tokens will also be discarded. FAGLE (Li et al.) 2024b) improves speculative decoding by constructing a tree-structured draft and performing parallel verification, the tree structure makes the rejection process still retain some information by leaving the tokens the rejected token's 's sibling subtree un-discadded.
Unlike EAGLE Li et al. 2024b). which uses a predefined static tree, EAGLE-2 (Li et al. 2024c)
and EAGLE-3 (Li al. 2025) improve speculative decoding by dynamically constructing tree- structured draft using the confidence score. The dynamic structure makes the inference much more data-dependent and performs much better. We will then briefly discuss some of the details.
3

---

Published as a conference paper at ICLR 2026
3.2.1 TREE EXPANSION PHASE
To organize the token sequence into a tree structure, one may have the following two definitions:
Each node corresponds token zu and its preceding context Cu.
The (confidence) value of node is:
v(u)
II wrpath(u)
PD Cw),
representing the confidence score by traveling along the draft path, and the root node will have probability of 1.
Starting from the root (initial context) node, EAGLE-2 (and EAGLE-3) dynamically expands the draft tree layer by layer, and the tree will be of depth H:
1. At each level except the last layer, select top- nodes with the highest v(u).
2. For each selected node u, generate K child nodes by sampling from Pp(- cu).
3.2.2 TREE RERANKING PHASF
In the expansion stage, the goal is to further develop the draft tree by exploring deeper paths. However, because node values- interpreeee diminish with depth. To address this, reranking over all candidate tokens will be performed, and the top tokens with the highest associated values will be selected. An important constraint is that the value assigned to any node does not exceed that of its parent. Therefore, after reranking, it still comprises a valid subtree within the original draft structure.
acceptance probabilities-Iit between 0 and they naturally
After selection, the subtree will be linearized into a flat sequence to produce the input for the verification stage. To maintain compatibility with standard autoregressive decoding, the attention mask will also be changed. Consequently, the attention mask is modified such that each token attends only to its ancestors, preserving the hierarchical dependencies encoded in the tree.
METHOD
Though EAGLE-2 (EAGLE-3) has constructed dynamic tree increase the inference performance, its construction rule is mostly based on heuristies and does not consider the intricate interplay of the inference algorithm and GPU hardware, especially in the case of batched processing. When using batching techniques, merely increasing the tree depth and node numbers may not always result in better performance. This is because the GPU utilization has already increased by using batching, and naively adopting the speculative decoding methods may result in competition in the GPU resources and slow down the process.
Therefore, should also consider the of inference during speculative decoding. Given batch of B samples, each with context of length c, the inference time of inputting length n sequence will depends on B, c, n, which denoted as f(B, n). To save the time of inference, we can precompute the time and maintain lookup table. To save the computation and storage, we only need to maintain the data of fBB c, n)) for kL (k 1....... .M) and 1,... .N. And also the associated select operator select(c) = (max( f M - 1) I)L.
Then, for each needed size B, one can maintain the following two lookup tables:
(B (fr(B,c,n) and Sp(B) (fD(B,c,n),
where fr is for target model and fo for draft model. For given context length (D(Ssseeec)(( will return an array of size N.
Given batch of B samples, w.l.o.g. we can assume they have the same context length no thanks to
the padding technique and denote them x 1.... .B.. As EAGLE has stages, namely the expansion stage and reranking stage, when constructing the draft tree, we will tackle these two one by one.
4

---

Published as a conference paper at ICLR 2026
4.1 DYNAMIC EXPANSION STAGE: BREADTH AND DEPTH PRUNING
The expansion stage of the draft tree construction involves two key dimensions: (1) the number of nodes per layer, and (2) the total number of layers in the tree. These two components inherently coupled. An illustrative example can be found in Figure 2 We first focus on determining the number of nodes retain in each layer, process we refer to breadth pruning.
(a) breadth pruning
(b) depth pruning
1.00
0.70
0.20
0.70
0.20
0.56
0.07
0.18
0.02
0.81
0.56
0.07
0.18
0.02
Avg
Cost
buffer
Yes
0.56
0.83
Generate the next layer or not No
Cost Algorithm
Stop generate
Figure An illustrative example for the dynamic expansion stage, we use batch size as for simplicity, general cases are tackled by averaging along batches. Each node will initially have branches in the example.
The primary objective is to minimize the average inference latency sequence. To this end, we aim to select draft tokens that are highly likely to be accepted by the target model. However, predicting excessive number of tokens cost. Thus, a tradeoff must be considered between the likelihood of token acceptance and the cost of incurring predictions.
increase overall latency, due to the additional computational
Empirically, the acceptance rate of a node u is strongly correlated with its confidence score v(u) Du al. 2024 Li et al.) (2024c), which from utility theory in economics, we frame node selection as a utility maximization problem.
proxy for acceptance probability. Drawing inspiration
Specifically, for the i-th layer, denote the confidence scores of the N, candidate nodes (sorted in descending order) for each sample j (1,..., ,,,,,,, B} as w/(0 (s), where E {I,...,N:] and ii (1) 2 2f2)) (N:). The cumulative utility of selecting the top nodes is defined as:
(
G) (s).
B
Let ny denote the number of nodes retained in layer j. for = 1,..,,i i 1. The context length for layer is then Σj= ny. The normalized cost of selecting k nodes layer i, using the draft model relative to the target model cost, is computed as:
Sp B)
(D)B (solect ( [g)][
(2)
(B) selecct (Ei=1 [1]
In economic theory, utility functions are typically concave, exhibiting diminishing marginal utility. For concave function u(c) defined R,, the marginal utility u(c)-u(co) decreases as increases. Based this principle, introduce threshold C₁ and retain nodes whose marginal utility exceeds
c-co

---

Published as a conference paper at ICLR 2026
this threshold. The intial number of nodes to be chosen in each layer will be determined by the top-K probability in the previous layer, similar to EAGLE.
Due to the discrete nature of our setting. the utility function may not be strictly concave. A robust selection strategy summarized Algorithm which takes as input the utility sequence {ur J. the associated cost sequence (0)) }. and the threshold C1 determine the number of nodes to retain at layer i. Notably, the node selection mechanisms in EAGLE-2 and EAGLE-3 can be viewed as special cases of this generalized formulation.
Theorem 4.1. EAGLE-2 and EAGLE-3's s selection algorithm in i-th layer is special case of the proposed selection Algorithm by setting ej = Äj + and C₁
En o
BA
Algorithm 1 Select Maximum Valid Index
I: Input: Arrays u ............... n], cf1 .n] strictly increasing; constant C > 0
2: Initialize mark ... nl +-
3: for = 1 to do
4: for j = ++++ to n do
5: if ] u(i
C' then
c|il cii
6:
mark(yl < 0
7: end if
8:
end for
9: end for
10:
Output: max{j markjj] = 1}
Next, we consider depth pruning, which determines whether an additional layer (i + 1) should be generated. This decision is based on the predictive relationship between successive layers. Let A; be a buffer that tracks predictive quality for layer i. We define: O; = Avg( where Avg denotes the average over the elements in A,. We proceed to generate layer (é only if the following condition
(4)
holds: ag To
2 C22 where C2 a predefined threshold. Once this condition is satisfied and the
number of nodes 7d,-1 has been determined via breadth pruning, we compute the confidence gain (1
ratio: Φ wi
We then update the buffer A; using a fist-in-iirs-outt (FIFO) policy, maintaining up to R recent values of Φ;. Each buffer A; is initialized with the value {1} to ensure stability in
early layers.
4.2 DYNAMIC RERANKING STAGE
1.00
Sorting
1.00 0.70 0.56
Adding
1.00 1.70 2.26
Algorithm 1
0.70
0.20
0.20 0 18 0.07
2.46 2.64 2.71
Cost
0.07
Figure 2: An illustrative example for the dynamic reranking stage.
After the dynamic expansion stage, a rooted draft tree is constructed, but with too many nodes that need be further trimmed. We first consider collecting data samples and calculating each sample's

---

Published as a conference paper at ICLR 2026
Cunulatiive Probabiltty VS Accept Length (EAGLE-2)
Currutative Probabillty VS Accepe: Length
(EAGLE-3)
Data Points
Data Points
Linear R
0.84c 1.4
Linear Rit
0.73x 60
Pecept Lengthh
Gumulztive
Cumurtivo Probability
(a) EAGLE-2
(b) EAGLE-3
Figure 3: The correlation of accept length and cumulative probability.
accept length and the cumulative probability score on the whole tree which is plotted in Figure From the Figure, it is clear that the accept length and the cumulative probability shares a linear trend. Therefore, in order to maximize the accept length of each sample, one should make the cumulative probability as big as possible. Thus, choosing the nodes with top probability score is the right choice. Suppose after the dynamic expansion stage, the (batch averaged) score on the whole tree is sorted as v(1) 2 ... 2 vNN) ) (N is the minimum of Ei= T4 and a predefined hyperparameter m)) By taking the inference cost into account, one also Algorithm to determine the number of nodes be [sssee]ctct( with threshold verified by the target model by setting u = Σラn v(j) and C₁. 3r (B(eeccccttnn)) constant C3.
5 EXPERIMENTS
Number of Draf: Takens
Speedlup Number Draft
68-
38
332
3154
333
335
30-
2854
733
106
100015
(a) Accept Length
(b) Speedup Ratio
Figure 4: The behavior of accept length and speedup ratio when varying the number of tokens to be verified by the target model using EAGLE-3
Following prior works, we perform experiments variety of models across diverse sizes, in- cluding Vicuna- 13B-v1.3 (V 13B) and Vicuna-33B-v1.. 13V (Chiang et al. 2023), Llama-333 8B-Instruct((331 8B) and Llama-3.3-70B 100.-33..370----tt 70B) (Meta 2024), -eepSeeek-R--isttiiee Llama-8B(DSL 8B), Siven2-B--Instruche conducted extensive evaluations across six different text generation tasks to show the applicability of our method under diverse scenarios, includng multi-turn conversation, code generation, mathematical reasoning, instruction following, summarization. and question answering, we used the MT-bench (Zheng et al. 2023). HumanEval (Chen et al.] 2021). GSM8K (Cobbe et al.) 202T). Alpaca (Taori al. al. 2023). CNN/Daily Mail (Nallapati et al. 2016). and Natural Questions (Kwiatkowski et al. 2019) datasets, respectively. In line with common practices in the community, we employed the same initial model weights for all tasks without any modifications. We will use the vanilla autoregressive decoding as the baseline for comparison, with speedup ratio of 1.00x. Our method will be com- pared with the most state-of-the- methods in speculative decoding and we will use their default hyperparameters, including standard speculative decoding (SpD) (Joao Gante 2023 Leviathan et al.)
7B). Following the standard benchmark in this area, we
2023 Chen et al. 2023), Medusa (Cai et al. 2024). PLD (Saxenal 2023). Lookahead (Fu et al.
2023). EAGLE (Li al. 2024b), EAGLE-2 (Li al. 2024c). and EAGLE-3 [Li et al. 2025)
Under
7

---

Published as a conference paper at ICLR 2026
non-greedy setting, methods like Medusa relax acceptance condition, will not compare with method like this. Given that speedup ratios are hardware- hardware-dependent, we ensured fairness by testing all methods identical devices, which the Nvidia A800 GPUs. All experiments relatively fäst, usually less than one hour, even for large datasets, as only inference is performed. The experiments more GPU types can be found in Appendix.
As we mainly consider the lossless acceleration technique that neither fine-tunes the original LLM nor alters its acceptance conditions. As a result, we focus on evaluating its acceleration performance using the following metric. Speedup Ratio: The actual increase in speed compared to standard autoregressive decoding single run and verification round.
We do not adopt the metric Average Acceptance Length (The average number of tokens generated per cnafting-verifiaatiio cycle, indicating how many tokens are accepted from the draft.). This is because this metric may be somewhat misleading, particular in the larger batch case. In Figure one can see that the maximum number of verified token number m increasing, the accept length increasing but sacrifices speedup when is relatively larger.
5.1
SINGLE SAMPLE CASE
Table 1: Comparison of Model Performance (Speedup Ratios) when batch size is 1.
Model
Method
MT-benchh
HumanEval
GSMBK
Alpaca
CNN/DM
Natural Ques.
Temperature=0
SpD PLD
1.93x
2.23x
1,77x
1.76x
1.93x
1.66x
V 13B
1.58x
1.85x
1.68x
1.16x
2.42x
1.14x
Medusa
2.07x
2.50x
2.23x
2,08x
1.71x
1.81x
Lookahead
1.65x
1.71x
1.81x
1.46x
1.46x
1.36x
EAGLE
2.61x
3.58x
3.08x
2.93x
2.80x
3.02x
EAGLE-2
3.02x
4,06x
3.35x
3.25x
3,40x
3.13x
EAGLE-3 3
3.70x
4,73x
4.00x
3.86x
3.68x
3.31x
CAST (Ours)
3,98x
5.18x
3,98x
3.80x
3.76x
3.40x
EAGLE33
4.13x
4.98x
4.63x
4.66x
3.50x
3.61x
L,3 70B
CAST (Ours)
4.23x
5.23x
4.65x
4.83x
3.56x
3.67x
RAGLE33
3.60x
4.27x
3.82x
4.00x
3.22x
3.06x
L33 8B
CAST (Ours)
3.77x
4.51x
3,95x
3.98x
3.32x
3.22x
EAGLE-3
3.47x
3.78x
3,68x
3.20x
2.90x
2.95x
DSL 8B
CAST (Ours)
3.63x
3.85x
3,98x
3,37x
3.02x
3.20x
Temperature=1
SpD EAGLE
1.62x
1.72x
1.46x
1.52x
1.66x
1.43x
V 13B
2.42x
2.75x
2.37x
2.43x
2.34x
2.04x
EAGLE-2
2,80x
3.22x
2.79x
2.71x
2.65x
2.27x
RAGLE-3
3.28x
3.94x
3.39x
3.25x
3.23x
2.74x
CAST (Ours)
3,51x
4.30x
3.76x
3.38x
3.32x
2.95x
EAGLE-3
3.96x
4,73x
4.37x
4.39x
3.42x
3.50x
L33 70B
CAST (Ours)
4.19x
4.93x
4.51x
4.66x
3.50x
3.50x
EAGLE-3
2.77x
3.58x
3.05x
3.26x
2.57x
2.32x
L31 8B
CAST (Ours)
3,06x
3.91x
3.36x
3.41x
2.89x
2.53x
EAGLE-3
2.58x.
3.15x
2.76x
2.42x
2.21x
2.37x
DSL 8B
CAST (Ours)
2.82x
3.43x
2.99x
2.65x
2.48x
2.66x
We begin our analysis by examining the usual setting in the literatures. namely when the batch size is 1. We term the proposed method as Cost-Aware Speculative Tree (CAST). To ensure fair and rigorous comparison with existing methods, we adopt the same target model configuration used in comparision with the respective SOTA EAGLE family models. This alignment in experimental setup allows us to attribute any observed performance differences solely to the algorithmic innovations of CAST, rather than to variations in model size, training regime, evaluation protocol.
The quantitative results are summarized in Table I which reports the speedup ratios achieved by CAST relative to prior baselines. As the table indicates, CAST usually yields higher speedup ratios across multiple evaluation tasks, underscoring its ability to more effectively utilize computational resources. This trend becomes increasingly evident the size of the target model grows, suggesting
8

---

Published as a conference paper at ICLR 2026
that method scales particularly well in large-model scenarios where efficiency considerations are most critical. The advantage of CAST is especially striking on the HumanEval benchmark, where specdup of 5.23 is achieved. These results collectively highlight the potential of our method as a practical solution for accelerating speculative decoding pipelines, particularly demanding real-world settings where inference latency and throughput remain key bottlenecks.
5.2
BATCHING CASE
Table Comparison of different methods across models and benchmarks when batch size is 8. All values are speedup ratios.
Model
Method
MT-Bench
IumaanEvl
GSM8K
Alpaca
CNN/DM
Natural Ques.
Temperature=0
EAGLE
1.18x
1.62x
1.76x
1.80x
0.84x
1.44x
7B
EAGLE-2
1.25x
1.49x
1.40x
1.11x
1.10x
CAST (Ours)
1.86x
2.16x
2.19x
2.06x
1.70x
1.72x
EAGLE
1.80x
2.14x
2.10x
2.09x
1.38x
1.76x
EAGLE-2
1.39x
1.60x
1.59x
1.63x
1.03x
1.32x
L31
EAGLE-3
1.72x
197x
1.92x
2.16x
1.34x
1.72x
CAST (Ours)
2.16x
2.62x
2.41x
2.62x
1.76x
2.11x
EAGLE
1.63x
1,91x
1.79x
1.72x
1.37x
1,51x
EAGLE-2
1,25x
1.42x
1.30x
1.28x
1.02x
1.03x
V 13B
EAGLE-3
1.59x
1.91x
1.67x
1.80x
1.37x
1.39x
CAST (Ours)
2.48x
3.12x
2.61x
2.76x
1,97x
2.27x
EAGLE
1.78x
2.09x
1.96x
1.75x
1,44x
1.47x
V 33B
EAGLE-2
1.27x
1,50k
1.37x
1.26x
1.05x
1.01x
CAST (Ours)
2.12x
2.48x
2.21x
2.09x
1.79x
1.84x
Temperature=1
EAGLE
0.80x
1.0xx
1.21x
1.15x
0.62x
1.00x
Q2 7B
EAGLE-2
0.93x
1.27x
1.30x
1.16x
0.83x
0.92x
CAST (Ours)
1.50x
1.96x
1.94x
1.82x
1.40x
1.57x
EAGLE
1.24x
1.53x
1.47x
1.57x
1.06x
1.23x
EAGLE-2
1.07x
1.48x
1.39x
1.47x
0.93x
1.07x
L31 8B
EAGLE-3
1.25x
1.70x
1.67x
1,90x
1.13x
1.32x
CAST (Ours)
1.73x
2.37x
2.26x
2.46x
1.69x
1.76x
EAGLE
1.25x
1.39x
1.39x
1.34x
1.
1.11x
EAGLE-2
1.14x
1.22x
1.22x
1.11x
0.94x
0.95x
V 13B
EAGLE-3
1.28x
1.56x
1.45x
1.34x
1.18x
1.28x
CAST (Ours)
2.08x
2.51x
2.22X
2.11x
2.16x
EAGLE
1.48x
1.66x
1.64x
1.49x
1.20x
1.26x
V 33B
EAGLE-2
1.18x
1.37x
1.34x
1.14x
1.01x
0.97x
CAST (Ours)
1.97x
2.16x
2.11%
1.95%
1.68x
1.79x
When moving beyond the single-sample setting to scenarios where multiple samples are processed simultaneously, batching becomes crucial factor in evaluating the practicality of speculative de- coding methods. In this regime, our study primarily focuses on comparisons with SOTA tree- rreebased speculative decoding approaches, which represent the most competitive baselines in this line of research. Table 2 provides comprehensive evaluation of CAST against these baselines under the batching setting where the batch size is fixed at 8. The evaluation spans a diverse collection of L.LMs, benchmark tasks, and decoding temperatures, ensuring that the reported results reflect broad and robust performance profile rather than being limited to a narrow set of conditions.
The empirical results reveal a clear and consistent advantage for CAST across the tested scenarios. Specifically, CAST achieves speedups of up to 3.12x in challenging tasks such as temperature 0, and up to 2.5 in V13B-MT-BB MTBBecchh-he at temperature 1. The results show the potential of our method under the batching cases. On average, CAST achieves relative improvements in the range of 5% to 20%, reflecting tangible efficiency gains without compromising correctness.
Eval

---

Published as a conference paper at ICLR 2026
CONCLUSION
In this work, we present cost-aware dynamic tree-based speculative decoding method that adapts to system-level factors such device type and batch size. By modeling the trade-off between accept length and inference speed, our method CAST dynamically adjusts the draft tree structure for more efficient decoding. Extensive experiments across diverse tasks and models demonstrate that our approach generally outperforms prior methods, achieving to 5.2 speedup and 5 20% efficiency gains over the best baselines.
ACKNOWLEDGMENTS
This work was supported by the Science and Technology Major Project of Yunnan Province (202302AF080006), the National Key R&D Program of China (2022YFB2703500)
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv: 2401. 10774, 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Raptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03372 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. |RLhttps:
-3-000---00000000000 30 vicuna/
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. arXiv preprint w0Xiv:2402.02082, 2024.
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404. 167100 10, 2024.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Breaking the sequential dependency of llm inference using lookahead decoding, November 2023. URL
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Gene, Kurt Keutzer, Amir Gholami, and Sophia Shao. Speed: Speculative pipelined execution for efficient decoding, arXiv preprint UrXiv:2310.12000 2023.
Joao Gante. Assisted generation: a new direction toward low-latency text generation, 2023. URL
https:://huggnggaaee co co/blog assi sted-generationn
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Minghan Li, Xilun Chen, Ari Holtzman, Beidi Chen, Jimmy Lin, Wen-tau Yih, and Xi Victoria Lin. Nearest neighbor speculative decoding for IIm generation and attribution. arXiv preprint arXiv:2405. /9325, 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, 2024b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16855. 2024c.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference celeration of large language models via training-time test. arXiv preprint 2503. 2025.
11

---

Published as a conference paper at ICLR 2026
Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding via double early exiting. arXiv preprint arXiv:2404.18911 2024.
Meta. LLaMA3.
fast/ 2024.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. SpecInfer: Accelerating generative LLM serving with speculative inference and token tree verification. arXiv preprini arXivv2305.09771, 2023.
Giovanni Monea, Armand Joulin, and Edouard Grave. Pass: Parallel speculative sampling. arXiv preprint arXiv:2311.1338/ 2023.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, al. Abstractive text summarization using scquence-to-sequeeee rmns and beyond. arXiv preprint (60.::::60200002... 2016.
OpenAl. Gpt-4 technical report. arxiv 2303.08774. View Article, 2(5), 2023.
Haifeng Qian, Sujan Kumar Gonugondla, Sungsoo Ha, Mingyue Shang, Sanjay Krishna Gouda, Ramesh Nallapati, Sudipta Sengupta, Xiaofei Ma, and Anoop Deoras. Bass: Batched attention- optimized speculative sampling. arXiv preprint arXiv:2404. 15778, 2024.
Apoory Saxena. Prompt lookup decoding, November 2023. URL. https::/github.coo com/
apoo http://proomtttloooo---eeeee
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autore- gressive models. Advances in Neural Information Processing Systems, 31, 2018.
Qidong Su, Christina Giannoula, and Gennady Pekhimenko. The synergy of speculative decoding and batching in serving large language models. arXiv preprint arXiv:2310. 200.:2330..881,,, 2023.
Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless
acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404. / 1912, 2024.
Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. Instantaneous grammatical error correction with shallow aggressive decoding. arXiv preprint arXiv:2/06.04970) 2021.
Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexee: Massively parallel speculative decoding for interactive I1m inference on consumer devices. arXiv preprint 14Xx:2240600253:: 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followe following Ilama model. /tetps://iithan com/ tttt/tatsss tat carabbsttafoodd _al_aaalpaaaaaaappaepp 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models (2023). arXiv preprint arXiv:2302. 13971,, 2023.
Zhaoxuan Wu, Zijian Zhou, Arun Verma, Alok Prakash, Daniela Rus, and Bryan Kian Hsiang Low. Tetris: Optimal draft token selection for batch speculative decoding. arXiv preprint 1517 2025.
Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailiopoulos, and Kangwook Lee. Pre- dictive pipelined decoding: A compute-latenge trade-off for exact Ilm decoding. arXiv preprint arXiv:2307.05900 2023.
Hanling Yi, Feng Lin, Hongbin Li, Peiyang Ning, Xiaotian Yu, and Rong Xiao. Generation meets verification: Accelerating large language model inference with smart parallel auto-correct decoding. arXiv preprint Ar0iv:2402. 1809, 2024.
12

---

Published as a conference paper at ICLR 2026
Jun Zhang, Jue Wang. Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via selfuspeculative decoding. arXiv preprint arXiv:2300.08116 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306. 05685, 2023.
13

---

Published as a conference paper at ICLR 2026
Appendix
A ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. In this study, human subjects animal exper- imentation was involved. All datasets used, including MT-bench, HumanEval, GSM8K, Alpaca, CNN/Daily Mail, and Natural Questions, ensuring no violation of privacy. We have taken care to avoid any biases or discriminatory out- comes our research process. No personally identifiable information was used, and experiments were conducted that could raise privacy or security concerns. We are committed to maintaining transparency and integrity throughout the research process.
sourced in compliance with relevant usage guidelines,
B REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper are reproducible. The experimental setup, for example the model configurations, and hardware details, is described in detail in the paper. We have also provided a full description of the algorithm details, to assist others in reproducing experiments.
Additionally, public datasets used in the paper, such as MT-bench, HumanEval, GSM8K, Alpaca, CNN/Daily Mail, and Natural Questions, are publicly available, ensuring consistent and reproducible evaluation results.
We believe these measures will enable other researchers reproduce work and further advance the field.
C LLM USAGE
Large Language Models (LLMs) were used to aid in the polishing of the manuscript. Specifically, we used LLM assist refining the language, improving readability, and ensuring clarity in various sections of the paper. The model helped with tasks such as sentence rephrasing, grammar checking, and enhancing the overall flow of the text.
It is important to note that the LLM was not involved in the idcation, research methodology, or experimental design. All research concepts, ideas, and analyses were developed and conducted by the authors. The contributions of the LLM were solely focused on improving the linguistic quality of the paper, with no involvement in the scientific content or data analysis.
The authors take full responsibility for the content of the manuscript, including any text generated or polished by the LLM. We have ensured that the L.M--generaeed text adheres to ethical guidelines and does not contribute to plagiarism or scientific misconduct.
D PROOF
Theorem D.1. EAGLE-2 and EAGLE-3's selection algorithum in i-th layer is special case of the
proposed selection Algorithm by setting = Aj + and C₁ -ー 155 (,)((K)
BA
Proof. Note (1) (s) is decreasing about & and u is constructed by prefix sum. Then we know k mean sequence larger than minimum, the maximum non-zero index will be K.
wi - c.. - cf max,>* 1 - -[k+]]=a[ > of (k+1) By also noticing that the
maxj>*
BA
E MORE IMPLEMENTATION DETAILS
In this section, we will present more details our implementation. And our method may have the potential limitation of pecomputing the inference cost.
14

---

Published as a conference paper at ICLR 2026
In experiments conducted with batch size of 1:
The Llama-333-70B-Insrrrtt and Vicuna-13B-vII3 models utilized a threshold of 4.
The Llama-3.1-8B-Inttee and DeepSeck-RI-Distill--e of 3.
models utilized a threshold
The 1et---..-770--nttttte model was run in a dual-card environment (2x A800 GPUs),
while the other three models were run in single-card environment (Ix A800 GPU).
For improved algorithm, all models used the following parameters: depth=13, to- tal token=72, and top_k=12
EAGLE-3 employed its default parameters, namely depth=7 and top_k=10, with total token configured according to the specific model (refer to the appendix of the EAGLE-3 paper for details).
In experiments conducted with batch size of 8:
We utilized a single-card A800 GPU environment.
For our method, we uniformly applied a threshold of 2.5, a depth 9, top_k 12, and count of 72.
For comparison, EAGLE, EAGLE-2, and EAGLE-3 were configured with their respective default parameters.
For the ablation studies:
Due to the involvement of large batch sizes, all experiments were conducted in a dual-card environment (2x A800 GPUs).
It is important to note that speedup ratios measured in single-card versus dual-card environ- ments can exhibit a little difference.
For more comprehensive hyperparameter settings, including specific values for each param- eter and detailed reproduction methodologies, please consult our supplementary materials.
E.1
THE EFFECT OF BATCH SIZE
Speedup vs. Batch Size for Speculative Sampling Algorithms
Algonith
EAGLE
ENGLE-3
CAST
Speedup
18
48
Batch Size
Figure The speedup under different batch sizes on HumanEval.
Figure plresents comparative analysis of speedup achieved by three speculative decoding algorithms under various batch Mizes-EAGLE EAGLE-3, and CAST-as a function of batch size. We observe that CAST consistently yields the highest speedup across all batch sizes, demonstrating strong scalability and robustness to increasing batch sizes. achieves peak speedup exceeding 5x at batch batch size 32. EAGLE-3 shows moderate performance,
size 1 and maintains over 2x speedup outperforming EAGLE at smaller batch sizes but converging toward similar performance as batch size increases. EAGLE, while providing stable gains at small batch sizes (e.g., 3.4x speedup batch
15

---

Published as a conference paper at ICLR 2026
size 1), suffers from rapid drop in efficiency batch size grows, eventually offering marginal speedup (close to Ix) beyond batch size 32.
This trend illustrates key limitation of bascline speculative decoding under large-batch settings and highlights the effectiveness of CAST in mitigating this degradation. The improved performance of CAST is attributed to its enhanced speculative mechanism, which more accurately predicts and validates multiple tokens parallel, thus reducing the need for fallback to the base model.
E.2 THE EFFECT OF EACH COMPONENT OF CAST
Table presents the results of an ablation study on CAST, our enhanced speculative decoding algorithm, which extends EAGLE-3 by progressively integrating three key optimization techniques: Dynamic Reranking (DR), Depth Pruning (DP), and Breadth Pruning (BP). The baseline EAGLE-3 demonstrates strong initial performance but degrades significantly as batch size increases, falling to 1.35x at batch size 16. Adding DR alone yields slight gains at larger batch sizes (e.g., 2.17x at batch size 16), while incorporating DP further improves performance consistently across batch sizes. The combination DR + DP BP (i.e., the full CAST system) achieves the best overall speedups, culminating in 4.14x speedup batch size 1 and maintaining robust 2.35x speedup at batch size 16. Notably, each additional component contributes marginal gains, confirming the cumulative effectiveness of the enhancements.
Table 3: Ablation study of CAST components.
Batch size
2
8
16
EAGLE3
3.99x
3.79x
2.98x
1.91x
1.35x
EAGLE3+DR
3.99x
3.74x
3.44x
2.77x
2.17x
EAGLE3+DR+DP
4.08x
3.82x
3.44x
2.84x
2.27x
EAGLE3+DR+BP
4.06x
3.80x
3.42x
2.79x
2.26x
CAST
4.14x
3.87x
3.48x
2.91x
2.35x
MORE RESULTS ON DIFFERENT GPUs
We present more experimental results on H20 and 4090 to show the flexibility of our methods on different GPU devices.
16

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparison of EAGLE-3 and CAST (Ours) on a single H20 GPU with Batch Size 1. Values represent speedup factors.
Model
Method
MT--eech
HumanEval
GSM8K
Alpaca
CNN/DM
Natural Ques.
Temperature=0
EAGLE-3
5.28x
6.46x
5.86x
5,80x
4.28x
4.59x
L33 70B
CAST (Ours)
5,40x
6.66x
5.86x
5.95x
4.35x
4.71x
EAGLE-3
3.13x
3.76x
3.14x
3.22x
2.85x
2.65x
V 13B
CAST (Ours)
3.30x
4.27x
3.26x
3.25x
2.95x
2.73x
EAGLE-3
3,57x
3,94x
3.67x
3.84x
3.04x
2.99x
1
CAST (Ours)
3.64x
4.24x
3.68x
3.90x
3.19x
3.16x
EAGLE-3
3.34x
3.95x
3.83x
3.11x
2.82x
2.99x
DSL 8B
CAST (Ours)
3.50x
3,95x
4.13x
3.27x
2.96x
3.15x
Temperature=1
EAGLE-3
5.09x
6.09x
5,56x
5.52x
4.17x
4.50x
L33 70B
CAST (Ours)
5,21x
6.36x
5.68x
5.78x
4.26x
4.61x
EAGLE-3
2.54x
3.25x
2,62x
2.65x
2.53x
2.33x
V 13B
CAST (Ours)
2.92x
3.57x
2.89x
2.89x
2.62x
2.52x
EAGLE-3
2.71x
3.68x
3.19x
3.18x
2.51x
2.28x
L31 8B
CAST (Ours)
2.87x
3.73x
3,22x
3.16x
2.70x
2.45x
EAGLE-3
2.65x
3.25x
2,83x
2.46x
2.15x
2.31x
DSL 8B
CAST (Ours)
2.78x
3.18x
3.16x
2.67x
2.35x
2.59x
Table 5: Performance comparison of EAGLE-3 and CAST (Ours) on two RTX 4090 GPUs with
Batch Size 1. Values represent speedup factors.
Model
Method
MT--Bench
HumanEval
GSM8K
Alpaca
CNN/DM
Natural Ques.
Temperature=0
EAGLE-3
4.28x
5.02x
4.17x
4.06x
3,90x
3.35x
V 13B
CAST (Ours)
4.54x
5.56x
4.38x
4.26x
4.07x
3.43x
EAGLE-3
3,83x
4.34x
3.98x
4.12x
3.34x
3.20x
L31
CAST (Ours)
3.97x
4.61x
4.08x
4.29x
3.40x
3.32x
EAGLE-3
3.68x
4.11x
4.07x
3.31x
3.04x
3.10x
DSL 8B
CAST (Ours)
3.75x
4.15x
4.13x
3.43x
3.19x
3.22x
Temperature=1
EAGLE-3
3,48x
3.89x
3.47x
3,44x
3.51x
3.12x
V 13B
CAST (Ours)
3.78x
4.49x
3.90x
3.68x
3.60x
3.30x
EAGLE-3
2.81x
3.82x
3.32x
3.37x
2.63x
2.41x
L31 8B
CAST (Ours)
3.14x
3.99x
3,55x
3,55x
2.92x
2.61x
EAGLE-3
2.66x
3.38x
3.08x
2.50x
2.37x
2.46x
DSL 8B
CAST (Ours)
2.85x
3.52x
3.16x
2.72x
2.57x
2.69x
17


Reviews:
[{"review_id": "V6FBDcXEhC", "reviewer": "Reviewer_ZSu1", "summary": "This paper argues existing dynamic tree speculative decoding methods (like EAGLE) are \"cost-agnostic\". They ignore system variables like GPU and batch size, which can paradoxically increase latency. The paper proposes CAST (Cost-Aware Speculative Tree). This method explicitly models these inference costs, often using a lookup table. CAST then dynamically prunes its tree by balancing token utility (acceptance probability) against the actual hardware cost. This cost-aware approach delivers up to 5.2x speedup over standard decoding and outperforms SOTA methods like EAGLE-3 by 5-20% , showing particular strength in batching scenarios.", "questions": "- What is the difference between c and n in Section 4?\n\nSee weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of using utility function to consider resource problem in the speculative decoding setting is novel and motivated. And the paper is well organized.\n\n- The way the paper formulates the utility function based on acceptance rate and how to choose the depth is new.\n\n- The experimental results are comprehensive and convincing. The authors validate their CAST method across a wide array of 6 distinct LLMs and 6 diverse tasks, ranging from multi-turn conversation to code generation.", "weaknesses": "- The method introduces precomputation overhead. If the hardware, batching strategy, or even the model (which changes the cost profile) is modified, this entire precomputation step must be redone.\n\n- There are multiple thresholds (at three new cost-utility thresholds) for tuning. What are the overheads?\n\n- The paper considers batch size as a factor and motivation but lacks more comprehensive experiments for that.\n\n- Theorem 4.1: Try to make it self-contained. What does j mean in the formula c_j. And also it is not clear what $\\lambda, \\delta$ means.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0wE9lzR3N9", "reviewer": "Reviewer_tm9i", "summary": "This paper introduces CAST, a dynamic tree-based speculative decoding method that addresses a key limitation in SOTA approaches like EAGLE-3: their failure to account for system-level inference costs. By pre-computing the actual hardware latency for different batch sizes and token counts, CAST reframes tree construction as a cost-utility optimization, dynamically pruning the draft tree's breadth and depth to maximize accepted tokens per unit of time. This cost-aware approach delivers state-of-the-art speedups (up to 5.2x), demonstrating a particularly strong advantage over prior methods in practical, high-throughput batched-inference scenarios where cost-agnostic heuristics fail.", "questions": "1. Cost of Pre-computation: Can the authors quantify the one-time profiling cost (e.g., in GPU-hours) required to generate the cost lookup tables for a single model and hardware setup?\n\n2. Hyperparameter Tuning Strategy: What is the recommended procedure for tuning the thresholds $C_1, C_2,$ and $C_3$, and how sensitive is the method's performance to these values?\n\n3. Cost Model Granularity: What is the performance impact of approximating the context length $c$ using the $select(c)$ function, and how sensitive is the method to this granularity?\n\n4. Intuition for Theorem 4.1: Can the authors provide more intuition for why the cost-agnostic EAGLE-2/3 algorithms are mathematically equivalent to Algorithm 1 with a specific linear cost model?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The method replaces prior heuristics with a novel and principled cost-utility framework. This formal optimization of acceptance \"utility\" versus hardware \"cost\" is a more robust and generalizable approach .\n\n2. Comprehensive and Rigorous Experimentation: Claims are exceptionally well-supported by extensive experiments across 6 models, 6 tasks, and 3 different GPU architectures. This thoroughness confirms the method's effectiveness and generality .\n\n3. The authors correctly use \"Speedup Ratio\" as the primary metric and insightfully argue against the misleading \"Average Acceptance Length,\" demonstrating a mature understanding of the evaluation problem. The ablation in Table 3 clearly isolates the individual contributions of the new components (DR, DP, BP), validating the paper's design choices.", "weaknesses": "1. Unquantified Profiling Overhead: The method relies on pre-computing cost lookup tables, but the paper never quantifies the one-time profiling cost (e.g., in GPU-hours), which could be a significant practical barrier to adoption.\n\n2. Lack of Hyperparameter Sensitivity Analysis: The new thresholds ($C_1, C_2, C_3$) are critical to the method, but their robustness and the strategy for tuning them are not discussed, leaving a key practical question unanswered.\n\n3. Unclear Intuition for Generalization Claim: The paper claims (Theorem 4.1) that prior work is a \"special case\" of CAST, but the intuition for why a \"cost-agnostic\" method maps to a specific linear cost model is not well-explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JI8Q34ErSr", "reviewer": "Reviewer_9P9F", "summary": "This paper addresses the inference latency of llms by improving speculative decoding. The authors argue that existing dynamic tree-structured methods, such as EAGLE-2 and EAGLE-3, are suboptimal because they ignore system-level inference costs, particularly the impact of GPU hardware and batch size. This paper introduces Cost-Aware Speculative Tree, a new dynamic tree approach that explicitly models this trade-off. CAST uses pre-computed cost look-up tables to guide the dynamic construction and reranking of the draft token tree, pruning branches where the computational cost outweighs the expected utility. Experiments across six models and six tasks show that CAST achieves state-of-the-art speedups, outperforming prior methods by 5-20% and autoregressive decoding by up to 5.2x .", "questions": "- How are the crucial thresholds $C_1, C_2, C_3$ and the FIFO buffer size $R$ determined? Please provide a sensitivity analysis for these new hyperparameters.\n- What is the practical overhead (e.g., in hours) of generating the $S_T(B)$ and $S_D(B)$ lookup tables for a new model and hardware configuration? How sensitive is performance to the accuracy of this precomputed cost model?\n- Algorithm 1 is used for both breadth pruning (Sec 4.1) and reranking (Sec 4.2), but with different cost functions ($c_k^{(i)}$ vs. $c_k$). Can you elaborate on the rationale for using the normalized draft cost for pruning but the normalized target cost for reranking?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- It correctly identifies that SOTA dynamic tree methods ignore critical system costs like batch size and GPU type, which can negate speedups . The proposed cost-utility model, which uses precomputed lookup tables to guide tree construction, is an good solution. \n- A key contribution is the generalization of prior SOTA (EAGLE-2/3), demonstrating they are special cases of this new framework (Theorem 4.1) . The empirical results are strong, showing consistent 5-20% gains over EAGLE-3 and demonstrating superior scalability as batch size increases—a vital metric for production systems.", "weaknesses": "- One of the weakness is the reliance on a new set of hyperparameters, specifically the cost thresholds $C_1$, $C_2$, and $C_3$ and the buffer size $R$, whose selection and sensitivity are not discussed or ablated. \n- The method's practicality hinges on pre-computing cost-lookup tables $S_T(B)$ and $S_D(B)$. While practical, the paper does not sufficiently analyze the cost and complexity of this profiling step, which must be run for different hardware and batching configurations. It is unclear how the $select(c)$ approximation for context length impacts the cost model's accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gz6LYymySV", "reviewer": "Reviewer_Aptz", "summary": "This paper addresses the latency challenge of autoregressive large language model inference by improving speculative decoding efficiency. Prior dynamic draft-tree methods such as EAGLE-2 and EAGLE-3 adapt token proposal structures, but they overlook system-level factors, including GPU characteristics and batch size. The proposed method models inference cost and dynamically adjusts tree depth, branching width, and token verification count to balance acceptance rates against computation overhead. Experiments demonstrate consistent gains and improvement over previous state-of-the-art methods.", "questions": "- General questions are given in the weakness part.\n\n- Is there a theoretical upper bound or lower bound under specific settings? Furthermore, do the experimental results align with the theoretical analysis?\n\n- Can you provide an example of how to represent hardware, such as parameters like memory capacity and FLOPS? Is there a unified quantitative representation for hardware characteristics?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Adeaute related background along the timeline and solid motivation analysis.\n\n- The paper explains its concepts clearly, and the writing is smooth and well-organised.\n\n- The paper presents clear experimental settings and includes a comprehensive set of comparison models.", "weaknesses": "- For the conclution \"merely increasing the tree depth and node numbers may not always result in better performance\", is there some quantitative analysis / results to support?\n- The results are mainly presented in table form. Is there a visualization of the performance trends? The paper also mentions a trade-off — are there additional experiments that explore multi-factor trade-offs in more depth? Just like Figure 5 in Appedndix E.1. Maybe more info in the Appendix could be included in the main body and cut off 1 table.\n- 3.2.2 typo \"valuesinterpreted\" \n- The core contribution of this paper lies in incorporating cost into the model’s decision process. However, the work does not provide concrete examples or detailed formulation for this component. Although the paper mentions that GPU hardware characteristics and batch size should be considered, it is necessary to further clarify the modeling procedure—for instance, which GPU parameters are included and how they quantitatively influence the cost function. This aspect is central to the method and requires empirical evidence to validate its effectiveness. Since the high-level idea is relatively straightforward, the design and justification of the cost function should serve as the main technical contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)