MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
Authors: Ranajoy Sadhukhan
Jian Chen
Zhuoming Chen
Vashisth Tiwari
Ruihang Lai
Jinyuan Shi
Ian En-Hsu Yen
Avner May
Tianqi Chen
Beidi Chen
Venue: ICLR 2025 Poster (2025)
Keywords: LLM Inference
Speculative Decoding
Performance Analysis
Primary area: generative models
TL;DR: Long-context serving, Speculative Decoding, Sparse KV Cache

Abstract: Large Language Models (LLMs) have become more prevalent in long-context applications such as interactive chatbots, document analysis, and agent workflows, but it is challenging to serve long-context requests with low latency and high throughput. Speculative decoding (SD) is a widely used technique to reduce latency losslessly, but the conventional wisdom suggests that its efficacy is limited to small batch sizes. In MagicDec, we show that surprisingly SD can achieve speedup even for a high throughput inference regime for moderate to long sequences. More interestingly, an intelligent drafting strategy can achieve better speedup with increasing batch size based on our rigorous analysis. MagicDec first identifies the bottleneck shifts with increasing batch size and sequence length, and uses these insights to deploy SD more effectively for high throughput inference. We leverage draft model with sparse KV cache to address the KV bottleneck, which scales with both sequence length and batch size. Additionally, we propose a theoretical model to select the optimal drafting strategy for maximum speedup. Our work highlights the broad applicability of speculative decoding in long-context serving, as it can enhance throughput and reduce latency without compromising accuracy. For moderate to long sequences, we demonstrate up to 2.51x speedup for LLaMA-3.1-8B when serving batch sizes ranging from 32 to 256 on various types of hardware and tasks.

Published as a conference paper at ICLR 2025
MAGICDEC: BREAKING THE LATENCY-THROUGH TRADEOFF FOR LONG CONTEXT GENERATION WITH
SPECULATIVE DECODING
Ranajoy Sadhukhan
Jian Chen
Zhuoming Chen'
Vashisth Tiwari!
Ruihang Lai!
Jinyuan Shi
Ian En-Hsu Yen? Avner May
Tianqi Chen' Beidi Chen'
*Camegie Mellon University Moffet AI Together Al
ABSTRACT
Large Language Models (LLMs) have become more prevalent in long-context applications such as interactive chatbots, document analysis, and agent workflows, but it is challenging to serve long-context requests with low latency and high throughput. Speculative decoding (SD) is a widely used technique to reduce latency losslessly, but the conventional wisdom suggests that its efficacy is limited to small batch sizes. In MagicDec, we show that surprisingly SD can achieve speedup even for a high throughput inference regime for moderate to long sequences. More interestingly, an intelligent drafting strategy can achieve better speedup with increasing batch size based on our rigorous analysis. MagicDec first identifies the bottleneck shifts with increasing batch size and sequence length, and uses these insights to deploy SD more effectively for high throughput inference. We leverage draft model with sparse KV cache to address the KV bottleneck, which scales with both sequence length and batch size. Additionally, we propose theoretical model to select the optimal drafting strategy for maximum speedup. Our work highlights the broad applicability of speculative decoding in long-context serving, as it can enhance throughput and reduce latency without compromising accuracy. For moderate to long sequences, we demonstrate up 2.51x speedup for LLaMA-2 3. 1 8B when serving batch sizes ranging from 32 to 256 on various types of hardware and tasks.
INTRODUCTION
The emergence of extremely long-context Large Language Models (LLMs) (AI@ 2024 Qwen'Team 2024 Liu et al. (023) has led to the popularity of long- long-context applications such as retrieval augmented generation Lewis et al. 2021), J, code generation (AWS) 2024 Chen al. 2021 and document summarization. Low latency and high throughput are both crucial for serving these long-context LLMs - low latency ensures a positive user experience interactive applications like chatbots (Achiam et al. 2023 Deepmind 2024).
while high throughput amortizes serving costs.
However, optimizing both latency and throughput in LLM serving presents significant challenges. Speculative decoding (SDyLeviathan et al. 2022 Xia et al. 2023 Chen et al., 2023) can reduce latency by using a smaller model to predict multiple tokens ahead followed by verification by the target model. But this approach becomes inefficient with large batch sizes because of increased verification cost(Liu et al.. [022aa] SSu et al. 2023), as shown in Fig. 7a| For small batches, the main performance bottleneck is the parameter loading cost, which can be amortized by the verification process across the tokens to be verified at the expense of increased computation. However, with large batches, LLMs become compute bound, making verification significantly costly because of its compute-hungry nature. Additionally, if the smaller model's predictions do not align well with the target model, frequent costly verifications are needed. Consequently, the usage of SD in high batch size regime is discouraged by existing Works(Liu et al. 20224a [Su et al.
2023 Miao et al. 2023). On the other hand, techniques like (Kwon et al. 2023 Yu et al. 2022 Agrawal al. 2024b) improve throughput by accom- modating larger batches, but at the cost of increased token-wise latency. While techniques such as quantization. pruning and KV cache eviction (Frantar al. [023] Xiao et al.] [024a]] (Hopper et al. [2024] Ma et al.) 2023) Sun et al. 2024b) can improve both throughput and latency, they typically result in lower quality model outputs.
Based on these challenges, we pose the following question:
"Equal contribution

---

Published as a conference paper at ICLR 2025
lead Activation
1101 Friftl 10189
100
20.0
KV load
3330
Accegtance Ratte
L1877
80
15.0
7020
70
18/708
Time
s0rte0
10.0
60
48N778
1
SelFapec Laaa--22
16 Pistch size
236
Aug. Tekonsiise Latenry Om1
0.00
0.05
Memory Ratio
0.10
0.15
0.20
0.25
(b)
(c)
(a)
Figure 1: (a) Time breakdown of LLaMA-3. 1-B8 vs batch size (input length= hardwarc=&xH100s) (b)
Throughput of autoregressive decoding and LeaamingLLM-baseee self-speculation of LLaMA-3. 1-8B against per-token latency for prompt length of 32k. (c) Draft token acceptance rate comparison for LLaMA-3 1-70B Self-speculation using Top-k attention achieves much higher acceptance rate than other draft-target pairs, even with lower memory ratio. The x-axis represents the ratio of draft memory footprint to target memory footprint.
Can we simultaneousty improve throughput and latency without sacrificing accuracy, particularly for long sequences?
We answer with a resounding yes! For large batches of long sequences, we show that SD can be used more effectively to improve both throughput and latency without degradation of the output quality. We base hypothesis on the following interesting insights:
(1) KV Cache Is The Dominant Bottleneck In Large batch size Long-conted Regime: In long-conted and large batch size regime, the KV cache outgrows memory footprint of the model parameter and continues to increase with batch size. Computation also increases with batch size, but due to the high peak PLooo-yo-memory bandwidth ratio of modern GPUs, the KV loading time increases much more than former for larger batches, making LLM inference more memory bound, as shown in Fig. la Yuan al.) 2024).
(2) SD Can Improve T Throughput Only Beyond Critical Sequence Length: While existing research(Liu cet al. 22224a Su et al. (023) suggests that SD is inefficient for large batches due to high verification costs, this limitation only applies to very short sequences. Because with short sequences, increasing the batch size makes computational costs the primary bottleneck, which is prohibitive for an efficient verification process. However, once sequences exceed a certain critical length (which varies based on the model and hardware), the KV loading cost becomes the dominant factor, even for large batches. At this point, SD becomes effective again because the computational overhead of verification becomes less significant compared to the KV loading costs, which can be amortized across the tokens to be verified.
(3) Compressed KV Cache Enables More Efficient Speculation: Token acceptance rate is crucial for SD in large batch processing to minimize costly verification steps. Our research found that compressing the Draft KV cache leads to higher acceptance rates than compressing model weights. To evaluate model compression only, we test different draft-target pairs on PG-19 (Rae et al. 2019) sequences of length only 256, to restrict the KV cache impact. For KV compression, we tested LLaMA-3 1-703 on longer sequences (4,000-100,000 tokens? ! Jusing Top-K selection for KV sparsification. Fig. Ic illustrates that model compression alone unable reach 90% acceptance rate, while KV compression achieved significantly higher rates under similar memory constraints. This advantage becomes even more significant with larger batch sizes, offering a promising new direction for improving the match-processing efficiency speculative decoding.
Building upon these insights, our work MagicDec illustrates that SD can improve speedup even for large batches by utilizing KV compression, contrary prior belief. As shown in Fig. Tb under long context-length. compressed KV-based self-speculation can improve throughput and latency at the same time in all spectrum, without hurting generation quality. Furthermore, MagicDec evaluates different KV compression-bart based drafting methods to determine the optimal approach based on the specific model, hardware, and task requirements. We structure the paper follows.
In Section 3.1 we theoretically analyze the factors that decide the efficiency of speculative decoding. Section 3.2 discusses how the performance bottlenecks in LLM inference shift with batch size and sequence length, and what are its implications on SD's batch- processing efficiency. In the light of this study, we discuss the challenges involved with conventional SD in large batch setting and how can be overcome
batch size is 1 nullify the effect of batch size on KV cache size
2

---

Published as a conference paper at ICLR 2025
by KV sparsification based drafting. Additionally, we introduce the concept of the critical sequence length beyond which SD can achieve higher speedups for larger batches contrary to prior studies (Liu et al. 2024a
Su ct al.
2023
Miao al. 2023).
we show why compressing the KV cache is crucial for effective batch processing. Our experimental results demonstrate that this approach achieves higher acceptance rates and, consequently, better batch performance compared to using parameter-efffiiene draft models. Section 4.4 discusses the trade-off between draft cost and acceptance rate for different static and dynamic KV sparsification algorithms on different kinds of tasks.
In Section 3.3
Finally, in Section [ we provide a comprehensive empirical evaluation across different hardware setups and tasks to show the effectiveness of our theoretical analysis and method. We demonstrate that our approach achieves 2.51x speedup in large batch settings for LLaMA-3. 1. 1-8B on 8xH100 GPUs, significantly improving both throughput and latency over traditional autoregressive decoding (
2 RELATED WORKS
Numerous efforts have been made to improve the latency and throughput of LLMs. Methods like Flash decoding (Dao 2023), and Flash decoding++(Hong
al.
2023) have performed system optimizations to
improve latency. KV compression methods (Li et al.
2024
Gupla et al. 2021 Xiao et al. 2024b Tang et al.,
2024 Cai.
2024 Zhang et al. 2023 Oren al.
2024
utilize attention sparsity to reduce the KV loading
cost. KV compression can improve both latency and throughput, but suffers from accuracy degradation.
Batching has been natural technique to improve GPU utilization by amortizing the model parameter loading cost across requests, thus boosting throughput. Recently continuous batching (Kwon et al. 2023 Yu et al.
2022 Prabhu al. (024) has been proposed to address the problems arising from heterogeneous batches with unequal context and generation lengths. In our work, we have considered the orthogonal direction of homogeneous batches, and the aforementioned methods are complementary to our observation.
Speculative decoding (Leviathan al., 2022; Xia et al. 2023 Chen et al., 2023 has emerged as an algorithmic novelty to improve latency without quality degradation. SD improves latency by using fast draft model generate multiple tokens, which are then verified in parallel by the LLM, thus maximizing GPU utilization. However, as the batch size increases and computation resources are saturated, the verification of speculated tokens becomes costly. Hence, existing research(Liu et al. [2024a] [Su al.) [023] Miao et al.) 2023. [Sun et al.] (2024a) has discouraged the use of speculative decoding to serve large batches of requests. In our work, we show that this claim only applies to short sequences.
To address the KV bottleneck for serving long sequences, we take inspiration from TriForce (Sun et al. 2024a), which demonstrates the effectiveness of self-speculation with compressed KV. While Trif is designed for small batches of extremely long sequences, we have focused on large batches of moderate to long sequences, which is more nuanced in terms of draft selection. For draft selection, we have considered a subset of KV compression techniques Xiao et al. 2024b Li et al. 2024 Zhang et al. 2024) to exhibit the trade-off between draft cost and acceptance rate. Our work does not advocate for a single KV compression technique, rather provides framework to choose the optimal strategy from suite of such techniques.
Many methods have been proposed to improve speculative decoding. For instance, Speculation Parallelism (SP) (Timor et al. 2024) overlaps target verification with draft speculation to enhance speedup. This method evaluates the drafter based on draft cost and acceptance rate, which is similar our approach: with the high acceptance rate and low draft cost of compressed K KV-based drafting, along with reduced verification costs provided by SP, speculative decoding can achieve even greater speedups in long-context serving scenarios.
analysis. SP complements
3 THEORETICAL ANALYSIS
In this section, we present our theoretical analysis of speculative decoding and LLM inference performance. We hegin by reviewing the mathematical formulation of speculative decoding speedup and identifying the key factors influencing it. Next, we analyze I.M inference in long-contex: scenarios, highlighting the bottleneck shift that enables speculative decoding to achieve speedup with large hatch sizes. Finally, we demonstrate the necessity of compressed KV-based drafting to achieve high speedup in long-context, large batch scenarios.
3.1 SPECULATIVE DECODING SPEEDUP ANALYSIS
The decoding time required by the target model and the draft model for a batch of size B and sequence length S are given by Tr (B.S) and Tp (B,S) respectively. The time taken by the target model to verify
3

---

Published as a conference paper at ICLR 2025
Prefill11000
Preill 4000
22
Profilla0000
Pocill
Droit Time (Togget Time
Pref18-16000 16000
Prefil-16000
32000
Feef0l=61000
Prefil660000
04 Batch Sizte
128
14
35.
Bottc Stone
Banokusko
(a)
(b)
(c)
Figure 2: Thcoretical analysis and expected speedup for L.LaMA-3, 1-888 deployed on 8xA 100s with y = (a) Theoretical TD/TT 'T versus batch sizes, (b) Theoretical Tv (7)/TT versus batch size, (c) Theoretical expected speedup of sell-speculation across different batch sizes ( draft KV budget = 512).
? tokens is given by Tv (B,S.) Given the draft token acceptance rate a E [0,1] and speculation length the expected number of tokens generated in one verification step is denoted by D( As described (Leviathan al. 2022), the expected number of generated tokens can be estimated as,
1-a7+1
2(T,,))== : El# generaledlokenn) ( 1-a
(I)
The total time taken for speculative decoding, T:S Totass is given by:
Titu (B..)
Hence, the expected latency token for speculative decoding is simply T =T -TF0///a/(())))) For brevity of notation, we will refer to these times as Tr, TD, and TV in the future, with the dependence on B and S implied, unless otherwise specified.
The speedup of speculative decoding and the factors regulating it can be understood from the following equation,
(T.DD Tv(1) TT
Aug
Tf
117,aa)
Tr
From equation 2 we can see that speed-up depends on three primary factors: (a) target verification to decoding cost ratio Tv (7) TT, (b) draft target cost ratio TD/TT, and (c) expected generation length 32(7,ar). For better speedups, we aim to achieve low (v)r)/T) (close to 1), low TD/Tr (close to 0) and high !2(7,a))
3.2 KV CACHE BOTTLENECK ENABLES SPECULATIVE DECODING SPEEDUP
In this section, we analyze how the inference bottleneck shifts as sequence length and batch size increase and how it affects the factors discussed in Section 3.1
For short sequence lengths, speculative decoding negatively impacts batch inference efficiency (Liu et al., 2024a| (Su et al. 2023). As batch size grows, the linear layers become compute-boume due to improved arithmetic intensity. This reduces the availability of compute resources that speculative decoding utilizes for parallel verification, essentially increasing the verification to decoding cost ratio.
In contrast, for moderate to long sequences, we observe transition towards memory- hound regime since with increasing batch size, the memory cost of loading the KV cache becomes the dominant factor. This shift from compute-bound to memory-bound inference makes the verification cost comparable to the target decoding cost. Because verification and decoding share the same KV budget, their KV cache loading costs are equivalent. The high ratio of peak FLOPS to memory bandwidth in modern GPUs causes the increase KV loading time with batch size outweigh the increase computation time (see Fig. [a]. As a result, although compute-bound linear layers increase verification cost, it is mitigated by the KV bottleneck.
Based on this shift in bottlenecks, identify critical sequence length Sinflection- beyond which speculative decoding achieves speedup for large batches. Moreover, its speedup tends to increase with batch size. This threshold depends on factors like the model architecture, hardware configuration, and drafting strategy.
4

---

Published as a conference paper at ICLR 2025
Liam---7b--0
2.25
Mata-Llama-3.1
2.00
FLOPS/Memory
Sgordup
200
1.50
150
325
1930
A310
Y200
1.00
1271
100-
0.75
2000
Prefill
10000 13800
1000
2000
Preftil Length
2000
6000
5009
6000
(a)
(b)
(c)
Figure 3: Theoretical analysis of sell-speculttion for LLaMA-2-7B- 3B3K and LLaMA-3. 1310--3311-8 with draft KV
budget of 512 and batch size of 256. We assume the acceptance rate 0.8 here. (a) Ratio of target-draft latency (r-Tbb (T'r ) and verification-tareel latency (TV(r)/TTr versus sequence length for LLaMA-2 ------2----25 with y (b) Theoretical speedup for different sequence lengths with a fixed a = 0.8. (c) Thcoretical arithmetic intensity for different sequence lengths and different models.
For S Simttection
In this regime, as batch size increases, decoding becomes more compute-bound, Large batches can saturate the available compute, making verification relatively more expensive, illustrated Fig. 2b The cost ratio (r)))/T increases significantly for 1000 token long sequences. If the draft token acceptance rate is low, the target model spends considerable time verifying incorrect speculations, reducing SD efficiency. Our theoretical estimate in this regime aligns with (Liu et al. (2024a). The expected speed-up with speculative decoding decreases with batch size for context lengths below the critical sequence length.
For S² iiitection
In this regime, speculative decoding can provide speedup for large batches, and this speedup even tends to increase with batch size when we use some intelligent drafting strategies. This happens a combined effect of how verification decoding cost ratio (Tv (T)(TTTT)) and draft target cost ratio (Tb/Tr)
evolve with increasing batch size, as shown in Fig. 2b and 2a
For long sequences, KV cache loading becomes the primary bottleneck rather than compute (Sun et al. 2024a Aminabadi et af., 2022) and the target model shifts towards memory bound regime, as shown in 3c Because KV memory bottleneck scales with batch-size, this shift is sustained even for large batches.
As the verification and decoding phases share the same KV loading cost, the cost ratio (r))//T remains
close to 1.
However, the cost ratio (v)))/ still increases monotonically with batch size and cannot explain how we can achieve higher speedups for larger batches. The draft to target cost ratio (Tb/Tr) plays an important role here, If the KV cache size of the draft model increases slower than target model, the cost ratio T /T will decrease for larger batches. That is because the target model inference will be more dominated by the KV cache bottleneck rather than the draft.
As Figure 2c illustrates in the case of LLaMA 3. 1-88 the theoretical speedup of speculative decoding is expected to improve with increasing batch size for longer sequence lengths. The speedup decreases with batch size for S 4000, but for 4000, the speedup increases with batch size.
As illustrated Figure 3c this critical sequence length Sinteztion depends on both the model's FLOPS-to- memory ratio and the GPU's FT.OPS-to-memee bandwidth ratio. For device with higher FL.OPS-to-mmoon bandwidth ratio, we expect lower Sinficction- Models also affect this critical sequence length. For instance, GQA model like LLaMA-3. 1-8B tends to have higher Siitection due to Grouped Query Attention (GQA), which requires a larger sequence length to achieve the same KV memory footprint.
3.3 COMPRESSED KV CACHE ENABLES MORE EFFICIENT SPECULATION
In this section, we explain why KV compression is preferred over lightweight draft models for speculation in long-context, large batch-size scenario. There are primarily two reasons,
KV cache grows beyond the parameter memory footprint: Unlike parameter memory, the KV cache size grows linearly with batch size. If we use LLaMA-3. 3.1-8 as draft for L.L.M--3. 1-70B and
5

---

Published as conference paper at ICLR 2025
Tredt Loces
97.5
Poatl 129ge
Unsh Koaaer Dharst Merucy
Peedi 12600
99.0
Douh Xrasity Traget
Acceptante Sttk
92.5
60.0
87.5
Tep-k
Seupky
Banch Stoe
Torch Stme
500 1000 1508 2000
2580
3009
3500 1060
KY Budget
(a)
(c)
Figure (a, Draftttrget memory ratio vs batch size across different sequence lengths for LLaMA 3. 3.1-88 1B /LLaMA 1000-311-777 1-B0B and LLaMA-22 2-7B /LLaMA 2- 2-70B models. self self-speculation acceptance rate of different drafting strategy versus KV budget. Target KV length: 32000.
(c) LLaMA-3. 3 1..1-BB
LLaMA-2-77 for LLaMA-2-70B the draft models can occupy up 38 140% memory footprint of target models (Figures 4a and 4by due to the fact that /lima//die /ditmode is higher. Hence, in this regime, small draft models are not sufficient and compressed KV-baeed drafting is quite beneficial (Sun et al., 2024a). This can be seen in Figure Bal which illustrates how TTr for fixed KV size draft self specuulcnnntion with LL aMA-3. 1-8B approaches 0 with increasing sequence length for batch size 256.
KV compression achieves a better token acceptance rate than model compression: A high draft token acceptance rate critical to restrict the number of costly verification steps while serving large batches. Interestingly, we see that KV cache compression can be more cost-effective way to improve the acceptance rate of draft tokens, especially a high batch size long-context regime. Figure Ic illlstrates this phenomenon that if target LLM speculates itself with sparsified version of its own KV cache, then it can achieve acceptance rates higher than those of small draft models with full KV cache.
In summary, a draft model with compressed KV cache achieves two important factors for higher speedup in long-contex: scenario: low draft cost and high acceptance rate. Figures 7b and 7c empiricllyy illustrate the efficacy of this drafting strategy over standard SD with a small draft model in achieving higher speedups.
MAGICDEC
In this section, we present the trade-off analysis MagicDec performs to identify the correct drafting strategy. Section 3.3 we have motivated the reason behind adopting compressed KV-based drafting in this regime. However, there are three different factors that we need to consider to effectively leverage KV compression (a) draft model size, (b) draft KV cache size draft KV budget, and (c) KV compression algorithm. All three factors are to be considered to strike the perfect balance between draft cost and acceptance rate.
4.1 GENERAL FORMULATION OF SPEEDUP WITH COMPRESSED K KV-BAC KV-BASED DRAFTING
To begin with, we give a general formulation of speedup obtained with compressed KV-based drafting. The following analysis considers sparse KV selection algorithmss however, can be easily extended to other KV compression methods (Hooper et al. 2024 Liu et al. 2024b Singhania et al., 2024). The draft cost for sparse-KV methods depends on two main components: (1) draft model decoding cost, and (2) the cost of KV selection. For given KV sparsification strategy (select) with a fixed KV budget of K, the selection cost is denoted as Teelecs (B,S,BBS K)), while the decoding time for tokens is (B,, The total time taken by the draft using this KV strategy with KV cache budget K is:
T D. (B.S)=T (B,S,)
(3)
Using this as the total draft decoding time in equation 2 final objective becomes
TSD Aug TT
Y: TT
Tv
Tr(,,)) ]
min
min K.
(4)
T K,jja
D(Tra)))
Tr(B,))

---

Published as a conference paper at ICLR 2025
Bad::20 SD
Budgg
Skg Leugth
(a) Speedup Comparison
(h) Trade-off Analysis
(c) Draft KV Budget Selection
3
cwe
qu-1
Snap-KV KV
PQ
Snap-KV Cost Ratio
PQ Cost Rusio
Figure 5: Comparative analysis of two KV selection algorithms SnapKV [i et al. 2024 KV selection) and PQCache (Zhang ct al. 2024) (dynamic KV selection) on Ruler tasks - needle in a haystack with passkeys 3, common word extraction, question answering (context length = 32.000). (a) Expected speed-up comparison between the two KV selection methods based on MagicDec evaluation framework. (b) Trade-off analysis between Draft-to Draft-to-target cost ratio and acceptance for SnapKV and PQCache methods. (e) Minimum acceptance rates required be achieved by self-speculation with different draft KV cache sizes to achieve 1. 1.8x specdup over standard autoregressive decoding by LLaMA-3.. 1 -8B The actual acceptance rates obtained for PG-19 dataset are marked with respective colors. The admissible budgets for each sequence length are ticked right.
Now we discuss in detail the three main factors that decide the total draft decoding time TD, Deleect, and the final speedup.
4.2 DRAFT MODEL SIZE SELECTION
Even with a compressed KV cache, the draft model weights can play a role in deciding the best performance. The draft model parameter loading is the major part of draft cost when KV cache size is small. Usually at lower batch sizes, small draft model with compressed KV cache can outperform self-speculation because of lower draft to target cost ratio. When batch size and sequence length are relatively small, the parameter loading cost can impede the draft performance. Moreover, for smaller batches, the token acceptance rate requirement can be relaxed to favor much more efficient draft model. However, beyond certain batch size, self-speculation can become more efficient because of its higher acceptance rate, as shown in Fig. 7c
4.3 DRAFT KV BUDGET SELECTION
For fixed draft model and KV compression algorithm, the optimal draft KV cache size varies across different batch sizes and context lengths. Hence, before selecting the optimal KV compression algorithm, we need to find the respective optimal KV budgets of the candidate algorithms. We illustrate the importance of optimizing the KV budget of static KV selection algorithms for self-speculation sequence lengths and batch sizes require different minimum acceptance rates to achieve any speedup via speculative decoding. Similarly, different KV budgets and different draft model would have different draft cost-acceptance rate trade-offs. This plot recommends the admissible draft KV budgets that reach the required minimum acceptance rate. This trade-off analysis is particularly useful for serving heterogeneous batches with different sequence lengths. Different sequences in the same batch can leverage different draft KV cache sizes to achieve the required speedup.
Figure 5c Batches of different
4.4 COMPARATIVE STUDY ON KV SELECTION STRATEGIES
Finally, MagieDec has to choose among differrntt kinds of KV selection algorithms to regulate the search cost Tretect- Although top-k attention can achieve very high acceptance rate with a much smaller KV cache budget, it not a practical draft option because of its prohibitively high KV selection cost.
There are many potential alternatives to top-k attention, but determining the optimal one is not straightforward. There are primarily two kinds of KV selection algorithms (a) dynamic KV selection algorithms such as Tang et al. 2024 Zhang et al. 2024). (b) static KV selection algorithms such as (Xiao et al. 2024b Yang Li et al. 2024). The first kind of algorithms dynamically searches the KV cache for each input query, attempting to find the top k nearest neighbors. Although these methods can achieve higher acceptance rates, they incur substantial search costs. Conversely, static KV selection methods pre-gathes sparse KV
al. 2024
7

---

Published as a conference paper at ICLR 2025
cache for attention approximation during generation. This approach eliminates search overhead but typically results in lower acceptance rates.
Static vs Dynamic:
We evaluate state-of-the-art KV selection strategies using both our theoretical framework and empirical acceptance rates from self-speculation with the LLaMA-3 1-8B model on various Ruler tasks (Hsich et al. 2024). Our analysis includes both static (c.g. StreamingLLM (Xiao et al. (2024b) SnapKV [L et al. 2024) ) and dynamic (e.g., PQCache (Zhang et al. 2024). TopK) KV selection algorthmss, exploring different K V budgets and speculation lengths to estimate optimal theoretical speedups.
Figure 5 illustrates the trade-off between two representative KV sparsification algorithms, SnapKV and PQCache, and their respective theoretical speedups on three distinct Ruler tasks: needle in a haystack with passkeys (niah-multikeys-3) common word extraction (cwe), and question answering I (qa-1), -1). Snapk a static algorithm, has a lower draft-tootargee cost ratio compared to PQCache, as PQCache incurs batch-size-deeeedeet KV selection cost select.
When the acceptance rates of static and dynamic methods are similar, the static method tends to dominate, as seen in the owe and qa-1 tasks. However, for the niah-multikeys-3 task. PQCache benefits significantly from its higher acceptance rate. With an acceptance rate close to 1, PQCache can leverage longer speculation lengths, which significantly reduces the objective function in equation + Nevertheless, with increasing batch-size, KV search cost dominates again and the static algorithm starts to outperform the dynamic one.
EVALUATIONS
In this section, we empirically validate our theoretical analysis and demonstrate the effectiveness of our drafting strategy selection modeling. Specifically, in Section [.1] we demonstrate the end-to-end speedup of self-speculation with sparse KV, showing that speculative decoding achieves speedup for moderate to- long sequences, with speedup increasing as batch size grows, when sequence length exceeds a critical threshold. In Section 5.2 compare the speedup of two drafting strategies, highlighting the effectiveness of our we perform an ablation study on the speedup of speculative decoding.
approach. In Section 5.3
5.1 END-TO-END SPEEDUP
We demonstrate the effectiveness of our analysis in Section throughput and latency for moderate-to-loog sequences.
3 that speculative decoding can improve both
Sctup: We use StreamingLLM (Xiao et al.) (024b) style sparse KV for drafting and conduct experiments across various batch sizes and sequence lengths to evaluate speculative decoding speedup. The system implementation details are shown in A. The evaluation is performed using the state-of-the-art long-context model 3--8 on the PG-19 dataset (Rae et al. 2019). Each run generates 96 tokens per sentence in the batch through greedy decoding on 20 batches. We tested two draft KV cache budgets to assess the trade-off between draft cost and acceptance rate.
Results: Fig. shows the speedup achieved by speculative decoding at the optimal speculation length across various batch sizes and sequence lengths. These experiments are conducted on 8xA GPUs.
SD can achieve speedup for moderate long context length. We can find that speculative decoding consistently outperforms autoregressive decoding except when batch size is large and sequence length is short, which indicate the conrectness of our analysis in Sec. 3.2
SD achieves better speedup with larger batch sizes. We find that on 8xA100, when the sequence length exceeds 4000, speculative decoding achieves speedup, which increases with batch size. This result aligns with our analysis in Sec. 3.2 To verify our analysis of factors affecting the critical sequence length, we ran experiments on higher-end GPUs (H100) and lower-cost alternatives (L40), and compared the results with LaMA-2 71--BB322 As shown in Table [ the H100 achieves higher speedup than the A100 and L40 under the same setting (sequence length, batch size, and drafting strategy). This is due the H100's higher FLOPS- to- to-memory bandwidth ratio, which lowers verification cost. Additionally, we can see for 8000 sequence length and the 32 batch size
without GQA achieves higher speedup than LLaMA 33.1-8 11-BBBB with 32000 sequence length, that's because Non-GQA model has lower FLOPSSto-memory ratio.
'SnapKV V was chosen for its superior acceptance rates among static algoritums, utilizing average pooling with kemel size of 5 and an observation window size of 32. PQCache employs product quantization with 16 sub-vectors and 8-hit quantization per key vector.
8

---

Published as a conference paper at ICLR 2025
1.8
1.9
1.8
Sanndap
1.4
Speedup
1.4
1.3
1,0
1.0
0.6
0.5
Batch 128
256
64
125
259
Batch
Prefil 2006
Precil tenoo
Prwiill s2000
Predill 100non
Profil 8000
Prefil 20000
Predil 64000
Figure End-to-end speedups for Streamingl M.wenninnggMM-MM-M-M MM--aaeed self-speculation with L...MA3.1- 13.1188 across various compressed KV budgets (left: 256, right: 512) on PG-19. Annotations indicate "Kysimal. which is the value conreponding to the highest speedup achieved. Experiments are conducted on 8xA100 with 8-way tensor parallelism. Raw data can be found in A.2
Table I: Results on L40 and H100, StreamingLLM budget for the draft model 512, each with the optimal ?
Target
Draf
Tisk
GPU
Prefill
Box
Y
<Tp(1)
Ty(1)
(7(.0))
TAR
TSD
x
Streamingl11.
PGL19
8s1.40
32
45.12
36662
30.32
Liama2 7BB32K
StreamingLLMM
PG-19
8x,.00
32
29,06
42.02
255
35.13
28.70
1.22
Llmm22 7D-32K
St
DG-19
8s1.40
8000
64
3
58.33
71,85
3.14
62.92
42.96
1.46
Litead. 1 8B
19
4x11:00
32000
32
15.09
18.30
282
17.32
12:16
1.42
Liamai2
StreamngLL.M
PG-19
4xH100
15.64
1485
10.29
Llam22 32K
Streminn
PG-19
4x11100
8000
64
23.63
27.90
3,37
26.17
15.58
1.88
5.2 COMPARING DIFFERENT KV COMPRESSION METHODS
this section, we compare two static KV compression methods for drafting, with results shown Fig. 7b and Fig. 7c The detail results are in Table 6 We perform sweep select the optimal speculation length and KV budget for each method. The best draft budget for Streamingl LL-M based self-speculation is 512, while for SnapK V-based approach, it is 2049. The results indicate that SnapK V-based drafting outperforms StreamingLLM for self-speculation in all the cases. Based on Fig. 4c and our analysis in Sec. 4 the key factor the acceptance rate. Both StreamingLLMM and SnapKV are static KV compression methods, neither incurs KV search overhead. However, SnapKV has a much higher acceptance rate, which increases rapidly with KV budget, mitigating the rise draft cost. In contrast, StreamingLLMM acceptance rate has a lower upper bound and increases more slowly with KV budget. As result, SnapKV achieves higher speedup due to the combined effect of acceptance rate and draft cost. We further evaluated SnapKV-based V-hased self-speculation across different batch sizes, sequence lengths, and tasks, with promising results. As shown in Table SnapK V- V-based self- speculation achieves up to 2.51x speedup, demonstrating speculative decoding's ability to improve throughput.
Table 2: Further Results of SnapKV Self-speculation on Different Tasks
Target
Draf
Tisk
GPU
Prefil
Bst
<Tb(())
Tv)
R())a))
T-B
TSD
x
Llama3.1 1-8B
SnupKV
PG-19
Sd11100
100000
41
34.34
28.50
5.61
25.96
11.35
2.29
11ams8.1-58
SuppVV
41
$3.90
7.91
25,00
10.6
2.43
1.mma.11- 1-8B
SnupKV
CWB
Su11000
100000
41
53.98
29.93
8.21
25.83
10.29
2.51
Liamad.1 1. 8B
SnupKV
PG-19 19
Sa11100
64000
64
32.89
25.80
5.41
25.52
11.5t
221
1Jama,.1158 1-5B
SupkV
Q4-1
SdH100
64000
64
38
20.11
6.08
25.43
2.27
Linme3.1158
SnapkV
6400
64
43.91
29.29
6,33
25.08
1081
236
5.3 ABLATION STUDY
In this section, we present ablation studies of our speculative decoding speedup analysis model.
Draft KV Budget. As modeled in Section 4 the selection of KV budget depends verification cost, acceptance rate, and draft cost. As shown in Fig. 66 when batch size and sequence length are large, a larger KV budget results in higher speedup. In this scenario, the L.LM is highly memory-bound verification

---

Published as a conference paper at ICLR 2025
14 rebioer bropletti
100
1.00
(a) Prompt Length=256
(b) Prompt Length=8192
(c) Prompt Length=32768
Figure 7: Comparison between different drafting strategy for LLaMB-3: 1-83 under short, medium and long context length across batch sizes. Hardware: 8xH100. Each with optimal gamma. Dataset: PG-19,
is low, but its absolute value is much larger than the draft cost with fixed KV size. Therefore, larger KV budget with a higher acceptance rate is preferred to increase the average generation length per step.
Draft Model Weights. Draft model weights loading is also a part of draft cost. We have several choices of drafting stategy with the trade-off of draft cost and acceptance rate. small draft model can have much lower model weights loading cost, but with sigmificant lower acceptance rate. We conduct experiments under prompt length 256, 8192 and 32768 to show the effect to speedup of different draft model selection. The results are shown in Fig. 7 We can see in Fig. 7 that when sequence length is not sufficient long and batch size is not very large, small draft model with the KV compression tends to outperform self-specclation. This is because, in these scenarios, KV doesn't 't fully dominate inference, and model weight loading makes draft costs of self-speculation lot higher. However, when both sequence length and batch size are very large, and the KV cache dominates LLM inference, specullann surpasses the small draft model, model weight loading contributes minimally to overall latency. The high acceptance rate of compressed KV self-speculation has higher specdup upper bound, and leads to better speedup when batch size is large, demonstrated in Fig. 7c
Models. Different models have different FLOPS Memory Ratio and acceptance rate. We also conducted experiments on Qwen2. 5- Qwen2. 5-333 and Mistra ratt-all- - 7B--0.. models to show the generalizability MagicDec. The results are shown in Sec. A.5 We can see speculative decoding works well for these models, achieving up to 2.06x speedup for Mistral-7B-B- 5-7B and 1.51x speedup for Qwen2. 5-32B on PG-19 dataset. The trend of speedup also matches our previous analysis and the LLaMA- 3. 1-B results.
3 1.89x speedup for Qwen2.
CONCLUSION AND LIMITATION
Optimizing both throughput and latency for LLM inference is challenging, especially for long-context large batch-size regime. Our analysis reveals that speculative decoding can be beneficial in this regime, with its efficacy increasing with larger batch-sizes contrary existing misconceptions. In search of effective drafting strategies, we discover that KV compression is easier than model compression to achieve higher acceptance rate at the same memory budget, which becomes more prominent in high batch-size and long context-length regime. Leveraging these insights, we explore different KV compression algorithms for drafting and present bottleneck-awand aware general formulation to select suitable drafting strategy based on task, batch-size and sequence-length MagicDec only focuses on decoding performance for long-com long-context LLM serving, while the prefill is also very challenging in this scenario. There has been some work focusing on improving the prefill performance (Agrawal et al. 2024a| Zhong et al. 2024), which could be integerated with MagicDec to improve both prefill and decode performance. MagicDec tends to achieve better speedup on high-end bandwidth ratio and large HBM size. Future work can explore the adoption of speculative decoding on offloading and distributed setting to reduce the communication overhead, thus better utilize the resource of commodity devices.
GPUs due to their higher FL.OPS-to-menne
ACKNOWLEDGEMENTS
We would like to thank Xinyu Yang, Yang Zhou, Harry Dong, Haizhong Zheng, Hanshi Sun, and the anonymous reviewers for providing us constructive feedback on our paper. This work was partially supported by Li Auto, Together Al and Moffett AI.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Amey Agrawal, Junda Chen, fnigo Goiri, Ramachandran Ramjee, Chaojie Zhang, Alexey Tumanov, and Esha Choukse. Mnemosyne: Parallclization strategies for efficiently serving multi-million context length Ilm in- ference requests without approximations, 2024a. URL BtLhhttps:/ wttpaaiiivv..ia....aaaaa org/ //bs///////// 2409 17264
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming throughput-latency tradeoff in Ilm inference with sarathi-serve, 2024b. URL https: arxi.. org/ abs 2403. 02310
Meta. The Ilama herd of models, 2024. URL cublications tteeetheeleee 1 11ama herd of models
http://ii.nttaa.oommeeeeeeee
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022. URL
btttps://axxv..oo/ abw/ 00032
Amazon AWS. Codewhisperer, 2024. URL ht tps: a com/ http://oodwwwippc/ sperer
Zefan Cai., Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic kv cache compression based pyramidal information funneling, 2024. URL /
org/ http///bb///....////// 02069
Charlic Chen, Sebastian Borgeaud, Geoffrry Irving, Jean- can-tBatiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrow, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Heb- gen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL httlltttps:/aaatt
org/ abs 2107.0003333
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL
https::/arxvv.org// org.a:aeeeeoootaoaaaooooaaoaaaaa
2307 08691
Google Deepmind. Our next-generation model: Gemini 1.5, 2024. URL htt.https://blog.ggg
[echhoooogy/aa/googgeeeggg otgp::/ew..eaaaiiiaooooeooaoaaaa februaa---- #bbuild experiment
flashinfer-ai. Flashinfer. URL httpp//s///ttttt..c.oaaaaaaaaaaa
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate pottinnnn quantization
for generative pre-trained transformers, /Rt.///sss///axx....e///////////
Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, and Jonathan Berant. Memory-efficient transformers via top-k attention, 2021. URL /ttpppsss///avvv..//.
arxiv. org/ abs 2ww..210....00 06899
Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Hanyu Dong, and Yu Wang Flashdecoding++: Faster large language model inference gpus. arXiv preprint arX0v:2311.0122,, 2023.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with ky cache quantization, 2024. URL htt.ttttss... /ttpssi/////vva// wwtp:/o..//oooooo.ooo......ossss 18079

---

Published as a conference paper at ICLR 2025
Cheng-Ping, Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Bonis Ginsburg. Ruler: What's ' the real context size of your long-context language models?, 2024.
URLtttss://aaxie org/ abs/ 2404 06654
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joseph E Gonzalez, Hao Zhang, and lon Stoica. vllm: Easy, fast, and cheap IIm serving with pagedattention. See https://vllm.ii/ (accessed ), 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. arXiv preprint arXiv:2211.71199 2022.
Patrick Lewis, Ethan Perez, Alcksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sehastian Riedel, and Douwe Kiela. Retrieval-augmeented generation for knowledge-intensive nlp tasks, 2021. URL
https://arxii..org//bat
Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapky: Llm knows what you are looking for before generation, 2024, URL
lttpps://arxiv.com
alos/ 2404. 2444444449499
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. URL
https://arxixs org/ abs 2304 08485)
Xiaoxuan Liu, Cade Daniel, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Xiangxi Mo, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Optimizing speculative decoding for serving large language models using goodput, 2024a. URL
/abs / 404.14066
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu, Kivi: A unnin--free asymmetric 2hit quantization for kv cache. arXiv preprint arXiv:240..02550. 2024b.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models, 2023. http:/:t/t//aiii....ot wtt..orgaaaaa//......// 21625.1662777
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng. Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative Ilm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781 2023.
Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi-state mns, 2024. /tl/tttps:://xxivorrgg/
06104
Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. vat- tention: Dynamic memory management for serving 1lms without pagedattention, 2024. URL
pytorch-labs. Gpt-fast, 2023. URL
wattsorrcchlas//ppppep fast
QwenTeem. Qwen2.5: A party of foundation models, September 2024.
URL
https
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling, 2019.
wwtp:/awwww...aa.aa.aoooooogooot o www.rrg/aaagr..oeeeeeeea 1911. 19500011110
Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele, Loki: Low-rank keys for efficient sparse attention, 2024. URL /ttp/tttpee//aarrr..ooo////////a 02542
Qidong Su, Christina Giannoula, and Gennady Pekhimenko. The synergy of speculative decoding and 18813
batching in serving large language models, 2023.
Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding, 2024a. URL
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models, 2024b. URL wtt.tttpsttt /tttttsss///axii/ http:///wxwwww/w//eaeeeee
absgg/b////aa abs 62006 11695
12

---

Published as a conference paper at ICLR 2025
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci,
Song Han. http:///hhhb.ccmmmmmmaaaaa//eee/
and
Quest: Query-aware sparsity for efficient long-context Ilm inference, 2024. URL
https: http://xvavoo/////.....oo// wttp:///////////..11111111
2406.10774
MLC team. MLC-LLM, 2023. URL
Nadav Timor, Jonathan Mamou, Daniel Korat, Moshe Berchansky, Oren Pereg, Moshe Wasserblat, Tomer Galanti, Michal Gordon, and David Harel. Distributed speculative inference of large language models is provably faster, 2024.
abs 2405 14105)
Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative de- coding: Exploiting speculative execution for accelerating seq2seq generation, 2023. URL
/tt/a///ww..rr////////333 16487
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models, 2024a. URL
https://arxiv.or wwt.aaiwww.....o///////aaaoaaaaa
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024b. URL/htcps://arxttet
17453
Dongjie Yang. XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. infer: Pyramid kv cache compression for high-throughput Ilm inference, 2024. URL
Pyramid-
https:/
/ttp///axvvvvvv.e//ee/eeeeeeeeve wwwpw..oow//a////ee/e/p// wwwpg//bs//40....m 12532
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for TYansformer-Bssed generative models. In 16th
USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521-
538, Carlsbad, CA. July 2022. USENIX Association. ISBN 978-1-939133-28-11 URL
https: //t:://wwww.. www.ix.ee.............. org/ http://coo.eeeencc/s///22///////
Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. Llm inference unveiled: Survey and roofline model insights, 2024.
Hailin Zhang, Xiaodong Ji, Yilin Chen, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, Weipeng Chen, and Bin Cui. Pqcache: Product quantization- based kveache for long context Ilm inference, 2024. URL
abss 2407 12820
Zhenyu Zhang. Ying Sheng. Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song. Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. Hạ0: Heavy-hitter oracle for efficient generative inference of large language models, 2023.
URL
https://arxiv.ot org. www.///....... 14048
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for moodput-optimizze large language model serving, 2024.
/ arriv..iiv orq/ abs/ httgg//b//2000/0 2401. 09670
A APPENDIX
A.1
SYSTEM IMPLEMENTATION
Prefill Phase
Decode Phase
Speculation
Full KV
Compressed KV
Overwrite
XV Compression
Speculated Tokens
Verification
Compressed
Figure 8: Self- Speculaacutiin System Design. We demonstrate using static KV compression method.
13

---

Published as a conference paper at ICLR 2025
The design of our speculative decoding system is shown in Fig. 8 demonstrating the use of a static KV compression method. The static compressed KV is generated during prefill phase and used for drafting. We implement the speculative decoding system both state-of-the-ar inference framework MLC-LLM (team 2023) and a self-implement inference backend. The main results are obatined from our self-implemented backend. The comparison of our backend and MI.C-LLM can be found in AA.3
The self-impleeent inference backend is built on GPT-Fast (pytorch- (023). with Flashinfer (laahhnfer-ai accelerating attention computation. We use torch.compile to compile the model and utilize Triton-based matrix multiplication to accelerate the MLP layers. We use Pytorch CUDA graphs to reduce CPU kemel launch overhead. These optimizations help minimize overhead and improve speedup. We also implement tensor parallelism for the embedding layer to further accelerate drafting.
A.2 RESULTS OF VARIOUS BATCH SIZE AND CONTEXT LENGTH ON A 100
We show the raw data points we collected when running speculative decoding on the self- implement backend to support our previous discussion. We sweep the hatch size and sequence lengths, and compare the speedup of different drafting strategy for different models. We ran all these experiments on 8 Nvidia A100 GPU with 8-way Tensor Parallelism.
(a) LLaHD. 2-7B- 32K
TinyL
(h) 1.aMA 2-78332 Self Spec-
(c) LLaNA 3-1-BB Self Specula-
Lama 1.18
alation
bin
TDD
Tv
TAR
TSD
YTD
Tv
TAR
TSD
TTD
Tv
TAB
T-D
1024
32
8.31
9.55
2.19
8.27
8.70
0.95
4000
32
15:42
13.17
2.56
11,89
11,00
1.02
4000
4000
32
64
13.16
16.48
13.55
10.32
2.54
2:51
10.07
883
12.36
9.78
0,90
0.81
1024
64
9.76
13,05
2.19
10.83
10,80
1.00
14.39
4000
128
23
19.77
2.54
13:42
17.70
0.76
1024
128
12.04
18.87
2.19
14.02
14.3
0.94
4000
64
19.75
22.01
2.57
19.28
16.82
1.15
256
35.05
2.19
11.89
4000
128
25.82
33.79
2.56
28.77
23,86
1.21
8000
32
13.28
11:34
2.50
9.90
10:40
0.95
4000
48
8.71
16.19
2.19
14.39
12,00
1.20
8000
64
16:98
16:06
2.51
14.16
13.72
1.08
4000
64
9.35
21.83
2.19
19:28
11.88
1.30
8000
15.70
18:23
2.53
16.53
13.99
1.18
8000
2.51
18.53
1997
26.77
8000
48
18:44
24.30
2.53
21,45
17.50
1.23
256
46.44
34.99
8000
8.61
18,40
2.18
16.53
13,02
1.27
31,49
22,05
16000
32
14.46
14.00
2.47
11.93
12.10
0.99
8.91
23.67
2.18
21
1.38
8000
128
10.10
53.69
2.52
49.89
30.05
1.55
16000
64
18,00
21.15
2.48
17.17
16.40
1.05
2.18
31,49
16000
128
25:77
34.82
28.00
25,36
1.10
3000
128
12:54
51.78
2.18
40.89
31.25
1.60
27.54
1.46
32000
32
2.46
17.13
14:05
1.22
16000
8.78
27.79
2.17
26.28
17,46
1.50
16000
48
19.75
39.03
2.50
35.83
24.15
1.48
32000
64
19:08
30.85
2:45
26.99
21.03
1.25
16000
9.33
38.29
2.18
35,83
22.52
1.59
16000
64
20.87
58.15
2,51
55.08
32,16
1.71
32000
128
28.26
$4.98
2.45
47.24
34.94
1.35
24000
16000
64
9.92
37.57
14.14
2.17
53.08
35.70
$1.99
1.62
1.72
24000
37.06
35,70
21.77
1.64
64000
69000
64
18.25
50.19
28
28
2.40
46.09
26.96
24.22
1891
1.58
1.43
32000
44.94
26.55
1,69
32000
32
16.19
46.55
2.50
44.94
25.64
1.75
10000
32
15.10
39.84
2:45
37.70
23.05
1.64
Table 3: Comparison of results for different LLaMA models and configurations (hudget=512 and 222 A100). Here
S and B represent prefill length and batch size, respectively.
A.3 COMPARISON WITH MLC-LLM RESULTS
We compare the results of SnapKV based self-speculation on MLC-LLM and our backend. As the measurement methods are different, we put them in two tables as shown in Table and The verification time of MI. CI..MMMMM includes one step of draft decode time. Our backend is highly optimized for speculative decoding setting, minimizing the drafting and verification overhead, thus, leading to better speedup. However, the trend that speedup increases with batch size is the same, aligning with our theoretical analysis in Section
Table 4: Results of Our Backend
Target
Backend
Task
7
(Tp((1)
Tv(:)
((%o))
TAR
TSD
x
LJama'
PG-19
8:5100
32000
10.96
691
3.42
541
1
Llamak1
1.1
Ous
P9-19
St1110
32000
32
4
16.09
10.39
4.10
9.2
6.75
1.37
L1mm3.1-50
Ouns
PG-19
8d11000
32000
64
5
23.96
17.45
4.59
14.85
9.17
1.22
14

---

Published as a conference paper at ICLR 2025
Table S: Results of MLC-LLM
Target
Backend
Tisk
GPU
Prefll
Bbx
Tb(()
Ty(n)
NumGen
ARTrput
SDTrput
x
PG-19
8AH100
L1mma51-888
MECALLM
S4H100
32000
32
16.3
36645
Lammá 1-8
MLC
PG-19
Sa1100
32000
64
5.27
28.26
2719
39300
1999.2
1,26
A.4 FURTHER SNAPKV AND STREAMINGI.LM RESULTS
We show the raw experiment data. We compare both BreamingLLLMMbure self-speculation and SnapKV-based self-speculation, and also small draft model with StreamingL.LM KV cache.
Table 6 Comparison of SnapKV, BieeamingLLLM, and Tiny Draft (Streaming KV) Speculation. Each with optimal and KV budget
Target
Draft
Tesk
GPU
Prefill
Biz
(T)
Tv(3)
fb(y.0)
TAR
TSD
x
L 1BB
PG-19
8s11100
32000
16
4.43
6.11
2.43
6.18
4.86
1.27
1-88
Sneaming
Liama 1-BB
SaspKV
PG-19
8sH100
32000
16
10.55
684
3.4
6.18
5,27
1.17
Lhmm3 1-88
I lamak 2-1R(5)
PG-19
8xH100
32000
32
4.71
9.70
2.43
9:10
6.22
1.46
Sneaming!
PG-19
8cH1000
33000
11.55
9:10
Linaa 1-88
SuppKV
4.0
T.mma3. 1-88
I Lima 2-1R(5)
PG-19
8xH100
32000
64
5.05
15.86
2.44
1484
8.88
1.67
Liumaš. 1-88
Smeamingi-LMM
PG-19
8cH100
32000
12.82
15.95
308
14.84
9.57
1.55
SuppKV
PG-19
8xH100
32000
22.91
4.5
1-86
2.1R(5)
8xH100
5.79
Lianaã. 18
SmeamingL_LM
PG-19
8cH100
32000
128
18:96
30.34
3.57
26,07
14.06
1.85
Lhmmã31
S
PG-19
8x11100
32000
128
33.33
31,60
5.07
26.07
12.96
2.01
A.5 RESULTS OF QWEN AND MISTRAL MODELS
Table 7: Results of Qwen and Mistral Models. Each with optimal 7 and KV budget
Target
Drrff
Task
GPU
Prefill
Bsz
TT(11
Tv(1)
(2((,0))
TAR
TS
x
Mistral78--003
SnapKV
PG-19
SxH1100
32
8.92
6.12
Misnal-7B-ut.3
PG-19
64
13.64
15.64
14,40
Masral 7B w0.3
SnapkV
PG-19
SxH100
32000
128
27.09
30.65
4.72
25.41
12:31
2.06
Qwmn-255-7B
SnapKV
PG-19
4x11100
30000
32
3
11:40
9.26
3.40
8.20
6.07
1.35
Quan-2.5-B
SnapKV
PGL19
15.77
Qwmn-2.577
SnapkV
PG-19
4xH100
128
12.06
Qwes-25.328
SnspkV
PG-19
SxH100
33000
8
23.67
11.98
3.50
10.42
10.19
102
Qwes-2.5-328
SnupKV
PG-19
SxH100
33000
16
25.27
15.20
3.52
13.36
11.52
1.16
Qwes-25.322
SnapKY
PG-19
Owes-25-328
Qwen-2.555 25.7B
PG-19
SxH100
10.42
Qwen: 2.5 32B
Qwen 2.5 7B
PG-19
Sx11100
32000
16
2
11.61
14.59
2.32
13.36
11.31
1.18
Owes-2.5-32
Qwen-25-7u 25-78
PG-19
SxH100
32000
32
2
16.22
20.87
2.31
19.83
16.27
1.19
Qwes-25-332
Qwan-2.5 (Bt.2/557////mmm/n (B/Sammn)gg)
PG-19
Qwes 2.5 32B
Qwen 2.5 (Deemmm)))
PG-19
Sx11100
32000
16
2
14.59
2.26
13.36
9.64
Qw5m-2.5-33 3-5
Qwan-2.5 7BB Smcaming)
PG-19
Sx11100
32000
32
3
11.78
21.82
2.60
19.43
12.85
1.51
15

---

Published as a conference paper at ICLR 2025
A.6
RESULTS
We also test the non-GQA model 1---AA---7-2--- for both StreamingLLM--asse self-speculation and small draft model with Streamingl.LMM KV cache. Due to the lower FLOPS to memory ratio of non-GQA model, tends achieve higher speedup than GQA model under the same setting.
1.6
Spoodup
Speadup
1.6
1.2
1.2
1.0
52
128
256
32
120
2.0
1.0
18-
1.6
Speedup
Speadup
1.6
32
04
128
Batch Size
Batch Size
Profill 1024
16000
Prefiil 32000
Prwfll
Profil 24000
Prwfil
Prwfill 24000
Protil Houp
(a) Draft: TinyLLama-1 1.18 1B 78-32K
Target: LLaMR-2-
(b) Draft: SteamiigLLMM based KV. Target: LLaMA-2 -7B-32K
Figure 9: End-to-end speedups for Streamingl M--aseed selfuspeoulttion across various compressed KV budgets (left: 256, right: 512) on PG-19. Annotations indicate "oppinot, which is the Experiments conducted 8xA100 with 8-way tensor parallelism. Raw data can be found in A.2
value corresponding to the highest speedup achieved.
16


Reviews:
[{"review_id": "4avedGW11A", "reviewer": "Reviewer_XfUb", "summary": "This paper introduces a speculative decoding technique designed to enhance throughput and reduce latency in long-context Large Language Models (LLMs), addressing a common bottleneck in high-batch, memory-bound inference tasks. By optimizing the Key-Value (KV) cache through sparse KV configurations and employing self-speculation, the method improves memory efficiency, making speculative decoding effective for large batch sizes and moderate-to-long sequence lengths. Empirical results on high-performance GPUs, such as A100s and H100s, show up to a 2.51x speedup compared to traditional autoregressive decoding, especially in scenarios with long sequences benchmarking. While promising, the approach lacks comparisons of some baselines, evaluating a limited spectrum of model sets, lack of case study or worse case analysis, and lacks a discussion of the scope/limitation section. If the authors could address the concerns in revision, I would be willing to raise the score.", "questions": "See weakness part.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The writing is pretty good and easy to follow.\n\n2. Sufficient experiments conducted on high-performance GPUs (e.g., A100, H100) show up to a 2x speedup compared to autoregressive decoding, demonstrating speculative decoding’s efficiency for long sequences.\n\n3. The authors provide a detailed mathematical analysis showing how speculative decoding can be effective even for large batch sizes in memory-bound regimes, particularly by addressing the KV cache bottleneck.", "weaknesses": "1. Missing discussion and comparisons of reasonable baselines. Although the authors briefly discuss TriForce, which demonstrates the effectiveness of self-speculation with compressed KV, they did not compare it with TriForce in experimentation for some reasons. Besides, there exists some self-speculation work aiming to accelerate LLM inference, e.g.,  Xia et al. [1] and Zhang et al. [2]. Could such existing works be applicable to / integrated with the proposed solution? Please either provide satisfactory reasons why forgiving such baselines or add comparisons in experiments and clear this work's novelty. While the paper references approaches like vLLM, it lacks a side-by-side comparison, especially regarding memory efficiency versus other batch-processing strategies. Including metrics from existing solutions in similar conditions would solidify the paper’s contributions.\n\n[1] SWIFT: ON-THE-FLY SELF-SPECULATIVE DECODING FOR LLM INFERENCE ACCELERATION\n\n[2] Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding\n\n2. Evaluation of a limited spectrum of models. The experimental settings focus on LLaMA series models, yet the performance of proposed methods on other models remains unknown.\n\n3. Lack of Case study or worse case analysis. The paper describes handling variability in token acceptance rates during speculation, but it could provide more details on failure cases or worst-case scenarios where rejection rates could impair throughput significantly. This is especially important for practical deployments in heterogeneous batches where sequence lengths vary widely.\n\n3. Lack of discussion and limitation. Providing a brief discussion of limitations would help clear the scope of this work and make broader impacts. For instance, this work focuses on high-end modern GPUs, so the proposed solution may not perform that well on desktop/low-end GPUs, or even worse at resource-constrained embedded systems. Including tests on more widely accessible hardware (e.g., T4 or V100 GPUs) would demonstrate the method’s practical viability across a broader range of settings. The authors could elaborate more directions in the revisions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ShXeAeYXjv", "reviewer": "Reviewer_PRVw", "summary": "The paper discusses the batched speculative decoding in the long context scenario. It finds a critical sequence length that is the threshold for batched speculative above which it can show speedups compared to the original autoregressive decoding. The paper further examines the compressed kv cache and tries to find the best compressed KV-based drafting with the best strategy.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Interesting insights on the significant sequence length and how the nature of memory bound translates to speedups in batched speculative decoding setting.\n\n- Insights that changing the KV drafting instead of the smaller draft models can potentially have better acceptance rate and thus better speedups.", "weaknesses": "- Perhaps on the incremental side since the main new ideas are not all that large. The idea of compression KV is already in the literature.\n\n- Perhaps the paper jump to some conclusions too fast without enough explanations, which makes it a little hard to follow.\n  \n  - What does Figure 1(b) try to show?\n\n  - Maybe consider adding more details about the effects of increasing batch size with the original speculative decoding.\n\n\n- Some terms are vague in the texts. Authors may consider more clearly define them.\n\n   - What does Figure 1(b) try to show?\n\n  - What does ``self-speculation\" in the texts mean?\n\n  -  What does KV budget mean?\n\n- Some important design details can be more explicitly clarified.\n\n  - Where does the time breakdown such as Figure 1(a) come from? GPU kernels can hide latency among threads so the end-to-end time does not necessarily equals to the combination of the memory loading and computing time. Nevertheless, the intuition of memory bound of KV cache when batch size is large makes sense.\n\n  - Is KV compression used for both the draft and target model?\n\n  - What is the baseline for results in Figure 4?\n\n  - As for this lossy KV compression selection method, perhaps consider adding the evaluation that shows the selection in KV compression strategy has advantage (e.g., speedups) over the original speculative decoding.\n\n  - How the framework guide the selection of KV compression method remains unclear to me.\n\n- The applications for the very long contexts prompts in large batch size still need further justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fp4mILcq1Q", "reviewer": "Reviewer_UdZh", "summary": "The paper addresses the challenge of improving latency and throughput in LLM inference for long-context tasks. Traditional speculative decoding (SD) literature tends to focus on smaller batch sizes, but MagicDec demonstrates how SD can also benefit high-throughput scenarios involving long sequences. By combining theoretical and empirical approaches, MagicDec identifies and mitigates bottlenecks in memory access using a draft model with sparse Key-Value (KV) caching. It introduces a theoretical model for selecting optimal draft strategies, achieving speedups of up to 2.51x in large batch settings for the LLaMA-3.1-8B model across diverse hardware configurations.", "questions": "## Questions\n1. Can you clarify the potential trade-offs in performance if MagicDec were applied to significantly smaller LLM models?\n2. Is the performance gain from MagicDec sustained across varying types of long-context tasks, particularly those that require variable batch sizes or non-standard hardware configurations?\n3. Have you considered additional draft model selection criteria that might further alleviate the KV bottleneck?\n\n## Comments\n### Comment on Lines 52-74 and Figure 1a\nIn the section describing how the \"KV Cache Bottleneck Enables SD Speedup Even For Large Batches,\" the authors assert that KV cache loading time increases significantly in long-context, large-batch scenarios, leading to a more memory-bound process. This serves as evidence for the memory bottleneck supporting speculative decoding at scale. However, Figure 1a appears to represent the combined \"KV load and store\" time rather than isolating the KV load time alone. Could you clarify what portion of this time is allocated to \"KV store\"? Understanding the distinction between load and store times would help validate the impact of the bottleneck more precisely.\n\n### Comment on Lines 203-211 (\"Expected Generation Length Per Step\")\nThe preprint by Timor, Nadav, et al., titled *\"Distributed Speculative Inference of Large Language Models\"* (arXiv preprint arXiv:2405.14105, 2024), appears to be highly relevant to your study, particularly in examining the regime where SD leads to either speedups or slowdowns based on the drafting latency budget. Consider discussing this work to strengthen the related literature section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "+ The work introduces novel improvements in speculative decoding for large-batch, long-sequence LLMs, challenging existing assumptions and offering a fresh perspective.\n+ Theoretical insights are well-validated through high-quality experimental results, with robust data supporting the effectiveness of MagicDec across diverse scenarios.\n+ While some sections could benefit from increased clarity, the paper's main findings are well-articulated, with sufficient depth to support the claims.\n+ MagicDec offers notable improvements in both latency and throughput for LLMs, with potential impacts across a range of long-context applications.", "weaknesses": "- Complexity in Explanation: Sections discussing KV caching and speculative decoding speedup factors could be streamlined to improve readability, especially for a broader audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6UvnKmeUny", "reviewer": "Reviewer_3wSP", "summary": "Conventional understanding suggests that **speculative decoding (SD)** enhances performance primarily in scenarios with small batch sizes. This paper presents a novel theoretical analysis demonstrating that SD can also yield performance improvements in settings with large batch sizes and extended prompt lengths. The analysis in this paper identifies how performance bottlenecks shift with increasing batch size and prompt length. To address these bottlenecks, this paper proposes using a draft model with a compressed key-value (KV) cache, effectively alleviating the new constraints. The theoretical framework provided by this paper enables an optimal drafting strategy tailored to specific draft-target model pairs, making this approach particularly valuable for SD applications in long-context scenarios.", "questions": "Please answer the following question.\n\n1. **Potential limitations in real-world application**: The theoretical analysis is thorough and effectively clarifies when the conventional view—that speculative decoding (SD) enhances performance primarily with small batch sizes—applies, as well as when the new discovery on SD’s performance benefits with larger batch sizes and extended prompt lengths holds true. However, in real-world cloud environments, request configurations are often limited to smaller batch sizes and shorter prompt lengths (see https://www.microsoft.com/en-us/research/publication/splitwise-efficient-generative-llm-inference-using-phase-splitting/). Could the authors kindly discuss how this theoretical analysis might be applied within the constraints of typical LLM request configurations?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper is well-written.\n2. The paper presents a clear and logical progression of ideas.\n3. The paper provides rigorous theoretical analysis supported by extensive experimental results.", "weaknesses": "I appreciate the quality of this paper, and I have only one minor suggestion.\n\n1. **Potential limitations in real-world application**: The theoretical analysis is thorough and effectively clarifies when the conventional view—that speculative decoding (SD) enhances performance primarily with small batch sizes—applies, as well as when the new discovery on SD’s performance benefits with larger batch sizes and extended prompt lengths holds true. However, in real-world cloud environments, request configurations are often limited to smaller batch sizes and shorter prompt lengths (see https://www.microsoft.com/en-us/research/publication/splitwise-efficient-generative-llm-inference-using-phase-splitting/). Could the authors kindly discuss how this theoretical analysis might be applied within the constraints of typical LLM request configurations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)