Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation
Authors: Jingyu Liu
Beidi Chen
Ce Zhang
Venue: ICML 2025 poster (2025)
Keywords: Inference Acceleration
TTFT
Large Language Model
Efficient Inference
Primary area: deep_learning->large_language_models

Abstract: Improving time-to-first-token (TTFT) is an essentially important objective in modern large language model (LLM) inference engines. Optimizing TTFT directly results in higher maximal QPS and meets the requirements of many critical applications. However, boosting TTFT is notoriously challenging since it is compute-bounded and the performance bottleneck shifts from the self-attention that many prior works focus on to the MLP part. In this work, we present SpecPrefill, a training free framework that accelerates the inference TTFT for both long and medium context queries based on the following insight: LLMs are generalized enough to preserve the quality given only a carefully chosen subset of prompt tokens. At its core, SpecPrefill leverages a lightweight model to speculate locally important tokens based on the context. These tokens, along with the necessary positional information, are then sent to the main model for processing. We evaluate SpecPrefill with a diverse set of tasks, followed by a comprehensive benchmarking of performance improvement both in a real end-to-end setting and ablation studies. SpecPrefill manages to serve Llama-3.1-405B-Instruct-FP8 with up to 7$\times$ maximal end-to-end QPS on real downstream tasks and 7.66$\times$ TTFT improvement.

Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation
Jingyu Liu
Beidi Chen
Ce Zhang'
Abstract
understand and generate human-like languages (Bubeck et al., 2023; Wei et al., 2022: Feng et al., 2024). Many SOTA models have been developed, such as GPT-4 (OpenAl ct al., 2024), the Llama family (Grattafiori et al., 2024), DeepSeek RI (DeepSeek-Al et al., 2025), Mistral (Jiang et al., 2023a), Gemini (Team al., 2024), and Qwen2 (Yang et al., 2024), meet the increasing expectations of users. In order to broaden their real-world applications, one essential requirement is to build an efficient serving engine that can satisfy various requirements (Miao et al., 2023; Kwon et al., 2023; Zheng et al., 2024; Shoeybi et al., 2020).
Improving time-to-first-token (TTFT) is an essen- tially important objective in modern large lan- guage model (LLM) inference engines. Opti- mizing TTFT directly results in higher maxi- mal QPS and meets the requirements of many critical applications. However, boosting TTFT is notoriously challenging since it is compute- bounded and the performance bottleneck shifts from the self-attention to the MLP part. We present SPECPREFILL' training free framework that accelerates the inference TTFT for both long and medium context queries based on the fol- lowing insight: LLMs are generalized enough to preserve the quality given only a carefully chosen subset of prompt tokens. At its core, SPECPREAILL leverages lightweight model to speculate locally important tokens based on the context. These tokens, along with the necessary positional information, are then sent to the main model for processing. We evaluate SPECPRE- FILL with diverse set of tasks, followed by a comprehensive benchmarking of performance im- provement both in real end-to-end setting and ablation studies. SPECPREFILL manages to serve Llama- 3. 3-1 050-058-88 instrrctt FP8 with up to 7x maximal end-to-end QPS on real down- stream tasks and 7.66 TTFT improvement.
There are several fundamental reasons why TTFT stands so pivotal: 1) many applications require a fast response time that directly influences how users perceive the responsive- ness of the system and 2) more importantly, TTFT deter- mines the scaling of maximal QPS an inference engine can support as shown in Figure 1. However, optimizing TTFT is an arduous task mostly because the prefill stage is largely compute-boundn bounded and the computational bottleneck can change depending on the prompt length and batch size. For example, many works focus on improving the self-attention speed (Dao et al., 2022: Jiang et al., 2024a), but in reality, there is still a huge traffic of large-batch short to medium context queries where it is the MLP part that clogs the whole system. Despite achieving impressive results, prior works that target the prefill phase either require a post-training adaptation (Qiao et al., 2024; Horton et al., 2024) or scale less efficiently (Shi al., 2024).
Inspired by those work, we found a key insight that LLMs can retain most of its performance when given only care- fully chosen subset of tokens from the prompt, and the model is able to adapt to that a zero-shot manner. SPECPREFILL optimizes the TTFT by leveraging secondary lightweight model speculate locally important tokens. Only these tokens are sent later to the base model. It reduces the total FLOPS by factor proportional to the percentage of token drop. SPECPREFILL does not require any fine-tuning and is ready to be deployed and scaled to larger models. We summarize our key contributions:
1. Introduction
Large Language Models (LLMs) represent a transformative innovation in artificial intelligence, enabling machines to
'epaarment of Computer Science, The University of Chicago, Chicago, IL. USA 'Department of Electrical and Computer Engi- neering, Carnegie Mellon University, Pittsburgh, PA, USA. Corre- spondence to: Jingyu Liu <jingyu6@ uch, Ce Zhang <cez@ ceoo@uchicago.cce edu.>.
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
hy the author(s).
'The code with exprriment is available https: thub. htt.b..cmm.eoooooooo////eeeeeeee
We present a conceptually simple, effective, and no- val framework called SPECPREFILL that significantly

---

Speculative Prefill
Multi-Doc QA
Few-Shot Learning
Summarization
Keep 10%
Keep 30%
Keep 50% Timeout 40s
Timeout 10s
Timeout 20s
Average Task Score
Average Task Score
Average Task Score
55.63
53.11
54,96
53.84
66.95
66.99
65:72
25:53
25:19
Average Query Latency
10
20
40
35
8
5x
2.1x
30
6
25
2
5
15
2.0
QPS
QPS
QPS
70B
SpecPrefill 70B
4058
SpecPrefill 4058
Figure 1. Speculatiiee Prefill QPS Improvement: In an end-to-end server-client setting with real world datasets, we benchmark the average query latency under given fixed timeout when sending queries constant QPS. SPEcPREFILL significantly improves the maximum QPS supported by the vLLM server as well as the latency compared to not using it. When we reach low keep rate, we can even serve the 405B model with SPecPrefilL to run more efficiently than the 70B model. As the base model size increases and keep rate drops, we can get 7x end-to-end QPS boost while only occurring < 5% accuracy.
improve the prefill phase, hence the maximal QPS, of LLM inference without any fine-tuning or adaptation.
the KV necessary for producing the output based on the query and the decoding phase where the model predicts new token auto-regressively.
We conducted comprehensive evaluations both on real and synthetic datasets to demonstrate its effectiveness and limitations, giving full picture of the expected benefits when deployed to productions.
2.2. Decoding Acceleration
The decoding phase is mostly memory- bounded, and there- fore reducing the amount of data to move around will ef- fectively help improve the latency. As result, explicitly manipulating the KV cache has been extremely success- ful with many strategies: H2O (Zhang al., 2023) and StreamingL (Xiao et al., 2024b) idenfitied key insights to KV dynamics which is used to evict less essential KV caches during decoding. CacheGen (Liu et al., 2024). Q- Hitter (Zhang et al., 2024b), and ShadowKY (Sun et al., 2024) apply efficient techniques to compress/quantize store, and transmit KV caches to reduce memory overhead. Spec- ulative decoding relies on the insight that hides the memory latency by concurrently verifying several speculated tokens from either draft model itself (Leviathan et al., 2023; Zhang et al., 2024a; Xia et al., 2024).
We implemented our method on industry standard serving engines, and benchmark its performance in both an end-to-end fashion and ablation experi- ments. The end result is system that serve with up to 7x maximal QPS under the same system specification and 7.66x reduced TTFT while maintaining decent accuracy.
I.1 ama-3. TTS
Our method can be easily combined with techniques from guantization, KV eviction, and speculative decod- ing, making it an ideal add-on to existing engines.
2. Background
Despite being crucially important, decoding speed is not the only factor that influences the overall inference pipeline and we will review why sometimes the prefill time optimization is even more essential in many cases.
2.1. Inference Bottlenecks
Improving LLM inference efficiency has been extensively studied in prior work (Miao et al., 2023; Yuan et al., 2024). We review works that focus on different aspects when deal- ing with real serving systems where the bottlenecks are quite different under various serving requirements (e.g. long con- text domains, latency sensitive applications, etc) (Kwon et al., 2023; Zheng et al., 2024).
2.3. Prefill Acceleration
The time-to-first-okkn (TTFT) is crucially important both from user experience but also the system serving per- spective (Sec 4.7). In many critical applications, the input token length can often eclipse that of the generation tokens (e.g. 10:1 ratio) in real traffic (Qiao et al., 2024). Unlike
LLM inference can be roughly divided into two major pro- cedures, namely the prefill phase where the model computes
2

---

Speculative Prefill
the decoding phase, however, the prefill phase is usually compute- hounded and the cost for the MLP calculation and the communication of tensor parallelism quickly becomes bottleneck.
SPEcPREAIL to speculative decoding, forming full small- model-assisted inference paradigm.
3.1. Overall Architecture
Many prior works have explored ways to make self- attention faster: Flash-attention series (Dao et al., 2022: Dao, 2023) compute the exact attention using carefully de- signed hardware-aware algorithms. Special (both static and dynamic) attention masks are designed for sparse calcula- tion such as LongFormer (Beltagy et al., 2020), MInfer- ence (Jiang et al., 2024a), FlexPrefill (Lai et al., 2025), Hip Attention (Lee et al., 2025), Sample Attention (Zhu et al., 2024), and Duo Attention (Xiao et al., 2024a). How- ever, none of these directly make the MI.P part faster like SPECPREFILL, SPECPREFILL achieves consistent efficiency improvements in various regimes because it skips parts of the attention MLP calculation and the all-reduce overhead, which proves to be effective especially when the ratio of is large (Xiong et al., 2023).
SpecPreALl follows a conceptually simple architecture where a usually less expensive model is chosen the spec- ulator model that predicts contextually important tokens given prompt. The speculated tokens, alone with the orig- inal position information, are then fed to the main model for processing. In the following section, we will discuss two central design choices in more details, namely the token estimation algorithm and the selection strategy. Note that SPECPREAILI can be seamlessly integrated with specula- tive decoding in which the small model can work both in the prefill stage for token selection and the decoding stage for drafting proposals, making our approach almost free to integrate and deploy.
batch size
soquence lengtti
3.2. Token Importance Speculation
Orthogonal to techniques such as prompt compres- sion/rewrite (Jiang et al., 2023b; 2024b; Li al., 2023), layer dropping (Elhoushi et al., 2024), and weight quan- explore selecting
The goal here is to select which tokens are contextually important for given query and send those along with nec- essary positional information for the main model. The pro- cedure starts with calculating the attention scores from the speculator, which uses the last token's 's attention score w.r.t. the context as the surrogate for measuring token importance:
tization methods (Lin et al., 2024), important prompt tokens to skip the full forward computa- tion. GemFilter (Shi et al., 2024) uses an extra pass to get model's own middle layer attention information that decides on what tokens to keep for the real forward. Contrast to this, we apply a separate and cheaper model to speculate locally important tokens via token transferability. which can scale more efficiently than GemFilter. Concurrent to ours, SwiftKV (Qiao et al., 2024) learns to skip later lay- ers by reusing the past layers' KV, which achieves up to 50% TTFT reduction (SPECPREFIL) can reach up to 87% TTFT reduction). Unlike our zero-shot requirement, they require extra leihht fine-tuning due to modified model behavior. It worth noting that our method approaches the problem in a different way, which makes them compli- mentary to each other. Finally, akin to our motivation, KV Prediction (Horton et al., 2024) proposes to adapt a cheaper system (i.c. learned auxiliary network and a KV predictor) to predict the KV cache of the base model, thus bypassing the original KV computation. We show that SPECPREFILL can accomplish better TTFT reduction than theirs, without introducing extra overhead when coupled with speculative decoding (Leviathan et al., 2023) while maintaining com- petitive quality.
a;j == Softmax Softmax(Qa+ VO S
where M the context length, N is the number of look- ahead steps, and asj is the attention score for the ith token in the prompt w.r.t. the jth decoded token, assuming we're looking at particular layer.
We build on top of this by aggregating the scores over the whole speculator model (Section 3.2.2) with potential look- ahead (Section 3.2.1) and select tokens based on chunks (Section 3.2.3). The subset of chosen tokens with their original positional information (Section 3.2.4) will then be used for the main model's inference.
3.2. MITIGATE POSITION BIAS VIA LOOK-ANEAD
Prior works have shown that there are many biases for atten- tion scores, such as the sink phenomenon (Xiao et al., 2024b) (the first couple of tokens tend to have higher weights) and the proximity bias (tokens closer to the output tend have higher weights (Lv et al., 2024)). To mitigate these issues, instead of relying on the attention score of the last token alone, we further decode the speculator by N steps and ob- tain the attention information from the new N tokens (Wan et al., 2024). N here serves as a trade-off between bias and budget, which can substantially increase the performance for shorter context queries.
3. Speculative Prefill
In this section, we present SPECPREFILL by first describing its high-level algorithm, followed by several design choices that mitigate various biases, and detailed implementa- tion account. Finally, we touch a bit on how to integrate
3

---

Speculative Prefill
3.2.2. AGGREGATED ATTENTION SCORE AS TOKEN IMPORTANCE
few line of code along with a configuration file to enable SPEcPREFILL. The KV cache is not necessary if we do not need to look-ahead for our speculator, which can save lots of memory allocation. However, we do need to explicitly store the queries of the decoded tokens (including the last token of the input query) which we later retrieve to compute the attention score. Note that a specific mapping (e.g. slot mapping in vLLM) might be kept track of retrieve the right data. For batched look-ahead, we only consider tokens that are valid by checking afterwards whether they are equal to the EOS tokens. Finally, we want to mention that despite being being a sequential implementation, we can actually split the process of speculation into a separate procedure and decouple from the inference of the main model by adding a new layer of scheduling, which we leave as a future work.
Given the full attention scores of the speculator, we decide to use max-meen aggregation strategy to map to scalar token importance. Formally, given an attention score tensor of shape [N, L. S, H] where N is the number of look-ahead tokens, L is the number of layers, is the sequence length, and is the number of heads, we take the maximum over H and L dimension to make salient tokens stand out, and average over N account for fair token contribution.
3.2.3. DENOISE ATTENTION SCORES BY CHUNK SELECTION AND POOLING
It has also been observed in concurrent works (Lv et al., 2024) that tokens that are positioned nearby share similar- ity in importance. We take this insights to select tokens by chunks in order to reduce the variance of our token im- portance estimation. Specifically, we chunk the context contiguously and average the token score within each block, and then we select the Top-K blocks. In order to eliminate the artifacts of chunkation, we apply a ID average pooling before this to smooth the cross block scores.
3.4. Relation to Speculative Decoding
Speculative decoding has been proven to be extremely suc- cessful at accelerating the decoding TPS. SPECPReFIlL can be seamlessly combined with speculative decoding by sharing the same draft model. Since speculative decoding itself requires a full forward pass of the context, SPECPRE- FILL will provide the necessary KV information required for subsequent decoding speculation, hence amortizing the overhead. This will open-up a huge space of possibilities, and lead to the first paradigm of an inference system that fully aided by smaller speculators.
3.2.4. RESTORATION OF POSITION IDs
Finally, when we select the subset of tokens based on our compute budget and query compressibility, we also need to restore the position information which are also sent to the main model. Basically, instead of using contiguous posi- tion ids as before, we send a potentially non-continuously increasing position ids which are obtained from tokens' po- sitions in the original context. In addition to that, we also need to explicitly set the decoding token position to the context length in case we dropped tokens before the first de- coding token. An example is shown below with ten prompt tokens and three decoding tokens (bold):
Algorithm 1 Speculative Prefill
Require: Base model M, speculator S, look-ahead steps
N, batch of mixed requests B, base model QKV cache
Co, speculator KV cache C.
1: Bp, Ba +- pwwpeeiillleedeeeeeee
(B)
2: {Section 3.2.1}
3: for i = to N do
4: B t model.forward (S, Bp, Cs, store_q=True)
Bp s update-reen requests (Bp. B²))
Original Pos Ids: [,, 3, 4, 6, 8.9]
6: Bp x check_for_eo (Bp)
Speculated Pos Ids: 1,,1,3,6,7 6,7]
7: end for
Decoding Pos Ids: [0, 1,3, 6, 7, 10, 11, 12, ...]
8: if is Jensor ceraale-ar O then
9: pa-gather-qk (C.)
where the bold indices are the decoding positions which are offset based on the original position information. We found this design choice to be crucially essential, especially
10: end if
11: Q, K + retrieve qk (Bp, C2)
12: A + ccompute teaiietetttettoncctt score (Q, K)
for position--eenitivee tasks such as synthetic tasks involving retrieval and counting.
13: {Section 3.2.2]
14: A <- agggregate attention score (A)
15: {Section 3.2.3}
3.3. Implementation Details
16: T + chunk select from smoothed attention (A)
17: (Section 3.2.4]
We describe both the high-level procedure and the imple-
ids 18: P + restore-por -arteeppss_ee ids (T, Bp)
mentation details of SPECPREFILL in this section. In Algo-
19: B t- merge requests (T, P, Bp. Ba)
rithm 1, we list the high-level steps of conducting SREC
20: Return forward (M, B, Ct)
FILL. Our implementation is based on creating a monkey patch on top of vLL.M (K al., 2023) which only needs
4

---

Speculative Prefill
4. Experiments
We hypothesize the reason behind the improvement might be that SPECPREFILLL helps remove noisy and distracting tokens in the prompt, hence projecting the prompt to the space where the main model performs better.
In this section, we start with our experiment setup for repro- ducibility, followed by categorizing prompt compressibility of different queries. We evaluate SPECPREFILL on down- stream long context, synthetic context probing, and standard short tasks. Finally, we conclude with a comprehensive effi ciency measurement of our system under the real end-to-end setting.
We will see examples in each categories the following evaluations. It is worth noting that we used a fixed keep percentage for our evaluation and it be tremendously helpful to automatically decide on the percentage based on the query, pushing the limit of SPECPREFILL, which we leave as future work.
4.1. Setup
We implement SPECPREFILL in vLLM that supports ten- sor parallelism with the same degree as the main model?. Due to its token dropping nature, we focus on evaluat- ing generative tasks this section and include a com- prehensive range of benchmarks to fully present its ap- plicability and potential pitfalls. We all of periments using tensor parallelism of 8 for both the speculator and the base model across either 8 NVIDIA H100s or H200s (full system specification in Appendix D and guidance on reproducing results in Appendix C). We choose 2024) with BF16 precision as our speculator for bal- ance of efficiency and attention transferability and couple it with either Llama-3, 1-1.-3...111--11 Instrrnnttttttttt (BF16) or Llama-3. 1-405B-Innttt wwtee-nnnttutttttee ruct-FF8* (fully quantized FP8) as the base model. In terms of token keep rate, we use a fixed percentage (i.e. the ratio of chunks when we do chunk selection) for a given task. In practice, we might devise more adaptive strategy for how many tokens to keep based on the query compressibility discussed next, or dele- gate the decision to users based on their needs. We leave all these possibilities for prospective applications.
4.3. Baselines and SPECPREFILL Variants
We aim to showcase both the quality and efficiency of SPECPREFILL under a comprehensive set of applications. To do so, we compare three variants of SPECPREFILL against baselines:
Baselines: We compare SPECPREFILL against four different baselines: 1) Base Llama instruct model. 2) Sentence RAG: SPECPREFILL can be framed as a special case of augmented (RAG) LLM with the granularity of tokens blocks and the rel- evance metric controlled by the speculator's internal knowledge (Li et al., 2024; Gao et al., 2024b; Lewis et al., 2021). Therefore, we implemented two sim- ple sentence-level RAG baselines and report the better one as RAG-LLAMA. 3) LLMLingua (Jiang et al., 2023b): SPECPREFILL can also be seen as context compression technique, and hence we test SPECPRE- FILL against text-level compression method. 4) MIn- ference (Jiang et al., 2024a): To understand the ben- efits of skipping the MLP part, we include sparse attention optimization approach for completeness.
1---...--BB-NNSTRUCC (Grattafiori al.,
4.2. Query Context Compressibility
We empirically found three types of queries during our evaluations based on the quality difference before and after applying SPECPREFILL:
SPECPREFILL: SPECPREFILL with raw attention scores and ignoring the techniques we discussed in Section 3.2.1 and 3.2.3.
1. Information-dense queries: These queries usually are short and information dense, which naturally makes token dropping less effective because there no redun- dancy the prompt.
SPecPrEFILL Full: SPECPREFILL with all techniques but no look-ahead.
SPeCPREFILL Full LAI:: SPECPREFILL with all tech- niques with 8-step look-ahead
2. Compressible queries: These queries are those that not get degradation after removing significant amount of tokens, often seen in long context tasks.
4.4. Real Long Context Tasks: LongBench
3. Noisy queries: These queries, perhaps surprisingly, get better results after dropping some "noisy" tokens.
We start with long context tasks using LongBench (Bai et al., 2024), which consists of six different categories focusing various aspects of long context modeling abilities.
WW expose the API so that it only takes a few line of code to apply SPECPRAFILL before initializing vLLM engines. 'https: / htt.//ugggngggg Meta- L.ama- 3. 405B- Instruct- FP8.
corneurallnn meuralmagiaa magic//
"We empirically found that going beyond 16 look-ahead gives minimal performance gain.
5

---

Speculative Prefill
Soggle-Dcuumntteeo QA TTFT Speeduup 1825
Multi-Docunm DiliDDDuummetttmmm QA
Summarization TIFT Speedup
TIFT Speedup 182%
069%
275%
133%
668%
275%6
133%
669%
275
182%
133%
65
65.65
70
GovReport
35 26
65
VCSUMcah
SpecPredii Ful 6558
55
30
SpecPrefil Full LAH 4058
50.17
51.81
26.5
50
25
45
Garrative
45
HotpobOA
Gusper
ZARIMAA
MultF-al9QA-on
Musque
DuRledeedh
36.03
18.0
35
Spechrent Full 405B
35
SpecPreftl Full 4058
SpecPrefill Full LAK 6058
SpecPreffl Full LAK 4058
15
10%
30%
50%
70%
90%
10%
30%
50%
10%
30%
50%
90%
Few-Shot Learning
Synthetic Tasks
Code Completion
6.9%
275%
182%
660%
275%
182%
133%
669%
275%
102%
133
100
76
100.0
90
74
80
72
Task Task Score
70
SAMSum
SpecPrent Full 405m
LSitT-tt
SpecPrent Full 4058
THiaGA
Thick Score
40
60
SpecPritll Full 425n
Passagehete seaa--hh
PassageCount
SpecProfil Full LAH 4058
www.teeeeeeeeeeeoooooe woneeen
Tosk Score
68
70
66
51.0
20
LCC
64
RephBenchh P
SpecPrenti Pull 4050
SpecPrefti 4058
62
10%
30%
30%
Keep Token Rate
Keep Token Rote
Keep Token Rate
Figure 2. LongBench Main Result on Llama 405B: In this figure, we showease the effectiveness of SPECPREFILL on Long- Bench, which consists of six categories of long context downstream tasks. In each plot, the dash lines are the results of baseline Llama-3. 1.ama-3.114053 10.-1--05--20 Instrrtttt FPP for each subtask and we benchmark SPECPREFILL with increasing token keep rates. We observe different behaviors such quality preservation, degradation, and improvement based the task type.
In Figure 2, we report the main results on LongBench for Llama-3. 1-1-4-0-Innstrutt----
between applying SPECPREFILL or not becomes smaller, which indicates that bigger models adapt better with our speculated subset of tokens.
whose length information is visualized in Appendix Figure 10. To compliment it, we also include the performance of in Appendix Figure 8. We vary the token keep percentage starting from 10% to 90% and draw the baseline model quality using the dash lines for each subtask. To ablate the effect of our design choices, we compare SPECPREFILL and SPECPREFILL Full LAHI with the 70B models in Appendix Figure and SPECPREFILL Full and SPECPReFILL Full LAH with the 405B model in Figure 2.
To ablate the effectiveness of techniques discussed in Sec- tion 3.2, we compare them separately in Figure 2 and 8 to avoid crowdedness. In both cases, we can see consis- tent improvement and the benefits of look-ahead are more consistent in shorter context tasks (more details in Sec 4.6).
Llama-3, 1- -1-70B-Instrun
Finally, we demonstrate the superiority of SPECPREFILL over three different baselines in terms of preserving the qual- ity of the inference for the 70B model. In Table 1, group tasks in LongBench into categories and list the results with varying degrees of compression rates. For RAG-LLAMA, we use the question to retrieve the relative information from the context (more detailed descriptions are given Ap- pendix B). For L.M.Lingua, we follow their official exam- ples and only compress the context, leaving the question and template intact. With the prior knowledge of separated context and question, these two methods are eclipsed by
As we can observe, for categories such as Single-Document QA, Multi-Document QA, Few-Shot Learning, SPECPRE- FILL can preserve most of the quality up to keeping only 10% tokens. For Summarization, we expect to see some degradation in performance as we drop more. Perhaps sur- prisingly, for the smaller 70B model, quality after we remove some tokens tasks like Code Completion. As the model size increases, the quality gap
achieve better
6

---

Speculative Prefill
Table LongBench 70B Model Comparison: We compare different methods based --a-ma---B---0 with varying compression rates on LongBench (grouped by task types). denotes models that not only require context-question separation but also have the true compression rates distinctive from the predefined ones, Among all tested methods, SPECPREAILL achieves superior average performance (underlined scores are the best under the comparable rate).
Model
Compression Rate
Single-Doc QA
Multi-Doe QA
Sum
Few-shot Learning
Code
Synthetic
Avg
Baseline
N/A
50.57
53.11
25.84
66.93
52.33
72.50
53.55
10.38%
32.32
41.17
18.86
45,40
44.76
30.42
35.49
27.68%
38.43
47.41
21.42
50.53
45.80
35.50
39.85
RAG
64%
40.53
46.64
22.45
49.52
46.00
43.15
41.38
63.42%
41.40
47.43
23,30
52.21
46.19
47.22
42.96
82.22%
43.25
48.16
23.56
45.92
53.04
~00%
26.50
32.94
20,95
37.40
45,00
16.33
29.85
38.83
44.02
23.37
42.23
47.27
37.00
38.79
LLLMLingua*
~500
43.64
50.67
24.77
50.96
49.05
60.33
46.57
70%
45.90
51.48
68.50
50.66
909%
45.94
53.91
25.87
60.46
$4.06
72.00
52.04
MInference
N/A (Section 4.7.2)
50.46
53.23
25.83
66.36
52.48
69.00
52.89
10%
47.64
52.96
21.74
64,52
63.33
66.25
52.74
30%
49.47
53.39
24.41
65.83
62.62
67.83
53.92
SpecPreaiL
50%:
59.91
68.17
70%
50.06
52.44
25.51
65.77
58.08
68.67
53.42
90%
50.26
53.25
25.66
53.47
70.67
53.27
SPECPrEFILL by large margin under the same rate. For MInference, we use the official searched optimal pattern, and SpecPreFILL can reach 99.77 average score with only 10% keep rate and outperform it with larger keep rate. Since the exact token-level "keep rate" of sparse attention ker- nel is not defined, we defer to Section 4.7.2 for more fair comparison between these two approaches. Overall, SPecPrEFILL achieves impressive performance without any fine-ture fine-tuning imput assumption, further supporting its effectiveness and flexibility.
SPECPREFILL even helps improve the quality', suggesting 1) that SPECPREFILL provides both efficiency and perfor- mance gains at the same time, and 2) the fact that there are lots of potential redundancy and noise these synthetic tasks.
4.6. Standard Short Tasks
Unlike prior works on prefill token dropping techniques (Lv et al., 2024; Shi et al., 2024) that do not include regular short context task evaluation, we present wide range of standard tasks to show the full spectrum of SPeCPREFILL' performance and potential caveats. We select tasks spanning general knowledge (Generative MMLU (Hendrycks et al., 2021) and Instruction Following Evaluation (Zhou et al., 2023)), math (GSM8K 8 Shots (Cobbe et al., 2021)), coding (HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021)), and reasoning abilities (Are Challenge (Clark et al., 2018) and GPQA 8 Shots (Rein et al., 2023)).
4.5. Synthetic Context Probing: RULER
In addition to LongBench, we also evaluate SPECPREFILL on synthetic context probing task to see if SPECPREFILI. can preserve effective context lengths. RULER (Hsich et al., 2024) a suite synthetically created tasks with con- trollable lengths, which ranges from retrieval, multi-hop tracking, real QA datasets, and context aggregation tasks. In Table 2. include the results for the 70B model with observe,
In Apendix Figure 6, showcase the performance of on these tasks. Non-
SPecPreFILL that keeps 10% context. As SPECPREFILL preserves the quality despite only using one tenth of the tokens except for aggregation tasks, which we queries
1.1 ama 3. surprisingly, prompts from standard tasks without few shot examples are very information dense, making SPECPREFILL less effective with low token keep rate. However, for certain tasks (e.g. MBPP and GPQA), we do observe improved performance when dropping certain tokens. On average. SPecPreFILL can maintain and even surpass the baseline when choosing the right token keep rate.
believe to fall into the category of information-desn that are not our main target application. Take CWE from ag- gregation category for example: CWE asks for the common words presented in the prompt, which becomes challenging to answer by token dropping. We hope explore in the future ways of potentially rewriting the queries instead of directly dropping the tokens to mitigate this type of limita- tion (Jiang et al., 2023b; 2024b). Averaging scores without the aggregation task, we can see that in most context lengths,
'For 4k Multi-hop Tracking, since we only keep around 400 kens, we might unintentionally ignore some essential information. But to keep experiment setup more consistent, we list the results here for clarity.

---

Speculative Prefill
Table 2. RULER Results on Llama 70B: We present results of SPECPREFILL with 10% token keep rate the effective context probing suite RULER with varying context length. SPECPREFILL can preserve the performance of all except for aggregation tasks, which are less compressible due the problem nature as each word in the prompt is important to reason about word frequency and commonality.
Model Name
Task Length
Retrieval Niah Variants
Multi-hop Tracking
QA
Aggregation CWE & FWE
Average w/o Aggregation
Variable Checking
SQuAD & HotpoiQA
4k
100.0
100.0
76.9
99.7
92.3
99.9
100.0
74.7
98.0
91.5
Llama-700 -0st1nst
16K
99.8
100.0
72.0
97.8
90.6
99.6
100.0
69.8
96,9
89.8
64K
98.5
99.9
65,1
65.6
87.9
128K
76.5
56.1
48.2
41.3
60.3
4K
99.7
89.6
75.2
77.9
88.2
99.6
100.0
75.6
79.7
91.7
SPEcPREFILL with 10% Keep Rate
16K
99.5
99.1
75.3
78.5
91.3
32K
99.7
100.0
72.6
70.0
90.8
64K
99.5
99.8
71.9
54.9
90.4
128K
85.8
$5.6
55.3
48.3
65.6
4.7. Efficiency Benchmarking
70B Instruct
Liama 4058 Instruct FP8
10%
SPECPREFILL offers great improvement to TTFT, a speedup almost proportional to the percentage of tokens we drop from the speculator, with almost ignorable overhead as we increase the base model size. In this section, benchmark both the 70B and 405B models under two settings: 1) un- derstanding the average query latency and QPS dynamics with real downstream datasets, and 2) evaluating TTFT with varying sequence lengths on synthetic data. We used one node consisting of eight NVIDIA H200s for all experiments unless separately specified (full system specification listed in Table from the Appendix D).
244%
352%
262%
367%
669%
694%
717%
766 %
Token Keep Rate
30%
206%
212%
219%
227%
275%
282%
296%
321%
50%
147%
151%
155%
158%
182%
183%
192%
204%
70%
112%
114%
116%
118%
133%
135%
138%
142%
128 4k
32 18k
16 33k
128 4k
32 Iok
32k
64
64
Batch Size Sequence Length
Batch Size Sequence Len 9th
Figure 3. SPECPREFILL TTFT Improvement: We present pre- fill TTFT speed-up using SPECPREFILL under different settings over 1.1-a-
4.7.1. AVERAGE QUERY LATENCY UNDER DIFFERENT QPS WITH REAL DOWNSTREAM DATASETS
and Llama-3. 1ommaa..1-0555 -nstts--nsttuut-EEE (achieving up to 7.66x faster TTFT when keeping 10% tokens for the 405B model).
We want to measure the real performance gain we can create in an end-to-end fashion. To do this, we launch a vLLM server with a given model, and an OpenAl API client 6 that sends asynchronous requests at a constant QPS with queries from datasets in LongBench. We measure the client-side per query latency that consists of the prefill stage with several decoding steps based on the maximum budget defined by the task. In Figure 1, we increase the QPS of our client and calculate the average query latency with a given fixed timcout to simulate real-world user needs. For each task category, we draw samples randomly from each subtask and shuffle them before starting the querying, making sure the same set of queries is used over all QPS. As we can observe, all models will follow a standard three-stage pattern: 1) the initial constant stage where latency remains almost un- changed as all queries can be finished before receiving new ones, 2) the middle linear stage where TTFT is small enough but the decoding step might not finish fast enough, and 3) the final timeout stage where the server can not even fin- ish the prefill stage before new requests and all subsequent queries are jammed thereafter. Since the maximum QPS
system can support is O(1/TTFT) given finite time- out, the acceleration from SPECPREFILL will drastically increases the maximal QPS under a fixed timeout, which pushes the transition from stage to stage further later.
We show that with the help of SPECPREFILL, the 405B model can convert to 7x QPS improvement QA suite from LongBench while maintaining > 95% accu- racy. The results vary as we change the model FLOPS ratio and drop rate. We believe that this type of analysis provides invaluable insights and tangible benefits of SPECPREFILL when deployed to real world systems.
Multi-Doc
4.7.2. TTFT IMPROVEMENT OVER DIFFERENT BATCH SIZE X SEQUENCE LENGTH PRODUCTS
We try to understand the dynamics of SPECPREFILL under products and keep per-
different batch- Mauch-size-sequee seize-seqeenc--eeeee
IIf we have finitely large timeout, we would expect SpECPRE- FILL to support around N times larger maximal QPS if reduce TTFT by times.
hhttps: thw.. com cpenaii /ammaai//ppnann /openai-eeaaaee//eeoeeee -python
8

---

Speculative Prefill
Llama 3.1 70B Instruct
Llama 3.1 4058 Instruct FP8
node of 8xH100s with TP-8). In Figure 4.7.2, we can see that SPECPREFILLL outperforms both the dense model and MInference when using large batch size and short to medium length prompts (i.e. less than 128K tokens). The main lim- itation for Minference under this scenario is the overhead introduced due to additional approximations, which gets amortized only when the ratio sequence length becomes large batch size enough (Jiang et al., 2024a). Overall, we found SPECPRE- FILL to be able to achieve 2.54x to 6.54 relative speedup over MInference with 99.5% quality performance. We be- lieve that this experiment provides more comprehensive guide to practitioners on which method to choose for spe- cific application needs (i.e. large batches plus less than 128K or ultra long prompts).
10%
264%
274%
287%
302%
573%
588%
584%
676%
30%
173%
101%
109%
200%
256%
264%
278%
303%
Token
50%
131%
134%
130%
144%
173%
176%
184%
197%
70%
102%
104%
107%
110%
129%
130%
134%
138%
128 dk
8k
32 16k
16 32k
64 dk
32 1ok
16 32k
128
Batch Sequence Length
Batch Size Sequence Length
Figure 4. SPECPREFILI with look-ahead TTFT Improvement: Complimentary to Figure 3, we also show the relative speedup when using a look-ahead 8 steps for both the 70B and 405B
model.
5. Limitation
The main limitation of SPECPREFILL is akin to all token- dropping based method: 1) they do not support explicit logit outputs for all input tokens, and hence we focus on genera- tive evaluation, 2) without explicit context recomputation, multi-ture conversation can potentially fail (Li et al., 2025), which poses a trade-off of whether we should compute and store all KV caches during the prefill. Given that we have the full knowledge of the speculator, we advertise for more principled method to estimate token importance beyond attention scores and efficient methods for dynamic KV re- computation. Finally, we believe that a robust algorithm that determines how many tokens are required for a given prompt will be of great use for SPECPREFILL
50
Proffll Labency
40
Pivienece
-30
Speccrotil 10% (91 7911
Scectrut1 20% (121 PP
SeecTrut1 50%
Batch Size Sequence Langth
Figure 5. SPECPREFILL v.s. MInference TTFT on 70B Models: The superiority of SPECPREFILL becomes more clear as we in- crease the batch size under 128K context lengths and MInference gradually improves the context length increases with smaller batch size due less overhead. Percentages in parenthesis are the relative average scores that of MInference on LongBench.
6. Conclusion
In this work, we introduce SPECPREFILL, training- free framework for accelerating the LLM inference by speculat- ing what tokens to drop with the help of a smaller specula- tor model. Leveraging the insight that models of different sizes within the same family can usually transfer token importance, SPECPREFILL not only achieves substantial improvement on TTFT, which leads to 7x maximal sup- ported QPS of an inference system, but also reduces the memory required. SPECPREFILL can also be readily com- bined with other techniques such as speculative decoding, the combination of which could result in the first unified mmall--mode-assiitted inference pipeline. With extensive evaluations, believe that SPECPREFILA will be one of the practical answers to large-scale LLM inference systems.
centage while isolating the advantage of TTFT. We use the official script from vLLM for latency benchmarking, In Fig- ure 3 and 4.7.2 (with look-ahead), we highlight the TTFT speedup against the vanilla base model without SPECPRE- FILL, which we produce by setting the maximum decoding step be 1. As we can see, for both the 70B and 405B mod- els, not only do we see more direct effects of SPECPREFILL but also imply the increasing scaling along the sequence dimension. As the relative FLOPS ratio between the spec- ulator and main model becomes larger, the overhead of speculation starts to become more negligible, which leads to more substantial improvement. In order to better understand the whole system, we include theoretical analysis of the overhead and the performance gains in Appendix E.
Impact Statement
One salient feature of SPECPREFILL is the ability to skip some MLP calculations, one of the major benefits of which is its strong performance under large batch size. To fur- ther understand the trade-offs, we compare the TTFT of SPECPREFILLL and MInference using the 70B model (one
This paper presents work whose goal is to accelerate large language model inference procedures. There are many po- tential societal consequences of our work, none which we feel must be specifically highlighted here.

---

Speculative Prefill
References
Dao, T. Flashattention-2: Faster attention with better par- allelism and work partitioning, 2023. URL https: / //arxiv..rr// wttp:/rrggbbbs/...oooo 08691.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language mod- els, 2021. URL https: : wttpiaiiiivvo/oo///.a///aaa org/ abs 21008 07732.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashat- tention:
Fast and memory-efficient exact attention with io- awareness, 2022. URL https: /tt///xxvvvv////e///ee/////eeelt wwwpwiiwworrass..oo. 2205.14138
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang. 7., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and J. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3119-3137, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10.18653ww1/2024.ao-oogg 172. URL https: http://ccaaantollloorog///////// acl-lol long. 172.
DeepSeck-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang. R., Xu, R., Zhu, Q., Ma, S., Wang, P. Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F, Gou, Z., Shao, Z. Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C.,, Ruan, C., Dai, D., Chen, D., Ji, D.. Li, E., Lin. F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Ding, H., Xin, H., Gao. H., Qu, H., Li, H., Guo. J., Li, J., Wang, J., Chen, J., Yuan, J., Qiu, J., Li, J., Cai, J. L.,, Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang. L., Zhang, L., Zhao, L.,. Wang. L.,. Zhang, L., Xu, L., Xia, L., Zhang. M., Zhang, M., Tang, M., Li. M., Wang. M., Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q., Chen, Q.. Du, Q. Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R. J., Jin, R. L.,, Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S. Wang. S., Yu, S., Zhou, S., Pan. S., Li, S. S., Zhou, S. Wu, S., Ye, S., Yun. T., Pei, T., Sun, T., Wang, T., Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Xiao, W. L., An, W., Liu, X., Wang, X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., Li, X., Su, X., Lin, X., Li, X. Q., Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song, X., Zhou, x., Wang, X., Shan, X., Li, Y. K., Wang, Y. Q. Wei, Y. X., Zhang, Y., Xu, Y., Li, Y., Zhao, Y.. Sun, Y., Wang, Y., Yu, Y., Zhang, Y., Shi, Y., Xiong, Y., Hc, Y., Piao, Y., Wang. Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Ou, Y., Wang. Y., Gong, Y., Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Tang. Y., Zha, Y., Yan, Y.. Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao, Z., Ma. Z., Yan, Z., Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li,
Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer, 2020. URL https: //arxivv org/ atggabs//200/ 05150.
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lund- berg. S., Nori, H., Palangi, H., Ribeiro, T., and Zhang, Y. Sparks of artificial general intelligence: Early experi- ments with gpt-4, 2023. URL attpps///arxv.... org/ abs/2303, 12712.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder. N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M. Chantzis, F. Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji. S., Jain, S., Saun- ders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J. Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Mc- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, 2021. URL https: / /twxxvixvvv./// org/ abs/ 2107. 03374.
Z. Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang, Z., and Zhang, Z. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https: / /////rivvv.o/ /t//aaxxvvvog///ae//
wrs./bs//////o.oooooooo.oe.esse abs 2501 12948.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. ArXiv, tttss/883.0555,,
Elhoushi, M.. Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., Aly, A., Chen, B., and Wu, C.-J. Layer- skip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- une 1: Long Papers). pp. 12622-12642. Association for Computational Linguistics, 2024. doi: 10. 18.53/5//v1/
Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021.
10

---

Speculative Prefill
2024.acl-long.681. URL hwtp:htpp///// doi. www../o/........... 18653/v1/2024.acl
R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Maheswari, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang. R., Hosseini, S., Chennabasappa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan. S., Borodinsky, S., Herman, T., Fowler, T. Sheasha, T., Georgiou, T., Scialom, T., Speck- bacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V. Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Gold- schlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y.. Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poul- ton, A., Ryan, A., Ramchandani, A., Dong, A., Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A.. Bharambe, A., Eisenman, A.. Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B. Montalvo, B., Parker, C., Burton, C., Mcjia, C., Liu, C., Wang. C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhoofrrr C., Gao, C., Civin, D., Beaty, D., Kreymer, D., Li, D.,, Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Mont- gomery, F., Presani, F.,, Hahn, E.,, Wood, F.,, Le, E.-T., Brinkman, E., Arcaute E., Dunbar, E., Smothers, E., Sun, F. Kreuk, F. Tian, F., Kokkinos, F., Ozgenel, F,, Cag- gioni, F., Kanayet, F., Seide, F., Florez, G. M. Schwarz, G., Badeer, G., Swee, G., Halpem,, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan. G., Inan, H., Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Damlaj. I., Molybog, I., Tufanov, I., Leontiadis, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Lam, J., Asher, J., Gaya, J.-B., Marcus, J., Tang. J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J.. Jin, J., Yang. J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang, J., Wu, K., U, K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K., Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, K., Huang, K., Chawla, K., Huang, K., Chen, L., Garg, L., A, L., Silva, L.,, Bell, L.,, Zhang. L.,, Guo, L.., Yu, L., Moshkovich, L., Wehrstedt, L.. Khabsa, M., Avalani, M.. Bhatt, M.. Mankus, M.. Hasson, M., Lennie, M., Reso, M. Groshev, M., Naumov, M., Lathi, M., Keneally, M.,
681.
Feng. T., Jin. C., Liu, J. Zhu, K., Tu, H., Cheng, Z., Lin, G., and You, J. How far are we from agi: Are Ilms all we need?, 2024. URL https: htapss//aaxiv...o// org/ abs/ 2405. 10313.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, L., Hsu, J., Le Noac"h, h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024a. URL https::/p/........... /enodo org/ / records/ 12608602.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. Retrieval -unmeettl generation for large language models: A survey, 2024b. https:::// /ta::///axiii aa/pxaxxiii.io//aa//aaa
10999
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dable, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshom, A., Yang. A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C. Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D., Starcia-Olaoo, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F,, Guzmán, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L.,, Thattai, G., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Ko- revaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I., Evtimov, I., Zhang, J., Copet, J., Lee, J.. Geffert, J., Vranes, J., Park, J., Mahadcokar, J., Shah, J., van der Linde, J., Billock, J., Hong. J., Lee, J., Fu, J., Chi, J.. Huang. J., Liu. J., Wang. J., Yu, J.. Bitton, J. Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K., Malik, K. Chiu, K., Bhalla, K., Lakhotia, K., Rantala- Yeary, L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L.,, Martin, L., Madaan, L., Malo, L., Blecher, L., Landzaat, L., de Oliveira, L., Muzzi, M. Pasupuleti. M.. Singh, M., Paluri, M., Kardas, M., Tsimpoukelli, M., Oldham, M.. Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Zhang, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishman, P., Koura, P. S., Xu, P., He, Q., Dong, Q., Srinivasan,

---

Z. Detoto, Z., Rosnbrick. Z.. Wen, Z., Yang, Z., Zhao, Speculative Prefill
Liu, M., Seltzer, M. L.,, Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M.. Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M.. Bansal, M., Santhanam, N.. Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N. P.. Dong, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P.,
LImlingua: Compressing prompts for accelerated infer- ence of large language models, 2023b. URL https: //arxiv.
05736.
Jiang H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, 7., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. Minference 1.0: Accelerating pre-filling for long- context Ilms via dynamic sparse attention, 2024a. URL https: /tEps://arxiv. org. abs 2407 02490.
Saab, P.. Balaji. P., Rittner, P., Bontrager, P., Roux, P.,
Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q. Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Patil, S., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S.. Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satter- field, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mi- hailescu, V. T. Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wu, X., Wang. X., Wu, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang. Y., Adi, Y., Nam, Y., Yu, Wang. Zhao, Y., Hao, Y., Qian, Y., Li, Y., He, Y., Rait,
Jiang, H., Wu, Q. Luo, X., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression, 2024b. URL wwt.hhttp////....mm / arxiv www.vo.oooo..o.ooooooooooooo.ooe orgg abs 2231 0..10.00 06839.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Lai, X., Lu, J., Luo, Y., Ma, Y., and Zhou, X. Flexpre- fill: A context-aware sparse attention mechanism for ef- ficient long-sequence inference, 2025. URL https:
//arxiv. wwwpxi..rrg/bb//25522 2576.20766. 20766.
Lee, H., Park, G., Suh, J., and Hwang, J. Infinitehip: Extending language model context up to million tokens on a single gpu, 2025. URL https: peattss://axiiii //axxixxxvogg///////////////../r abs/ abs0 08910.
Z., and Ma, Z. The llama herd of models, 2024. URL https: / aariiviiv...//.. org/ abs/ /
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding, 2023. URL
17199.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive multitask language understanding, 2021. URL https: //arxiv. org/ /2099 03300.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., tau Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval- augmented generation for mhowledge-intensiig nlp tasks, 2021. URL tttps://arrxe org/ wwwp/rrg//////o///e/...o/eeeeeee abs/2005. 2005. 11401.
Horton, M., Cao, Q., Sun, C., Jin, Y., Mehta, S., Rastegari, M., and Nabi, M. Kv prediction for improved time to first token, 2024. URL https:://arxii org/ abs/ 2410. 08391.
Li, Y., Dong, B., Lin, C., and Guerin, F. Compressing con- text enhance inference efficiency of large language models, 2023. URL https: https/://axxvvoooo 2310 06201.
Hsich, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. Ruler: What's the real context size of your long-context language models? arXiv preprini ar arXiv:2404.06654 2024.
Li, Y., Jiang, H., Wu, Q., Luo, X., Ahn, S., Zhang, C., Abdi, A. H., Li, D., Gao, J., Yang, Y., and Qiu, L. Scbench: A kv cache-centric analysis of long- long-context methods, 2025, URL https:tt///....... wtwpaaxivvvvvvo... org / abs /abs///0 /411..4122.1. 031 9.
Jang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023a. URL https: //arxiv.or orgo abs /2300 0825.
Li, Z., Li, C., Zhang, M., Mei, Q., and Bendersky, M. Re- trieval augmented generation or long-contex llms? a comprehensive study and hybrid approach, 2024. URL /ttes///rxivee http://axxxvv..r//bbee/s/
Jiang H., Wu, Q. Lin, C.-Y,, Yang, Y., and Qiu, L.
16833.
12

---

Speculative Prefill
Lin, J., Tang, J., Tang. H., Yang, S., Chen, W.-M., Wang. W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for Ilm compres- sion and acceleration, 2024. URL https: arxiv. org/abs, 2306. 00978.
Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S.,
Jonn, B., Jun, H., Kaftan, T., Lukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim. J. W., Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M., Kokotajlo, D., Lukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung. J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M. Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang. L.., O'Keefe, C., Pachocki,
Liu, J., Xia, C. S., Wang, Y., and Zhang. I.. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https: / http:///ppeeeeeeeeeeeemee net. corummid=qqvxff OCu?.
Liu, Y., Li, H., Cheng, Y., Ray, S., Huang. Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., Maire, M., Hoffmann, H., Holtzman, A., and Jiang, J. Cachegen: Kv cache compression and streaming for fast large language model serving, 2024. URL /tttps://axxiv..e arg.v.org/ abs/2310.00 07240.
J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E.,, Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M.. de Oliveira Pinto, H. P., Michael, Pokomy, Pokrass, M., Pong. V. H., Powell, T., Power, A., Power, B., Prochl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakooo,, T., Shieh, J.. Shoker, S., Shyam, P.. Sidor, S., Sigler, F.,, Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song. Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A.,, Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng, I.., Wi- ethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S.. Yu, K.,, Yuan, Q.. Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph. B. Gpt-4 technical report, 2024. URL https:tt///// /tt.//aivvv/... www.wwwwwww..aaaaa.aaooaaa.a.aae http://bbb2333.087773
Loper, E. and Bird, S. Nitk: The natural language toolkit, 2002. URL htRtt::/://axti org/ abs/ cs/ 0205028.
Lv, J., Feng, Y., Xie, X., Jia, X., Peng, Q. and Xie, G. Critiprefill: A segment-wise criticality-basel approach for prefilling acceleration in Ilms, 2024. URL ht //arxiv. org/ abs / 220/ 12490.
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., and Jia, 7. Towards efficient generative large lan- guage model serving: A survey from algorithms to systems, 2023. URL httts:://aaiii org/ abs/ 2312.15230
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, I.., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Bal- aji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett- Shaaaidirtt-Shiioo...oo G., Berner, C., Bogdonoff,, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K. Cai T., Camphell, R., Cann, A., Carey, B., Carlson, C. Carmichael, R., Chan, B., Chang, C., Chantzis, F,, Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings. D., Cur- rier, J., Dai, Y., Decareaux, C., Degry. T,, Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Lontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C. Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S.,
Qiao, A., Yao, Z., Rajbhandari, S., and He, Y. Swiftkv: Fast prefill-opimmzzed inference with knowledge-preserving model transformation, 2024. URL https::/arxxt org/ abs 24140.0900000
Reimers, N. and Gurevych, I. Sentence-bert Sentence em- beddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computational Linguistics, 11 2019. URL attps://arxiv.org abs 0bs//190..0 11088.0088 10084.
Rein, D., Hou, B. L.., Stickland, A. C., Petty, J., Pang, R. Y.,
13

---

Speculative Prefill
Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark, 2023.
Balaguer, J., Xiang, J., Wang, P., Ashwood, Z., Briukhov,
A., Webson, A., Ganapathy, S., Sanghavi, S., Kannan,
A., Chang, M.-W., Stjerngren, A., Djolonga, J., Sun, Y., Bapna, A., Aitchison, M., Pejman, P., Michalewski, H., Yu, T. Wang. C., Love, J., Ahn, J., Bloxwich, D., Han, K., Humphreys, P., Sellam, T., Bradbury, J., Godbole, V., Samangooei, S., Damoe, B., Kaskasoli, A., Arnold, S. M. R., Vasudevan, V., Agrawal, S., Riesa, J.. Lepikhin. D., Tanbur, R., Srinivasan, S., Lim, H., Hodkinson, S., Shyam, P., Ferret, J., Hand, S., Garg, A., Paine, T. L., Li, Li, Y., Giang, M., Neitz, A., Abbas, Z.,, York, S., Reid, M., Cole, E., Chowdhery, A., Das, D., Rogozińikaa, D., Nikolaev, V., Sprechmann, P., Nado, Z., Zilka, L., Prost, F., He, L.,, Monteiro, M., Mishra, G., Welty, C., Newlan, J., Jia, D., Allamanis, M., Hu, C. H., de Liedek- erke, R., Gilmer, J., Saroufim, C., Rijhwani, S., Hou, S., Shrivastava, D., Baddepudi, A., Goldin, A., Ozturel, A., Cassirer, A., Xu, Y., Sohn, D., Sachan, D.. Amplayo, R. K., Swanson, C., Petrova, D., Narayan, S., Guez, A.. Brahma, S., Landon, J., Patel, M., Zhao, R., Villela, K., Wang. L., Jia, W., Rahtz, M., Giménez, M., Yeung, L., Keeling, J., Georgiev, P., Mineu, D., Wu, B., Haykal, S., Saputro. R., Vodrahalli, K., Qin, J., Cankara, Z., Sharma, A., Fermando, N., Hawkins, W., Neyshabur, B., Kim, S., Hutter, A., Agrawal, P., Castro-Ros, A., van den Driess- che, G., Wang, T., Yang, F., yii Chang, S., Komarek, P., Mcllroy, R., Lučić, M., Zhang, G., Farhan, W., Sharman, M., Natsev, P., Michel, P., Bansal, Y., Qiao, S., Cao, K., Shakeri, S., Butterfield, C., Chung. J., Rubenstein, P. K., Agrawal, S., Mensch, A., Soparkar, K., Lene, K., Chung, T., Pope, A., Maggiore, I.., Kay, J., Jhakra, P., Wang, S., Maynez, J., Phuong, M., Tobin, T., Tacchetti, A., Trebacz, M., Robinson, K. Katariya Y., Riedel, S., Bailey, P., Xiao, K., Ghelani, N., Aroyo, I.., Slone, A., Houlshy, N., Xiong. X., Yang, Z., Gribovskaya, E., Adler, J., Wirth, M., Lee, L., Li, M., Kagohara, T., Pavagadhi, J., Bridgers, S., Bortsova, A., Ghemawat, S., Ahmed, Z., Liu, T., Pow- ell, R., Bolina, V., linuma, M., Zablotskaaaa,, P., Besley, J., Chung, D.-.., Dozat, T., Comanescu, R., Si, X., Greer, J., Su, G., Polacek, M., Kaufman, R. L.., Tokumine, S., Hu, H., Buchatskaya, E., Miao, Y., Elhawaty, M., Sid- dhant, A., Tomasev, N., Xing, J., Greer, C., Miller, H., Ashraf, S., Roy, A., Zhang, Z., Ma, A., Filos, A., Besta, M., Blevins, R., Klimenko, T., Ych, C.-K., Changpinyo, S., Mu, J., Chang, O., Pajarskas, M., Muir, C., Cohen, V., Lan, C. L., Haridasan, K.. Marathe, A.. Hansen, S., Dou- glas, S., Samuel, R., Wang, M., Austin, S., Lan, C., Jiang, J., Chiu, J., Lorenzo, J. A., Sjösund, L. L., Cevey, S., Gle- icher, Z., Avrahami, T., Boral, A., Srinivasan, H., Selo, V., May, R., Aisopos, K., Hussenot, L., Soares, L. B., Baumli, K., Chang, M. B., Recasens, A., Caine, B., Pritzel, A., Pavetic, F., Pardo, F., Gergely, A., Frye, J., Ramasesh, V., Horgan, D., Badola, K., Kassner, N., Roy, S., Dyer, E., Campos, V. C., Tomala, A., Tang, Y., Badawy, D. E.,
Shi, Z., Ming, Y., Nguyen, X.-P., Liang, Y., and Joty. S. Discovering the gems in early layers: Accelerating long- context Ilms with 1000x input token reduction, 2024.
URL https: arxiv. org/ abs/2409. 17422.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J.. and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model par- allelism, 2020. URL https: http://xxvvvvoo/ abs/ 1909.08055
Sun, H., Chang, L.-W., Bao, W., Zheng, S., Zheng, N., Liu, X., Dong, H., Chi, Y., and Chen, B. Shadowkv: Kv cache in shadows for tigg-throughput long-context Ilm inference, 2024. URL https: /ttp/a/aivvvv/////e//eee/e/eeele org/ abs/ 2410. 21465.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.. Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Johnson, M., Antonoglou, I., Schritwiesse, J. Glaese, A., Chen, J., Pitler, E., Lillicrap, T., Lazari- dou, A., Firat, O., Molloy, J., Isard, M., Barham, P. R., Henmigan, T., Lee, B., Viola, F., Reynolds, M., Xu, Y., Doherty, R., Collins, E.,, Meyer, C., Rutherford, E., Mor- eira, E., Ayoub, K., Goel, M., Krawezyk, J., Du, C., Chi, E., Cheng, H.-T., Ni, E., Shah, P., Kane, P., Chan, B., Faruqui, M., Severyn, A., Lin, H., Li, Y., Cheng, Y., It- tycheriah, A., Mahdieh, M., Chen, M., Sun, P., Tran, D., Bagri, S., Lakshminarayanan, B., Liu, J., Orban, A., Güra, F., Zhou, H., Song. X., Boffy, A., Ganapa- thy, H., Zheng, S., Choe, H., Ágoston Weisz, Zhu, T., Lu, Y., Gopal, S., Kahn, J., Kula, M., Pitman, J., Shah, R., Taropa, F., Merey, M. A., Baeuml, M., Chen, Z Shafey, L. E., Zhang, Y. Sercinoglu, O., Tucker, G., Pi- queras, E., Krikun, M., Barr, I., Savinov, N., Danihelka, I. Roelofs, B., White, A., Andreassen, A., von Glehn, T., Yagati, L., Kazemi, M., Gonzalez, L., Khalman, M., Sygnowski, J., Frechette, A., Smith, C., Culp, L., Proleev, L., Luan, Y., Chen, X., Lottes, J., Schucher, N., Lebron, F., Rrustemi, A., Clay, N., Crone, P., Kocisky, T., Zhao, J. Perz, B., Yu, D., Howard, H., Bloniarz, A., Rae, J. W., Lu, H., Sifre, L., Maggioni, M., Alcober, F., Garrette, D... Barnes, M.. Thakoor, S., Austin, J., Barth Marth.Maron.C.om G., Wong, W., Joshi, R., Chaabouni, R., Fatiha, D., Ahuja, A., Tomar, G. S., Senter, E., Chadwick, M., Kornakov, I. Attaluri, N., Iturrate, I., Liu, R., Li. Y., Cogan. S., Chen, J., Jia, C., Gu, C., Zhang, Q., Grimstad, J., Hart- man, J., Garcia, X., Pillai, T. S., Devlin, J., Laskin, M., de Las Casas, D., Valter, D., Tao, C., Blanco, I.., Badia, A. P., Reitter, D., Chen, M., Brennan, J., Rivera, C., Brin, S., Iqbal, S., Surita, G., Labanowski, J., Rao, A., Winkler, S., Parisotto, E., Gu, Y., Olszewska, K., Addanki, R., Miech, A., Louis, A., Teplyashin, D., Brown, G., Catt, E.,
14

---

Speculative Prefill
White, E., Mustafa, B., Lang, O., Jindal, A., Vikram, s., Gong, Z., Caelles, S., Hemsley, R., Thornton, G., Feng. F., Stokowiec, W., Zheng, C., Thacker, P., Çağlar Unlü, Zhang, Z., Saleh, M., Svensson, J., Bileschi, M., Patil, P., Anand, A., Ring, R., Tsihlas, K., Vezer, A., Selvi, M., Shevlane, T., Rodriguez, M., Kwiatkowski, T., Daruki, S., Rong, K., Dafoe, A., FitzGerald, N., Gu-Lemberg, K. Khan, M., Hendricks, A.. Pellat, M., Feinberg, V., Cobon-Kerr J., Sainath, T., Rauh, M., Hashemi, S. H., Ives, R., Hasson, Y., Noland, E., Cao. Y., Byrd, N., Hou, L., Wang, Q. Sottiaux, T., Paganini, M., Lespiau, J.-B., Moufarek, A., Hassan, S., Shivakumar, K., van Amers- foort, J., Mandhane, A., Joshi, P., Goyal, A., Tung, M., Brock, A., Sheahan, H., Misra, V., Li, C., Rakićević, N., Dehghami, M., Liu, F., Mittal, S., Oh, J., Noury, S., Sezener, E., Huot, F., Lamm, M., Cao, N. D., Chen, C., Mudgal, S., Stella, R., Brooks, K., Vasudevan, G., Liu, C.,, Chain, M., Melinkeri, N., Cohen, A., Wang, V., Seymore, K., Zubkov, S., Goel, R., Yue, S., Krishnakumaran, S., Albert, B., Hurley, N., Sano, M., Mohananey, A., Joughin, J., Filonov, E., Kepa, T., Eldawy, Y., Lim, J., Rishi, R., Badiezadegan, S., Bos, T., Chang, J., Jain, S., Padman- abhan, S. G. S., Puttagunta, S., Krishna, K., Baker, L., Kalb, N., Bedapudi, V., Kurzrok, A., Lei, S., Yu, A., Litvin, O., Zhou, X., Wu, Z., Sobell, S., Siciliano, A., Papir, A., Neale, R., Bragagnolo, J.. Toor, T., Chen, T., Anklin, V.. Wang, F., Feng, R., Gholami, M., Ling, K., Liu, L., Walter, J., Moghaddam, H., Kishore, A., Adamek, J., Mercado, T., Mallinson, J., Wandekar, S., Cagle, S., Ofek. E. Garrido, G., Lombriser, C., Mukha, M., Sun, B., Mohammad, H. R., Matak, J., Qian, Y., Peswani, V., Janus, P., Yuan, Q. Schelin, L., David, O., Garg, A., He, Y. Duzhyi, O., Ålgmyr, A., Lottaz, T., Li, Q., Yadav, V., Xu, L.,, Chinien, A., Shivanna, R., Chuklin, A., Li, J., Spadine, C., Wolfe, T., Mohamed, K., Das, S., Dai, Z., He, K., von Dincklage, D., Upadhyay, S., Maurya, A., Chi, L.. Krause, S., Salama, K., Rabinovitch, G., M. P. K. R., Selvan, A., Dektiarev, M., Ghiasi, G., Guven, E., Gupta, H., Liu, B., Sharma, D., Shtacher, I. H., Paul, S. Akerlund, O., Aubet, F.-X., Huang, T., Zhu, C., 7hu, E., Teixeira, E., Fritze, M., Bertolini, F., Marinescu, L.- E., Bölle, M., Paulus, D., Gupta, K., LAtkar, T., Chang, M., Sanders, J., Wilson, R., Wu, X., Tan, Y.-X., Thiet, L. N., Doshi, T., Lall, S., Mishra, S., Chen, W., Luong, T., Benjamin, S., Lee, J., Andrejezuk, E., Rabiej, D.,, Ranjan, V., Styre, K., Yin, P., Simon, J., Harriott, M. R., Bansal, M., Robsky, A., Bacon, G., Greene, D., Mirylenka, D., Zhou, C., Sarvana, O., Goyal. A., Andermatt, S., Siegler, P.. Horn, B., Israel, A., Pongetti, F., Chen, C.- W. L., Sel- vatici, M., Silva, P.. Wang, K., Tolins, J.. Guu, K., Yogev, R., Cai, X., Agostini, A., Shah, M., Nguyen, H., Don- naile, N. O., Pereira, S., Friso, L., Stambler, A., Kurzrok, A., Kuang, C., Romanikhin, Y. Geller, M., Yan, Z., Jang, K., Lee, C.-C,, Fica, W., Malmi, E., Tan, Q. Banica,
D., Balle, D., Pham, R., Huang, Y., Avram, D., Shi, H., Singh, J., Hidey, C., Ahuja, N., Saxena, P., Dooley, D., Potharaju, S. P., O'Neill, E., Gokulchandran, A., Foley, R., Zhao, K.. Dusenberry, M., Liu, Y., Mehta, P. Kotikala- pudi, R., Safranek-Shrader, C., Goodman, A., Kessinger, J., Globen, E., Kolhar, P., Gorgolewski, C., Ibrahim, A., Song, Y., Eichenbaum, A., Brovelli, T., Potluri, S., La- hoti, P., Bactu, C., Ghorbani, A., Chen, C., Crawford, A., Pal, S., Sridhar, M., Gurita, P., Mujika, A., Petro- vski, I,, Cedoz, P.-L., Li, C., Chen, S., Santo, N. D., Goyal, S., Punjabi, J., Kappaganthu, K., Kwak, C.,, L.V, P., Velury, S., Choudhury, H., Hall, J., Shah, P., Figueira, R., Thomas, M., Lu, M., Zhou, T., Kumar, C., Jurdi, T., Chikkerur, S., Ma, Y., Yu, A., Kwak, S., Ähdel, V., Ra- jayogam, S., Choma, T., Liu, F., Barua, A., Ji, C.,, Park, J. H., Hellendoorn, V., Bailey, A., Bilal, T., Zhou, H., Khatir, M., Sutton, C., Rzadkowski, W., Macintosh, F., Shagin, K., Medina, P., Liang, C., Zhou, J., Shah, P., Bi, Y., Dankovics, A., Banga, S., Lehhman, S., Bredesen, M., Lin, Z., Hoffmann, J. E., Lai, J., Chung, R., Yang, K., Balani, N., Bražinskas, A., Sozanschi, A., Hayes, M., Alcalde, H. F., Makarov, P., Chen, W., Stella, A., Snijders, L., Mandl, M., Kärrman, A., Nowak, P., Wu, X., Dyck, A.. Vaidyanathan, K., R. R., Mallet, J., Rudominer, M., John- ston, E., Mittal, S., Udathu, A., Christensen, J., Verma, V., Irving, Z., Santucci, A., Elsayed, G., Davoodi, E., Georgiev, M., Tenney, I., Hua, N., Cideron, G., Leurent, E., Alnahlawi, M., Georgescu, I., Wei, N., Zheng, I., Scandinaro, D., Jiang, H., Snoek, J., Sundararajan, M., Wang. X., Ontiveros, Z., Karo, I., Cole, J., Rajashekhar, V., Tumeh, L.,, Ben-David, E., Jain, R., Uesato, J., Datta, R., Bunyan, O., Wu, S., Zhang, J., Stanezyk, P., Zhang, Y., Steiner, D., Naskar, S., Azzam, M., Johnson, M., Paszke, A., Chiu, C.-C., Elias, S., Mohiuddin, A., Muhammad, F., Miao, J., Lee, A., Vieillard, N., Park, J., Zhang, J., Stanway, J., Garmon, D., Karmarkar, A., Dong, Z., Lee, J., Kumar, A., Zhou, L., Evens, J., Isaac, W., Irving, G., Loper, E., Fink, M., Arkatkar, I., Chen, N., Shafran, I., Petrychenko, I., Chen, Z., Jia, J., Levskaya, A., Zhu, Z., Grabowski, P., Mao, Y., Magni, A., Yao, K. Snaider, J., Casagrande, N., Palmer, E., Suganthan, P., Castaño, A., Giannoumis, I., Kim, W., Rybinski, M., Sreevatsa, A., Prendki, J., Soergel, D., Goedeckemeyer, A., Gierke, W., Jafari, M., Gaba, M., Wiesner, J., Wright, D. G., Wei, Y.. Vashisht, H., Kulizhskaya, Y., Hoover, J., Le. M., Li, L., Iwuanyanwu, C.,, Liu, L., Ramirez, K., Khorlin, A., Cui, A., LIN, T., Wu, M., Aguilar, R., Pallo, K., Chakladar, A., Perng, G., Abellan, E. A., Zhang, M., Dasgupta, I., Kushman, N., Penchev, I., Repina, A., Wu, X., van der Weide, T., Ponnapalli, P., Kaplan, C., Simsa, J., Li, S., Dousse, O., Yang, F., Piper, J., Ie, N., Pasumarthi, R., Lintz, N., Vijayakumar, A., Andor, D., Valenzuela, P., Lui, M., Paduraru, C., Peng, D., Lee, K., Zhang, S., Greene, S., Nguyen, D. D., Kurylowicz, P., Hardin, C., Dixon, L.,
15

---

Speculative Prefill
Janzer, L., Choo, K., Feng, Z., Zhang, B., Singhal, A., Du, D., McKinnon, D., Antropova, N., Bolukbasi, T., Keller, O., Reid, D., Finchelstein, D., Raad, M. A., Crocker, R., Hawkins, P., Dadashi, R., Gaffney, C., Franko, K., Bu- lanova, A., Leblond, R., Chung, S., Askham, H., Cobo, L. C., Xu, K., Fischer, F,, Xu, J., Sorokin, C., Alberti, C., Lin, C.-C., Evans, C.,, Dimitriev, A., Forbes, H., Ba- narse, D., Tung. Z., Omernick, M. Bishop. C., Sterneck, R., Jain, R., Xia, J., Amid, E., Piccinno, F., Wang, X., Banzal, P., Mankowitz, D. Polozov, A., Krakovna, V., Brown, S., Bateni, M., Duan, D., Firoiu, V., Thotakuri, M., Natan, T., Geist, M., tan Girgin, S., Li, H., Ye, J., Roval, O., Tojo, R., Kwong, M., Lee-Thorp. J., Yew, C., Sinopalnikov, D., Ramos, S., Mellor, J., Sharma, A., Wu, K. Miller, D., Sonnerat, N., Vnukov, D., Greig, R., Beattie, J., Caveness, E., Bai, L., Eisenschlos, J., Ko- rehemniy, A., Tsai, T., Jasarevic, M., Kong, W., Dao, P., Zheng. 7., Liu, F., Yang, F., Zhu, R., Teh, T. H., Sanmiya, J. Gladchenko, E., Trdin, N., Toyama, D., Rosen, E., Tavakkol, S., Xue, L.,, Elkind, C., Woodman, O., Carpen- ter, J., Papamakarios, G., Kemp, R., Kafle, S., Grumina, T., Sinha, R., Talbert, A., Wu, D., Ofusu-Afriyie, D., Du, C., Thortoo,, C., Pont-Tuset, J., Narayana, P., Li, J., Fatehi, S., Wieting. J.. Ajmeri, O.,, Uria, B., Ko, Y., Knight, L., Héliou, A., Niu, N., Gu, S., Pang, C., Li, Y., Levine, N., Stolovich, A.,
J., Zhang, H., Pruthi, G., Bauer, J., Yang, F., Mansour, R., Gelman, J., Xu, Y., Polovets, G., Liu, J., Cai, H., Chen, W., Sheng, X., Xue, E., Ozair, S., Angermueller, C., Li, X., Sinha, A., Wang, W., Wiesinger, J., Koukoumidis, E., Tian, Y., Iyer, A., Gurumurthy, M., Goldenson, M., Shah, P., Blake, M., Yu, H., Urbanowicz, A., Palomaki, J., Fernando, C., Durden, K., Mehta, H., Momchev, N., Rahimtoroghi, E., Georgaki, M., Raul, A., Ruder, S., Red- shaw, M., Lee, J., Zhou, D., Jalan, K., Li, D., Hechtman, B., Schuh, P., Nasr, M., Milan, K., Mikulik, V., Franco, J., Green, T., Nguyen, N. Kelley, J., Mahendru, A., Hu, A., Howland, J., Vargas, B., Hui, J., Bansal, K., Rao, V., Ghiya, R., Wang, E., Ye, K., Sarr, J. M., Preston, M. M., Elish, M., Li, S., Kaku, A., Gupta, J., Pasupat, I,, Juan, D.-C., Someswar, M., M., T., Chen, X., Amini, A., Fab- rikant, A., Chu, E., Dong, X., Muthal, A., Buthpitiya, S., Jauhari, S., Hua, N., Khandelwal. U., Hitron, A., Ren, J., Rinaldi, I.,, Drath, S., Dabush, A., Jiang, N.-J., God- hia, H., Sachs, U., Chen, A., Fan, Y., Taitelbaum, H., Noga, H., Dai, Z., Wang, J., Liang, C., Hamer, J., Ferng, C.-S., Elkind, C., Atias, A., Lee, P., Listík, V., Carlen, M., van de Kerkhof, J., Pikus, M., Zaher, K., Müller, P., Zykova, S., Stefanec, R., Gatsko, V., Hirnschall, C., Sethi, A., Xu, X. F., Ahuja, C., Tsai, B., Stefanoiu, A., Feng, B., Dhandhania, K., Katyal, M., Gupta, A., Parulekar, A., Pitta, D., Zhao, J., Bhatia, V., Bhavnani, Y., Alhadlaq, O., Li, X., Danenberg, P., Tu, D., Pine, A., Filippova, V., Ghosh, A., Limonchik, B., Urala, B., Lanka, C. K., Clive, D., Sun, Y., Li, E., Wu, H., Hongtongsak, K., Li, I., Thakkar, K., Omarov, K., Majmundar, K., Alver- son, M., Kucharski, M., Patel, M., Jain, M., Zabelin, M., Pelagatti, P., Kohli, R., Kumar, S., Kim, J., Sankar, S., Shah, V., Ramachandruni, L., Zeng, X., Bariach, B., Weidinger, I.., Vu, T., Andreev, A., He, A., Hui, K., Kashem, S., Subramanya, A., Hsiao, S., Hassabis, D., Kavukcuoglu, K., Sadovsky, A., Le, Q. Strohman, T., Wu, Y., Petrov, S., Dean, J., and Vinyals, O. Gemini: family of highly capable multimodal models, 2024. URL https: / arxiv org/ abs 2212. 11805.
Sewammarria-eemaaded R., Goenka, S., Yustalim, W. Strudel, R., Elqursh, A., Deck, C., Lee, H., Li, Z., Levin, K., Hoffmann, R., Holtmann- Rice, D., Bachem, O., Arora, S., Koh, C., Yeganeh, S. H., Pöder, S., Tariq, M., Sun, Y., Ionita, L., Seyedhosseimi, M., Tafti, P., Liu, 7.,, Gulati, A., Liu, J., Ye, X., Chrzaszez, B., Wang, L., Sethi, N., Li, T., Brown, B., Singh, S., Fan, W., Parisi, A., Stanton, J.. Koverkathu, V., Choquette-Choo C. A., Li, Y. Lu, T., Ittcceerihh, A., Shroff, P., Varadarajan, M., Bahargam, S., Willoughby, R., Gaddy, D., Desjardins, G., Cornero, M., Robenek, B., Mittal, B., Albrecht, B., Shenoy, A. Moiseev, F.. Jacobsson, H., Ghaffarkhah, A., Rivière, M., Walton A., Crepy, C., Parrish, A., Zhou, Z., Farabet, C., Radebaugh, C., Srinivasan, P., van der Salm, C., Fidjeland, A., Scellato, S., Chimoro- E., Klimczak-Plucinssk, H., Bridson, D., de Cesare, D., Hudson, T., Mendolicchio, P., Walker, L.,, Morris, A., Mauger, M., Guseynov, A., Reid, A., Odoom, S., Loher, L., Cotruta, V., Yenugula, M.. Grewe, D., Petrushkina, A., Duerig, T., Sanchez, A., Yadlowsky, S. Shen, A., Globerson, A., Webb, L., Dua, S., Li, D., Bhupatiraju, S., Hurt, D., Qureshi, H., Agarwal, A., Shani, T., Eyal, M., Khare, A., Belle, S. R., Wang, L., Tekur, C., Kale, M. S., Wei, J., Sang, R., Saeta, B., Liechty, T., Sun, Y., Zhao, Y., Lee, S., Nayak, P., Fritz, D., Vuyyuru, M. R., Aslanides, J., Vyas, N., Wicke, M., Ma, X., Eltyshev, E., Martin, N., Cate, H., Manyika, J., Amiri, K., Kim, Y., Xiong, X., Kang, K., Luisier, F. Tripuraneni, N., Madras, D., Guo, M., Waters, A., Wang, O., Ainslie, J., Baldridge,
Wan, Z. Wu, Z., Liu, C., Huang, J., Zhu, Z., Jin, P., Wang, L., and Yuan, L. Look-m: Look-once optimiza- tion in kv cache for efficient multimodal long-context inference, 2024. URL https: //aa/aixvvorrrg...e /rb/bb////////// 18139.
Wei, J.. Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D.,, Bosma, M., Zhou, D., Metzler, D... Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abili- ties of large language models, 2022. URL https: / / //arxiv.co org/ www..//./.......op
Xia, H., Yang, Z., Dong, Q., Wang. P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language
16

---

Speculative Prefill
model inference: A comprehensive survey of speculative decoding. 2024. URL htttps:://aaaiitt 0.01.078500
Zheng, L.,, Yin, L., Xie, Z.., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, E., Bar- rett, C., and Sheng, Y. Sglang: Efficient execution of structured language model programs, 2024. URL https: /ttaa/xii....gaab.ee.aa
abs/
Xiao, G., Tang. J., Zuo, J., Guo, J., Yang. S., Tang, H., Fu, Y., and Han, S. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads, 2024a. URL https: www.v.orgg/oaaooaaaoo.t..o.ooorr ab///bs///// / 222241000 10819.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-folloiing evaluation for large language models. arXiv preprint arXiv:231 23.:29911 2023.
Xiao, G., Tian, Y. Chen, B., Han, S., and Lewis, M. Effi- cient streaming language models with attention sinks, 2024b. URL https: /ttaas//aavvvv.///a httgg/abs/2300 17453.
Zhu, Q. Duan, J., Chen, C., Liu, S., Li, X., Feng, G., Lv, X., Cao, H., Chuanfu, X., Zhang, X., Lin, D., and Yang, C. Sampleattention: Near-lossless acceleration of long context 1lm inference with adaptive structured sparse attention, 2024. URL https: / /arxiv.....//////////a/////aiav/ org/ /bb// 2406.15488
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L.,, Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang. S., and Ma, H. Effective long-context scaling of foundation models, 2023. URL https://arxa org/ 2309. 6039.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J. Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao,
R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T.,
Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang,
X., Wei, X., Ren, X., Liu. X.. Fan, Y., Yao, Y., Zhang,
Y. Wan, Y. Chu, Y. Liu, Y., Cui, Z., Zhang, Z., Guo,
Z. and Fan, Z. Qwen2 technical report, 2024. URL https:/ // //aa/aivvvorrrre///ae abs/ /anss/2400 2407 10671.
Yuan, Z., Shang, Y., Zhou, Y., Dong, Z., Zhou, Z., Xue, C., Wu, B., Li, Z., Gu, Q., Lee, Y. J., Yan, Y., Chen, B., Sun, G., and Keutzer, K. Llm inference unveiled: Survey and roofline model insights, 2024. URL https: //arxiv. org/ abs/ / 022224422222 6363.
Zhang, J., Wang, J., Li, H., Shou, L.,, Chen, K., Chen, G., and Mehrotra, S. Draft & verify: Lossless large language model acceleration via self-speculative decoding, 2024a. URL https:::p///.. / aawiiiiiivv.////vv////v/////aaav v. wwwp:/wvvoorr/oooooo......oo abs / 9309. 08 68.
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H20: Heavy-hitter oracle for efficient generative inference of large language models, 2023. URL https: //arxiv..../////////a wrgg abs/ /2002336222 14048.
Zhang, Z., Liu, S., Chen, R., Kailkhura, B., Chen, B., and Wang, A. Q-hitter: A better token oracle for efficient Ilm inference via sparse-quantized ky cache. In Gibbons, P., Pekhimenko, G., and Sa, C. D. (eds.), Proceedings of Machine Learning and Systems, volume 6, pp. 381-394, 2024b.
17

---

Speculative Prefill
A. Standard Short Task Performance of SPECPREFILL
In Figure 6, we report SPECPREFILL when applied to Llama- 3. 3.11-00--110 Instrutt on standard short tasks as discussed in Sec 4.6. It is worth noting that for shorter tasks, the queries are more likely to become information dense, rendering SPECPREFILL less effective especially for certain tasks.
B. Comparing SPECPREFILL with RAG Based Systems
In this section, we first detail the algorithm behind two of our RAG baselines and present results comparing SPECPREFILL against them. Both variants split the context of the prompt into sentences using nitk library (Loper Bird, 2002). After splitting the context, all sentences are encoded using pretrained sentence embedding models (Reimers Gurevych, 2019). specially chosen query is used to select relevant sentences based on similarity scores without exceeding the predefined budget. Finally, the new context re-assembled and fed the main model. We highlight the key differences in various steps of the pipeline in the following table 3:
Model Name
Embedding Model
RAG Query
Reassemble Method
RAG-LLAMA LS RAG-LLAMA EQ
gtr-t5-large all-mpnet-base-v2
Last sentence in full prompt
Original order
Provided by the dataset
Ordered by relevance
Table 3. RAG Baseline Specification: We implemented two RAG baselines with different trade-offs to compare SPECPREFILL
In Figure 7, we compare Ll both RAG variants are based on sentence chunkation, and hence we calculate the final real token keep percentage for visualization and a fair comparison.
amam 3.
with RAG-LLAMA-70D LS and RAG-LLAMA-70N EQ. Since
It is worth noting that both two RAG variants can in principle fall short under given tasks due the fact that their strategy for selecting the query for retrieval is not flexible enough. For example, RAG-LLAMA-7700 LS will become less effective when the real query is not placed at the end of the prompt, and RAG-LLAMA-70B EQ not only assumes that the real query is separated from the context and given to the model but also needs special catering when it is not obvious how to design the query for certain tasks (e.g. summarization). Therefore, we consider RAG systems and SPECPREFILL, to be useful for different cases with varying degrees of requirements for efficiency, cost, performance, and generality.
C. Experiment Details
There are some details for our experiments that we wish to give some accounts for:
1. For standard short task evaluation in Sec 4.6, we use EEAE-EALLAAANRA et al., 2023). For several tasks in L.M-EVLLHARRNESS we include task configuration files for L.1 amlam--... based on its templates in our code base, which should be placed in the right place for reproducing experimental results.
(Gao et al., 2024a) and EVAL-PLUS (Liu
2. In the QPS experiment Sec 4.7, we add an extra seconds to the timeout in order to avoid potential system instability. The final results are reported with the original timeout. We set the number of samples for each category based on the maximal QPS we want to evaluate, which makes sure we have constant QPS during the duration of querying.
3. When running all experiments in vLLM (0.6.3.postl), we set enforce eager= and enable chunked prefill=False to avoid any unexpected behaviors.
D. System Specification
In Table 4, we list the detailed specification of the system which we test the efficiency of models.
E. Overhead Analysis
SPECPREFILL uses a smaller model as a speculator to help accelerate the larger model. Although proven to be effective, there is no free lunch. In this section, we analyze and quantify the overhead incurred by the speculator so that practitioners are more informed when they choose to use it.
18

---

Speculative Prefill
MMLU Gen
IF Eval
GSMBK
HumanEEal
0.9
0.8
0.855
0.4
0.7
0.6
0.7
08
Taakk Store
Taak Score
0.5
0.6
Taak
0.5
0.5
0.3
Bescline
Basdiic SpecPrefll
Baselino
0.2
Baseline
Spechelll
8.70
Specirrfil
8- Specielll
0.3
SpecPreft CS
SpeePrettl + C5
SpecPrett + CC
-8 Spectrel + CCC
SpecProftl CS LAH
0.3
SpodProffl + CS LAH
0.65
SpocPrafil CS LAH
SpacProlll CS LAH
30%5
50%
70%
90%
30%
50%
70%
90%
3096
50%
70%
90%
30%
50%
70%
90%
MBPP
Challenge
GPQA
0.43
0.8
0.9
0.8
0.785
0.423
0.675
0.42
0.7
0.6
0.41
0.5
0.6
Task Score
Task Score
0.4
0.5
0.3
0.4
0.39
0.4
Bakelin
Basediee SpedFreftl
Biseiine
Daxeline SpecPreftl
SpecPreft1
SpecPreft1
0.1
SpecProfil
C5
SpecPruffl
E5
SpecPrafil CS
Spechrolil C5
SpecPreffl+
SpocProffl C5
SpecProftl
CS
SpacProftl
30%
50%
90%
32%
50%
70%
90%
30%
50%
70%
M0%
30%
5.%
70%
90%
Taken
Taken
Token Rate
Taken Rate
Figure 6. Standard Short Tasks on Llama 70B: We include results on popular regular context tasks ranging from common knowledge, math, reasoning, and coding ability. Unlike prior works on token eviction and prompt compression, we wish to give comprehensive evaluation on domains where SPeCPRefILL becomes less effective due to the fact that short and knowledge rich prompts are less compressible.
We start by calculating the FLOPS of a transformer model based on a standard implementation, and then compute the theoretical overhead when using a specific speculator for a specific main model with the official Llama model configuration. Since the prefill phase is mostly compute-bound, it is fair to use the FLOPS ratio as a decent approximation of latency improvement. We introduce several parameters for an Llama-like transformer architecture and we calculate the FLOPS of each module separately using these parameters. We ignore less essential computations (e.g. normalization, RoPF, etc) and the formula are shown in Table 5.
In Figure 9, we calculate the theoretical upper-bound of the TTFT speedup for SPECPREFILL with sequence length being 32K and batch size equal to 16. As we can see from the theoretical analysis, the real speedup we obtain from the implementation is very close to it, as shown in Figure 3, which suggests that our implementation is highly efficient (i.e. measured 7.66x compared to analyzed 7.72x). On each bar, we annotate the overhead of the speculation process, which we define as:
FLOPS(spec)
overhead(a) = FLOPS(speec) + a FLOPS(base)
Ya E (0, 1]
Within our expectation, the overhead is higher when we have lower keep rate and lower when we have higher keep rate. Table E reports the theoretical relative FLOPS between the speculator and the base model.
F. LongBench Task Length Distribution
We visualize the LongBench suite's average length for each task, which, when coupled with the token keep rate, provides a more clear picture of the model's efficiency gain.
19

---

Speculative Prefill
Single-Documann
Documentt QA
Summarization
62.55
70
25.01
54.28
60
26.77
Took: Score
Task 3elly
30
20
bataliveAA
intporQA 2hik MGA
Gelakaprr
Ouupet
20
GKSum
MuE.He suuk-n
Vusieue
MullMows
NuBiFla d0A-th
OiRescerah
VCSUt--a
Regilama 15 702
Tag Lama 728
Bag 4Jama 728
Ragetaaaa 60788
sap- lama 738
10
Rag Limma 50
Spectoett Bull AM
Sper frelll RETLAN
Spectiefl (AH
Shot Learning
Synthetic Taskks
Code Completion
70
80
60
60
FassageC.on
Usima
40
Spectreell hull 700
60
40
Spechrolt Full LAY 708
20
SAMMSum
LSifiihh
20
20.0
Uaga 15 100
Rage 603
Spothelll Pull LAM 735
10%
Keep 70%
40%
30%
5.%
70%
10%
Keep
Keep Rata
Figure 7. LongBench Baseline Comparison: We compare SPECPREFILL against the baseline and RAG-LLAMA in this separate figure for clarity. R RAG--LLAMA could he effective on certain tasks but for the majority of the tasks, hut SPECPREFILL does not require any prior knowledge about the prompt while still being accurate with finer-control.
System Hardware/Software Name
Value
CUDA Version
12.7
vLLM Version
0.6.3.post
GPU Type
8 x NVIDIA H200
Total GPU TFLOPS
428.2
Total RAM
1123.2 GB
Per GPU Memory Bandwidth
4052.8 GB/s
Per GPU NVLink Bandwidth
478.1 GB/s
Per GPU PCIe Bandwidth
52.8 GB/s
Per GPU PCIe Lanes
16 x PCIe 5.0
Disk Bandwidth
4730 MB/s
Internet Upload Speed
605.5 Mbps
Internet Download Speed
733.4 Mbps
Table 4. System Specification for Efficiency Benchmarking:
Efficiency scores can vary when benchmarked on different platforms, and therefore, we list detailed specification the system we' using for better reprodueibility and understanding.
20

---

Speculative Prefill
Single-Document TIFT
Multi-Documant QA
rization Speedup 147%
TTFT Speedup 147%
3441
200%
347%
132%
344%
200%
112%
344%
200%
11225
62 25
60
60
55
54.28
150
50,01
50
47.9 58
Teol
Taok
Narmiivedt Gusper Mito/c//00o on
Holpengs
20
40
17.38
30
SpecPreffl Full LAH
10%
30%
50%
70%
90%
10%
30%
50%
70%
50%
10%
30%
50%
70%
50%
Few-Shot Learning 147%
Synthetic Tasks
Code Co Completion
344%
200%
344%
206%
147%
344%
54.04
100
90
80
Score
700
SAMS um
SpecPrettl
TREE TinkaDA
703
Task
60
Spsctrcfll
Pessige wetsiggeebiicaa-oo
Pessageletiieaa oh
Spsctrufil 702
PassageCourt
133
50
50,25
50
36192
20
45
LCC
RepoBtenchh
Socchelll
40
40
10%
30%
50%
70%
50%
10%
30%
70%
50%
50%
70%
90%
Keep Token Rate
Keep Rate
Keep Rate
Figure 8. LongBench Results on 70B Model: We supplement the evaluation of 1.1ama-3. Intuuuu, under the same setup as Figure and 7.
Parameters
Value
Number of layers
L
Hidden Size Intermediate Size
D
T H H'
Number of Query Heads
Number of KV Heads
Vocabulary Size
V
Sequence Length
S
Batch Size
B
MLP FLOPS
3BSDI
QK Projections FLOPPS
BSD + 2H' (H) 2BS²D
Self-Attention FLOPS
LM-Head FLOPS
BSDV
Total FLOPS
LBSD(31 + D(2 + 222 HH) HH) + 25) + BSDV
Table 5. FLOPS Estimation of Llama Models: We estimate the FLOPS (MACS) for Llama model with a given configuration. We ignored lower order terms of computations such as vector addition, RMSNomm, ROPE, and treat all matrix operation FMA for simplicity.
21

---

Speculative Prefill
Base Model
7.72x
Llama 3.1-708 Instruct
Llama-3 1aama-3.114055. 1---059-Insttuct-PP
Theoretical TTFT Speedup
3.03x
2.2
87
1.19x
1.37
92.2%
22.2%
16.9%
10%
Keep Rate
Figure Theoretical TTFT Specdup of SPECPREFILL: We show the theoretical upper-bound of the TTFT speedup that SPECPREFILL can achieve assuming no other system overhcad. The percentage of cach bar is the pereentage of FLOPS spent on speculation w.r.t. the total FLOPS. Comparing it with results in Figure our real measurement has very little gap the thcoretical maximum, which suggests the efficiency of the implementation.
Base Model Size
FLOPS spen
/ FFOP Stase
70B
14.24%
405B
2.96%
Table Relative FLOPS SPECPREFILLA calculate the theoretical FLOP ratio between speculator model size and the
base model.
carsapee retrieval, (56/8))
(3379)
pasaaga retrivall en (12522)
ACE
malrattveds 198620
pa5sage count
Sun Complction
(34970)
Sota
QA
Learning
(18243)
musKiue (15616)
Figure 10. LongBench Prompt Token Lengths: We visualize the average token lengths of prompts for each task spanning the five major categories in LongBench.
22


Reviews:
[{"review_id": "bweHeFhlGJ", "reviewer": "Reviewer_pMkJ", "summary": "This paper propose SpecPrefill, that identifies the important tokens during the pre-fill stage via accumulating the attention scores in a 8-steps look-ahead window, calculated with a small model as the speculator. This approach can achieves a speedup of about 3x and also maintains a comparable performance compared to full attention.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Summary\nWe sincerely thank the reviewer for the time and constructive feedback! We addressed all feedback and believe that it makes this paper significantly stronger (**added experiment result numbers in reviewer USDK’s rebuttal**). Specifically, we addressed the feedback by: \n\n1. We moved the original main results for our RAG-baselines to the main body and created a table to compare all methods including two newly added baselines: LLMLingua and MInference. SpecPrefill achieves higher quality and better efficiency than all of them. \n2. We agree with the application of multi-turn situations and believe that SpecPrefill will be able to solve it with the full knowledge of the speculator. We will add results in the camera-ready version (running the experiments). \n3. We added more context and discussions for the essential references in our camera-ready version. \n\n### Per-feedback detailed response: \n\n> Not a single baseline was included in the experiment… Try comparing spec-prefill against prompt compression methods, and dynamic sparse attention approaches.\n\nIn the previous draft, we compared two RAG-based methods in the Appendix. We should have done a better job in making them more visible -- to do this, we will add a summary of baseline comparison in T1 (reviewer USDK's comment) in the main paper body. \n\nIn addition, we follow the advice from the reviewer by adding LLMLingua (text-level prompt compression, https://arxiv.org/abs/2310.05736) and MInference (sparse attention, https://arxiv.org/abs/2407.02490) to complement our original RAG baseline. \n\nWe evaluate all three baselines on LongBench, and the result is shown in T1 (reviewer USDK's comment). **SpecPrefill outperforms RAG and LLMLingua by margins when keeping the same tokens from the prompt** (i.e. same tokens ~= similar speedup). Comparing MInference requires a more careful design due to the fact that it is a sparse attention optimization that does not change the computation of the MLP and needs to compute all KVs. Therefore, we benchmark MInference’s prefill latency in T2 (reviewer USDK's comment) where we vary the batch size x sequence length products from 128 * 4K to 4 * 128K. We observe that MInference has substantial overhead when the batch size is large and only gets amortized when the sequence length gets ultra long (at least > 128K with small batch size < 4, this is also observed in paper https://arxiv.org/abs/2502.05431 and confirmed by the author in issue https://github.com/microsoft/MInference/issues/18). **Overall, SpecPrefill can achieve 2.54x to 6.54x relative speedup compared to MInference with 99.7% quality and surpasses MInference’s quality when keep rate is at least 30%**. \n\n> Try test spec-prefill on multi-turn long context benchmarks such as SCBench. \n\nWe fully agree that multi-turn tasks are important applications to test SpecPrefill. **Unlike many prompt compression methods, SpecPrefill maintains the full KV for the speculator, which makes it possible to revive dropped tokens during the first round of conversation. This allows us to re-estimate token importance when a new turn comes in and let the base model fill-in the missing KVs as needed**. We are currently running evaluations and will include results for the camera-ready version (due to new implementation and evaluation time, we can’t finish it during the rebuttal window). In addition, here’s our detailed description of what we’re running: \n\nWe outline the high-level algorithm here: \n1. Check if it is the first turn. If so we just call our standard SpecPrefill and return the results. If not, move to step 2. \n2. Let the speculator first process the new N-token context and estimate important tokens just like in the first prefill. \n3. Identify missing KVs based on slot-mapping and position information. \n4. For positions missing KVs, we simply keep those ids. \n5. For the base model, we will recompute the KVs if some ids are not in the cache. Return the new results. \n\nThe total number of KVs that will be computed and stored will not exceed the context length, which means there is no waste of “recomputation”. \n\nNonetheless, we’d like to mention that our initial starting goal for SpecPrefill is to tackle large-batch offline (no further interaction) prefill acceleration that targets a large traffic of requests such as bulk document summarization, QA, etc. \n\n> Essential reference not mentioned. \n\nAll of these are awesome add-ons and have been included in appropriate places for reference. \n\nWe are confident that SpecPrefill can be applied and benefit many real world applications and the contributions are novel and fully reproducible. We’d love to hear from the reviewer on further suggestions if any to make the paper better!", "claims_and_evidence": "-", "methods_and_evaluation_criteria": "-", "theoretical_claims": "-", "experimental_designs_or_analyses": "This is method aims to optimize the pre-filling stage unlike the existing sparse attention, instead it uses a speculator to find the critical tokens and only involve these in the actual attention computation.\n\nThe strength of the paper:\n1. good system design. the proposed method reuse the speculative decoding framework, which extend the speculative decoding to the per-fill stage, so the large model not only benefit from the decoding proposal, but also the sparse pre-fill. This also makes this method to be easily adopted in systems with spec-decoing supported.\n\n2. good performance on most of tasks. spec-prefill achieves comparable performance against full attention on long-bench and ruler, with 50% tokens perserved. it achieves good speedup (2x speedup when 50% tokens perserved).\n\n3. a novel attempts for pre-fill acceleration that may inspire future works.\n\nThe limitation of the paper:\n1. not a single baseline was included in the experiment. This proposed method uses a draft/small model to evaluate the importance of the tokens from the prompt, is pretty much similar to the idea of prompt comparession and kv cache compression. Prompt compression, in particular, also aims to accelerate the pre-filling acceleration, which should definitely be compared against spec-prefill. In fact, spec-prefill is simply a prompt compression technique with a 8-step look-ahead window. In addition, dynamic sparse attention that build sparse index online for faster pre-fill is also very relevant to spec-prefill, which is a must to compare with. the missing of these baselines is a big limitation of this paper.\n\n2. this token-level sparse attention is previously studied to be less robust in real production environment, especially in multi-round scenarios. SCBench demonstrates that the critical tokens of a prompt can be dramatically different for different queries. In other words, the look-ahead window in spec-prefill can be sub-optimal when the conversation goes on and move to the next query. The benchmarks used in the experiments fail to prove that the proposed method can tackle such a limitation.", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "You should cite:\nLi, Y., Dong, B., Lin, C., & Guerin, F. (2023). Compressing Context to Enhance Inference Efficiency of Large Language Models. Conference on Empirical Methods in Natural Language Processing.\nfor prompt compression in addition to llmlingua series of model.\n\nYou should test the proposed method in SCBench to see how well the method perform under multi-turn scenarios:\nLi, Y., Jiang, H., Wu, Q., Luo, X., Ahn, S., Zhang, C., Abdi, A.H., Li, D., Gao, J., Yang, Y., & Qiu, L. (2024). SCBench: A KV Cache-Centric Analysis of Long-Context Methods. ArXiv, abs/2412.10319.\n\nIt may also help to show how well the method works on more challenging long-context tasks such as gsm-infini.\nZhou, Yang et al. “GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?” (2025).", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "-", "questions_for_authors": "1. try compared spec-prefill against prompt compression methods, and dynamic sparse attention approaches.\n2. try test spec-prefill on multi-turn long context benchmarks such as SCBench.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OkNT0sX4Q1", "reviewer": "Reviewer_USDK", "summary": "This paper introduces Speculative Prefill, an innovative training-free framework that elegantly addresses efficiency challenges in language model inference. By leveraging a lightweight model to speculate on important tokens based on context, the approach impressively enhances time-to-first-token performance. The results are quite remarkable, achieving up to 7x higher queries per second and a 7.66x improvement in TTFT.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Summary\nWe thank the reviewer for the constructive and insightful feedback! We addressed all feedback and believe that incorporating all comments makes this paper significantly stronger: \n\n### Per-feedback detailed response\n> It would be interesting to see more exploration of how it compares with alternative attention sparsity methods such as Minference…\n\nAs the reviewer has mentioned, SpecPrefill not only reduces the computation on the attention but also on the MLP while sparse attention primarily focuses on the attention module alone. In order to provide a more comprehensive comparison between the two routes, we conducted experiments comparing ours against MInference as requested by the reviewer. \n\nWe adopt the pre-searched optimal pattern from MInference’s official repo for the 70B model and benchmark’s its quality on LongBench in T1 (see below). As we can see **SpecPrefill can achieve 99.7% quality of MInference while keeping only 10% tokens and surpasses it when using 30% or more tokens**. Because MInference is an attention based sparsity optimization, we benchmark its latency for a fair comparison in terms of acceleration and quality. In T2 (see below) where we vary the batch size x sequence length products from 128 * 4K to 4 * 128K, we found that **SpecPrefill is particularly effective in efficiency compared to MInference with similar or better quality (getting 2.54x - 6.54x relative speedup compared to MInference)**. The main reason why MInference might fall short is due to the overhead in dynamic index building, which gets significant when the batch size is large. We do want to emphasize that Minference is an excellent work and its advantage starts to shine when the sequence becomes ultra long, at least 128K+ (or the ratio between sequence length and batch size gets large enough). Another trade-offs we want to mention between our methods and sparse attention approaches are the decision of whether we want to compute / store all the KV caches. Besides, MLP generally dominates the prefill part for large batch size + decent context lengths (at least within our tested 128K). This becomes essentially important based on the application scenario (e.g whether the prefill/decode is disaggregated and hence requires KV transmission). Overall, we believe that since both methods require no-training and can be switched on and off on the fly, they are complementary to each other and can hence meet a wider range of requirements when combined together. \n\nWith the added results, we are confident that SpecPrefill can be applied and benefit many real world applications and the contributions are novel and fully reproducible. We’d love to hear from the reviewer on further suggestions if any to make the paper better!\n\n### Newly added experiment results:\n\n**LongBench comparison with baselines (T1):**\n| Model | Compression Rate | Single-Doc QA | Multi-Doc QA | Sum | Few-shot Learning | Code | Synthetic | Avg |\n|---|---|---|---|---|---|---|---|---|\n| baseline 70B | N/A | 50.57 | 53.11 | 25.84 | 66.93 | 52.33 | 72.50 | 53.55 |\n| RAG | 10.90% | 32.32 | 41.17 | 18.86 | 45.40 | 44.76 | 30.42 | 35.49 |\n|  | 28.60% | 38.43 | 47.41 | 21.42 | 50.53 | 45.80 | 35.50 | 39.85 |\n|  | 46.80% | 40.53 | 46.64 | 22.45 | 49.52 | 46.00 | 43.15 | 41.38 |\n|  | 65.10% | 41.40 | 47.43 | 23.30 | 52.21 | 46.19 | 47.22 | 42.96 |\n|  | 83.40% | 43.25 | 48.16 | 23.56 | 51.44 | 45.92 | 53.04 | 44.23 |\n| LLMLingua | ~10% | 26.50 | 32.94 | 20.95 | 37.40 | 45.00 | 16.33 | 29.85 |\n|  | ~30% | 38.83 | 44.02 | 23.37 | 42.23 | 47.27 | 37.00 | 38.79 |\n|  | ~50% | 43.64 | 50.67 | 24.77 | 50.96 | 49.05 | 60.33 | 46.57 |\n|  | ~70% | 45.90 | 52.88 | 25.44 | 59.77 | 51.48 | 68.50 | 50.66 |\n|  | ~90% | 45.94 | 53.91 | 25.87 | 60.46 | 54.06 | 72.00 | 52.04 |\n| Minference | N/A | 50.46 | 53.23 | 25.83 | 66.36 | 52.48 | 69.00 | 52.89 |\n| SpecPrefill | 10% | 47.64 | 52.96 | 21.74 | 64.52 | 63.33 | 66.25 | 52.74 |\n|  | 30% | 49.47 | 53.39 | 24.41 | 65.83 | 62.62 | 67.83 | 53.92 |\n|  | 50% | 50.18 | 52.56 | 25.10 | 65.60 | 59.91 | 68.17 | 53.59 |\n|  | 70% | 50.06 | 52.44 | 25.51 | 65.77 | 58.08 | 68.67 | 53.42 |\n|  | 90% | 50.26 | 53.25 | 25.65 | 66.35 | 53.47 | 70.67 | 53.27 |\n\n**Latency comparison (T2):**\n| | 128 * 4k | 64 * 8k | 32 * 16k | 16 * 32k | 8 * 64k | 4 * 128k |\n|-----------------|----------|---------|----------|----------|---------|----------|\n| 70B Instruct | 22.6 | 23.7 | 25.9 | 30.1 | 38.7 | 56.0 |\n| Minference | 46.6 | 45.3 | 42.5 | 40.8 | 38.3 | 34.6 |\n| SpecPrefill 10% | 7.1 | 7.0 | 7.2 | 8.0 | 9.8 | 13.6 |\n| SpecPrefill 30% | 11.8 | 11.7 | 12.1 | 13.2 | 15.7 | 20.9 |\n| SpecPrefill 50% | 16.2 | 16.4 | 17.1 | 19.0 | 22.7 | 30.7 |\n\n**RULER 64K (T3):**\n|  | Retrieval | Multi-hop | QA | Agg | Avg w/o Agg |\n|---|---|---|---|---|---|\n| baseline 70B | 98.5 | 99.9 | 65.1 | 65.6 | 87.9 |\n| SpecPrefill 10% | 99.5 | 99.8 | 71.9 | 54.9 | 90.4 |", "claims_and_evidence": "The paper presents its claims with well-supported evidence throughout, providing a convincing foundation for the proposed approach.", "methods_and_evaluation_criteria": "The method strikes an excellent balance between simplicity and effectiveness, offering an elegant solution to the prefill optimization problem.\nThe evaluation framework thoughtfully considers both task performance scores and TTFT metrics, which provides a comprehensive view of the inevitable trade-offs between accuracy and efficiency that such approaches entail.", "theoretical_claims": "The theoretical underpinnings of the work appear sound and well-reasoned, with clear connections between the conceptual framework and practical implementation.", "experimental_designs_or_analyses": "The experimental design demonstrates careful consideration across multiple dimensions. The comparison of different token keep percentages on LongBench and Ruler offers valuable insights, while the efficiency evaluations provide a holistic view of the method's practical benefits. Overall, the experimental approach nicely balances breadth and depth.", "relation_to_broader_scientific_literature": "Not addressed in this review.", "essential_references_not_discussed": "I haven't identified any significant omissions in the paper's references.", "other_strengths_and_weaknesses": "Strength:\n1. The speculative prefill concept is particularly compelling, offering an insightful approach to identifying important tokens using a smaller model's predictions as input for the larger model.\n2. The paper presents its ideas with clarity and thoughtful organization.\n3. The experimental section offers comprehensive and nuanced analysis of the method's performance.\n\nWeakness:\n1. While the approach is quite innovative, it would be interesting to see more exploration of how it compares with alternative attention sparsity methods such as minference, especially considering the different optimization strategies they employ.", "other_comments_or_suggestions": "No additional comments.", "questions_for_authors": "1. I'm curious about how this approach might compare with attention sparsity methods like minference. Both approaches optimize different aspects of the inference process, and while this method also addresses MLP computation, I wonder if you could share any insights or perhaps preliminary comparisons on the relative advantages of each approach? This could help readers better understand the positioning of your contribution within the broader landscape of inference optimization techniques.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WcYroFgSSm", "reviewer": "Reviewer_JHsj", "summary": "SpecPrefill proposes a prompt token pruning technique using speculation by a small model before forwarding the large model. This method is especially beneficial when the token length is medium (latency of attention and MLP are similar, so sparse attention methods do not do a good job) because it reduces the number of tokens.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Summary\nWe sincerely thank the reviewer for the effort in providing constructive comments. We addressed all the feedback from the reviewer, and believe that with these changes, the results in our paper become significantly stronger (added experiment result numbers in reviewer USDK’s rebuttal). \n\n### Per-feedback detailed response\n> I cannot find any comparison between baseline.\n\n**In the original submission, we included two RAG baselines where we put the conclusion in the experiment section 4.4 and detailed results in the Appendix B. We now move all results in T1 (reviewer USDK's comment) to the main body, along with two more baselines as suggested by the reviewer**: \n\n1. We evaluate LLMLingua 2 using LongBench. Similar to RAG, LLMLingua does not support query-aware compression, and therefore, we separate the context from the query and send it to the compression model. **In T1 (reviewer USDK's comment), SpecPrefill outperforms LLMLingua and RAG Llama**. \n\n2. We benchmark MInference's quality on LongBench and its inference latency in vLLM for a more comprehensive view. In T1 (reviewer USDK's comment), the quality score of MInference can already be matched very closely with SpecPrefill with 10% token keep rate (0.3% loss) and is outperformed when we increase the rate to 30% and higher. **In T2 (reviewer USDK's comment) where we vary the batch size x sequence length products from 128 * 4K to 4 * 128K, we found that SpecPrefill is particularly effective in efficiency compared to MInference with similar or better quality (getting 2.54x - 6.54x relative speedup compared to MInference)**. The main reason why MInference might fall short is due to the overhead in dynamic index building, which gets significant when the batch size is large (this is also observed in paper https://arxiv.org/abs/2502.05431 and confirmed by the author in issue https://github.com/microsoft/MInference/issues/18). \n\n> I think an ablation study on N should be in the paper. \n\nWe agree that ablation on N will provide more insights and are running a full ablation on N for the camera ready version. **For our current results as in Figure 2 and Figure 5, we showcase the benefits of using LAH with steps = 8 (with LAH as the suffix in model names), which proves its usefulness especially for shorter context tasks**. \n\n> I strongly suggest authors submit the code as a supplementary to find potential practical issues rather than showing it in only papers. \n\nWe agree on this and we will upload code whenever OpenReview allows uploading again. We’re confident that our experiments are reproducible and implementations are highly optimized (see Appendix E). \n\n> I cannot fully agree that MLP is a significant bottleneck, so speeding up attention is not important, as declared in lines 108 and 121. I partially agree that MLP is a bottleneck in a somewhat short context (less than 8K tokens). \n\nSorry for the confusion! We fully agree that attention is a significant bottleneck (in both short-context and especially long context) and SpecPrefill also helps skip the attention part as well, as shown in T2 (reviewer USDK’s comment) when we move on to 128K. **What we hoped to convey was that there are many existing techniques to optimize attention in various regimes but optimizing MLP hasn't attracted too much attention, which is where SpecPrefill stands out**. We revised the wordings to make this point clear. \n\n> I think we should try at least up to 128K tokens... The RULER and InfintieBench benchmark will be a great choice to test the effectiveness. \n\nWe understand the reviewer’s concern on longer context performance. **We increase the RULER results in the original paper from 32K to 64K and observe consistent trends (T3 reviewer USDK’s comment)**. We will present the result on 128K for the camera-ready version and believe the same conclusion to hold (it is still currently running due to large models and limited cards). \n\n> Essential References Not Discussed.\n \nWe thank the reviewer for providing relevant works:\n1. MInference, SnapKV, HiP Attention, SampleAttention, DuoAttention, and FlexPrefill are awesome work. **However, they focus primarily on the attention module while keeping the MLP part intact. We include those in our camera-ready version for more context**. \n2. **For MInference, we added additional experiments to show that SpecPrefill achieves better speedup with better quality**. \n3. **MInference, SnapKV, and DuoAttention were originally in our draft** and we added more context and discussion in our camera-ready version. \n\nWe truly agree with the concerns and feedback from the reviewer and hope our responses can help address some of those. With the added results, we are confident that SpecPrefill can be applied and benefit many real world applications and the contributions are novel and fully reproducible. We’d love to hear from the reviewer on further suggestions if any to make the paper better!", "claims_and_evidence": "They have some evidence, but I cannot find any comparison between baseline. They have only vanilla attention, while they still can compare their method with other pruning methods (such as sparse attention, MInference: https://github.com/microsoft/MInference, FlexPrefill: https://arxiv.org/abs/2502.20766)", "methods_and_evaluation_criteria": "I think the method looks correct. However, the evaluation is weak due to a lack of baselines.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "I think an ablation study on N should be in the paper, which is used in algorithm 1 line 3.", "relation_to_broader_scientific_literature": "The key contribution is about speeding up the QPS of the LLM serving framework, which is widely used in various NLP research. I think this kind of topic is pretty important, and this method is tackling that issue.", "essential_references_not_discussed": "I think they should compare it with the sparse attention method, even if they claim that MLP is a bottleneck in the middle and a little bit longer in context. I cannot fully agree that MLP is a significant bottleneck, so speeding up attention is not important, as declared in lines 108 and 121. I partially agree that MLP is a bottleneck in a somewhat short context (less than 8K tokens). However, these days, many applications require longer context to perform well (e.g., RAG, Image (costs 1-4K tokens per image), and Video (costs 100-1K tokens per frame) model). Therefore, I suggest the  following works (that can speed up prefill) to compare:\n- MInference: https://github.com/microsoft/MInference\n- SnapKV: https://github.com/FasterDecoding/SnapKV\n- HiP Attention: https://github.com/DeepAuto-AI/hip-attention\n- SampleAttention: https://arxiv.org/html/2406.15486\n- DuoAttention: https://github.com/mit-han-lab/duo-attention\n- FlexPrefill: https://openreview.net/forum?id=OfjIlbelrT", "other_strengths_and_weaknesses": "I think the tested context lengths are too short compared to other modern LLM (Qwen 2.5=1M tokens, Llama 3.1=128K tokens ...) I think we should try at least up to 128K tokens, which is the context length of Llama3.1. The RULER and InfintieBench (https://github.com/OpenBMB/InfiniteBench) benchmark will be a great choice to test the effectiveness", "other_comments_or_suggestions": "Please refer to other sections.", "questions_for_authors": "Please refer to other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: I lean toward accepting this paper. While concerns about baseline comparisons were valid, the authors thoroughly addressed these in their rebuttal by adding LLMLingua and MInference comparisons, extending evaluations to longer contexts, and providing their code. The paper tackles the important TTFT problem with a novel approach that achieves substantial speedups while maintaining quality. The method's effectiveness across model scales and its focus on optimizing both attention and MLP computation represent valuable contributions to efficient LLM inference.
Decision: Accept (poster)