Draft-based Approximate Inference for LLMs
Authors: Kevin Galim
Ethan Ewer
Wonjun Kang
Minjae Lee
Hyung Il Koo
Kangwook Lee
Venue: ICLR 2026 Poster (2026)
Keywords: long-context
sparse attention
KV cache eviction
prompt compression
Primary area: foundation or frontier models, including LLMs

Abstract: Optimizing inference for long-context large language models (LLMs) is increasingly important due to the quadratic compute and linear memory cost of Transformers. Existing approximate inference methods, including key-value (KV) cache dropping, sparse attention, and prompt compression, typically rely on coarse predictions of token or KV pair importance. We unify and extend recent work by introducing a framework for approximate LLM inference that leverages small draft models to more accurately predict token and KV pair importance. We provide novel theoretical and empirical analyses justifying lookahead-based importance estimation techniques. Within this framework, we present: (i) **SpecKV**, the first method to use lookahead with a small draft model to enable precise KV cache dropping; (ii) **SpecPC**, which leverages draft model attention activations to identify and discard less important prompt tokens; and (iii) **SpecKV-PC**, a cascaded compression strategy combining both techniques. Extensive experiments on long-context benchmarks demonstrate that our methods consistently achieve higher accuracy than existing baselines while retaining the same efficiency gains in memory usage, latency, and throughput.

Published as a conference paper at ICLR 2026
DRAFT-BASED APPROXIMATE INFERENCE FOR LLMs
Kevin Galim 1+ Ethan Ewer?* Wonjun Kang
Minjae Leeb
Hyung Koo
Kangwook Lee2.5,6
FuriosaAl PUW-Madison
SSoul National University
"joou University
KKRAFOON
Ludo Robotics
(kevin. galim, kangwj1999 1995, minjae. lee, @ (eewer, kangwook Iee) @wisc.
Code:
att.auuriiiiaaaaaaataaaaaaaaaaaa approx-11 11m
ABSTRACT
Optimizing inference for long-comien large language models (LLMs) is increas- ingly important due to the quadratic compute and linear memory cost of Trans- formers. Existing approximate inference methods, including key-value (KV) cache dropping, sparse attention, and prompt compression, typically rely on coarse pre- dictions of token or KV pair importance. We unify and extend recent work by introducing a framework for approximate LLM inference that leverages small draft models to more accurately predict token and KV pair importance. We provide novel theoretical and empirical analyses justifying lookahead- based importance estimation techniques. Within this framework, we present: (i) SpecKV, the first method to use lookahead with a small draft model to enable precise KV cache drop- ping; (ii) SpeePC, which leverages draft model attention activations identify and discard less important prompt tokens; and (iii) SpecK V-PC, cascaded compres- sion strategy combining both techniques. Extensive experiments on long-context benchmarks demonstrate that our methods consistently achieve higher accuracy than existing baselines while retaining the same efficiency gains in memory usage, latency, and throughput.
INTRODUCTION
The demand for longer context lengths in large language models (LLMs) (Achiam et al. 2023 Google
DeepMind] 2025) continues to grow (Liuu al. 2024b), driven by applications such as dialogue
systems (Achiam et al., 2023 Google DeepMind 2025). document summarization (Liu et al. 2020), and code completion (Du al, (202)). Modern models like GPT-4 (Achiam et al. 2023) and Gemini 2.5 Pro (Google Mind (225)) have pushed context windows to over million tokens. However, scaling Transtormers (Vaswani et al.] 2017) to these lengths remains difficult due to significant computational and memory constraints. Attention computation scales quadratically with context length, increasing inference latency, while key-value (KV) cache memory grows linearly, straining GPU resources. For example, caching the KV states for 128K tokens in Llama-3. 1-8B (Grattafiori et al. 2024) can consume over 16GB of memory, limiting the practical scalability of LLMs.
To address scalability challenges, recent work introduces approximate LLM inference techniques that reduce latency and memory usage at inference time. Techniques include sparse attention for prefilling (Jiang et al.) (2024)) and decoding (Tang al. (024), which speed up inference by having each query attend to only a subset of keys. Sparse prefilling shortens time to the first token, while
sparse decoding boosts generation throughput. KV cache dropping (Cai al. 2024b Li et al. 2024 Xiao et al., 2024 Zhang et al. 2023) reduces memory and increases throughput by shrinking the cache after prefilling or during decoding. Prompt compression (Choi et al. 20244) Jiang et al.) 2023b
Liskavets et al. 2025) further improves efficiency by removing less important tokens before inputting the prompt, reducing both attention and MLP computation, as well as decreasing KV cache size.
"Equal contribution.

---

Published as a conference paper at ICLR 2026
100
SpreKV (ound
Current approsch
Use iepus to ideatity Importart Impur tokrms
Onaele
mue ourpuss idennity
RULER Score
LAQ- (W2255)
impoctant loput tekens
1220 (2h/23)
inspoctatice
80
SmagY (L'
Target Model
Pyrky (Ca/24))
ipput
cutpui
Ours
Use appcocinatee suiputs
Wh
idmilfy importani impui tokens
4k
8k
16k
32k
64k
Draft Model
Sequence Length
approsimatc
(a) Proposed Framework.
(b) Impact of lookahead (LA) on accuracy.
Figure 1: (a) Overview of our Draft-based Approximate Inference framework for input token im- portance estimation in comparison with current approaches and the oracle approach. Prior methods use input tokens to estimate input token importance. Our approach incorporates draft model pre- dictions of future output tokens, yielding more accurate importance estimates. This better aligns with the hypothetical oracle setting, where the true output is known and influential tokens can be precisely identified. (b) On RULER with Llama- 3-B0B (Grattafiori et al. (002)), lookahead- based methods (LAQ++ (Wang al. 2025), SpecKV) significantly outperform non-lookahead approaches (H20 (Zhang et 2023), SnapKV (Li et al.) 2024). PyramidK (Cai et al. (024b)), with our proposed SpecKV achieving the best overall downstream score.
Orthogonally, speculative decoding (Cai et al. 2024a Chen et al. 2023 Hu et al. 2025 Leviathan
at.l. 2023) accelerates L.LM inference by using small draft model propose sequence of multiple tokens, which the target model verifies in parallel. This improves throughput without altering the output distribution and is particularly effective for autoregressive models, where sequential generation is a bottleneck. However, unlike approximate inference, speculative decoding does not lower the total memory or computation requirements and struggles with increasing context length.
In contrast, approximate LLM inference improves efficiency by reducing the amount of computation the model performs. This is often done by estimating the importance of each token or KV pair for future generation and discarding less important ones from attention or feedforward computations. Existing methods (Cai et al., 2024b, Feng et al. 2025 Tiang al., 2024a Li et al.) (2024) use attention activations from input tokens to predict which tokens or KV pairs future tokens will attend to, as future tokens are not yet available. However, input attention activations alone do not reliably identify the tokens or KV pairs most relevant for future token generation. Recent methods (Liu et al. 2025 Wang et al. 2025) address this by leveraging an approximate output to improve importance estimates. In this work. we unify and extend these techniques by introducing Draft- based Approximate Inference, a lookahead-based framework that uses an inexpensive draft model to approximate future outputs with minimal overhead (Fig. Tab. Our main contributions
follows:
1. We present for enhanced approximate inference.
Approximate Inference, a framework using draft model lookahead
2. We present theoretical and empirical analyses justifying lookahead-based KV cache dropping and the use of draft model token importance approximate target model token importance.
3. Within the Draft-based Approximate Inference framework, we develop two concrete algo- rithms targeting three LLM inference optimizations: Speculative KV Dropping (SpecKV) for KV cache dropping with sparse prefill, and Speculative Prompt Compression (SpecPC) for prompt compression. Notably, SpecKV V is the first to use draft model lookahead for KV cache optimization. Additionally, we introduce SpecK V-PC, a cascaded pipeline integrating both algorithms to achieve superior accuracy, latency, and memory efficiency.
4. We perform comprehensive experiments on long-context benchmarks, demonstrating that our methods attain state-of-the-art accuracy under fixed KV cache or prompt size constraints. Our results consistently outperform prior baselines, underscoring the potential of draft models for fast and accurate approximate inference in large language models.
2

---

Published as a conference paper at ICLR 2026
Table 1: Summary of prior work. Complexity reported without auxiliary/draft model. nin and nout denote the number of input and output tokens, respectively. Cimx represents the maximum KV cache or prompt capacity. Sprefill and Sdecode indicate the number of keys each query attends during the prefill and decoding phases, respectively. L is the number of attention layers. Since all time complexities are linear with respect to L, we only include L for space complexity. Highlighted cells indicate improved complexity.
Type
Method
Spurse Alta
KV Deupping
Prefil Time
Decoding Time
Prefill Space
Decoding Space
Dessc
Olosa)
Mow.))
O(Low)
ChAAjs Noal)
Miteermess FeePrelll
Poetill
Olouhwas)
Olnailta + Noel)
O(Lon)
Cilipaa muel
Spass ateenion
Quer, RetizvalAncation
Decod:
X
Olsa²)
O(low)
Chlisa+ +
LM
Postill
Decole
OhouCiua
Olnabiual
Olaan(ta (CCana)
O(
O(naa)
Oina
Ofract F.C..
Oltfinml
KV dupppnn
Stap&V, PyrmmiKV, AddKV LAQ
X
After prefil
Olina)
Olteaa + Hewa)
Ofzaxitta LCmual
OLL(Cou + Real))
x
After prefil
Oleaa)
Oineal Hoal)
OLdiCiun heal))
SprcKY (fies)
Poetill
Ova Buall
Ofrrantt, MCa.ll
Of Roal)
Prouutt commpeaciion
ingux-2,
SpeePC (Ours)
Of
Ofrealoin Foal))
O(tCra)
OL1(C Newl)
2 RELATED WORK
Sparse Attention One way to improve inference efficiency is through sparse attention with static patterns. For example, sliding window attention (Beltagy et al., 2020), used in models like Mis- tral 7B (Jiang et al. (023a). Gemma 3 (Gemma Team 2025). GPT-3 (Brown et al.) (020), and gpt-oss (Agarwal al.] 2025), restricts each query to attend only fixed-size window of recent keys, reducing computation and KV cache size during decoding. StreamingI.LM (Xiao et al.) 2024) improves on sliding window by using initial tokens, called attention sinks, along with the sliding window. MInference (Jiang et al. (2024a), adopted by Qwen2.5-1M (Yang al. 2025b), further boosts prefill efficiency by searching offline for adaptive sparse attention patterns (A-shape, Vertical-Slas and Block-Sparse assigned per head. FlexPrefill (Lai et al. 2025 extends this idea by determining sparsity rates for each input prompt. In contrast, Quest (Tang et al., 2024) and RetrievalAtteentin (Liu al. 20244) target the decoding stage by only retrieving the most important KV pairs from the cache, reducing both memory bandwidth and computational demands during generation.
KV Cache Dropping KV dropping reduces computation and memory during decoding. Sliding window attention (Beltagy et al. (020) and StreamingLLM (Xiao et al.) (024) are examples of KV dropping methods (as well as sparse attention) as they permanently evict KV pairs from cache. H2O (Zhang et al., 2023) improves on this by dynamically selecting attention sinks, termed heavy- hitters, using attention scores at each decoding step, while also maintaining sliding window. SnapKV (Li et al.) 2024) compresses the KV cache at the end of the prefill stage by dropping unimportant KV pairs. Subsequent work extends this idea by allocating KV cache budgets non- uniformly across layers (PyramidKV (Cai et al., 2024b) ) and attention heads (AdaKV (Feng et al.. 2025), HeadKV (Fu et al.) 2025)). However, these approaches drop tokens based only current information, making them less robust to changes in token importance over time (Nawrot et al. 2025). Recently, Wang et al. (2025) proposed Lookahead Q-Cache (LAQ++), which addresses this by generating draft queries with a sparse approximation of the target model, using them to compute more accurate importance scores, though this comes at the cost of no reduction in peak memory usage.
Prompt Compression Prompt compression removes tokens before reaching the model, reducing compute and memory usage during both prefill and decoding, unlike KV dropping, which speeds up only decoding. It also surpasses sparse attention by saving both attention and MI. computation. Prompt compression works seamlessly with all types of inference setups, such as APIs or inference engines like vLLM (Kwon et al. 2023). since it does not require any modifications to the model. However, KV dropping can achieve higher compression because it selects tokens per head, while prompt compression drops the same tokens across all layers and heads.
In a question- answer setup, prompt compression may be question-agnosti. (compressing context without considering the question) or question-awee (factoring in the question). Selective context (Li ct al. (023) and LLMLingua (Jiang et al., 20236) are training-free free, -oeestionaanossicc approaches using small I.LM keep only key tokens. LongI.LML.inuua (Jiang al.] 2024b) adapts this
3

---

Published as a conference paper at ICLR 2026
for longer contexts in question-awan manner. LLMLingua-2 (ann al., 2024) trains small model (Conneau et al. 2020) to score token importance without using the question. CPC (Liskavets) et al. 2025) uses trained encoder to compute sentence importance via cosine similarity with the question, while R2C (Choi et al. 2024) splits the prompt into chunks, processes each with the question using fine-tuned encoder-decoder Transformer (FiD (Izacard & Grave 202T)), and ranks them via cross-attention. Similar to our proposed SpeePC, SpecPrefill (Liu et al., (2025) leverages attention scores from smaller draft model to identify important tokens, using the draft model for lookahead to improve token importance estimates.
Speculative Decoding Speculative decoding Cai et al. 2024a; Chen et al. 2023 Hu et al. 2025 Leviathan et al., 2023) accelerates LLM inference by using small draft model propose multiple tokens that the target model verifies parallel. This increases decoding throughput without changing the output distribution, addressing the bottleneck of autoregressive generation. Previous work further accelerates speculative decoding by enabling approximate inference in the draft model, using techniques such as sparse attention (Sadhukhan al. 2025), KV cache dropping (Sun al., (024), or KV cache quantization (Tiaari et al. 2202)) all while preserving exact inference. In contrast, our approach leverages draft models to enable fast, approximate inference directly in the target model.
Table I
summarizes prior work, highlighting their prefill, decoding time, and memory complexities.
3 PROPOSED FRAMEWORK: DRAFT-BASED APPROXIMATE INFERENCE
Previous LLM approximation methods (Cai et al. 2024b Feng et al. 2025 Jiang et al. 2024a Li
et al. 2024) estimate the importance of current tokens on future generation by analyzing current attention patterns. While this can be effective, it provides only rough estimate of each token's importance. contrast, if future tokens were available, we could make substantially better importance estimates by directly identifying which input tokens contribute generating those output tokens. However, these future tokens are inaccessible before generation.
Recent work has explored using approximate future information improve token importance estimation. LAQ++ (Wang et al. 2025) extends SnapKV (Li et al. 2024 by generating draft queries and then using them compute more accurate importance scores. SpecPrefill (Liu al. 2025) generates lookahead tokens with a small draft model and relies on the draft model's 's attention to those tokens to estimate input token importance. We bring these ideas together under a unified framework. Draft-based Approximate Inference that leverages approximate future information to improve token importance estimation. We further extend this framework with two new algorithms: SpecKV for KV cache dropping and sparse prefilling (Section 4.1 and SpecPC for prompt compression (Section 4.2) Finally, we introduce cascaded compression with VVPCKV-PC, combining both approaches into single pipeline for superior accuracy and efficiency (Section
While our methods use draft models, technique also common in speculative decoding, our objective fundamentally different. Speculative decoding improves hardware utilization by having draft model propose tokens that the target model verifies, accelerating generation without changing the output distribution. However, it does not reduce total computation memory usage. In contrast, framework reduces computation and memory costs by approximating the target model.
3.1 JUSTIFICATION FOR LOOKAHEAD-BANE BASED KV CACHE DROPPING
KV cache dropping requires estimating the importance of each input KV pair. We define importance as the average attention activation from output queries to each input key. Specifically, the vector of importance scores and its approximation are given by
Nost
Softmax
(o) W, W,w? x
and ST
Rest
Softmax
L W. WT
X
Vd
(1)
4

---

Published as a conference paper at ICLR 2026
Crax
85
Gmax 1024
el.sff 0B
SpecKV
A = 0.97
1.31
1:1
RULER Score
80
LAQ++
Token Importance (14B)
0.8
85.0
716
3B
0.6
RUILER Score
75
82.5
1024
0.4
80.0
70-
256
11
m128
0.2
0.58
65
0.0
75.0
-158.4bit
10
15
0.0
0.5
1.0
0.85
0.90
0.95
1.00
Token Importance (1.58)
(a) Score vs. e (32B Target)
(b) Importance Corr. (1.5B vs. 14B)
(c) Score vs. R? (14B Target)
Figure Experimental validation RULER-32K tasks samples each) using Qwen2.5 models. (a) Lower enror (Eq. (2) yields higher downstream scores. Increasing the draft model size (SpecKV) or initial eache size (LAQ++] reduces c, with SpecKV outperforming LAQ++. (b) Importance scores T
(as used in SpecPC) of the draft and target models are highly correlated. (c) For SpecPC, larger draft model improves both the token importance correlation (R²) and the final task performance.
where X - E Roaxd is the matrix of input embeddings, _.!) E Rd the input embedding from the ith output token, and a!")) Fd is its approximation. 8; and & denote the importance of the ith KV pair.
To understand when bookahead-basee KV cache dropping algorithms provide reliable importance estimates, we analyze how error in approximate input embeddings impacts enror importance score estimates for single attention head.
2| S e for all and || S for all j. then I|s 8 S < <|IW, elW,, WT11 12 Theorem 1. If [|= -!
This result shows that for a single attention layer, the worst-case error in the approximate importance scores is proportional to the worst-case error in the approximate input embeddings, implying that lookahead-based KV cache dropping algorithms provide reliable importance estimates as long as the draft model remains reasonably accurate (see Section B. for proof).
To evaluate the quality of different draft outputs, we compute e for two algorithms (LAQ++ and SpecKV). While Theorem assumes strict token-wise alignment (where each draft token a!()) (o)) ), in practice, the draft and target models often generate sequences of different lengths (Nour * nout). To address this mismatch, we relax the strict point-wise condition and empirically evaluate draft quality using the global distance between the centroids of the output sequences. We compute e as:
corresponds to a target token o
(0)
(2)
Pouul
nout
This metric serves as a practical proxy for the distributional mismatch between the target and draft outputs. We report averaged across all attention layers. As shown in Fig. [2 increasing draft quality (decreasing e) leads to higher downstream accuracy. Furthermore, Section E.4 demonstrates how S'ecKV's s lookahead mechanism improves importance score correlation and accuracy, particularly for long outputs.
3.2 JUSTIFICATION FOR DRAFT- DRAFT-BASED BASED PROMPT COMPRESSION
Most KV cache dropping algorithms estimate token importance using attention scores from the target model itself. While effective, this approach is too computationally expensive for tasks like prompt compression, where we need to estimate token importance without forwarding the entire
For LAQ++, initial cache size refers the value of Cmax used during the sparse draft generation.
5

---

Published as a conference paper at ICLR 2026
input through the target model. A more efficient alternative is to smaller draft model for this estimation. To justify this approach, we study how the similarity between the draft and target models' outputs correlates with the similarity of their attention activations in single attention layer.
The target model attention layer uses weights Wg. Wk. and W,, and the draft model attention layer uses We, Wr and W,. Let the input prompt be X = [Z],.... x,]" E Raxd The outputs of the target attention layer and its approximation are
Y Softmax
XW, X Vd
XW, AXW,
Ý Softmax
XW. X Vd
XW, AXW.
where X [21,- Tnn ! E RRad A [a
On T is the attention matrix, and À =
[a1, (0,,...;,,. On)
[T is the approximate attention matrix.
the scaled inputs satisfy the Restricted Isometry Property (RIP CCandes & Tao, 2005 a condition widely studied in compressed sensing to ensure the stable recovery of sparse signals, can establish the following bound:
Theorem If there exists a constant such that eXT satisfies the Restricted Isometry Property with parameters (2k, 5), where å is the restricted isometry constant and l is the approximate sparsity of a, and à, and the output envor satisfies 18 - Uill2 I <|| then the attention envor satisfies ||a: - à: <
20e|| 1
Omin Omin(W.)))
This result offers a surprising and elegant connection: it reveals that mathematical tools developed for compressed sensing can also bound the error in attention approximations. Specifically, it shows that the worst-case error in the approximate attention activations is proportional to the worst-case error in the approximate outputs, with the constant depending the conditioning of the weight matrices and the maximum input embedding norm. This implies that if the draft model provides a reasonable approximation of the output, also gives a reasonable approximation of the attention activations (see Section B.2 for proof). Furthermore, even if the scaled inputs do not satisfy the RIP, we can still bound the attention approximation error by applying Theorem 3 (see Section B.3 for proof).
In addition to the theoretical analysis, we examine the correlation between the importance scores (as computed by SpecPC) for Qwen2.5-Instrucc (1.5B draft, 14B as target). In Fig. 2b we plot the draft importance scores against the corresponding target importance scores. The results reveal a strong correlation, supporting the use of draft attention activations to approximate target token importance. Furthermore, this correlation strengthens the draft model size increases (Fig. 2c
4 DRAFT-BASED APPROXIMATE INFERENCE METHODS
4.1 SPECKV: ROBUST IMPORTANCE ESTIMATION FOR KV CACHE DROPPING
Existing sparse attention and KV cache dropping methods, such as SnapKV, estimate token importance by analyzing recent attention activations. This approach can be inaccurate when the set of important KV pairs shifts during generation, as past patterns do not always predict future ones. We argue that a more robust estimate be derived from the attention activations of draft queries for future tokens.
LAQ++ attempts this by generating draft queries from the target model using an initially compressed cache. These queries are then used to compute more accurate importance scores for second, more informed compression pass, However, this two-pass method provides no reduction in peak memory because it must store the entire original KV cache of the target model avoid recomputing the expensive prefill stage.
To overcome this limitation, we propose SpecKV. Our method employs lightweight draft model to generate the draft output, which substantially reduces the cost of the lookahead step. This enables an accurate compression of the KV cache without sacrificing peak memory reduction.
The input embedding matrix may satisfy the RIP its entries are approximately uniformly or normally distributed, RIP can also hold with positional embeddings constructed from Fourier basis. II || denotes the maximum &a norm of X's rows; oain (W.) the smallest singular value of W,.

---

Published as a conference paper at ICLR 2026
SnapKV Importance Scores
SpecKV Importance Scores (Ours)
Draft Model
prompt
window
prompi
prompi
oulpul
Figure 3: Overview of SpecKV: Instead of using only the last prompt tokens like SnapK SpecKV employs a lightweight draft model to generate lookahead tokens, providing richer context for more accurate KV importance estimation. Tokens in window are always retained.
SpecKV (Algorithm!] [T) begins by generating a draft output of length Toookahead using a small draft model, which acts proxy for the target model's s future outputs. During prefilling, both the input tokens and the draft tokens are passed through the target model. For each attention head, we compute token importance scores by measuring the cross-attention between the queries from the last Awindow input tokens and the draft output tokens to the remaining input keys (Fig. 3}. We apply local pooling with kernel size k the attention scores maintain continuity. These scores guide two optimizations: sparse prefilling and KV cache dropping. For sparse prefilling, we use a variation of the Vertical-Slash kernel pattern introduced in (Jiang al. 2024a). For KV cache dropping, we retain the top Caa Awindow KV pairs with the highest importance scores, along with the final Rwindow KV pairs from the most recent tokens.
4.2 SPECPC: LEVERAGING DRAFT MODELS FOR EFFICIENT PROMPT COMPRESSION
SpecKV leverages the draft model outputs to enable more effective KV cache dropping. However, greater efficiency gains are possible by leveraging more information from the draft model. As demonstrated in Section 3.2, draft model attention scores serve as reliable estimates of target token importance. Building on this insight, we introduce SpecPC, an extension of SpecPrefill (Liu et al. 2025), which compresses the prompt to reduce latency and memory usage during both prefilling and decoding, surpassing the efficiency benefits of traditional KV cache dropping.
SpecPC (Algorithm 2 feeds an input prompt (length nin) to the draft model and directly extracts its attention activations E IR "luyer X Rieed x Piodieed x)x, where nayer and Thead denote the number of layers and heads. These activations indicate token importance and are used to drop less relevant tokens from the prompt.
SpecPrefill (Liu et al., 2025) uses window size of Tewindow = 1, meaning it relies on attention scores from queries associated with the last input token and Mlookahead draft output tokens to compute token importance. However, input queries used for importance estimation) lask-dependent, with some tasks benefiting from additional input queries. To address this, adopt large window with non-uniform token weights, placing greater emphasis on tokens near the prompt's 's end achieve robust, task-wide performance.
experiments show that the optimal choice of Rowinow (the number of
Window tokens are reweighted so that the jth token from the end receives weight aggregation is performed across layers, heads, and queries to produce a single importance score per token (excluding the always-kept last Towindww tokens). Additionally, as PyramidKV (Cai et al., 2024b) showed that attention is more focused in deeper layers, we exclude the first lkip layers during aggregation.
Thendow - ((--)) Max Rheadow
We apply average, then max pooling, so selected tokens also include nearby context, avoiding static chunking that could split related tokens (e.g., key-value pairs). This maintains the local context LLMs require. Unlike other methods that select entire sentences, we avoid sentence-level pre- pre-processing to support non-text inputs, such as images. We then select the top-Cmax tokens with the highest scores, always including window tokens, form the compressed prompt. We pass the compressed prompt to the target model with reassigned contiguous position IDs, enabling SpeePC to surpass the model's ''s maximum context length.
7

---

Published as a conference paper at ICLR 2026
output
SpeeKV
Target Model
lookahead tokens
SpecPC
Draft Model
token importance
compressed input
input
Figure 4: Overview cascaded compression with SpecKV-PC: First, the draft model produces token importance scores and lookahead tokens. Next, SpecPC uses these scores compress the initial input prompt. Finally, the target model prefilled using both the compressed prompt and the lookahead tokens, while SpecKV compresses its KV cache
4.3 SPECKV-PC: CASCADED COMPRESSION WITH SPECKV AND SPECPC
SpPcKV-PC integrates SpecKV and SpecPC into a highly efficient, two-stage compression pipeline. The core strategy leverages cascaded approach, which first compresses the prompt with SpecPC and then further compresses the KV cache with SpecKV (Fig. 4.. Because of fewer target model activations. SpecKV-PC achieves substantially lower latency and smaller memory footprint than SpecKV alone, as the computationally intensive target model processes only fraction of the original prompt.
EXPERIMENTS
5.1 SETUP
We benchmark SpecKV and SpecPC against baselines on RULER (Hsieh et al. (024) and Long- Bench (Bai et al. 2024) using two model pairs: Qwen2.5-Instre (Yang et al.) 2024) (14B target with BB drafts for KV dropping/prompt compression) and Llama-3-Instrict (Grattafiori et al. 2024 (3.1-70B 4-bit target with 3.2-3B/IB drafts, respectively).
RULER is synthetic benchmark with 13 tasks of varying complexity, including tasks such as key-value retrieval (NIAH), mulli--hop tracing, and aggregation. It can be generated at any sequence length to assess model's effective context window. We evaluate at 4K, 8K, 16K, 32K, and 64K (Qwen is excluded at 64K due to its 32K sequence limit). LongBench contains 12 English, five Chinese, two code, and three synthetic tasks across five categories. We exclude the Chinese tasks (unsupported by Llama) and synthetic tasks (already covered by RULER). We select 50 examples from each task, yielding 700 examples from LongBench and 650 examples each context length for RULER.
For SV we compare against KV dropping methods H2O (Zhang al. 2023), SnapKV Li et al. 2024), PyramidKV (Cai et al. 2024b). and LAQ++ (Wang et al. 2025), using Cmax = 256. Additionally, we assess SpecK V-PC- that first compresses to 2048 tokens via SpecPC before applying SpecKV (Cmax 256). For SpecPC, we benchmark against LLMLingua-v2 (Pan et al.)
2024). CPC (Liskavets et al. 2025), R2C (Choi et al. 2024), and SpecPrefill (Liu et al. (025) with C 1024. Based
ablation studies (Fig. 19 we set Floookhhedd the maximum token limit for SpecKV and to one for SpecPC. For SpecPrefill and LAQ++, we use Nigokabeed = as in the official paper. Sections Coo contain datasets, details, and additional results, including cross-family evaluations, different max settings, and multimodal experiments on MileBench (Dingjie et al. 2024
with Qwen2.5-VI. (Bai et al. 2025
8

---

Published as a conference paper at ICLR 2026
Qwen2.5 32B
Llama-3 70B (4-bit)
Qwen2.5 32B
Llama-3 70B (4-bit)
100
90
90
Score
Sowce
80
70
70
60
4k
4k
8k
16k
32k
4k
Sk
16k.
32k
4k
8k
16k
32k
6/k
Sequennce I onggth
Sequence I Lengtt
Sequence Length
Sequence Length
SprckCy
Dmiht
1E00
Pysamdyy
SpeePC
Target
CPC
SpecPretti
SpccKY-PC-298
Tarnet
SrapkV
LAQ-+
Deat
LIMiLingsa2
R2C
KV dropping (Caax 256)
(b) Prompt compression (Cmax 1024)
Figure 5: Performance of SpecKV and SpecPC. Both methods consistently outperform all baselines across sequence lengths, maintaining strong results longer contexts. SpecK V-PC further improves upon SpecKV to achieve state-of-the-ar results for KV dropping. Note that H2O and PyramidKV are not plotted for Qwen2.5 32B as their performance falls outside the visible range.
5.2 RESULTS
Fig. 5 (RULER) and Table 2 (LongBench) compare our methods with baselines. All our methods consistently outperform other baselines, demonstrating superior KV cache and prompt compression. Their performance far exceeds the draft model, highlighting robustness even with weaker drafts. Performance improves further with better drafts (Fig. [8).. On RULER and LongBench, SpecKV and SpecPC consistently surpass existing baselines across multiple model families and context lengths. Interestingly, SpecK V-PC outperforms SpecKV V alone, suggesting that the initial SpecPC-based prompt compression acts
effective pre-filter by removing casy-to-identify, unimportant tokens. For larger Cmax. our methods remain superior (Figs. [O] [T] Tables too [T). Finally, we provide additional results regarding V (Section
B.2 multimodal tasks (Section B.3 cross-family
settings (Section E.5). and SpecK V-PC prompt compression ratios (Section E.6
Table 2: LongBench performance with Qwen2.5 and Llama-3.
Qwen2.5 32B
Llama-3 70B (4-bit)
SingleG4
MatrOA
Fow-sbot
SingleQA
MutigA
Fencabot
Summ.
Code
Samm
Codee
Group
Method
Dense
Turget
56.01
43.99
25.90
64.06
44.74
47.78
55.02
47.06
28.61
70.47
48.19
49.99
H2O
46.63
30.81
19.88
56.03
39.27
39.32
54.07
41.30
23.55
49.10
54.14
43.52
SnupKV
KV
PyramidKV
50.92
37.26
18.90
63.24
40.20
43.19
55.41
45.59
22.50
59.06
49.90
46.25
LAQE
55.15
44.14
22.24
63.25
11.19
45.79
$4.90
46.48
22.83
64.31
55. 10
48.43
SpecKV
SpecKV-PC-2045
52.60
44.52
24.11
63.38
48.45
46.48
61.42
47.15
26.51
66.94
$8.19
$1.60
33,83
26.39
22.85
32,46
43.01
30.90
37.95
28.20
23.35
42.37
37.63
33.63
R2C
50.49
40.37
23.26
53.45
34.11
39.88
48.93
42.01
25.38
58.91
40.19
43.29
SpeePrefill
45.94
39.32
23.16
62.04
43.17
42.70
$4.62
46.43
25.63
64.30
44.92
48.37
SpeePC
41.40
62.26
5.3 EFFICIENCY
We evaluate latency and memory on a single NVIDIA H200 (141GB) GPU. The target model Qwen2.5-32B, with draft models of Qwen2.5-1.5h for KV dropping and Qwen2.5-0.5B for prompt compression. Latency is measured as the time to generate tokens including all draft stages, with Tloookahaad 64 for SpecKV and SpecKV-PC" 8 for SpecPrefill and LAQ++, and for SpecPC, with SpecKV-PC prompts compressed to 2048 tokens. For KV dropping methods, we report peak system memory, while for prompt compression, report memory for the compression stage only,
WWilee tasks can generate up 128 tokens, benchmark outputs typically under 64 tokens.

---

Published as a conference paper at ICLR 2026
because the target model uses the same amount of memory for all prompt compression algorithms. A detailed breakdown of latency and memory is included in Section E.7
SpecKV outperforms both SnapKV and LAQ++ in speed, driven by efficient sparse prefilling and low-latency draft model (Fig. 6). Notably, SpecKV-PC achieves a 75% reduction in latency compared to LAQ++ 64k context, it significantly reduces the target prefill bottleneck through prompt compression. Regarding memory, while SpecKV requires more memory than SnapKV V to store draft weights, this overhead is constant and can be further reduced by offloading to a CPU. Crucially, SpecKV is far more memory-efficiee function, matching the memory footprint of the dense target model. Thus, SpecKV offers superior combination of accuracy and memory efficiency. Furthermore, SpecKV-PC yields substantial peak memory savings (around 25GB compared to LAQ++ 64k context) by feeding shorter prompt to the target model. Other KV dropping methods are omitted as their performance is similar to SnapK V.
than LAQ++ which needs the entire target KV cache to
SpeePC achieves the lowest latency among all baselines. avoids the CPU preprocessing overhead of CPC and R2C and is faster than SpecPrefill due to shorter lookahead. SpecPC is also the most memory-efficient, using substantially less memory than R2C. Overall, prompt compression faster than KV dropping because only the compressed prompt tokens (Cmax) are passed to the target model.
Memory
Latency
Memory (Compress)
Latency
Peak Mcnsay (GBB)
90
Peak Memory (GB)
10.0
80
Time (4)
10
7.5
5
5.0
60
2.5
0k:
25k
50k
25k
50k
25k
50k
25k
50k
Sequence Length
Sequence Length
Sequence Length
Sequence Length
SprrKV
SprcKWPCC
Targe SupRV
1.Q4+
SpeeIC
SpecPhoffl
SpecKV
Deal.
LLMLingus
R2C
(a) SpecKV memory and latency.
(b) SpeePC memory and latency.
Figure 6: Peak memory usage and latency. SpecK denotes SpecKV without sparse prefill.
DISCUSSION
In this paper, we present Draft-based Approximate Inference, a framework that leverages draft model lookahead for approximate inference. Within this framework, introduce concrete algorithms: Speculative KV Dropping (SpecKV) for KV cache dropping with sparse prefill, and Speculative Prompt Compression (SpecPC) for prompt compression. We further propose SpecK cascaded pipeline that synergizes these approaches for improved efficiency and accuracy. Our approach is grounded in theoretical and empirical analyses that justify lookahead based KV cache dropping and the use of draft model token importance to approximate target model importance. Through comprehensive experiments on long-context benchmarks, we show that our methods consistently achieve state-of-the-art accuracy under fixed KV cache and prompt size constraints, surpassing prior baselines. These contributions establish draft model lookahead as a powerful tool for efficient long-context inference, extending the role of draft models beyond speculative decoding.
Limitations and Future Work For SpecKV, draft generation causes minimal latency even for larger Teookaheud (Section E.7). However, very long outputs or large Nhookaheud values may reduce performance. In these cases, lowering Mlookahead could maintain speed with little loss in accuracy. For SpecPC, increasing Ylookahead generate more tokens led to only minor accuracy gains; better leveraging longer drafts remains future work. Additionally, while our methods are robust using very small (Fig. 18 or cross-family (Section E.5) draft models, a reasonably accurate draft model is still required. Currently, Draft-basel Approximate Inference supports sparse prefill, KV dropping, and prompt compression. Extensions such as lookahead-baat based sparse decoding iterative KV cache dropping, where KV entries are periodically removed using draft lookahead, could further improve support for reasoning models with long outputs.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide a link to our public code. All algorithms, datasets, and experimental details, including hyperparameter settings, can be found in Section 3 and the appendix (Sections
C and D
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint, abs/2303.08774, 2023. URL https: arxiv. org/ abs http://ab///333.33 2tt.:/333...7777
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, al. GPT-OSS- GPT-OSS-20B model card. arXiv preprini, abs/2508. 10925, 2025.
http///wwwwi..rrra/////////5/510 hbtp://b//////// 10925
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query Transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 4895 4901, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report. arXiv preprint, abs/2502. 13923, 2025. URL httph::: //apxi:///xvvv
org/ abs /502 13923
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang. Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong. Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 3119-3137, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10. 10.18863///22004acllooooo 172.
URL http::ttps://t htRL/httts:///cccla http::///acaaatth/oa ogy. org/ 2024 acl.lllllnnnoooo 172
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document Transformer. arXiv preprint, abs/2004.05150, 2020. URL httpst////a/iii..coomm http//bb///0000 05155 05150
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Ties Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL hittpss//proceee//
nwwrinu..p....m htccc//ppa/ http:/paap/220////2 1457 (457c0dm 70d77c006bffccoc cb496 96741 1179166677888e8 Bbfb8ac1 42f 4211444444444444 Absaa .html
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In International Conference Machine Learning, pp. 5209-5235. PMLR, 2024a.
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong. Yue Dong, Baobao Chang, Junjie Hu, and Xiao Wen. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling. arXiv preprint, 0bs/2406.02069) 2024b. URL. https
7/arxiv. org /246 02069
E.J. Candes and Tao. Decoding by linear programming. IEEE Transactions on Information Theory, 51(12):4203-4215, 2005. doi: 10. 10.1100911.2000088888888 858979
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, arXiv preprint, abs/2302.01318, 2023. URL https::::: //arxi/axxii wtt.vvvorggabbbvvvveve 01318

---

Published as a conference paper at ICLR 2026
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and and--aad inference acceleration for large vision- language models. In European Conference Computer Viston, pp. 19-35. Springer. 2024.
Eunseong Choi, Sunkyung Lee, Minjin Choi, Jun Park, and Jongwuk Lee. From reading to compress- ing: Exploring the multi-document reader for prompt compression. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 14734-14754, 2024.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, 2020. As- sociation for Computational Linguistics. doi: 12.18663/112022.acllmml 747. URL https
//aclanthology.o org/ 2020 mall-main..44 747
Tri Dao. FlashAttention-22: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024.
Song Dingjie, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. MileBench: Benchmarking MLLMs long context. In First Conference on Language Mod- eling, 2024.
Xueying Du, Mingwei Liu, Kaixin Wang. Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 97984000001774 doi: 10. 00.114553597500336392999 3597503 3639219
Yuan Feng. Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou, Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference. In The Thirty- Annual Conference on Neural Information Processing Systems, 2025. URL https:://pperevize met/forund wtt.ormmmmittiissuuueeeeeeeeeeee suhGszZ
Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level KV cache compression method with integrated retrieval and reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https
7/ //openreview.oo net
Gemma Team. Gemma 3. 2025. URL http:::/goo.gle/Geemaaaa 3Report
Google DeepMind. Gemini 2.5 Pro: Advanced reasoning Al model, 2025. URL https://
Iddepm.n.. wttpg/ggeeeeeecooooooge/////e//e
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint, abs/2407.21783. 2024. URL https: attps://arxxvv www.xxii.../gaaaa. 2400
21783
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's the real context size of your long-context language models? In First Conference on Language Modeling. 2024. URL
net forum?
[d=kIoBbc7653 76yy
Yunhai Hu, Zining Liu, Zhenyuan Dong, Tianfan Peng, Bradley McDanel, and Sai Qian Zhang. Spec- ulative decoding and beyond: An in-depth survey of techniques. arXiv preprint, 2is./2002.1932...
2025. B httpt::///// httttp::://xiivi worgor//aaaa.o abs 2502 .0.0002..00 19732
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874-880, Online, 2021. Association for Computational Linguistics. URL httpt://////// http://aaaaatthooooooooooo U wwqp/yy.....g/oooo/ 202l maal-maann
74
12

---

Published as a conference paper at ICLR 2026
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Ama Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée I Lacroix, and William EI Sayed. Mistral 7B. arXiv preprint, abs/2310.0688 2023a. /tL/
06825
Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Compressing prompts for accelerated inference of large language models. In The 2023 Conference on Empir- ical Methods in Natural Language Processing, 2023b. URL htttps://openzeviiiet net/
Forum? A= 32n.
Huigiang Jiang. YUCHENG LI, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yerr Lin, Yuqing Yang, and Lili Qiu. MInference 1.0:
Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https //openreview. net/ Eorummidd--- Eorummid-EPSAAAAaSSS EPBACAbaSN
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin- Lin, Yuqing Yang, and Lili Qiu. LongL.LMinguu Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1658-1677, 2024b.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez. Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. FlexPrefill: context-aware sparse attention mechanism for efficient long-sequence inference. In The Thirteenth International Conference on Learning Representations, 2025.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from Transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In The 2023 Conference on Empirical Methods in Natural Lan- guage Processing, 2023. URL
net / forum id=c JDRNNBYBBBBB
Yuhong Li, Yinghing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In The -ightt Annual Conference on Neural Information Processing Systems, 2024. URL /ttttss://opnrevieere wtw.:eeeeeeormmmmdopp55555.. 4
Barys Liskavets, Maxim Ushakov, Shuvendu Roy, Mark Klibanov, Ali Etemad, and Shane K Luke. Prompt compression with context-aware sentence encoding for fast and improved LLM inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 24595-24604 2025.
Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chen- gruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang. Yuqing Yang, and Lili Qiu. RetrievalAttention: Accelerating long-context LLM inference via vector retrieval. arXiv preprint, 10512 2024a.
10516
Fei Liu et al. Learning to summarize from human feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 583-592, 2020.
Jingyu Liu, Beidi Chen, and Ce Zhang. Speculative prefill: Turbocharging tift with lightweight and training-free token importance estimation. In Forty-second International Conference on Machine Learning, 2025.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the 2024b. 2024b..
Association for Computational Linguistics, 12:1
13

---

Published as a conference paper at ICLR 2026
Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, and Edoardo M. Ponti. The sparse frontier: Sparse attention trade-offs in Transformer LLMs. arXiv preprint, abs/2504.17768, 2025.
www.aaivvvv.ooo org. abs wttp://oo/g/b//...111777
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, et al. LLMiingua-:: Data distillation for efficient and faithful task aask-agnostic prompt compression. In Findings of the Association for Computational Linguistics ACL 2024, pp. 963 0811981,222 2024.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS- W, 2017.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text Transformer. Journal of machine learning research, 21 21(140):1-67, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, 2016. Association for Computational Linguistics. doi: N0.18533/1/D166 1264. /tRL:/tttpa///caaatooooo.
org/ 64
Ranajoy Sadbukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En- Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. MagieDec: Breaking the datency-througge tradeoff for long context generation with speculative decoding. In The Thirteenth International Conference on Learning Representations, 2025. URL TtIl.tttstt. /ttp:///pppevvvieeeee ew net/ tetee/foruse forum? id-CS2JWaziYr
Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. TriForce: Lossless accel- eration of long sequence generation with hierarchical speculative decoding. In First Conference on Language Modeling, 2024. URL
net http://wwwweeeeotuut/eeee f
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST: Query- aware sparsity for efficient long-contexs LLM inference. In Forty-first International Conference on Machine Learning, 2024.
Rishabh Tiwari, Haocheng Xi. Aditya Tomar, Coleman Hooper, Sehoon Kim, Maxwell Horton, Mahyar Najibi, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. QuantSpec: Self- speculative decoding with hierarchical quantized KV cache. arXiv preprint, abs/2502. 10424, 2025.
URL /tttss://aaxiiss org/ abs/ 10424
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL https: / htttps::/prrcceediinss neurips cc/pppper/211/haah/ Abstract atsteiccaaa.aaaaaaatttt.t...... html
Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, and Wanxiang Che. Lookahead Q-cache Achieving more consistent KV cache eviction via pseudo query. In Christos Christodoulopoulls,, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.). Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 34158-34174, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176- 332-6. doi: 10.18653/v1/2055. www.eeennnmaaaeeeeee 173in.1732. URL https: https://aclaattooooryy.o 2025 www.ppmmminn....mm..mm.mmmmmmmmm wwt.mmaiinn............... 1732/]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace's Transformers: State- of-the-art natural language processing. arXiv preprint, abs/1910.03771, 2019. URL https 7/ Warxivv. www./bog///aaeoo 1tt.a/////11111111111111
14

---

Published as a conference paper at ICLR 2026
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayi- heng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint, abs/2412.151155, 2024. URL
org httpaa///22.1111..1.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025a.
An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, et al. Qwen2.5-1M technical report. arXiv preprint, abs/2501.15383, 2025b. URL
http:///rrraa////011./.// 15383]
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi:
18 1259, URL https: //acl wnw.oaannttoooooooo ogy. wwwpy//-g/1111220000 1259
Zhenyu Zhang. Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: Heavy- hitter oracle for efficient generative inference of large language models. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023. URL https:://penrreveew. http///eeeeee.eee...... Eorum?id Rerum?id=RKRRPpP71
15

---

Published as a conference paper at ICLR 2026
Appendix
Algorithm Pseudocode
17
B Mathematical Prools
17
Proof of Theorem
19
B.2 Proof of Theorem
19
B.3 Proof of Theorem
20
Benchmark Dataset Details
23
LongBench
23
C.2 RULER
23
MileBench
24
Experimental Setup
24
D.I Hyperparameter Settings
24
D.2
Implementation Details
25
Extended Experimental Analysis
26
Additional Results on RULER and LongBench
26
Integration and Comparison with the AdaK Baseline
26
Multi-modal Evaluation
26
E,4
SpecKV: Impact of Moookahedd Parameter on Importance Score Correlation
27
E.5
Performance of Cross-Family Models
27
E.6
Extended Results for S-PcKV-PC
28
SpecKV: Extended Latency and Memory Analysis
29
Ablation Studies]
29
E.8.1
SpeeKV: Analysis of Enhanced Draft Models
30
E,8.2
SpecKV and SpeePC: Impact of Riookaheadd
30
E.8.3
SpeeKV: Accuracy Analysis of Sparse Prefil]
30
16

---

Published as a conference paper at ICLR 2026
A
ALGORITHM PSEUDOCODE
Algorithm SpecKV
I: Imput:
Input sequence with length
Parameters: Number of lookahead tokens Yelokahends Maximum cache capacity Cmnes
Compression window size Rlwindww Kernel size k, Prefill window size Titush
Number of global tokens in pretill Rlae
2: Generate draft output Ydnn of length Rioaaaeedd using the draft model.
3: Forward and jimn through the target model.
4: for each attention head in target model do
5: X <- Target model hidden states from prompt at current layer
D X E R
Y Target model hidden states from draft output at current layer
Y E R Ximbhas
m Vhnn Twwndoww
CrossAtrention (Q from X
K/V from X.m)
Compute attention
9:
MaxReduce(AA)
R X
R
10:
(vg
b Smooth attention
11:
Evest + topk(s, Rwven
D Select global tokens (Sparse prefill) D. Sliding window (Sparse prefill)
12:
13:
feuche +- lopk(s, Caux Tlwindow) ) U [m + 1, m + 2,... Rim)
> Select KVs
output VerticalSlash(X, Evert, Astesh)
Sparse prefill attention
15:
eache K V Viaae
KV cache dropping
16:
end for
Algorithm 2 SpecPC
I: Input:
Draft attention tensor A E IR "laeer X (my t-ooaaell Xmi
Parameters: Window size Y-window Kernel size k, Number of neighbors Thniihhbrrs
Number of selected tokens Caux. Number of skipped layers lasp
2:
Rwinddw
3: A <- Ata.ps.mm..
Skip layers and only consider window queries and non-window keys
4: for E (1,2,......
Twwidow } do A...fe
5:
A...g.
Assign more weight to later tokens
6: end for
s +- MaxReducceA))
th X X R
8: AvgPoolI
Smooth attention
9: + MaxPoolID(s, Raeggttaaa
Keep neighbor tokens
10: Asdlected topk(s, Cmax) U [m + 1, + 2,..., Nin] > Keep most activated tokens and window tokens
11: return Eselected
B MATHEMATICAL PROOFS
Lemma 1. |Sotmmax(22 - Softmax(y)lla S || yll
Proof. Let J be the Jacobian matrix of Softmax. Then, J(v) = diag(p) - -ppT, where p Softmax(v). Note that is probability distribution, p; 0 for all i and Σ, P: 1. For any
17

---

Published as a conference paper at ICLR 2026
vectors and
|J(r)=112
[[(iag(()) pp") )211
E(iis - pip" 2)²
Ept(e - -1=
Epils - -)²2)²
E (piz) - -2pizp pi 2)²)
Epie - ()²²)
$ Σp Piet
Epill=la
|
Thus, [[(())=122 < |0| for all o and z. From the fundamental theorem of line integrals,
Softmax(x) Softmax(y) = f J(y 1xx y2) y)dl.
Finally,
11,0 J(y + 1(x 4))(3 y)dl
(Softmax(x)) Softmax(y)lla
S.
<
|| + 1xx - w))( - 12
[
| ull dt
I| yl1
Lemma Let y = Softmax( r) and y' Softmax(2'). If lly - y'llp S e, then there exists a sealar such that |x x' cllpp where min,(min(y:.) v/)) and p (1,2..... .... )0).
Proof. From the mean value theorem, there exists & E (26, 1/6 such that
log w: log v/
dogg de
(5)
9. 4/
E-
1==6
Note that >>> 0. Then,
log y:
log 3/1
v/l s 1 1/: -%1l-
(6)
c = log E, " log E, e²), so
log ei Eye'5
log c Σ, 2
24 d
log Σ = log Σ /
= |x
cl
(7)
18

---

Published as a conference paper at ICLR 2026
for all i. Thus,
10 - 2/- cl <
1 1/ - vil
= 24 - x c|p
<
1
1/ y
mp
Σ x3 x - c|
Σ m22 2 1// 3/4
1
|| x' 11 <
1y v/ $1p
mF
| x' c1l|
<
y 1/ 1p
|| x' c1 1|p -
B.1 PROOF OF THEOREM I
We define the vector of importance scores as and approximation as
T
Plout
Softmax
(0) W 2i
and
S
Softmax
2
W.WFX vd
Than
vd
Yhout
(8)
where X = [ Rn 1T Roaxd is the matrix of input embeddings, z Rd is the ith output
embedding, and #/ E Rd is the ith approximate output embedding (from the draft model). 8; and 3; denote the importance of the ith KV pair. In practice, SpecKV estimates importance using queries from recent input and draft output tokens. This is omitted from the theoretical analysis for clarity.
Proof. We assume || - =!!0 1l2 for all i and ||ا S v for all ||
2((0)
12 I c,
10) W. W.W.e my
20
W. W,WTas dd
Ja (x) 20 W WT,
Ja WW, Hil,W W 1laevd
wlW. WT 11z.
Thus,
W.
W.W?x' X
LOTT W, W,W.T. x
SelW,WEF1
(9)
vd
Va
Applying Lemma I and the triangle inequality, we get
LoYT W. W x
Phatl
W, W x Vd
||s
$112
Softmax
Softmax
That
Jd
Rost
[ofT W.W. x
LoIT W.WTxx X
Softmax
Softmax
Tosi
Vd
Vi
W. W,W.TX x
200 W. W X
17umt
W WWW,lll..
B.2 PROOF OF THEOREM 2
[I....., Fnal T
(10)
Y (y..... mn T Sofumax
XW.WTX? XW,
(11)
19

---

Published as a conference paper at ICLR 2026
Y .i...., Inal" Sootmax
XW, XT
XW,
(12)
A [01,--.. Ore T
Softmax
XW,WTX x vd
(13)
XW,WIx"
Á = [0 amal
Softmax
(14)
vd
Proof. We assume ||s: 8:ll2 S olo, where ||x |loo, is the maximum l2 norm of the rows of X. Additionally, we assume that there exists a constant such that eX has the Restricted Isometry Property (Candes & Tao. 2005) with parameters (2k, 5), where is the restricted isometry constant and & is the approximate sparsity of a, and â,.
Recall that a matrix satisfies the Restricted Isometry Property with constant E0,,1) if for every k-sparse vector v, the following inequality holds:
(1 - 6)11113 & Bull - (1 0)11112
(15)
Let AW, = W. W. and Da; = a; - âū.
If nin 1, then A Å E R with A1,1 A 1, so AX Ax = X zT. which implies
|111 أn 22
|| Wv - W. = || AW. 1ll S xx = =|l=ill
(16)
for all Z1. | W 22 S ||اا||| || for all x1 is the definition of the matrix &g norm, so ||WW S E.
luu úil12
|| XW. oF x.xW2ol2
|1af XW. (T,'W, af XAW.
|1f XW. - (afx,, -f XW, -WW.-axxAM xAWW)l2
|| XW, + of XAW.12
I ||,,2
Then, X.. & lo2 | XAW 12
Since a." X is convex combination of the rows of X, || X 1l2 |
Thus, | XWW,ll - <|| + ||]" XAW, 2 - < x|| x0.2 + || Xl2/AAWWllllll S 2e
Attention scores are approximately sparse (Jiang et al. 2024a), especially for long sequences. Therefore, we assume and à; are k-sparse. Then, Aa; is at most 2k-sparse. Since cX has the Restricted Isometry Property with parameters 2k, 8,
(1 - --)llaa.ll < Aa (cX))l2 & (1 + 5) Aa.ll2
(17)
Then.
1 - 511-))1Aaf1+ W | X < |
(18)
Guin (.)
so
Aa/2/1 <
2cr XX 1lo.
(19)
Tuin ()((
B.3 PROOF OF THEOREM 3
Theorem 3. ! 1122 S <|| for all X and the column space of Wg, Wt. W,, We is a subset of the column space of W then ||a; lell I cō, where
8 = 2dl Tmax (W.)²
(w)) exp (2 max
[Walla Wallz) [WWallalWaalla (W.)
|
(20)
Omin (W,)²
20

---

Published as a conference paper at ICLR 2026
Proof. We assume ||Y - Yll22 I c||X||2- Additionally, assume that the column space of W: W5y We, W& is subset of the column space of W To get norm bound on Aa; a; ag. we will bound the norms of the error in approximate weight matrices. We will find these bounds by using specific inputs, taking ad advantage of the fact that | Yll - <|| 2l2 for all X.
We will start by bounding AW, = W. W,, by choosing input that fixes A and A. If = 1, then A Á = [1] so AX- AX = X = =f. which implies
c|| 1| Y 1 2 = |AXWY AXW || of 12 |ol Al S <||1|12
(21)
for ø1. Thus, ||W.l12 e,
Next, we will bound the norm AB B B, where = W and B W.W.T. We will choose the X so that the values are the identity matrix. Then Y Let UEV the singular value decomposition of W.. We set
X = VVV-
(22)
where & is an arbitrary orthonormal basis spanning Rdxd
Note that min ())
mmax ( 1. so
|| maax (X)
mmia (W.) and m (X)
mmax (w,)
Now,
|| F12
|AXWW - AxWallz
IAXXW (AXW, AXAWx)l
|AXX (AXW. AAXW. AXAW.)
I|АAX,, AXAW,ll
S I|АXW 112 ||AAA
DOOIVE-"UTTEV" Il2 + IJÁXAW.II,
||ΛΦ 12 + -|ÁXAW,.12
||AI|22 + |(ÁXAW, l12
<||x1112 1|
W
2c
Guin (W.)*
Note that each row of À is probability distribution (non-negative entries summing to 1), so left- multiplying X by Å forms a convex combination of the rows of X. From Jensen's 's inequality get | is convex function.
|| || because
Let 8, max
[Walla||
|| Walla
Since ||X||2 [W.] and ||l||2 <
||W,la||WWlla
Gann W.)²
aan (W.)²
|XBX" 1|a = 81. Consequently, |o: Br, 82 for all i, This implies that each attention weight satisfies
aij
Fd 19
-26
28
(23)
The same argument applied to à gives
-6
261
(24)
Ed e9
Applying Lemma to each row, there exists IRd such that
21

---

Published as a conference paper at ICLR 2026
XBX² vd
xBx²
+ c1 vd
S 21 Softmax
XBX*
Softmax
[XBXT
vd
12
XBX* - xBXT + vdc11
S 2 201 ||A/|
(25)
2000/2
Juin (W.)
Minimizing over e, we obtain
min |XBX* -XXX" -BXX" 12 min miin|XABXX Vdc1
=||XABX" - -IXABX"1111
(26)
(I- =|IXABX" (11))1))1
Substituting in the definition of X, we get
- (
26, miin (W.)
(27)
Each multiplication by E-1 decrease the norm by at most "m.)(W.)) when removing both instances of 1-1 we scale the bound by max (W.)² giving us
DVUT ABUV p"((I"TT---T" (I- d 11" S
2/d 26, O max (W,) 2
(28)
aain (W.)
Then, since & and V are orthonormal and preserve spectral norm under multiplication, we conclude
JUTABUVTO TJ -1-d-'11') S
226 aanin (W.) Tmax (W,)?.
(29)
Finally, since the column space of Wq. Wk. Wg, We is a subset of the column space W., the column space of AB is a subset of the column space of U. Thus, left multiplication by U does not impact the spectral norm,
|ABUV T (I---
22 aminn (..) maxx (W.)²
(30)
Note that the matrix 1 111" is a projection onto the subspace orthogonal to the all-ones vector. Its singular values are [1...., 1, 0]. its spectral norm is
11 1 =1
(31)
and an arbitrary orthonormal basis of 12d, it follows that for any fixed P E Rddd we can choose such that the largest component of PΦ lies entirely in the subspace orthogonal to 1. In this case.
Moreover, since ||z 1 11"" à
(T
||
(32)
Thus, ||BB|2 82 where 82 3/2 xmax (W.)² Omial (Min(W..)) W.)
261
Now that we have bounded |ABl|, 2, we will consider any input X. Then, |r/ABr,1 S 8g |X
so | ABXT aloe 6z || || |loc is the maximum l norm of the rows of X.
From Lemma I
TTBXT V
|a à @1 = Softmax
Softmax
l12
Vi
<
B
Vd
2 Vd
Fa
||
ABX*
<
5al1X 2
Vd
22

---

Published as a conference paper at ICLR 2026
||4 - 10:22 8a11X11
2nd Tmax (W,)²
mmin (W.) exp (2 maxxx [[alll Wallz
[W Walla
|| 1
mmin (W.)
mmin (W,)²
(33)
C BENCHMARK DATASET DETAILS
C.I LONGBENCH
Table 3: LongBench tasks.
Task
Dataset
Source
Avg. Words
Metric
Language
Size
Single-Document QA
NarrativeQA
Literature, Film
18,409
English
200
1-2
Qasper
Science
3,619
F1
English
200
1-3
MultiFieldQA-en
Multi-field
4,559
F1
English
150
1-4
MultiFieldQA-zh
Multi-field
6,701
F1
Chinese
200
Multi-Document QA
2-1
HotpotQA
Wikipedia
9,151
FI
English
200
2-2
2WikiMultihoPQA
Wikipedia
4,887
FI
English
200
2-3
MuSiQue
Wikipedia
11,214
FI
English
200
2-4
DuReader
Baidu Search
15,768
Rouge-L
Chinese
200
Summarization
GovReport
Government report
8.734
Rouge-L Rouge-L.
English
200
3-2
QMSum
Meeting
10,614
English
3-3
MultiNews
News
2,113
Rouge-L
English
200
3-4
VCSUM
Meeting
15,380
Rouge-L
Chinese
200
Few-shot Learning
4-1
TREC
Web question
5.177
Accuracy (CLS)
English
200
4-2
TriviuQA
Wikipedia, Web
8.209
Fi
English
200
4-3
SAMSum
Dialogue
6,258
Rouge-L
English
200
4-4
LSHT
News
22,337
Accuracy (CLS)
Chinese
200
Synthetic Task
PassageCount
Wikipedia
11,141
Accuracy (EM)
English
200
5-2
PussagcRetrievaleen
Wikipedia
9,289
Accuracy (EM)
English
200
5-3
RussugcRetrival-zhh
Dataset
6.745
Accuracy (EM)
Chinese
200
Code Completion
6-1
LCC
Github
1,235
Edit Sim
Python/C#/Java
500
6-2
RepoBench-P
Github repository
4,206
Edit Sim
Python/Iava
500
LongBench (Bai et al. 2024 a benchmark suite designed for long-context evaluation, comprising 14 English tasks, five Chinese tasks, and two code tasks. As Llama does not support Chinese, we excluded the corresponding tasks. Furthermore, we removed the synthetic tasks, as these are already covered by the RULER benchmark. The remaining tasks grouped into five categories: single-document question answering, multi-document question answering, summarization, few-shot learning, and code completion. For each category, the overall score calculated the average of all its subtasks. The final LongBench score is computed as the average across all included tasks. Table provides an overview of all tasks, adapted from Bai al. (2044).
C.2 RULER
RULER HHiieh et 2024 is a synthetic dataset designed evaluate the true supported context length of LLMs. It comprises 13 tasks, including eight needle-in-a-haysliee (NIAH) retrieval tasks, two aggregation tasks, two question answering (QA) tasks, and one multi-hop tracing task.
The NIAH tasks involve hiding random key-value pairs within generated text and challenging the model retrieve them. Aggregation tasks simulate summarization by asking the model to extract the
wwwpaaassttttttHtt///Saa HHO/H/HHUMMMMO// HHH.////oooooooeennccc (MIT License)
https:///tttub..cmm//ii
(Apache License 2.0)
23

---

Published as a conference paper at ICLR 2026
most frequent common words from given passage. The QA tasks require the model to answer question about a randomly selected paragraph within the context, serving as real-world analog to NIAH tasks. In the multi-hop tracing task, the model must identify all variable names that reference the same value within a chain of assignments.
RULER is generated for range of sequence lengths using randomly generated texts drawn from Paul Graham essays, SQuAD (Rajpurkar et al., 2016), and HotPotQA YYng et al. (2018) datasets. This approach enables comprehensive assessment of language model's capability to process varying context lengths. Evaluation is conducted based on accuracy, considering a response correct contains the requested value associated with the specified key.
C.3 MILEBENCH
Table 4: Overview of the MileBench datasets. Average tokens are computed using Qwen2.5-VV VL
Bai
al. 2025).
Category
Dataset
Avg. Words
Avg. Images
Avg. Tokens
Metric
Size
EgocentricNavigation
85
45
3,079
Accuracy
200
Temporal
MovingDirection
62
1,042
Accuracy
200
SceneTrannsiionn
5,125
Accuracy
200
SlideVQA TQA
2,053
Accuracy
200
Semantic
50
5,536
Accuracy
200
WehQA
146
1,706
Accuracy
200
MileBenct Dingjie et al., 2024 a long-context benchmark designed to evaluate Multimodal Large Language Models (MLLMs). It comprises 29 multi-image-text datasets, organized into 12 tasks, which are further grouped into four categories: Temporal Multi-Image, Semantic Multi-Image, and two diagnostic categories-NIAN and Image Retrieval.
we selected three datasets each from the Temporal Multi-Image and Semantic Multi-lmage categories: EgocentricNavigation, MovingDirection. and Trensstratti for the Temporal Multi- Image category, and SIid TQA, and WebQA for the Semantic Multi Image category. [able//ppovice overview of the selected datasets.
For our additional experiments in Section E.3
D EXPERIMENTAL SETUP
D.1 HYPERPARAMETER SETTINGS
Table 5: Prompt compression backbones and parameter counts.
Method
Backbone
Parameters
LLMMLnnga 2 Pan et al. 2024
xlm roberta- large Conneau al. 2020
560M
CPC [iskkaees
Llama-3 3.2-BB Graitation T5- base Rafiel el al. 20.00
IB
R2C
Clu et 2024
220M
Table 6: Summary of hyperparameters for various methods.
Hy perparameeer
StreamingL
H20
SnapKV
PyramidKV
Ada-Smapky
LAQ++
SpickY
SpecPreliit SpecPC
Window size
32
32
32
32
32
32
32
64
Pool
Max
Reduction
Mean
Mean
Mean
Max
Max
Mexn-Max
Max
lookahead tokens
All
Compression window Ratut flon
2018
global
neighhorrs Haugter / chunk size skipped layers hap Initial cache size
64/32
8
Cen
https://mi lebench. www.ch.giituu..........
(Apache (Appahe License 2.0)
24

---

Published as a conference paper at ICLR 2026
Table 5 lists the backbone models employed by each prompt compression method, while Table details the hyperparameters used in our experiments. Generally, we select hyperparameters for each method based their respective codebases. We observe that using max aggregation improved performance compared to mean aggregation for SpecKV and SpecPC. For SpecK setting Nalash and Oleen to 2048 resulted in minimal accuracy loss but substantially reduced latency (Section E.8.3).
For SpecKV, we always generate tokens until the draft model produces the EOS token, which yields the best performance. For latency measurements, we Pllokaheadd 64 tokens, reflecting the average sequence length in our benchmarks. In SpecPC, prompt compression drops tokens uniformly across all layers and heads (unlike SpecKV, which prunes per head), so a larger Cmax is needed to retain relevant information. While larger Aoooaabadd can boost performance, in practice, generating only one token per prompt (Rlookaheud = 1) usually sufficient. Strong alignment between the draft and target model attentions enables SpecPC PC to outperform methods like R2C and CPC. For an ablation on Polookahaad- see Fig.
Retaining the local context for prompt compression proved essential. This observation aligns with the design existing prompt compression methods, which typically aim to preserve entire sentences within the prompt. Consequently, we increase both the pooling kernel size (k) and the number of neighboring tokens (Tinsighbor) to 64. For Llama, slightly better results are achieved by reducing both k and Reighhor to 32, though the performance difference marginal.
For all remaining methods not explicitly mentioned, we use the default configurations provided in their respective codebases.
D.2 IMPLEMENTATION DETAILS
For our experimental results, we employ the following large language models: Llama-3.2-11 Instruc 8 ----m---------nstrreee Llama-331-70B-Instruct (4-bit Qwee2.5-005B-Insttrus
Swee2.5-14B-Instruu 12 Qwen2.5532BBIInttuu [3] and
For MLLM evaluation on MileBench (Dingjie et 2024), we utilize Qwen225-VV-3B-Instruee
AWO 15 and
Our implementation is based PYTORCH (Paszke al. 2017) (BSD-3 License) and Hugging- face's TRANSFORMERS (Wolf et al., 2019) (Apache License 2.0). All experiments leverage FLASHATTENTION-2" Dao 2024). Latency measurements are performed using vLLM wherever possible (i.e., where attention map outputs are not required). For implementing the sparse prefill mechanism of SpecKV, V, we use kernels from MINFERENCE 19 All methods are evaluated via greedy decoding. Experiments are conducted on NVIDIA H100 80GB GPUs, with runtimes varying by context length; for a maximum context length 64K tokens, experiments take up to 2 hours.
For evaluating STREAMINGLLM, (Xiao et al. 2024). H20 (Zhang et al. 2023), SNAPKV et al. 2024), and PYRAMIDKY V (Cai et al.) (024b), we use implementations from KVCACHE-FACTOR In this library, the STREAMINGLL than at each decoding step, differing from their original codebases. This adjustment enables fairer comparison to SNAPKV V and others. We extend
and H2O implementations drop KV once after prefill, rather
CCCACHE-FACTORE to support Grouped Query Attention (Ainslie et al. 2023) by repeating keys and values for each KV head, computing attention within the window, and averaging across KV heads. This approach avoids duplicating the KV cache.
(Ilama 3.2 license)
http://////uugnnneeeoooooee////a Instruct (Ilama 3.1 license)
(Illama 3.1 license)
(Apache License 2.0)
Apache License 2.0)
Apache License 2.0)
GPTC- PeprQ-Int44 (Apache License
2.0)
(Apache License 2.0)
(Apache License 2.0)
htteer//iiuuucon//cccal///faaaar 3-Clause License)
httpes//aattbooooo/////ooooooood 2.0)
httts://4aabbbacoo///ooooottttfr Licens))
https//aathhcccc///feea///aaaaer (MTT License)
25

---

Published as a conference paper at ICLR 2026
For other baselines, their official implementations. For Lookahead Q-Cache (LAQ++), provide our own implementation since their code not publicly available.
E EXTENDED EXPERIMENTAL ANALYSIS
E.1 ADDITIONAL RESULTS ON RULER AND LONGBENCH
In this section, we present results for RULER and LongBench using various Cma values of 256, 512. and 1024 for KV dropping, and 1024, 2048, and 3072 for prompt compression. Specifically, we employ Sowen..5-22--nnstree
(4-bit quantized with bitsandbyted 2T and as target model.
-wee2.5-72B--lnstuur--GPP
Figs. [0 [ show the RULER results, and Tables 9 to present the LongBench results. Overall, our methods achieve higher accuracy than the baselines most settings, especially with small Cmax- Specifically, SpecKV significantly outperforms SnapKV and LAQ++ RULER in most cases. Similarly, SpecPC consistently achieves strong results, particularly at longer sequence lengths on RULER. On LongBench, both of our methods also surpass the baselines.
E.2 INTEGRATION AND COMPARISON WITH THE ADAKV BASELINE
Fig. 12 (RULER). Tables 12 and (ongBench)) present the performance of our proposed methods on Qwen2.5 (0.5B draft, 14B target) and Llama-3 (IB draft, 8B target). We include additional baselines, notably V (Feng et al. 2025), for various values Caax- AdaKV is an extension to SnapKV and allows different KV budget per attention head. We apply AdaKV to SpecKV in similar fashion, which further boosts performance in our experiments.
E.3 MULTI-MODAL EVALUATION
We conduct additional experiments using www.---VL--B--sstrrct-eee (draft) and Qwen2.5-VL. I1B-Instrutt-WW (target) on six MileBench (Dingjie et al., 2024) datasets. We select three datasets each from the Temporal Multi-Image Image (EgocentrieNavigattion, MovingDirectiion SceneTransie Transition) and Semantic Multi-Image VQA, TQA, WebQA) categories. We focus on these datasets because, for Qwen2.5-VI., the performance gap between draft and target models is most significant; in other cases, the models perform too similarly or the draft even outperforms the target.
For KV dropping, evaluate H2O, SnapKV, and PyramidKV from our prior experiments. We do include AdaK in our evaluation as it is dependent on an older Transformers (Wolf et al. (219)) ver- sion incompatible with Qwen2.5-VL. For prompt compression, we compare with FastV (Chen et al.) 2024) method specialized for dropping image tokens inside LLMs. FastV uses a hyperparameter k: runs all tokens up layer k. then drops less-attended image tokens based on the attention map, processing only the top tokens thereafter. This makes FastV less efficient than SpecPC, since all tokens must be processed up with the full model, requiring considerable memory. Notably, FastV must compute the entire attention map at layer k, preventing the of FlashAttention and leading to out-of-memory errors, even for moderate sequence lengths, As a result, many MileBench datasets limit our analysis these six datasets.
exceed 80GB VRAM,
Since the selected MileBench datasets have relatively short average context lengths, we conduct experiments using reduced Cmmxx values for both KV cache dropping (64, 96. and 128) and prompt compression (512, 768, and 1024).
Fig. 13a presentss presents results for various KV dropping methods. Our proposed method, SpecKV, demon- strates performance comparable to existing approaches, while significantly outperforming the others on the WebQA task.
Fig. 13b compares the performance of SpecPC and FastV under two configurations (k and k 5). Our method consistently outperforms FastV in most cases.
26

---

Published as a conference paper at ICLR 2026
28
GovReport
27
Mulli-uews
26
512
LongBench Score
25
414
24
666
MB2
21-
6
20
0.3
0.4 0.5
0.6
Figure 7: Impact of Ripokaheud SpecKV importance score accuracy (R²) and LongBench down- stream performance (Neut 512). The plot shows a strong positive correlation: as Plookahead increases (with Tookkheal 0 being equivalent to SnapKV), both the R (correlation with the true target model scores) and the downstream task score improve. Notably, SpecKV improves correlation and accuracy even with a small Plookaheadd 32, which only 6.25% of the output length. Experiments use Qwen2.5-0.5B draft model and a Qwen2.5-32B target model on two LongBench tasks.
E.4 SPECKV: IMPACT OF PLLOOKAREAD PARAMETER ON IMPORTANCE SCORE CORRELATION
We analyze the impact of the Niookabedd parameter the correlation (R²) between SpecK estimated importance scores and the ground-truth scores from the target model. For this analysis, we use 50 examples from the Multi-news and GovReport tasks from LongBench, selected for their long maximum output length (512 tokens). We employ Qwen2.5-0.5B as the draft model and Qwen2.5-32B as the target model, with Caax 256.
To establish the ground-truth scores, we use the target model's generated output (up to Four as perfect lookahead sequence and record the resulting importance scores. In Figs. 14 and compute the R correlation between these ground-tree -ound-truth scores and the scores estimated by SpecKV using various Mloookhhaad values.
As shown in Figs. 14 and 15 the R² correlation steadily increases with Mljockheadd- confirming that a larger lookahead provides more accurate importance estimation. This benefit is particularly pronounced for longer output sequences (Nau)). We also test Plonkahead 0 (equivalent to SnapKV), which yields a significantly inferior correlation.
Furthermore, we connect this score accuracy to downstream performance. Fig. plots the R² value against the final LongBench downstream score. We observe that these two metrics are highly correlated: higher R (better score accuracy) generally yields better downstream performance. Consequently, increasing Tlookahead improves both the R² correlation and the final task score.
E.5 PERFORMANCE OF CROSS-FAMIIE MODELS
In this section, we evaluate the effectiveness of our methods using cross-family draft models. We set Llama-3. 3.1B70B as the target model and employ Qwen2.5-0.5B and Qwen2.5B-115B 5..5BB as draft models. The evaluation is conducted on the RULER and LongBench benchmarks, using Cmax = 256 for SpecKV and Cmux 1024 for SpecPC.
key challenge in this setup that the Qwen and Ilama families use different tokenizers, neces- sitating token translation step. For SpecKV, de-tokenize the draft model's 's output and then re-tokenize it with the target model's tokenizer. For SpecPC, we aggregate attention scores from tokens to words before re-tokenizing, avoid tokenizing partial words, and replace the draft model's chat template with the target's.
(MIT License)
27

---

Published as a conference paper at ICLR 2026
3lama-3 70B (4-hit)
Llama-3 70B (4-bit)
100
100
90
Sate
4k
8k
16k
32k
Sequence Length
Sequence Longth
Specky (Llama
1.58)
StapKy
SpacCC (Llamm-
SprePC (Qwen-L5B)
CPC
SpecKTV (Qwca-0.8B))
LAQ
Spachhutil
(a) KV cache dropping.
(h) Prompt compression.
Figure 8: Cross-family model results for SpecKV and SpecPC on RULER. SpecKV demonstrates robust performance even with draft models from different families, outperforming the strongest baseline, LAQ++.
Table Cross- Cross-family model results for SpecKV and SpecPC on LongBench using Llama-3.1-700 (4-bit) as target model.
Single- doc QA
Multi- doc QA
Few-shot Learning
Code
Caax
Method
Summary
Completion
All
SnapKV
55.88
45.30
22.49
62.15
55.49
47.75
LAQ++
54.90
46.48
22.83
64.31
55.10
48.43
KV
256
SpecKV (Llama-3B)
51.80
47.23
25.53
64.02
58.75
48.80
SpecKV (Qwen-0.5B)
53.18
45.70
24.40
63.69
$7.36
48.26
SpecKV (Qwen-1.5B)
51.20
47.07
24.98
65.81
57.53
48.73
CPC
45.14
39.41
24.86
61.40
37.58
41.97
R2C
48.93
42.01
25.38
58.91
40.19
43.29
SpecPrefill (Llama- IB)
54.62
46.43
25.63
64.80
44.92
48.37
PC
1024
SpecPC (Llama-1B)
56.84
44.48
25.91
67.37
47.15
48.44
SpeePC (Qwen-0.5B)
36.73
36.42
24.68
64.51
50.75
42.04
SpecPC (Qwen-1.5B)
51.58
39.53
25.87
66.43
50.25
46.48
The results, shown in Fig. (RULER) and Table (LongBench), indicate that both SpecKV and SpecPC achieve good results even with cross-family drafts. Notably, SpecKV outperforms the strongest baseline, LAQ++. This aligns with the intuition that SpecKV, which relies only on the draft model's 's output, should generalize well to mroel scenarios. SpecPC experiences performance drop because relies on similar attention patterns between the draft and target models, condition that may not hold in cross-family setups.
E.6 EXTENDED RESULTS FOR SPECKV-PC
This section presents extended results for the cascaded compression strategy, SpecKV-PC. By restricting the large target model to process only small fraction of the original prompt, this approach achieves substantially better latency and memory efficiency than SpecKV alone.
We evaluate the impact of varying prompt compression ratios on accuracy using Qwen2.5-1.55//33 and Llama-3-3B/700 as draft/target pairs, with a final KV cache size of Ciax = 256. Our combined method, denoted as SpecKV-PC-X (where the prompt is compressed X tokens), is benchmarked against standard SpecKV, SpecPC, LAQ++, and SnapKV.
As illustrated in Fig. 9 (RULER) and Table g (LongBench), SpecKV-PC proves to be both efficient and highly accurate. Notably, we find that moderate pre-compression (e.g.. SpecKY-PC-2044 yields accuracy superior standard SpecKV alone, particularly the longer sequence lengths (RULER). This suggests that the initial SpecPC stage acts as an effective pre-filter, discarding irrelevant information to help the target model focus on the most important tokens.
28

---

Published as a conference paper at ICLR 2026
Conversely, extreme prompt compression (e.g.. SpecPC-256) results in significant performance
degradation. This indicates that for aggressive compression targets, coarse prompt-level reduction is insufficient.
Qwen2.5 32B
Llama-3 70B (4-bit)
100
90
Score
80
70
Sequence Length
Soquence Length
SpecKy
SpeckV-PC-512
LAQ-+
SpecKV-PC-204 P-94-334
SpvcPC-256
SaspXV
SpecKV-PC-1824
Figure 9: Performance of the combined SpecKV and SpecPC methods RULER (final Caux 256). Our cascaded approach (SpecK V-PC-X) pre-compresses the prompt to X tokens, achieving higher accuracy than standard SpecKV (which sees the full prompt) longer sequences. This suggests the initial pre-filtering allows SpecKV V to make more effective final selection.
Table 8: LongBench performance of cascaded compression with SpecKV and SpecPC using Qwen2.5 and Llama-3. SpecK V-PC-2048 achieves superior performance.
Qwen2.5 328
Liamn-3 70B (4-bit)
SingleAA
MatiOA
Summn
Fotesbot
Code
All
Single
MutirAA
Summ.
Few-sbok
Cole
All
Group
Method
Donse
Turget
56.01
43.99
25.90
64.06
44.74
47.78
55.02
47.06
28.61
70.17
48.19
19.99
SnapKV
40:21
LAQ++
55.15
44.14
22.24
63.25
41.19
45.79
$4.90
46.48
22.85
64.31
55.10
48.43
SpecKV
53.48
43.77
24.02
63,79
44.90
46.06
51.80
47.23
25.53
64.02
58.75
48.80
Speek PPccKV.PC-2088
SpecKV-PC-1024
55.37
42.21
24.14
63.09
39.72
45 28
58.75
47.33
25.
64.28
55.19
49.89
SpecKV-PC-512
45.77
36.57
22.36
56.57
39.57
40.21
51.73
46.11
23.01
63.64
50.00
46.68
SpeePC-256
27.05
45.0
35.07
27.97
19.09
E.7 SPECKV: EXTENDED LATENCY AND MEMORY ANALYSIS
We analyze the latency and memory usage of baselines and our algorithms, breaking down latency into three stages: draft generation, target prefill, and target decoding. For L.Q++, draft generation occurs after target prefill to share the same KV cache.
All experiments are run on an H200 GPU (141 GB VRAM) using the Qwen2.5 model (3B draft, 32B target). We evaluate across range of input sizes, Vin E (4k, 8k, 16k, 32k, 64k), and output sizes, Nout E [64, 128, 256, 512). global Cimax = 256 used for all experiments. For algorichm-specific settings, we set Tolookahcad Nout for SpecKV and use the default Tolookahead 8 for LAQ++. For the SpecKV-PC variant, we employ SpecKV-PC-204 which pre-compresses the prompt to 2048 tokens.
Our results demonstrate that draft lookahead introduces negligible latency (Fig 16] and memory overhead (Fig. [17) relative to the total cost of target model inference. Notably, cascaded compression with SpecKV and SpecPC (SpecKV-PC) significantly reduces prefill time and peak memory. At 64k context, SpecKV-PC is about 40% faster and 25 GB more memory efficient than LAQ++.
E.8 ABLATION STUDIES
In this section, we conduct a series of ablation experiments to further analyze the effectiveness of our two proposed methods: SpecKV and SpecPC. For consistency, fix Gmax to 256 for KV dropping
29

---

Published as a conference paper at ICLR 2026
and 1024 for prompt compression across all experiments. We utilize Qwen2.5 (Instruct), employing the 0.5B model as the draft and the 14B model as the target. We sample 100 random examples per task from the LongBench and RULER benchmarks.
E.8.1 SPECKV: ANALYSIS OF ENIIANCED DRAFT MODELS
Fig. 18 illustrates the impact of using draft models of different sizes and versions on the RULER score. As anticipated, both newer (Yang et al. 2025a) and larger draft models lead to improved performance of SpecKV.
E.8.2 SPECKV AND SPECPC: IMPACT OF PLOOKAHEAD
Fig. 19 19illustrates how varying the number of generated draft tokens, Nloekahead- affects the performance of SpecKV and SpeePC. Overall, increasing Wlookhend generally results higher final accuracy for Sp whereas yields only marginal improvements for SpecPC. We attribute this to the larger Cmax budget SpecPC, which allows it to capture all important tokens without needing to generate long drafts.
E.8.3 SPECKV: ACCURACY ANALYSIS OF SPARSE PREFILL
In this section, we experimentally evaluate how the sparsity of SpecKV's prefill procedure affects downstream task performance (Fig. 20). Specifically, we set Tiver equal to Tislash and compare several values for these parameters. As anticipated, reducing sparsity (i.e., using a higher thvnn) generally results higher accuracy; however, accuracy improvements plateau at fvenn 2048, which we therefore adopt for main experiments.
Interestingly, for certain LongBench categories, increased sparsity (i.c., lower nen) can actually lead to improved performance. This counterintuitive result suggests that, for some tasks, sparser prefill may serve as form regularization, preventing overfitting to irrelevant context.
LLM USAGE DISCLOSURE
We utilized Google's Gemini 2.5 Pro (Google DeepMind 2025) and OpenAl's GPT (Achiam et al. 2023) to assist with improving the grammar, clarity, and readability of this manuscript. The authors reviewed and edited all LLM- L--generrtedd suggestions to ensure the final text accurately reflects our scientific contributions and claims. The authors retain full responsibility for the content of this paper.
30

---

Published as a conference paper at ICLR 2026
Cmax 256
Cuax 512
Cmx 1024
100
Score
70
4k
8k
16k
32k
4k
8k
16k
32k
4k
8k
16k
32k
Sequence Length
Sequence Length
Sequence Length
SpecKV (1.58)
Target (328)
Snap
LAQ
Draf: (1.58)
1120
PyramidVV
(a) Qwen2.5: Draft model (1.5B), target model (32B
Cma
256
max
512
Cax 1024
100
Score
70
4k
8k
16k
32k
64k
4k
8k
16k
32k
64k
4k
8k
16k
32k
64k
Sequence Length
Sequence Length
Sequence Length
SpecKY (3B)
Target (70B)
SnapKV
LAQ++
Deaft (3R)
PymmidKV
Llama-3: Draft model (3.2-3B), target model (3.1-70B, 4-b))
Cmax 256
Cmax 512
Cmax 1024
100
90
Score
60
4k
8k
16k
32k
4k
8k
16k
32k
4k
16k Sequence Length
8k
32k
Sequente Length
Sequence Length
SpecKV (1.58)
Target (72R)
StapKV
L.AQ++
Dratt
H2O
PysmidKV
(c) Qwen2.5: Draft model (I.5B,, target model (72B, 4-bit)
Figure 10: Extended RULER results for KV cache dropping. Our proposed SpecKV method consistently outperforms SnapKV and LAQ++ across the majority of evaluated settings, often by substantial margin.
2H20 and PyramidKV are not plotted for Qwen2.5 at Canx 256 as their performance falls outside the visible range.
31

---

Published as a conference paper at ICLR 2026
Caa 1024
Cmax 2048
Cmax 3072
100
90
Score
60
4k
32k
4k
16k
32k
4k
8k
32k
Sequence Length
Sequence Length
Sequence Length
SpecPC
Torget
SpecPrefill
Doaft (0.58)
LIMMLingus-2
R2C
(a) Qwen2.5: Draft model (0.5B), target model (32B)
max
1024
Cmax 2048
Cmax 3072
100
Score
70
4k
8k
32k
64k
4k
32k
64k
4k
32k
64k
Sequence Length
Sequence Length
Sequence Length
SpeePC (1B)
Trrget (708)
CPC
SpecPretill (1B)
Deaft (1B)
LLMLingus
R2C
Llama-3: Draft model (3.2-1B), target model (3.1-70B, 4-bi))
max
1024
Cmax 2048
Cmax 3072
100
90
Serre
70
60
4k
8k
16k
32k
4k
8k
16k
32k
4k
8k
16k
32k
Sequence Length
Sequence Length
Sequence Length
SpecIC (0.58)
Target (728)
R2C
SpecPrefill (0.5B)
CPC
(c) Qwen2.5: Draft model (0.5B), target model (72B, 4-bit)
Figure 11: Extended RULER results on prompt compression. Our proposed SpecPC method consistently outperforms CPC, R2C, and SpecPrefill, maintaining strong performance long sequences.
32

---

Published as a conference paper at ICLR 2026
Table 9: LongBench results for Qwen2.5, featuring 0.5B (SpecPC) and 1.5B (SpecKV) draft models and 32B target model.
Single- doc QA
Multi- doc QA
Few-shot Learning
Code Completion
max
Method
Summary
All
Draft (0.5B)
19.07
26.58
20.90
53.51
32.48
30.37
Dense
Draft (1.5B)
36.16
35.01
22.79
63.92
36.62
39.06
Target (328)
56.01
43.99
25.90
64.06
44.74
47.78
H20
46.63
30.81
19.88
56,03
39.27
39.32
SnapKV
52.54
40.21
19.89
61.18
40.12
42.98
PyramidKV
50.92
37.26
18.90
63.24
40.20
43.19
LAQ+
55.15
44.14
22.24
63.25
41.19
45.79
SpecKV (1.5B)
53.48
43.77
24.02
63.79
44,80
46.06
H20
48.62
36.72
21.31
59.10
40.99
42.27
SnapKV
55.24
42,21
21.47
63.36
39.69
44.73
KV
512
PyramidKV
54.79
41.87
20.28
62.00
40.08
46.55
LAQ+
55.77
45.30
23.96
62.37
41.86
46.44
SpecKV (1.5B)
52.78
43.97
24.80
64.72
46.77
46.60
H2O
51.86
41.30
23.02
60.18
42.93
44.84
SnapKV
55.32
44.04
23.08
66.15
44.42
46.76
1024
PyramidKV
55.91
42.67
22.28
64.76
42.52
48.31
LAQ+
56.33
45.18
25.17
62.13
43.06
46.61
SpecKV (1.5B)
55.72
43.95
25.20
63.31
42.38
46.38
CPC
45.60
40.62
23.09
60.08
32.31
40,91
R2C
50.49
40.37
23.26
53.45
34.11
39.88
1024
SpeePrefill (0.5B)
45.94
39.32
23.16
62.04
43.17
42.70
SpeePC (0.5B)
51.23
41.40
23.37
62.26
38.23
43.66
CPC
51.01
42.31
23.74
60,92
35.83
43.26
R2C
50.32
42.66
24.08
59.11
40.54
44.19
PC
2048
SpecPrefill (0.5B)
$1.60
40.72
23.71
64.20
45.29
45,09
SpecPC (0.5B)
55.40
42.60
24.12
61.46
48.05
46.20
CPC
56,80
42.05
24.77
62.79
37.98
45.37
R2C
51.67
42.88
24.09
62.45
27.01
42.66
3072
SpecPrefill (0.5B)
53.85
42.92
24.48
64.03
45.76
46.24
SpecPC (0.5B)
56.89
42.42
24.71
62.66
47.49
46.79
33

---

Published as a conference paper at ICLR 2026
Table 10: LongBench results for Llama-3, featuring 3.2-IB (SpecPC) and 3.2-3B (SpecKV) draft models and 3.1-70B (4-bit) target model.
Single- doc QA
Multi- doc QA
Few-shot Learning
Code Completion
C max
Method
Summary
All
Draft (1B)
28.37
28.62
26.12
59.10
33.59
35.27
Dense
Draft (3B)
45.53
40.52
27.46
64.82
46.73
44.89
Target (70B)
55.02
47.06
28.61
70.47
48.19
49.99
H2O
54.07
41.30
22.55
49.10
54.14
43.52
SnapKV
55.88
45.30
22.49
62.15
55.49
47.75
256
PyramidKV
55.41
45.59
22.50
59.06
49,90
46.25
LAQ++
54.90
46.48
22,83
64.31
55.10
48.43
SpecKV (3B)
51.80
47.23
25.53
64.02
$8.75
48.80
H2O
55.01
43.86
23.85
58.48
52.00
46.26
SnapKV
56.08
46.94
24.40
63.34
52.14
48.32
KV
512
PyramidKV
54.73
47.07
24.20
63.62
47.05
47.35
LAQ+
55.06
47.45
24.58
63.39
51.75
48.36
SpecKV (3B)
53.31
47.47
27.33
67.49
54.19
49.66
H2O
54.19
45.77
26.03
63.62
49.58
47.71
SnapKV
54.94
47.40
25.97
67.56
48.13
48.85
1024
PyramidKV
56.32
46.82
26.20
62.58
48.28
48.02
LAQ++
55.19
47.17
26.43
67.41
47.16
48.61
SpecKV (3B)
53.32
46.02
27.41
66.40
50.66
48.63
CPC
45.14
39.41
24.86
61.40
37.58
41.97
R2C
48.93
42.01
25.38
58.91
40.19
43.29
1024
SpecPrefill (IB)
54.62
46,43
25.63
64.80
44.92
48.37
SpeePC (1B)
56.84
44.48
25.91
67.37
47.15
48.44
CPC
55.97
46.00
26.72
64.78
41.31
47.36
R2C
53.62
46,70
26.27
62.41
47.90
47.34
PC
2048
SpecPrefill (1B)
58.39
45.37
27.13
66.60
45.45
48.81
SpecPC (1B)
59.39
46.25
27.60
68.42
46.70
49.88
CPC
56.64
46.69
27.29
64.92
44.75
48.30
R2C
56.11
45.15
27.51
61.76
47.17
47.57
3072
SpecPrefill (1B)
57.75
46.15
27.15
64.70
42.50
48.02
SpecPC (1B)
58.47
48.07
27.72
65.43
41.28
48.69
34

---

Published as a conference paper at ICLR 2026
Table 11: LongBench results for Qwen2.5, featuring 0.5B (SpecPC) and 1.5B (SpecKV) draft models and 72B (4-bit) target model.
Single- doc QA
Multi- doc QA
Few-shot Learning
Code Completion
max
Method
Summary
All
Draft (0.5B)
19.07
26.58
20.90
53.51
32.48
30.37
Dense
Draft (1.5B)
36.16
35.01
22.79
63.92
36.62
39.06
Target (728)
58.70
45.89
26.24
64.83
51.08
49.22
H20
$3.68
35.05
21.10
50.84
48.87
41.41
SnapKV
55.85
40.12
21.65
55.82
50.41
44.37
PyramidKV
55.13
38.89
20.14
$3.02
42.91
LAQ+
57.26
43.54
22.95
62.53
55.79
46.98
SpecKV (1.5B)
53.94
42.71
25.07
66.58
46,93
47.06
H20
55.77
39,52
22.61
58.01
44.94
SnapKV
59.08
44.13
23.04
61.33
51.77
47.59
KV
512
PyramidKV
56.98
42.52
22.03
55.43
50.29
45.10
LAQ+
58.28
34,94
24.38
63.74
54.79
49.08
SpecKV (1.5B)
55.74
43.39
25.85
64.56
47 47.80
47.44
H2O
55.37
43.50
24.11
62.09
46.90
SnapKV
59.56
44.06
24.41
62.85
52.87
48.46
1024
PyramidKV
58.34
43.68
23.18
59.66
$1.45
46.96
LAQ+
58.66
35.37
26.35
60.90
53.56
49.59
SpecKV (1.5B)
56.04
44.27
25.71
63.46
49.90
47.73
CPC
46.37
38.36
24.19
52.54
28.30
38.64
R2C
55.78
40.10
24.40
47.62
33.17
40.72
SpeePrefill (0.5B)
49.04
40.32
24.41
63.53
50.12
45.16
SpeePC (0.5B)
48.52
45.34
24.14
61.04
48.30
45.26
CPC
55.14
43.53
24.71
49.46
33.21
41.78
R2C
54.61
45.70
24.99
53.65
42,43
44.41
PC
2048
SpecPrefill (0.5B)
$3.39
43.46
25.53
65.89
50.18
47.51
SpecPC (0.5B)
58.36
46.38
25.32
64.43
51.11
48.98
CPC
55.78
44,41
25.47
42,71
39,16
41.68
R2C
59.22
44.70
25.69
55.22
44,04
45.90
3072
SpecPrefill (0.5B)
55.75
45.71
25.93
64.22
51.63
48.44
SpecPC (0.5B)
59.84
44.95
25.91
64.18
46.88
48.46
35

---

Published as a conference paper at ICLR 2026
Qwen2.5 (14B)
Llama-3 (8B)
Qwen2.5 (14B)
Llama-3 (8B)
100
100
90
90
80
Score
Score
80
70
60
70
.0
60
4k
1nk
32k
4k
8k
16k
32k
64k
8k
16k
32k
4k
8k
16k
32k
64k
Sequence Length
Sequence Length
Sequence Length
Sequence Length
SpecKV
Tamps
PynmiKY
SpeePC
L.M.appud
Spee-
H20
StapKV
Deaft
CPC
Spachelll
StopKV
LAQu+
Torgsi
(a) KV dropping (Cmxx 256)
(b) Prompt compression (Cmax 1024)
Qwen2.5 (14B)
Llama-3 (8B)
Qwen2.5 (14B)
Llama-3 (8B)
90
901
80
Score
Score
80
70
60
70-
50
60
40
16k
32k
8k
32k
64k
8k
16k
32k
4k
8k
16k
32k
64k
Sequence L Length
Sequence I Length
Sequence I Length
Sequence Length
Specky
Target
ParuiiKV
SpeePC
Ads-SpecKV
H20
Sto-SoppKV
Domf
CPC
SpecPhotil
Druft
SnapKV
LAQ
Target
(c) KV dropping (Cmix 512)
(d) Prompt compression (Cmma 2048)
Qwen2.5 (14B)
Llama-3 (8B)
Qwen2.5 (14B)
Llama-3 (8B)
100
100
90
80
Scote
Szore
80
70
60
70
60
40
4k
8k
16k
32k
4k
8k
16k
32k
61k
4k
8k
16k
32k
4k
8k
16k
32k
6tk
Sequence Length
Sequence Length
Sequence Length
Sequence Length
SpecKY
Torga H20
PynmiKVV
SpeePC
M.Milngued
- RBC
Ada-Secck
Ada SnopKy
CPC
SpecPrettl
Dnit
StopkV
1.Q++
Target
(c) KV dropping (Gixx 1024)
(f) Prompt compression (Cman -0722
Figure 12: Performance of the proposed SpecKV and SpecPC on RULER compared to various baselines including V for Dwven2.5-0.5B-Instru and 2.ama-3.2-1B-Instttee and Qwen2.5- Qwen2255144B-nnstnee Inn-Itruttt and Llama-3.1-8B 1-51--8--nnsstue as target models. AdaKV can further boost the performance of SpecKV. V. Notably, SpecPC performs nearly on par with the target model. Low- performing methods are omitted.
draft models,
36

---

Published as a conference paper at ICLR 2026
Table 12: Results for LongBench performance with Qwen2.5-0.5B (draft) and Qwen2.5-14B (target).
Single- doc QA
Multi- doc QA
Few-shot Learning
Code Completion
Cmax
Method
Summary
All
Draft
21.04
24.4
21.23
54.07
33.39
30.64
Dense
Target
53.19
42.83
24.99
65.34
51.75
47.33
StreamingILLM
38.66
24.13
19.10
49.75
35.23
33.24
H2O
46.98
29.66
19.82
50.88
47.86
38.41
SnapKV
49.07
33.19
19.49
54.49
47.34
40.25
PyramidKV
47.07
31.84
18.32
54.81
43.26
38.76
256
Ada Ada-SnapKV
50.92
34.57
19.63
55.07
49.58
41.41
LAQ++
52.45
39.99
22.12
61.46
51.32
45.05
SpecKV
51.07
38.76
23.20
59.68
49.58
44.09
Ada-SpecKV
51.92
39.38
24.90
61.05
53.43
45.61
StreamingLLMM
38.97
27.04
21.14
53.78
36.52
35.41
H10
47.56
33.16
21.02
53.45
49.78
40.37
SnapKV
50.74
38.33
20.97
57.81
49.94
43.10
PyramidKV
50.19
36.37
20.25
58.17
47.88
42.19
KV
512
Ada-SnapKV
51.69
38.61
21.48
59.84
51.85
44.18
LAQ++
52.31
42.06
23,75
61.62
51.64
45.89
SpeeKV
51.98
41.51
24.11
62.60
52.33
46.09
Ada-SpeeKV
51.00
42.18
25.74
63.74
55.28
48.46
Streaming
40.08
32.01
22.12
55.42
38.36
37.54
H20
49.73
37.27
22.38
55.94
51.26
42.75
SnapKV
51.56
40.98
22.65
63.90
51.48
45.73
PyramidKV
51.42
40.93
21.76
59.96
49.92
44.43
1024
Ada-SnapKY
51.81
40.96
22.83
64.14
51.97
45.94
LAQ++
52.27
42.59
25.04
61.56
51.68
46.27
SpecKV
52.92
41.85
24.30
63,46
52,50
46.61
Ada-SpecKV
52.27
43.22
26.39
64.83
54.40
47.78
IL.MLingua-
27.10
23.74
22.57
35.85
37.62
28.79
CPC
42.76
36.05
22.58
48.38
35.57
37.17
1024
R2C
46.43
37.42
22.64
47.21
29.48
37.15
SpecPretill
46.21
37.21
22,82
62.18
48.34
43.00
SpeePC
47.70
38.30
23.30
59.74
52.52
43.73
LLMLingua-2
35.10
32,22
23.51
39.82
44,80
34.40
CPC
48.74
39.70
23.39
55.77
42.07
41.92
PC
2048
R2C
50.68
40.65
23.47
56.72
45.62
43.27
SpecPrefill
49.75
38.58
23.70
65.05
50.40
45.15
SpeePC
53.23
41.43
23.68
63.26
54.92
46.76
LLMI.nngua22
43.50
35.67
24.10
45.66
47.32
38.67
CPC
51.59
41.08
23.84
58.14
44.82
43.83
3072
R2C
51.71
41.39
23.89
62.09
48.58
45.31
SpecPrefill
51.81
41.27
24.26
64.28
50.66
46.15
SpeePC
53.25
41.48
24.49
64.51
54.41
47.14
37

---

Published as a conference paper at ICLR 2026
Table 13: Results for LongBench performance with Llama-3.2-1B (draft) and Llama-3, 1.1-BB (target).
Single- doc QA
Multi- doc QA
Few-shot Learning
Code Completion
Cmax
Method
Summary
All
Draft
28.08
27.27
25,65
60.16
31.11
34.69
Dense
Target
45.85
43.79
28.68
66.65
50.46
46.84
StreamingILLM
38.69
27.12
21.64
50.75
34.74
34.58
H2O
43.54
36.81
22.62
55.64
47.81
40.82
SnapKV
43.79
37.31
21.96
56.29
47.34
40.91
PyramidKV
43.62
37.79
21.75
55.15
46.32
40.54
256
Ada Ada-SnapKV
44.04
37.86
22.34
59.95
50.39
42.38
LAQ++
45.81
41.72
22.93
64.18
47.21
44.17
SpecKV
43.23
39.73
24.43
60.90
51.09
43.36
Ada-SpecKV
42,40
40.73
25.74
57.94
52.51
43.25
StreamingLLMM
38.80
29.15
24.16
52.59
37.25
36.33
H10
44.82
39.36
23.95
58.70
49.30
42.79
SnapKV
45.00
41.31
23.61
61.36
49.92
43.83
PyramidKV
45.26
41.22
23.36
61.15
48.06
43.51
KV
512
Ada-SnapKV
45.09
41.17
24.19
61.81
$1.74
44.30
LAQ++
46.17
43.44
24.17
63.13
48.74
44.87
SpeeKV
43.48
41.86
26.17
62.34
51,36
44.59
Ada-SpeeKV
43.80
42.56
26.59
58.72
54.41
44.56
Streaming
38.45
32.54
25.03
58.09
38.60
38.54
H20
45.45
42.50
25.42
59.17
50.10
44.13
SnapKV
45.46
43.15
25,42
62.06
$2.37
45,22
PyramidKV
46.10
42.78
25.17
49.55
45.11
1024
Ada-SnapKY
46.06
43.34
25.48
63.47
50,50
45.43
LAQ++
46.21
44.14
25.71
64.10
50.00
45.75
SpecKV
43.73
43.39
26.95
63.14
51.28
45.30
Ada-SpecKV
44.83
43.72
27.50
61.04
54.23
45.70
IL.MLingua-
29.61
24.83
23.43
24.18
40.66
27.68
CPC
35.67
36.61
25.26
34.01
43.58
34.42
1024
R2C
38.41
39.07
25.28
43.26
43.99
37.58
SpecPretill
41.51
39.38
25.82
63.51
44.68
42.86
SpeePC
44.83
39.94
25.85
63.70
44.82
43.76
LLMLingua-2
34.00
32.51
24.90
24.76
47.27
31.64
CPC
40.02
39.41
26,83
39,02
46.66
37.80
PC
2048
R2C
44.53
38.97
26.63
54.62
46.67
41.97
SpecPrefill
42.35
42.11
27.23
63.56
44,46
43,91
SpeePC
44.92
40.71
27.30
64.77
46.89
44.78
LLMILnngua22
39.13
35.44
25.98
29.73
49,92
35.05
CPC
41.73
39.52
27.27
42.12
49.13
39.30
3072
R2C
44.77
40.97
27.35
60.85
48.08
44.14
SpecPrefill
43.86
42.45
27.82
63.17
45.25
44.46
SpeePC
47.12
41.95
28.02
65.61
45.52
45.65
38

---

Published as a conference paper at ICLR 2026
EgocentricNavigation
MovingDirectiio
Scene Transition
92.5
50
Senre
45
90.0
40
40
96
128
96
96 Cmax
Casa
Cinax
WebQA
TQA
VQA
80
85
Score
60
40
70
128
96
128
CanA
Cinx
Cmax
SpecKV
Target H20
SmupKV
Draft
PymmidKV
(a) KV dropping
EgocentricNavigationn
MovingDirection
50
90
Score
40
80
30
512
768
1024
512
768
1024
512
768
1024
Cmxx
Cmux
Cmxx
WebQA
TQA
VIi
85
80
Score
80
70
70
512
768
1024
512
768
1024
512
768
1024
Cmxx
Cmux
Cmxx
SpeePC
Target
FasVV k
Draft
FastV k
(b) Prompt compression
(draft) and Qwen2.5- (target). SpecKV demonstrates competitive performance across most tasks and achieves substantial improvement on WebQA. SpecPC consistently outperforms both FastV configurations the majority of datasets.
Figure 13: MileBench multi-modal results using
39

---

Published as a conference paper at ICLR 2026
Roui 16
R² 0.90
Aº = 0.90
32
R² 0.82
0.84
R' 0.84
Poot 64
R² 0.69
0.74
0.77
Aº² 0.76
128
R² 0.54
R² 0.59
R² 0.63
R² 0.68
R² 0.67
256
R²
0.48
R²
Rº 0.58
512
R² 0.35
0.40
R² 0.43
R² 0.49
R 0.53
0.57
R² 0.57
0
16
32
64
128
256
512
Figure 14: Correlation of importance scores from SpecKV against ground-truth scores from the dense target model for various output lengths (Now) and lookaheads (Mookkhheaa) Increasing Tiookahend improves correlation, especially for longer output lengths (Now). SnapK V (Mookkhhed = 0) consistently shows the lowest correlation. Experiments use a Qwen2.5-0.55 draft model and a Qwen2.5-32B target model on Multi-news from LongBench.
40

---

Published as a conference paper at ICLR 2026
Rooi 16
R² 00911
Rº = 0.94
32
R² 0.82
0.88
R' 0.87
Poot 64
R² 0.64
0.70
0.75
Aº 0.71
Rout = 128
R² 0.48
R² 0.54
R² 0.59
R² 0.61
R 0.61
Bast 256
R²
0.42
A
R²
0.48
512
R² 0.26
0.31
R² 0.35
R² 0.39
R² 0.42
0.44
R² 0.45
0
16
32
64
128
256
512
Figure 15: Correlation of importance scores from SpecKV against ground-truth scores from the dense target model for various output lengths (Now) and lookaheads (Moonkhheat). Increasing Tiookahead improves correlation, especially for longer output lengths (Now). SnapK (Rlookaheud = 0) shows the lowest correlation. Experiments use Qwen2.5-0. 2-0.BBB draft model and Qwen2.5-32B target model on GovReport from LongBench.
41

---

Published as a conference paper at ICLR 2026
40
30
Time (6)
20
10
Tappy
Sugk?
Lag0e
Speokt
Speck Pepe
Tarpor
Stantry
LAtk-
Spoak Specch
Speck
Tagar
fuanty
LADee
Speck
Speckt
Spock PC
Pepor
Sount
Lagke
Spo
Sjork?
Spock
Tayen
Seupl?
Lao
Spor Sperky
Speck PoP
mi
nl
Rn 16k
fie
Ple
6%k
Tagget Picfill
Tareet Dovode
Target Lookaabaaaaaa.o..oo.o
Druft Lockabead (SpeeEV)
Figure 16: End-to-end latency comparison across five input sequence lengths (Ni)) and four output sequence lengths (Nom E (64, 128, 256, 512)). All experiments use the Qwen2.5 model (3B draft, 32B target) with Caua = 256. For SpecKV algorithms, we use Alookahead Routt Each bar segment breaks down the latency by processing stage, and the four bars for each method correspond the four Toorr values. SpecK 8 denotes SpecKV without sparse prefill, while SpecK VVPC combines SpecK V with SpecPC. Notably, when vinn 2 16k, the efficiency gains from sparse prefill (SpecKV) V) and prompt compression (SpecKV-PC) effectively offset the lookahead overhead for Rlookahead I 64 and < 512, respectively. Overall, SpecK V-PC is the most efficient method; by precompressing the prompt, significantly cuts target prefill time, resulting in speedup of about 40% to 75% (depending on output length) over LAQ++ at 64k input context.
80
Peak Memory (GB)
40
20
Tager Soonts Specki Spoc
Swantt Speckte Spocky E.p
Snaptt
Luoks Spcck Fer
Spokt
Speak PC
Shapks Rt
Spec Speek
speak SC
Tagger RV
SpecK2 Sweck Speck S.RkF-pe ?
4k
Rio 16k
Ain
32k
nn 64k
Total Peak Memcey
Draft Model Monory (SpecKV)
Figure 17: Peak memory usage across five input sequence lengths (nin) using the Qwen2.5 model (3B draft, 32B target) with Cmax 256 and Tloookahcad = Rour SpecKV* denotes SpecKV without sparse prefill, while SpecKV-PC combines SpecKV with SpecPC. LAQ++ offers no peak memory savings over the target model, as it must store the full cache. SpecKV is slightly more memory-hungy than SnapKV because also stores the draft model weights (indicated in green), though this constitutes small fraction of the total memory. SpecKV-PC is the most memory-efficient method; it first compresses prompt (to 2048 tokens here) before the target model's prefill phase, significantly reducing peak usage. At 64k context length, SpecKV-PC saves approximately 9 GB over SnapKV and 25 GB over LAQ++.
42

---

Published as a conference paper at ICLR 2026
95
Score
85
75
4%
8k
16k
32k
Sequence Length
Targei
Qwar3-0.46
5-31
Owen2.5.0.8
Qwea2.
Figure 18: Effect of draft model quality RULER score. The target model is Qwen2.5-144 (Instruct). Consistent with expectations, employing more capable draft models boosts performance. For reference, also evaluate oracle setting where the draft model is identical the target (Qwen2.5-14B), representing an empirical upper bound for SpecKV.
1.00
0.8
Relative Scote
0.95
0.7
0.80
64
128
Number of Draft Tokens (Mascakedd)
Number of Dralt Tokens (Mockaeea))
Stngledecanest QA
Sumurnicatioo
Corpistion
RULLER SK.
Muli-decataon QA
Fey dar Louning
(a) SpecKV on RULER
(b) SpecK on LongBench
0.99
0.98
Relative Score
1.00
0.97
0.90
0.95
128
Namber of Draft Tokens (Mtastahew))
Number of Draft Tokens (Buokabea))
Stegle-dcssmerro Mast-decameen UA
Compution
BULLMSE
Leaning
(c) SpeePC on RULER
(d) SpeePC on LongBench
Figure 19: Impact of the number of generated draft tokens, Niookahead- on the relative performance of SpecKV V and SpecPC, using Qwen2.5-64 3-0.5BBInsttruc the draft model and Qwen2.5- target model. The relative score calculated as the score each SpecKV V configuration divided by the score of the full dense target model. Increasing Plookatead substantially boosts SpecK score, whereas SpeePC shows only minor improvement with higher 7ijokahend-- SnapKV (Niookahed 0) has the lowest performance in most cases. co denotes lookahead to the EOS token.
as
43

---

Published as a conference paper at ICLR 2026
0.90
1.05
0.85
Reluive Score
1.00
0.80
256
512
1024
2048
4096
255
512
1024
2048
4096
Spunitty OTen and Rewaa)
Spunily Chua and Raaa)
RUTER
RUSER
RIEER
Singledreamee
Summortriion
Coda
Loming
(a) RULER
(b) LongBench
Figure 20: Impact of different sparsity levels in SpecKV's sparse prefill relative performance, using Qwen2.5-0.35 1...5-00--nnttruee as the draft model and Qwen2.5- 1eten2.5--4--Issttiie InB-tnstruct as the target model. We vary Thven (set equal Nisash)) and observe the impact on accuracy. Accuracy improves as sparsity decreases (higher Rwenn) up to 2048, beyond which gains saturate, hence our choice of 2048 for main results. corresponds to fully dense prefill. Notably, for some LongBench tasks, higher sparsity actually benefits accuracy. The relative score is calculated as the score of each SpecKV configuration divided by the score of the full dense target model.
44


Reviews:
[{"review_id": "tjbk7eRDN9", "reviewer": "Reviewer_zgDL", "summary": "This paper proposes Draft-based Approximate Inference, leveraging lightweight draft models for lookahead-based token/KV importance estimation. It introduces SpecKV for KV cache dropping with sparse prefill and SpecPC for prompt compression, both supported by theoretical proofs and extensive experiments showing superior accuracy–efficiency trade-offs over baselines.", "questions": "1. The core premise is that approximate future information improves token/KV importance estimation. Could the authors present a quantitative comparison of importance score distributions obtained with lookahead versus current-token-only methods? If the distributions differ substantially, would SpecKV’s advantage over methods like SnapKV diminish when output length greatly exceeds $n_{lookahead}$?  \n2. The acceleration analysis is limited. Since SpecKV and SpecPC incur additional draft-model overhead, can the authors provide a detailed latency breakdown (draft inference, dense/sparse prefill, decoding) for varying input/output lengths?  \n3. Given that SpecKV employs sparse prefill, has its speed been compared directly to optimized prefill approaches such as MInference [1]?  \n4. Can SpecKV and SpecPC be combined in a single pipeline, and if so, could the authors include ablation studies showing the individual and combined contributions to speed, memory reduction, and accuracy?\n\n[1] MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. First integration of draft-model lookahead into KV dropping and prompt compression, with theoretical justification. \n2. Strong empirical gains across diverse benchmarks, models, and compression budgets.  \n3. Clear motivation, concise algorithms, and well-presented results.", "weaknesses": "1. Lacks analysis of importance score differences with/without lookahead\n2. Limited breakdown of the latency trade-off\n3. Unclear whether SpecKV and SpecPC can be effectively combined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N8Slj3gPbO", "reviewer": "Reviewer_bFv3", "summary": "This paper proposes an effective \"Draft-based Approximate Inference\" framework that leverages a draft model for lookahead to more accurately estimate the importance of tokens and KV pairs, thereby improving long-context approximate inference. The authors further introduce SpecKV and SpecPC for KV cache dropping and prompt compression, respectively, and provide a thorough error analysis. A comprehensive set of experiments further demonstrates the effectiveness and application potential of the proposed method.", "questions": "1. I am curious whether using different types of draft models and target models would affect token selection (e.g., Qwen-2.5-0.5B + Llama-3-70B). Does this imply that draft models must be selected from the same model family as the target model?\n2. As shown in Figure 2, the benefits gained from the draft model vary across different tasks. Any explaination?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By using a draft model to estimate the importance of tokens in the KV cache and prompt, the method achieves strong performance under controllable complexity.\n2. This work provide a clear theoretical analysis, demonstrating how embedding errors influence KV importance estimation errors (Theorem 1), and how output approximation under RIP or more general assumptions can upper bound attention approximation errors (Theorems 2 and 3).\n3. The experiments on LongBench and RULER Benchmarks are solid, with results that convincingly demonstrate the effectiveness of the proposed methods.", "weaknesses": "1. For different input embeddings, are there any limitations to the applicability of Theorem 2?\n2. There appear to be some typo errors in Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vkGtve9tCR", "reviewer": "Reviewer_QfmW", "summary": "This paper proposes a framework called draft-based approximate inference to optimize the efficiency of LLM in long context decoding. Existing methods, which approximate KV-cache discarding, sparse attention, and prompt compression, usually rely on rough prediction when estimating the importance of tokens or KV pairs. In contrast, the core idea of this paper is to use a small and cheap \"draft model\" to generate approximate future output (lookahead), so as to more accurately predict the importance of the current token or KV pair.\n\nIn this paper, two methods are introduced: \n\n1. SpecKV, whose goal is to use the \"look ahead\" capability of the draft model to more accurately discard unimportant parts in the kV cache, and combine sparse pre-filling to improve efficiency.\n\n2. SpecPC, whose goal is to use the draft model to directly determine which tokens in the input prompt are not important, and compress the prompt before sending it to the target model.\n\n\nA large number of experiments on the benchmark of rule and longbench show that speckv and specpc continuously achieve higher accuracy than the existing baseline methods under the fixed KV cache or prompt size limit.", "questions": "How does SpecKV reduce the peak memory than LAQ++? In Algorithm 1, the target model still needs to store all the KV cache of the input sequence x and draft output y_draft.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides theoretical (Theorems 1 and 2) and experimental evidence to support the effectiveness of the \"look ahead\" based importance estimation. \n\n2. This framework unifies and extends the idea of using approximate future information to improve token importance estimation.\n\n3. The paper claims that its method achieves the current state-of-the-art accuracy in the long context benchmark under the constraint of a fixed KV cache or prompt size.\n\n4. Even if a weak draft model is used, this paper can perform well, far exceeding the performance of the draft model itself. Also, using a better draft model will further improve performance.", "weaknesses": "1. The core of the whole framework is to use the draft model to approximate the behavior of the target model. Both theoretical analysis (Theorem 1) and experimental results (Fig. 10) show that the accuracy of the draft model directly affects the final performance. If a draft model that is small (low overhead) and similar enough to the target model (high accuracy) cannot be found, the effect of SpecKV and SpecPC may be compromised.\n\n2. Compared with methods such as SnapKV, which only pre-fills and compresses the target model once, the pre-filling steps of SpecKV are more complex, and the calculation cost is higher. Although the paper claims that the overall delay is reduced, this is mainly the benefit of the decoding stage.\n\n3. This paper also introduces a new memory occupation: it needs to load and store the weight of the draft model. Although this overhead is fixed and does not increase with the sequence length as KV cache, it is still an additional memory burden compared with methods that do not require a draft model.\n\n4. The main idea is similar to speculative decoding and previous LAQ++. It is more likely a technical extension.\n\n\nI have a borderline opinion on this paper and hope to see the rebuttal and other reviewers' comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tbLZTt5HH4", "reviewer": "Reviewer_tK1Y", "summary": "This paper proposes Draft-based Approximate Inference (DAI), a framework that leverages a small draft model to generate a short sequence of lookahead tokens, which are then used to estimate token or KV importance in large language models (LLMs).\nUnder this framework, the authors instantiate two methods:\n\nSpecKV — for lookahead-guided KV cache dropping\n\nSpecPC — for prompt compression\n\nThe key idea is to use the draft model’s predicted future queries to obtain a more accurate estimate of which tokens will be attended to in future steps, enabling more effective prefill-time cache compression.\nEmpirical results on RULER and LongBench show that the proposed methods outperform prior works such as SnapKV, LAQ++, and SpecPrefill, with reduced memory and latency.\nThe paper also provides theoretical bounds relating draft embedding quality to attention approximation error.", "questions": "How sensitive are the results to $n_{lookahead}$? The ablation in Appendix E.4 is informative, but further scaling analysis would be valuable.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.  Clean and implementable idea:\nExtends speculative decoding to approximate inference in a conceptually neat way, requiring minimal system changes.\n\n2.  Strong empirical results:\nOn both synthetic and long-context benchmarks, SpecKV yields clear gains (1–2 points improvement) over existing KV compression baselines, with ~40–50% memory savings.\n\n3.  Solid theoretical justification:\nProvides error bounds connecting the draft model’s embedding deviation to importance estimation error, filling a theoretical gap that earlier methods (e.g., SnapKV, LAQ++) lacked.\n\n4.  System efficiency:\nThe overhead of lookahead generation is minimal (<10% in prefill), while decoding latency and memory are significantly reduced.", "weaknesses": "1. Limited conceptual novelty:\nThe main idea—using a smaller model to predict future attention patterns—is a natural extension of speculative decoding (FastGen, Medusa) and prior KV-dropping methods (SnapKV, LAQ++).\nThe improvement lies mainly in integration and theoretical refinement, not in introducing a new paradigm.\n2. Marginal accuracy gains:\nImprovements on benchmarks are moderate (1–2%), suggesting the practical benefit mainly comes from efficiency rather than substantial modeling advance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)