LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation
Authors: Jinwoo Ahn
Ingyu Seong
Akhil Kedia
Junhan Kim
Hyemi Jang
Kangwook Lee
Yongkweon Jeon
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Efficiency
Key-Value Cache Compression
Long-Context LLM
Inference Optimization
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a novel method that augments the LLM with parameter-efficient modules to perform fast and accurate KV cache eviction by predicting the attention pattern of the model's future response.

Abstract: Transformer-based large language models (LLMs) rely on key–value (KV) caching to avoid redundant computation during autoregressive inference. While this mechanism greatly improves efficiency, the cache size grows linearly with the input sequence length, quickly becoming a bottleneck for long‑context tasks. Existing solutions mitigate this problem by evicting prompt KV that are deemed unimportant, guided by estimated importance scores. Notably, a recent line of work proposes to improve eviction quality by “glimpsing into the future”, in which a draft generator produces a surrogate future response approximating the target model's true response, and this surrogate is subsequently used to estimate the importance of cached KV more accurately. However, these approaches rely on computationally expensive draft generation, which introduces substantial prefilling overhead and limits their practicality in real-world deployment. To address this challenge, we propose LookaheadKV, a lightweight eviction framework that leverages the strength of surrogate future response without requiring explicit draft generation. LookaheadKV augments transformer layers with parameter‑efficient modules trained to predict true importance scores with high accuracy. Our design ensures negligible runtime overhead comparable to existing inexpensive heuristics, while achieving accuracy superior to more costly approximation methods. Extensive experiments on long-context understanding benchmarks, across a wide range of models, demonstrate that our method not only outperforms recent competitive baselines in various long-context understanding tasks, but also reduces the eviction cost by up to $14.5$×, leading to significantly faster time-to-first-token. Our code is available at https://github.com/SamsungLabs/LookaheadKV.

Published as a conference paper at ICLR 2026
LOOKAHEADKV: FAST AND ACCURATE KV CACHE EVICTION BY GLIMPSING INTO THE FUTURE WITHOUT GENERATION
Jinwoo Ahn'
Ingyu Seong"
Akhil Kedia
Junhan Kim
Hyemi Jang
Kangwook Leef
Yongkweon Jeon"
Samsung Research
(jinwoo. ingyu.seong, kw.brian lee,
dragwon. amonngsamsung.com
ABSTRACT
Transformer-based large language models (LLMs) rely on key-value (KV) caching to avoid redundant computation during autoregressive inference. While this mechanism greatly improves efficiency, the cache size grows linearly with the input sequence length, quickly becoming bottleneck for long-context tasks. Existing solutions mitigate this problem by evicting prompt KV that are deemed unimportant, guided by estimated importance scores. Notably, recent line of work proposes to improve eviction quality by "glimpsing into the future", in which draft generator produces surrogate future response approximating the target model's true response, and this surrogate is subsequently used to estimate the importance of cached KV more accurately. However, these approaches rely on computationally expensive draft generation, which introduces substantial pre- filling overhead and limits their practicality in real-world deployment. To ad- a lightweight eviction frame-
dress this challenge, we propose work that leverages the strength of surrogate future response without requiring explicit draft generation. LOOKAΙHADK augments transformer layers with parameter-efffcient modules trained to predict true importance scores with high accuracy. Our design ensures negligible runtime overhead comparable to existing inexpensive heuristics, while achieving accuracy superior to more costly approx- imation methods. Extensive experiments long-context understanding bench- marks, across a wide range of models, demonstrate that our method not only outperforms recent competitive baselines in various long-context understanding tasks, but also reduces the eviction cost by up to 14.5x, leading to significantly faster Lookahheak)) SamsungLabs/
Our code available https:/ htaps:://gtthbb. //ggtbttbbb.tt////.////g com/
INTRODUCTION
Extending the context length of Large Language Models (LLMs) is becoming increasingly crit- ical for many emerging applications: processing long documents (Bai et al. 2024; Wang et al.
2024 Hsieh et al.] 2024), repository-level code understanding and generation (Luo al. 2024
Liu et al.] 2024] Jimenez al. 2024). in-context learning (Li et al. 2025 Agarwal et al. 2024). etc. However, a central challenge in enabling these applications is that the key-value (KV) cache size grows linearly in sequence length, which rapidly becomes bottleneck for inference. restrict- ing scalable deployment of such applications. For example, even for moderate-sized models, such as L.LMA3.1 1-70B (Dubey et al. (024)) in half-precision, storing single 128K-token sequence al- ready takes up 40GB of memory, while scaling to 1M tokens requires 320GB, exceeding the memory capacity of high-end consumer hardware.
A growing line of work addresses this challenge by identifying salient tokens to achieve effective
KV cache eviction without loss of performance (Li et al. 2024 Cai al. 2024 Galim al. 2026
Equal contribution Corresponding authors.

---

Published as a conference paper at ICLR 2026
Lockahead LoRA on Training
Trainable
Frozen
W
DLARA
Input
Model Response (pre- genersted)
Ingut
Leumable Lookabeed Tokens
Input
Leannd Lookahuud Tokuss
Lookahead LoRA on Prefill
W
Model
Lestable Looksheed Tokees
Leared
DIARA
Ruegerso
Looksined Tolers
Σ
Σ
KL Div. Loss
Lookahead LoRA on Decode
True importance scores
Predicted importance scores
Compressed KV
W
(a) LookaheadK V training
(b) LookaheadKV inference
Figure An overview of LOOKAHEADKY (a) Training. During training, lookahead tokens and lookahead LoRA trained predict the ground-truth importance scores obtained with pre- generated model response via KL divergence loss. (b) Inference. During prefill, LOOKAHEADKY utilizes the learned modules to identify essential tokens and compress the KV cache, facilitating memory-efficient decoding.
Wang et al. 2025 Zhang al. 2023 Early methods often rely simple heuristics, in which token importance is estimated based on the self-attention scores of a subset of the input tokens. SnapKV (ii et al. 2024), for instance, leverages the attention weights between the suffix of the input and the preceding context to estimate the importance of each prompt token. More recently, several studies (Galim al., 2026 Wang al.. 2025) reveal that leveraging the model's response, rather than the input prompt, can greatly improve the eviction quality. Furthermore, they show that low-cost generated draft response (generated using a smaller draft model (Galim al., (2026), for instance), which closely approximates the true response, can serve a powerful proxy for accurately estimating the importance scores.
While these draft-based methods substantially improve eviction quality, they still face trade-off between per- formance and latency, since their draft token generation step computationally expensive. Figure presents the trade-off between accuracy and overhead of different ap- proaches using the QASPER benchmark (Dasigi al. 2021) and LLaMA3.1-88 1-..3.1----IInttur (Dubey al. 2024).
QASPER score FulkKV
preseettt
40.5
LookaheadKV
60.0
LAQ
OASPER
While simpler approaches like SnapKV induce minimal latency overhead, they suffer
performance degra-
dation under highly constrained budget settings. On the other hand, Lookahead Q-Cache (LAQ) (Wang 2025), draft-based approach, shows impressive results even extremely limited budget settings. However, this approach incurs prohibitive computational overhead by generating an extra draft response, which limits its prac- ticality in latency-sensitive applications such as mobile devices.
360
SnapkV
Time to- Ros-irs-lloknnn TTFT (ms)
Figure 2: Accuracy-overhead trade-off
acToss KV cache eviction methods.
To overcome this limitation, we introduce LOOK AOOKAH.A novel KV cache eviction method that augments LLMs with parameter-efffcient modules, capable of accurately predicting future attention patterns, eliminating the need for costly draft token generation. As shown in Figure 2 our method effectively overcomes the eecuracy-overhead trade-off, achieving minimal performance loss with negligible overhead. LOOKAHEADKV as depicted in Figure I employs tokens, together with lookahead LoRA modules, novel selectively activated low-rank adapters, to produce queries that can reliably estimate token-importance scores. By fine-tuning them predict
of learnable special
2

---

Published as a conference paper at ICLR 2026
the true importance scores, LOOKAHEADKY effectively minimizes the quality loss incurred by KV cache eviction with marginal inference overhead.
To rigorously assess the effectiveness of LOOKAHEADKV.
evaluate it diverse set of
long-context benchmarks (Bai et al. 2024 Hsich et al. 2024 Ye et al. 2025 Zheng et al. 2023) across multiple models of varying sizes (Dubey
2024 Yang al. 2025) Experimental
sults consistently demonstrate that LOOKAHEADKY outperforms strong baselines across multiple budgets and context lengths while incurring significantly less eviction latency.
To summarize, our contributions are as follows:
We propose LOOKAHEADKY, a novel KV cache eviction framework that employs learn- able lookahead tokens and special LoRA modules to predict the importance scores from the model's 's true response without explicitly generating costly approximate response.
Through extensive experiments, we demonstrate that the proposed approach is effective and robust across different models and context lengths. It remains superior in low-budget settings, providing useful solution in resource-constrainee environments.
By conducting rigorous analysis of eviction latency, both theoretically and empirically, show that our method incurs negligible eviction overhead of less than 2.16% at 32K context length, which is up 14.5x lower than the overhead incurred by
approaches.
BACKGROUND
The primary objective of the KV cache eviction methods considered in this work, including our proposed approach, is to accurately estimate the importance score of individual key-value pairs of prompt tokens using attention weights, in order to guide the eviction process. In the following section, we formally define the problem of KV cache eviction and briefly discuss how prior methods have approached it.
KV Cache Eviction Using Importance Scores.
Let X (111, (..,,-.,TTa)) } be input token sequence (e.g., a user instruction, part of a code snippet, etc.) and Y = (V1, Unon } the model's generated
response X, For given layer and attention head an LLM, the attention scores of the complete sequence are given by:
X Y
X
QK Vd
Q
W
K
W
A Softmax
(1)
where X
[1, x,, IT E Rd and Y = [y1...., T E Rdaaxd are the hidden states omit
of the input prompt and model-generated response, respectively. For better readability, the layer and head index. We define the ground-truth importance scores SGT = [81,..., 8n,, of the KV cache the average cross-attention scores between the queries of Y and the keys of X, i.e., A i.j. Intuitively, these scores quantify the relative contribution of each prompt token's 's key-value pair to the model's response generation. Based these scores, the pruned KV cache can be obtained by retaining subset of (e.g., TopK) important KV pairs to minimize the attention output perturbation, such that Attn(x, KVorgg) Attn(z, KVGT). where KVorg and KVer are the original and evicted KV cache using the ground-truth importance scores, respectively.
8j - Now E. Putt ther
However, since the model's 's true future response is unknown during the prefill phase, such scores cannot be computed directly. Consequently, prior methods resorted to constructing surrogate re- sponse sequence Y = [i1...., Draname Roisto x to approximate the model's (partial) future response and predict the attention pattern:
X Y
QKT Vd
=
W
K
WE
Á = Softmax
(2)
resulting in the estimated importance score vector Supprox [5 .... S.,, whose entries are computed
8j Reppoow 2 S À j.j- In short, these methods aim obtain the estimated score vector m,+1
whose ranking is similar that of the ground-truth, such that the overlap between the retained KV
3

---

Published as a conference paper at ICLR 2026
400%
Method
Method
Ours
Ours
Overheed
150%
Specky
Overhved (S)
800%
SpeckY
SnapKV
20056
150%
50%
100%
50%
5,000
15,000
15,000
(bokenss)
(tolkens)
(a) Thcoretical lateney overhead
(b) Actual lateney overhead
Figure 3: Time-to-First First-Token (TTFT) latency overhead ratio across context lengths. Similar to Vn BOOKAHEADK? introduces negligible TTFT overhead across all tested context lengths; draft-based methods (LAQ, SpecK incur substantial latency, especially for shorter contexts.
pairs and KVer er is high. Various approaches have been suggested approximate the future response for effective KV cache eviction.
Prompt-based Approaches. SnapKV (Li al. 2024) uses the suffix of input prompt to compute the estimate of the future importance scores. It has been widely adopted as a simple and effective KV cache eviction method because can reuse attention weights from the prefill forward pass, requiring only marginal extra computation.
Recently, several works have proposed to use a low-cost generator to generate a (partial) approximate response first, and subsequently use it to estimate the future impor- tance scores. For example, SpecKV (Galim et al. (026) employs a smaller LLM to generate a draft response, while Lookahead Q-Cache (LAQ) /Wang al. 2025) first applies SnapKV to the target model to generate a draft response, which is in turn used to approximate the future salience.
Draft-based Approaches.
These draft-based methods have consistently shown superior performance compared to simple heuristics (Li et al.) 2024), demonstrating the effectiveness of employing surrogate future response. However, the explicit draft generation step still incurs substantial additional compute, resulting in significant increase in latency, as shown in Figure 3 In summary, existing methods face a clear trade-off: inexpensive heuristics are fast but less accurate, whereas draft-based techniques improve performance at the cost of increased inference time.
PROPOSED METHOD: LOOKAHEADKV
To overcome the challenge of fast and accurate importance prediction, we introduce LOOKA- HEADKV, framework that augments the I.LM with a set of lightweight learnable modules which are optimized to predict ground-truth importance scores. LOOKAHEADKY V achieves the best of both worlds by glimpsing into the future without generation: 1) eliminates the need for the explicit draft generation step, resulting in significantly faster KV cache eviction. 2) it employs learned spe- cial tokens that serve as implicit future response for importance estimation, leveraging the strength of draft-based methods without their computational overhead.
3.1 MAIN COMPONENTS
Learnable Lookahead Tokens. LOOKAHEADKY performs KV cache eviction using set of learn- able special tokens during the prefill phase, followed by auto-regressive decoding with the preserved KV cache. For given input sequence X, our framework appends sequence of trainable soft looka- head tokens = {P1, Presates) whose queries each attention head are used to estimate the attention pattern of the true model response. In other words, these tokens are trained to compress the attention information of the true response to serve as the "observation window" in the eviction phase. These are initialized randomly and added to the vocabulary before training, Note that the lookahead tokens are used during the prefill stage only for eviction, and introduce no overhead for the decoding stage.
4

---

Published as a conference paper at ICLR 2026
To enhance the quality of estimation, we introduce lookahead LoRA, novel low-rank adapter module that only activates for the lookahead tokens. Lookahead LoRA provides complementary performance gains by allowing these tokens to learn richer representations, enabling their queries to more accurately predict token importance. The selective activation mechanism the LoRA modules ensures that the outputs of normal input tokens are unchanged, preserving the origi- nal model behavior. Since the original model weights remain unaltered, LOOKAHEADK V modules can be selectively enabled disabled depending on the particular requirements of given applica- tion, thereby broadening the method's applicability.
Lookahead LoRA.
Combining the modules together, LOOKAHEADKV computes the queries and keys of the complete sequence as follows:
x P
0 P
X
0 P
QLKV
W
+
AW
KLKV = P
W +
AW
(3)
where P E Ti xd denotes the hidden states of the lookahead embeddings, and AW g. AW, are the lookahead LoRA modules for query and key projections. Similar to prior meth- ods (Li et al.) 2024] Cai et al., 2024, Zhang et al. 2023). we use the attention matrix ALKV = Soltmax( Quxy Kisy Va
), estimate the importance score 3j Plockabaa Ei Enat ALKV i.j and
retain Top-K K KV pairs with the highest importance scores.
3.2 LOOKAHEADKY TRAINING
We train LOOKAHEADKY modules to compress the attention pattern of the true future response, using the gelee--gnerated responses target. Given data pair (), one iteration of LOOKA- HEADKV training consists of the following steps:
1. GT Forward Pass. For each layer I 1,..., and head h 1...., H, the ground-truth importance scores s between the input prompt X and model-generated response Y are computed.
I.b
2. Lookahead Forward Pass. For each layer / and head h, we obtain the importance score estimates SLVV between the input prompt X and the lookahead tokens P.
I.A.
3. Loss Computation. We first normalize all score vectors such that they sum to 1, and compute the average KL divergence loss between the GT and LOOKAHEADKV importance scores across all heads and layers:
H DEL
LLKV L. ΣΣ
1
Il sL
where is the noromaalleeed importance scores such that å s/ /|s|||. The loss is back- propagated to update the both the lookahead embeddings and LoRA modules, while all other L.LM layers remain frozen, as shown in Figure The pseudo-code for LOOKA- HEADKV training and eviction is given in Algorithm and Algorithm
Training Objective. Inspired from works on distilling attention scores (Wang et al. 2020 Izacard & Grave 2021), ) we minimize the KL. divergence between these normalized attention scores. As our attentions scores are normalized, this KL divergence is equivalent to the popular ListNet (Cao et al., 2007) ranking loss, with of ListNet as identity instead of exp.
Lookahead LoRA Overhead. In principle, one can apply lookahead LoRA to any subset of the linear layers to tradeoff accuracy and latency. However, even when lookahead LoRA applied to every linear layer, there is minor increase (<1.3%) in latency compared to not using lookahead LoRA at all (see Table for ablation results), while significantly boosting performance compared to not using LoRA. Consequently, train LOOKAHEADKV with LoRA modules applied to all linear layers.
To avoid materializing the full attention score matrix, FlashAttention DDao al. 2022) in the forward pass, coupled with eager attention for importance score computation and loss backprop- agation, detailed in Section
5

---

Published as a conference paper at ICLR 2026
1amaa2-Sttstuc
Lama33 instrdd
Gwen3-40
Owan3-Dn
MeggBaenc Scom
1
8 Cade Judget
Cathi Bagge
Gode tudget
Cocna Bugge.
30
40
Oe
Ca
SnapkV
PyramidKy
Streaming
SpecKV
FullKV
Figure 4: Top row: Average LongBench results across multiple budgets and models. Bottom row: Average RULER results across varying context lengths with a fixed budget 128. Across all tested models, budgets and context lengths, LOOKAHEADKY consistently demonstrates superior perfor- mance.
EXPERIMENTS
4.1 TRAINING
Dataset. To encourage the model to learn from diverse attention patterns, curate training samples of varying lengths and sources, comprising both instruction- honloiionn datasets as well as pretraining texts. We collect 50K samples from the long sft subset of the ChatQA2 (Xu et al. 2025) dataset, 20K samples from the Tulu (Lambert et al. 2025) enttuctiionfollowiiie dataset, 7K samples from the Stack (Kocetkov al., 2023), and 9K few-shot completion data samples that we create based on the training splits of the MetaMath, ARC, and HellaSwag datasets, originally curated in Pal et al. (024). For instruction-followingg data, remove the last assistant response and the target model to obtain the (X, Y) pairs of input prompt and model response. For pretraining documents, we first truncate the text at random positions obtain X. and use the target model to complete the sequence to obtain Y. We limit the maximum input sequence length to 16K, and generate all training responses using greedy decoding and max generation length of 512.
Training Details. We apply LOOKAHEADKV on two widely used open-source architectures, LLaMA (Dubey let al.) 2024) and Qwen (Yang et al. 2025). covering three model sizes each: LLaMA3.2-IB, LLaMA3.2-3B. 1.aM33.1-BB, Qwen3-1.7B, Qwen3-4B, and Qwen3-8B.
Table Additional trainable pa- rameters introduced by LOOKA- HEADKV.
Trainable Params
Model
For all models, we set the lookahead size Mookkahead 32, and apply LoRA to all projection and feed-forward modules (W g. Wr. W.. W.. W up. W dowm: and W gatte) with rank 8 and scaling factor = 32. This configuration intro- duces less than 0.5% additional trainable parameters across all models, as summarized in Table [ Full hyperparameter
Params
% of Model
1.L.MA32-1BB
5.4M
0.44
LLaMA332-3B
11.9M
0.37
LLaMA3.1-8B
20.6M
0.26
Qwen3-1.7B
8.5M
0.49
Qwen3-4B
16.2M
0.40
settings are provided in Table
Qwen3-8B
21.5M
0.26
4.2 EVALUATION SETUP
We evaluate our method on comprehensive suite of benchmarks: LongBench (Bai et al. 2024), RULER (Hsich et al. 2024), LongProc (Ye et al. 2025). and MT-Bench (Zheng et al. 2023). Long- Bench is multi-task benchmark that assesses the long-context understanding capability across di- verse tasks, such as question answering, summarization, and code completion. We report results on the 16 English tasks, and the average score the main metric. RULER is another multi-task synthetic benchmark, primarily comprising 13 Needle-in-a-Hayee Haystack-stylle subtasks. Each sample can be constructed varying sequence lengths, allowing systematic evaluation of scaling behav-

---

Published as a conference paper at ICLR 2026
Table 2: MT-Bench evaluation results. Bold and underlined values indicate best and second best
scores, respectively.
Model
Budget
PyramidKV
SnapKV
Streamingl..MM SpeckV
LAQ
LOOKAHEADKY
FullKV score: 3.72
LLaMA3.2-1B
4.64
4.70
4.54
N/A
5.03
128
5.10
5.39
4.94
N/A
5.45
5.60
256
5,49
5.67
5.37
N/A
5.64
5.62
FullKY score: 7.35
LLaMA322-3B
6.30
6.28
5.96
6.52
6.48
6.87
128
6.93
7.03
6.42
7.02
6.93
7.26
256
7.19
7,50
7.20
7.28
7.43
7.30
FuliKV score: 7.77
1JMMA3.-88
64
6.85
6.80
6.17
6.77
7.1
7.26
128
7.39
7.50
6.84
7-34
7.54
7.63
256
7.76
7.72
7.41
7.84
7.72
7.92
FullKV score: 7.19
Qwen3-1.7B
64
6.38
5.81
5.95
5.83
N/A
N/A
6.19
6.70
256
6.90
6.94
6.91
N/A
7.02
7.20
FulKKV score: 8.02
Qwen3-4B
128
64
6.85
7.55
7.71
6.60
7.24
6.24
7.78
7.05
7.70
7.06
8.12
7.69
256
7.90
8.20
7.87
8.11
8.12
8.06
FaNKV 8.48
Qwen3-8B
64
7.33
7.26
6.81
7.69
7.58
8,04
128
7.85
7.94
7.64
7.97
8.24
8.41
256
8.34
8.45
8.56
ior. Similar to LongBench, we use average score as the main metric, and report the results 4K, 8K, 16K and 32K context lengths. We further evaluate the model's long-form output generation capability on the HTML to TSV task from LongProc, which involves converting structured infor- mation from long HTML documents into TSV format. Finally, MT-bench provides comprehensive multi-turn question set, spanning various domains such as writing, coding, and math.
Baselines. We evaluate method against popular KV cache eviction methods: 1) SnapKV (Li et al., 2024), 2) PyramidKV (Cai et al.] (0024), and 3) StreamingI.LM (Xiao et al.) 2024). We also compare our approach stronger, more recent baselines that require costly approximate future response generation, such as 4) Lookahead Q- Q-Cache (LAQ) (Wang et al.) (025). and for 8B-scale models, 5) SpecKV (Galim al., 2026). all experiments, Llama3.2 2- 1B-Itruutttt and Qwen3- 1.7B are used as draft models for Llama3.1 1------Instuu and Qwen3-8 -B,, respectively. We follow the standard eviction configuration settings for all baseline methods, which we detail in Section F
4.3 PERFORMANCE RESULTS
Bench Evaluation. Figure shows the average LongBench scores of LOOKAHEADKY and baselines, across cache budget settings ranging from 64 to 2048. Our method consistently demon- strates superior performance across all models and all budgets tested, demonstrating the effective- ness and robustness of our approach. Overall, results show that expensive draft-based methods (c.g.. LAQ and SpecKV) V) outperform simple baselines, corroborating that employing approximate future response for importance estimation effective. Nevertheless, our method significantly outperforms draft-based approaches, especially at lower budget settings, highlighting that learning to estimate future importance crucial for performance preservation. Due to space limitations, we report results of IB-sccle models in Section F
RULER Evaluation. We report the RULER evaluation results of methods with fixed budget of 128 in Figure (IB-cclle results provided in Section E). LOOKAHEADKY consistently outper- forms other baseline approaches here as well, maintaining strong performance across all evaluated context lengths. Despite being trained effectively generalizes to a longer context length of 32K. We conduct additional experiments on the impact of training context length in Section 5.4
maximum sequence length of 16K, LOOKAHEADKY
Long-form Output Evaluation. To further validate LOOKAHEADKV's ability to generate long- form outputs, evaluate our method on the HTML to TSV task from LongProc. We assess

---

Published as a conference paper at ICLR 2026
Table 3: Theoretical and empirical cost analysis of LLaMA3.1-8B at C 128.
Theoretical Cost
Empirical Cost
Context Length
Compute (TFLOP))
Memory Traffic
TIFT (ms)
TTFT Overhead (ms)
TTFT
TTFT
Method
(GB)
(ms)
Overhead (ms)
Forward Pass Only
1.36
N/A
N/A
LOOKAHEADKV
137
1.03
302
8K
SnapKy
136
0.01
SpecKV
159
79.53
LAQ
137
445
492
234.59
800
Forward Pass Only
928
1754
N/A
1760
N/A
LOOKAHEADKY
929
1755
1.74
1798
32K
SnapKV
928
1754
0.01
1838
SpecKV
1115
106
2156
402.80
2263
503
LAQ
930
1993
239.26
2314
554
LOOKAHEADKY and baseline methods under two input-output settings: 12K-0.5K and 23K tokens, both at fixed cache budget ratio of 30%.
Figure presents the results on the HTML TSV using I.LMMA331-88h Across both sequence-length configurations, LOOKA- HEADKV consistently outperforms prior ap- proaches. We hypothesize that LOOKAHEADKY learning to predict the attention pattern of the en- tire future response, is particularly superior in long-form generation tasks compared to draft- based methods that rely only on partial future sponse as the observation window.
S
Dutput Langth
0.5k
42-
F1-Scome
30
16-
Multi-turn Evaluation. To test method under multi-turn conversation setting, we eval- uate LOOKAHEADK and baselines MT- Bench (Zheng et al., 2023). The generated re- sponses are evaluated using Qwen3-235B-AD2 as the LLM judge. The results in Table Sopleeindicate
0.rk
LAO
Spectt
Seaptt/
Figure 5: HTML-tO-TSV evaluation results us- ing L.LMMA31-88B.
that LOOKAHFADKY V either on par or superior across all models and budgets tested. LOOKA- HEADKV is particularly robust in lower budget settings (e.g., C = [64, 128). where it consistently outperforms other methods.
5 ANALYSIS
EFFICIENCY COMPARISON
Efficiency is assessed by measuring the Time-To-First-D (TTFT) for LLaMA-3.1-8FB across multiple context lengths. While we utilize official codebases for most baselines. LAQ was re- implemented due to the absence of an official release. Because the latency of method can vary significantly depending on the implementation, we conduct rigorous analysis and derive the theo- retical latency for each method, based on the analytical model proposed in Davies et al. (2055. We discuss further details Section B
Table 3 presents the results of the TTFT analysis for SK and 32K context lengths (see Table 15 for 4K and 16K results). Overall, we observe that based methods incur significant overhead, either due to increased computation (SpecKV) memory traffic (LAQ). On the contrary, LOOKAHEADK requires marginal additional cost across all tested context lengths, reducing eviction overhead by 14.5x compared LAQ at 32K sequence length.
8

---

Published as a conference paper at ICLR 2026
Table Average LongBench performance at different temperature settings on LLaMA3.1-8B, with C 128. LOOKAHEADKY outperforms baselines across all tested temperature settings.
Method
FullKV
SnapKV
SpecKV
LAQ
LOOKAHEADKV
Greedy
49.88
43.50
45.45
46.61
47.72
T 0.2
49.58 (-0.60%)
43.29 (-0.48%)
44.99 (-1.01%)
46.73 (+0.26%)
47.75 (+0.06%)
T 0.8
47.82 (-4.13%)
41.39 (-4.85%)
43.43 (-4.44%)
45.27 (-2.87%)
45.81 (-4.00%)
Table 5: 2D ablation across lookahead sizes and trainable modules, on LLaMA3.2-IB, Average LongBench scores with cache budget of 64 and TTFT overhead are reported.
Plockshoal
Phockssoal
16
Plockaical 32
Plocksioal 64
Risiided 128
Module
score
overhcad(
score
overhead(
score
overhead(
score
(verlacdd(
overhcad(
score overhead(%)
emb-only
25.5
25.7
26.4
4.2
7.3
26.2
10.7
ov
26.5
3.7
26.4
4.1
26.9
4.0
26.9
4.4
26.7
7.7
27.1
10.7
al1
26.6
4.2
27.0
4.2
27.0
4.7
27.1
5.0
27.1
8.5
27.0
11.0
5.2
EFFECT OF STOCHASTIC DECODING
We analyze the effect of stochastic generation on LOOKAHEADKY' performance by evaluating our method using two temperature settings: [0.2, 0.8). Results in Table 4 show that LOOKAHEADYY maintains superior performance over all other baselines across all temperature settings. Further, we observe that performance degradation at high temperature setting (3-4% at T 0.8) is consistent across all methods. including FullK V, indicating that stochasticity in inference affects all approaches similarly. We further discuss the interplay between stochastic decoding for training data generation and LOOKAHEADKY performance in Section
5.3 ABLATION ON TRAINABLE MODULES
We study the impact of lookahead size Poookahead and LoRA placement through 2D ablation across six lookahead sizes (4, 8, 16, 32, 64, 128) and three infiiuuaation:: emb-only (No LoRA applied), QV (LoRA applied to Q and V), and all 1ll (LoRA applied to all linear layers). The results indicate that both larger lookahead windows and broader LoRA coverage generally improve average LongBench performance. However, performance gains saturate at Tloookhhaad size beyond this point yields diminishing returns while incurring a noticeable increase in inference overhead. On the other hand, applying lookahead LoRA to all layers results in relatively minor rise in TTFT while significantly improving the performance across all lookahead sizes. Based on this analysis, we set Plookahend 32 and apply LoRA to all linear modules our main experiments.
32: increasing the lookahead
5.4 ROBUSTNESS TO TRAINING CONTEXT LENGTII
Transformer- based language models trained with fixed context lengths often struggle to generalize be- yond their training window. Similarly, one may raise concern about the context length generalization of our method. To examine this effect, we apply LOOKA- HEADKV training to LLaMA-3B with limited train- ing context lengths of 2K, 4K, and 8K, and evaluate on RULER (Figure (). We observe that while longer training context length yields better performance as expected, training on shorter contexts still remains ef- fective with relatively minor degradation in perfor- mance, demonstrating that our method generalizes bustly to unseen sequence lengths.
RUILER Score
(x220)
2K
4K
LAQ
Figure 6: RULER evaluation LOOKA-
HEADKV trained with shorter contexts.
6 RELATED WORK
KV Cache Eviction. Early analyses revealed that attention scores tend to be sparse (Zhang et al. 2023 implying that only small subset of KV entries substantially contributes the attention out-

---

Published as a conference paper at ICLR 2026
put. Further, subsequent work showed that the importance of these tokens remains stable throughout generation, i.c., tokens deemed important early on tend to stay important (Liu et al. (023). These
observations motivated range of eviction methods aimed at discarding unimportant KV entries
while preserving model performance. Representative methods such as H2O (Zhang et al. 2023),
NACL (Chen et al., 2024) and TOVA (Oren al., 2024) rely attention scores estimate token importance and evict low-importance KV pairs. In contrast, other works propose to use alternative importance metrics for eviction (Park al., 2025 Guo al. 2024; Geng et al. 2025) improve importance estimation or address the challenge of materializing the full attention matrix.
Prefill KV Cache Eviction. A specific line of work, which we discuss extensively in our paper, focuses on eviction of prefill KV cache. SnapKV (Li et al.] (2024) introduced the notion of an "ob- servation window" consisting of the suffix of the input prompt, which used to predict important tokens to keep for subsequent response generation. SpecKV V (Galim et al.) 2026) proposed to gen- erate an approximate response with a smaller model and use the resulting tokens as a more reliable observation window for future importance prediction. Lookahead Q-Cache (Wang et al. applies a simple eviction method, such as SnapKV, to obtain a partial low-cost draft response, then re-evicts KV entries based on the importance scores derived from the draft. KVzip (Kim al. 2025) adopts a query-agnosee strategy by inserting a repeated prompt and measuring which KV entries essential for accurately reconstructing the input. Orthogonal to these approaches, several works proposed to allocate non-uniform budgets for each layer (Cai et al. 2024) and head (Feng et al., 2024 to further improve performance.
2025 first
Prompt Tuning for Task Adaptation. Another line of work closely related to ours is parameter- efficient finetuning through learned prompts. Prompt Tuning (Lester al. 2021) inserts a sequence of continuous, learnable embeddings into the frozen LLM for downstream task adaptation, while Prefix-Tuning (Li & Liang/ 202T) extends this idea by pre-pending learned vectors across multiple layers. Further, P- P-Tuning v2 (Liu et al.] 2022) demonstrated that prompt-based adaptation scales well across a wide range of model sizes. Unlike conventional prompt-tuning methods that aim to improve task performance, our work leverages learned prompts to predict internal model statistics, thereby enhancing computational efficiency rather than accuracy.
Training objectives similar to ours have been used in distillation (Wang al. 2020), in rank- ing/retrieval (Cao et al. 2007 Izacard & Grave 2021). Some contemporaneous works (Greenewald
et al. 2025 Peng et al. 2025 Sammagh et al. 2025) also propose LoRA modules that selectively
activate only for some tokens.
7 CONCLUSION AND LIMITATION
We introduce LOOKAHEADKV, a trainable KV cache eviction framework that accurately predicts token importance without relying explicit draft generation. The method augments a frozen LLM with a small set of learnable lookahead tokens and lookahead LoRA modules that activate only on these tokens. Trained match ground-truti importance distributions across layers and heads, LOOKAHEADKV achieves performance superior to costly draft-based approaches. Across a wide range of model families and long-context benchmarks, our approach consistently outperforms prior methods, especially in low-budget regimes. It introduces less than 0.5% additional parameters and incurs only marginal increase prefill latency.
Due to limited compute resources, we were unable to conduct experiments on larger-sized models. However, experimental results indicate that AH improves both performance and la- tency of KV cache eviction across variety of model sizes. LOOKAHEADK currently focuses on the prefill KV cache eviction; extending LOOKAHEADKY to also perform decoding-stage eviction remains an interesting future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
We would like to thank Daehyun Kim, Ph.D., Hyeonmok Ko, Ph.D., Ho-young Kim, Anshumann, Mohd Abbas Zaidi, and Harshith Goka for their helpful discussions and generous support in this work.
REFERENCES
Rishabh Agarwal, Avi Singh. Lei Zhang. Bemd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning- In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang Advances in Neural Information Processing Systems, volume 37, pp. 76930-76966. Curran Associates, Inc., 2024.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pp. 3119-3137, Bangkok, Thailand, 2024. Association for Computational Linguistics.
Zefan Cai, Yichi Zhang. Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyra- midal information funneling. ArXiv preprint, (bs22406.02069, 2024.
Zhe Cao, Tao Qin, Tie Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th International Conference on Machine Learning. ICML '07, pp. 129-136, New York, NY, USA, 2007, Association for Computing Ma- chinery.
Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. NACL: A general and effective KV cache eviction framework for LLM at inference time. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7913-79222 1726, Bangkok, Thailand, 2024. Association for Computa- tional Linguistics.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neurul Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Pradeep Dasigi. Kyle Lo. Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seekingg questions and answers anchored in research papers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cot- terell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4599-4610 Online, June 2021. Association for Computational Linguistics.
Michael Davies, Neal Crago, Karthikeyan Sankaralingam, and Christos Kozyrakis. Efficient 11m inference: Bandwidth, compute, synchronization, and capacity are all you need, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Yuan Feng, Junlin Lv, Yukum Cao, Xike Xie, and S Kevin Zhou. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient 1lm inference. ArXiv preprint, abs/2407.115500 2024.
11

---

Published as a conference paper at ICLR 2026
Kevin Galim, Ethan Ewer, Wonjun Kang, Minjae Lee, Hyung Il Koo, and Kangwook Lee. Draft- based approximate inference for 1lms. In International Conference on Learning Representations (ICLR), 2026.
Zijie Geng, Jie Wang, Ziqi Liu, Feng Ju, Yiming Li, Xing Li, Mingxuan Yuan, Jianye HAO, Defu Lian, Enhong Chen, and Feng Wu. Accurate KV cache eviction via anchor direction projec- tion for efficient LLM inference. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Kristjan Greenewald. Luis Lastras, Thomas Parnell, Vraj Shah, Lucian Popa, Giulio Zizzo, Chulaka Gunasekara, Ambrish Rawat, and David Cox. Activated lora: Fine-tuned Ilms for intrinsics, 2025.
Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in KV cache reduction: Value also matters. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 21158-1116, Miami, Florida, USA, November 2024. As- sociation for Computational Linguistics.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's ss the real context size of your long-context language models? In First Conference on Language Modeling, 2024.
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimban. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Jang-Hye Kim, Jinuk Kim, Sangwoo Kwon, Jae W Lee, Sangdoo Yun, and Hyun Oh Song. Kvzip: Query-agnostic kv cache compression with context reconstruction. In Advances in Neural Infor- mation Processing Systems, 2025.
Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harn de Vries. The stack: 3 TB of permissively licensed source code. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang. Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Christo- pher Wilhelm, Luca Soldaini. Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training. In Second Confer- ence on Language Modeling, 2025.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.
Cheng Li. Llm-analysis: Latency and memory analysis of transformer models for training and inference. https: /tttps://githbb com/ wwwii//9///// 111i19/mmm-aaalysisg 2023.
Tianle Li, Ge Zhang, Quy Due Do, Xiang Yue, and Wenhu Chen. Long-context LLMS struggle with long in-context learning. Transactions Machine Learning Research, 2025. ISSN 2835-8856.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li. and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 0897, Online, 2021. Association for Computational Linguistics.
12

---

Published as a conference paper at ICLR 2026
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: LLM knows what you are looking for before generation. In Amir Globersons. Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Sys- tems 38: Annual Conference Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Tianyang Liu, Canwen Xu, and Julian J. McAuley. Repobench: Benchmarking repository- code auto-completion systems. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7- 2024. OpenReview.net, 2024.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning be comparable to fine-ture fine-tuning across scales and tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anmual Meeting of the Association for Compulational Linguistics (V (Volume 2: Short Papers), pp. 61-68, Dublin, Ireland, 2022. Association for Computational Linguistics.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, Xiaoyin Che, Zhiyuan Liu, and Maosong Sun. RepoAgent: An L.M-ppoeree open-source framework for repository-level code documentation generation. In Delia Irazu Hernandez Farias, Tom Hope, and Manling Li (eds.), Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 436 464, Miami, Florida, USA, 2024. Association for Computational Linguistics.
Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi- state RNNs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18724-18741, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. ArXiv preprint, abs/2402.11228,, 2024.
Junyoung Park, Dalton Jones, Matthew J Morse, Raghavv Goel, Mingu Lee, and Christopher Lott. Keydiff: Key similarity- based KV cache eviction for long-context LLM inference in resource- constrained environments. In The Thitty-ninth Annual Conference on Neural Information Pro- cessing Systems, 2025.
Yuqi Peng, Lingtao Zheng, Yufeng Yang, Yi Huang, Mingfu Yan, Jianzhuang Liu, and Shifeng Chen. Tara: Token-aware lora for composable personalization in diffusion models, 2025.
Mohammad Samragh, Arnav Kundu, David Harrison, Kumari Nishu, Devang Naik, Minsik Cho, and Mehrdad Farajtabar. Your Ilm knows the future: Uncovering its multi-token prediction potential, 2025.
Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. Leave no document behind: Benchmarking long-context LLMs with extended multi-doc QA. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5627-5646, Miami, Florida, USA, 2024. Association for Computational Linguistics.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In H. Larochelle, M. Ranzato. R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. 5776-5788. Curran Associates, Inc., 2020.
13

---

Published as a conference paper at ICLR 2026
Yixuan Wang, Shiyu Ji, Yijun Liu, Yuzhuang Xu, Yang Xu, Qingfu Zhu, and Wanxiang Che. Looka- head Q-cache: Achieving more consistent KV cache eviction via pseudo query. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 34158-34174, Suzhou, China, November 2025. Association for Computational Linguistics.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Peng Xu, Wei Ping, Xianchao Wu, Chejian Xu, Zihan Liu, Mohammad Shoeybi, and Bryan Catan- zaro. Chatqa 2: Bridging the gap to proprietary Ilms in long context and RAG capabilities. The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025.
Xi Ye, Fangeong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao, Greg Durrett, and Danqi Chen. Longproc: Benchmarking long-context language models on long procedural generation. In Second Conference on Language Modeling, 2025.
Zhenyu Zhang. Ying Sheng. Tianyi Zhou, Tianlong Chen, Lianmin Zheng. Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2O: heavy-hitter oracle for efficient generative inference large language models. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 46595 46623.. Curran Associates, Inc., 2023.
14

---

Published as a conference paper at ICLR 2026
A
PSEUDO-CODE
The pseudocode for LOOKΑΠЕADRY training and eviction described in Algorithm and Algo- rithm 2
respectively.
Algorithm LOOKAHEADKY Training
Require: dataset D of input-response pairs
scores <
GT importance scores score estimates using LOOKAHEADKY
2: estimates <
3: for each training sample (X, () in dataset D
4: for each layer / do
D GT pass
5: for each head h in layer do
6:
$ - GT importance score for head h1
7:
scores.append(
8:
end for end for
9:
for each layer / do
> lookahead pass
11:
for each head & in layer do - importance scores using lookahead embeddings for head (1, h)
12:
13:
estimates.apppen(55
14:
end for end for
15:
16:
L+0
D compute loss
17:
for all (5,5) in scores, est imates do
18:
L L+
DKL.
8
$
sh
|
19:
end for
20:
L
scoress
21:
L.backward
end for
Algorithm 2 LOOKAHEADKV Eviction
Require: Input prompt X {a1, (..,...., F
Require: cache budget &:
1: Append learned lookahead tokens to input and compute the sequence embeddings
X P|T
shape: (nmm + 7ljokkhhadd x d
2: Perform a prefill forward pass with X:
3: for each layer / do
4: for each head h do
QQKT Vi
5:
A +- Softmax
shape: (nin + Mlookhhead)
X (Ni + Tiookahend
6:
Å A fm 77mll
attention between lookahead tokens and input prompt
7:
MeanReduce(A)
8:
4 (ooling(s)
>> score vector, shape: 1 x Rin
9:
<- TopK k)
10:
K"kept <- Z
kkept +- V
12:
Cache keel /deett)
evict unimportant KV pairs
13:
Compute attention output for MLP layer
14:
end for
15:
Compute MLP output for next layer
16:
end for
17:
return
15

---

Published as a conference paper at ICLR 2026
B THEORETICAL ESTIMATION DETAILS
This section details our methodology for the theoretical estimation of the Time-to-Fing Faa---iisttokee (TTFT) latency for various KV cache eviction algorithms. Our analysis is based on the analytical model for FLOPs and memory traffic proposed by Davies et al. (2025). To align configurations of theoretical estimates with those of actual measurements, we simulate the execution 8B on single NVIDIA H100 80GB GPU with a batch size of 1, assuming all weights and activations are in half- calf-precision. We set KV cache budget size of 128, lookahead size as 32, and window size as 32. We only consider tensor operations which are dominant parts of the computations. To provide estimates that closely reflect real-world performance, our calculations incorporate practical hard- ware utilization by assuming flops efficiency of 0.7 and memory efficiency of 0.9, as described in Li 2023).
LLaMA3.1-
To isolate the specific overhead introduced by each eviction algorithm, we first establish baseline by calculating the theoretical latency single forward pass. The TTFT overhead for each eviction method is then determined by subtracting this baseline forward pass latency from the method's total estimated TTFT. For LAQ, the total latency calculated by summing the costs its three con- secutive steps-the the first eviction, low-cost generation of pseudo response, and the second eviction. Similarly, the total latency of SpecKV is estimated by aggregating the latencies of its draft prefill, draft decode, and target model eviction phases. A comprehensive implementation of the code to derive theoretical estimates of all baselines is available in the Supplementary Materials.
C IMPLEMENTATION OPTIMIZATION
Efficient attention implementations such as FlashAttention (Dao et al. 2022) do not materialize the full attention score matrix, but is required in our setting compute importance scores and enable gradient backpropagation. A possible solution is to compute the complete attention matrix using native PyTorr (i.e., eager attention), but this quickly leads to an out-of-memory error as the matrix size grows quadratically with the sequence length, which is incompatible with our training setting (up to 16K sequence length). Fortunately, for our objective, we only require the cross-attention scores between the generated response and the entire input sequence, and the response length is typically much shorter than the input prompt.
Leveraging this observation, adopt the following approach: for the attention layer's 's forward pass, we use flash attention, while for importance score computation and backpropagation, we employ eager attention. This reduces the memory requirement of eager attention from [([[)] [Y()²) to O[ [Y] + [Y where X and Y denote the lengths of the input prompt and model response, respectively, with X >> |Y|
16

---

Published as a conference paper at ICLR 2026
D NEED FOR DATA GENERATION
One of the requirements of LOOKAHEADKY training is that the target model's generated responses must be available as training data. However, generating these responses from the model can some- times be costly, c.g., when applying LOOKAΙEADKY across multiple models. Hence, to assess whether this requirement of can be relaxed, we evaluate an alternative setting where training uses the responses from the source datasets instead of goneel-gnerated outputs.
We observe in Figure that this substitution leads to relatively minor drop average LongBench performance in lower- lower-budget regimes. We hypothesize that if the attention distribution of the model- generated responses and that of the source dataset responses are moderately similar, our method can still successfully learn to accurately predict the importance scores. Overall, these results suggest that, scenarios where training data generation impractical, using source responses provides viable and effective alternative.
1.005
1.000
0.995
0.900
0.980
64
128
256
512
1024
2048
Cache Budget
Llama3. 1-88-Intuct
L'ma3.2-38-instee
Figure 7: Performance ratio of training using -ener-generated data vs. source data.
17

---

Published as a conference paper at ICLR 2026
E ADDITIONAL RESULTS
In this section, we provide additional experimental results excluded from the main text due to page
limitations.
E.1 RULER EVALUATION ON LONGER CONTEXTS
RULER at 64K and 128K context lengths using longer contexts, evaluate method with a cache budget of 128. We randomly sample 50 examples per task from the RULER benchmark. As shown in T LOOKAHEADKY achieves the best performance at these context lengths as well, showing that the effectiveness of our method scales even longer context lengths.
To explore the capability of LOOKAHEADKY
Table 6: RULER evaluation results on longer context lengths using Llama3. 1-8B-Instruct at C'
128.
Context Length
FullKV
LOOKAHEADKV
SnapKV
SpecKV
LAQ
64K
84.15 73.72
69.45 54.83
39.64 30.56
64.02 52.62
64.10
128K
50.67
E.2 EFFECT OF COMBINING SUFFIX WINDOW
To test the effect of incorporating suffix window, proposed in SnapKV (Li al. 2024). augment LOOKAHEADKY by also including queries of the last 32 prompt tokens for importance score estimation. As shown in Table 7 we observe a slight drop in performance when SnapK importance scores are included. The degraded performance when averaging LOOKAHEADKY im- portance scores with SnapKV scores, compared to using LOOKAHEADKV scores alone, indicates that the importance predicted by our method is superior to SnapK
Table 7: Average LongBench scores using LOOKAHFADKY window only and LOOKAHEADKY SnapK suffix window, evaluated using I.LaMA3.2-1B-IIssmuu with = 64.
FullKY
LOOKAHEADKV
LOOKAHEADKV
Suffix Window
32.01
29.10
28.52 (-199%)
E.3 DISCUSSION OF GENERATION STOCHASTICITY IN LOOKAHEADKV TRAINING
For LOOKAHEADKY training, various stochastic decoding methods may be employed to generate training data. One may hypothesize that the attention matrices induced by responses generated with higher stochasticity may diverge significantly from those from greedy responses, potentially limiting the generalizability of KOOKAHEADKV modules trained exclusively on greedy responses to stochastic inference scenarios. To investigate this, we quantify the similarity between importance score vectors induced by greedy responses and those generated under varying temperature settings.
Table 8 presents presenns recall @ 512 and Kendall rank correlation coefficients comparing importance scores induced by greedy decoding against stochastic decoding at multiple temperatures using L.aMA3.1 1- 8B. The scores are averaged over 30 randomly selected samples from our training data, across all layers and heads. Even at relatively high temperature (T 0.8), we observe strong persistence of attention patterns. Notably, the deviation is smaller than that induced by responses of speculative model (Llama3.2-1 BBB equivalent to the SpecKV setting). This indicates that the ground-truth impor- tance scores derived from stochastically generated responses highly similar to those from greedy responses, which in turn indicates that learning signals for stochastic settings.
training data provides sufficiently robust
18

---

Published as a conference paper at ICLR 2026
Table Importance score similarity with stochastic response using various temperatures vs. greedy response on LLaMA3.1-8B. 1-8B. LLaMA3.2-1B presents the similarity of importance scores using greedy response generated with LLaMA3.2-IB vs. LLaMA3.1-8B.
Generation Method
T 0.2
T 0.4
T 0.6
= 0.8
LLaMA3.2-IB
Recall @ 512 (%)
95.06
93.73
91.40
91.37
88.66
Kendall's ' Tau
91.44
88.63
84.61
84.79
80.05
E.4
RESULTS ON LONGBENCH
U 18-nstruct
Lama33223 Instruct
Lbma3 1-88 Instruct
Lono LepoTrechh COOm
2600
<0.
2003
Owen3-1.77 1.70
Qwen3-40
Qwen3-
Longtarch Soore
Longtere SCCER
Ours
LAQ
SnapKV
PyramidKV
StreamingLLM
SpecKV
FullKV
Figure 8: Full Longbench results across multiple cache budgets. IB-scale results included.
19

---

Published as a conference paper at ICLR 2026
Table 9: LongBench evaluation results for Llama3.22111
Single-Dncument QA
MulisDDeumentt QA
Summerizatio
Fooellai Loening
Syntaatic
Code
Avg
Qusper
MFF-c
HiqpurQA
ZWILMMQA
Munique
CovRepoit
QMSum
MuliiNess
TREC
SAMSum
FCout
KB-P
PunKy
19.24
15.96
42.47
25.53
29.42
19.87
26,34
22.15
25.64
64.00
80.85
38.83
3,50
4.78
38,63
43.25
32.01
Corhrr Sige
Straaning LM
14.42
12.30
14.04
24.05
15.50
20.00
4.00
33.61
$2.38
27
StapKy
14.24
12.18
30.55
25.66
12.44
14.27
19.27
15.00
73.57
29.29
2.00
4.69
58.01
36.34
24.22
Paramily
13.33
11:36
26.05
25.34
24.01
10.57
13.64
19.40
11.96
15.00
69.95
27.75
1.50
4.33
85.25
56.01
22:85
LAQ
17:21
12.76
37.30
30.27
27. 27.36
14.23
16.42
20.00
14.25
39.50
76002
51.19
3.00
4.58
39-15
38.37
26.36
LOOKABEADRY
17.09
13.30
40.80
33.56
29.80
16.95
18.76
20.85
18.97
45.50
80.12
34.77
2.50
5:22
33.09
36.47
27.93
LM
14.84
15.48
23.56
14.71
15.50
34.00
7161
51,82
3.50
3.79
39.03
24.14
Suupyy
15.74
12.59
35.87
29.37
26.43
14.17
16.17
20.35
16.47
36.50
75.04
31.84
3.50
4.57
59.13
40.15
26.32
PyramsKK
14:84
12.10
53.63
17.73
23.05
11:77
15.27
15 79
13 19
15.50
74.17
50.50
1.50
4.71
17.99
24.71
LAO
18.63
13.65
41.78
34.75
29.59
16.57
18,89
20.85
19.19
44.00
79.29
34.89
2.54
4.25
39.43
41.06
28.71
LOOKABEADKY
17.35
35,46
17.47
20.13
20:75
$1.50
36.19
37.68
29.10
Stramming LM
14.74
12.30
25.26
36.71
23.57
5.88
16.99
19.67
13.97
44.50
24.85
35.96
3.50
5.00
40.5
42.
25.74
Stapky
10.59
13:78
38,80
12.54
28.11
16.55
18.35
20:00
19.09
41.50
79.31
33.70
4.00
4.58
39.15
41.26
28.01
PytamoVV
15:11
13.08
37.31
12.03
12.00
16.91
20.
17.78
40.50
15.61
32.34
3.50
4.65
17.13
25.56
LAQ
61
21.37
7962
3006
ABEADKY
18.25
36.52
21.75
20.91
38,05
KV Cache
512
StreamingLLM
14.62
12.55
26.37
27.00
24.19
3.96
19.12
19.03
20.99
02.50
76.92
36.50
2.54
3.64
40.76
42.83
26.90
StapXy
17.55
14.17
2091
34.57
20.19
16.74
20.32
20.78
22.10
$2.50
M0.29
34.65
3,00
4.58
38.20
42.59
29.51
PyramiKV
39.7
19.63
78 18
15.59
18. 50
20.87
50
90
34.1
4.00
3857
LOOKABEADKY
18.32
36.05
30.30
23,05
21.49
80.40
38.73
KV Cache SIze
1624
15.07
13:40
29.51
35.46
25.1
11.51
20.66
19.85
23.62
$7.00
70.36 16366
37.90
4.00
5.64
39.55
44.5
28.11
Stapky
18.12
14.56
40.52
35.01
29.85
15.86
22 38
21.12
24.16
00.50
$1.15
35.51
3,00
4.62
39.55
43.47
30.75
PyramdKV
16.05
35.3
LAO
19:15
34.43
30.52
23.96
21.47
61.50
31.40
LOOKABEADKY
18.51
15.41
41.49
35.41
29.74
19.29
24.93
21
24.58
64.00
81.07
39.96
3.50
4.80
$7.99
43.55
31.53
KV Cacha Stzr
2648
StramingL
17.10
14:71
31.30
11:33
26.0
11.20
20.21
24.89
59.00
7981
38.00
4.04
4.00
36.49
44.55
29.32
StupXy
36.12
49
19.34
34.30
21.75
38.23
LAQ
19.03
15.61
40.9%
14. 10
10.35
18.99
25. 75
21 85
25.40
64.55 50
79.7%
38.66
1.50
3.38
40.35
45.65
51.76
LOOKABEADKY
18.18
16.08
42.13
35.45
30.13
19.89
26.34
21.23
25.03
64.00
$0.90
39.52
3.00
4.70
38,05
44.13
31.84
Table 10: LongBench evaluation results for Qwen3-1.7B
Single Documenz QA
Mult- Decument QA
Summarizatin
Pew-shor Leaming
Synshetic
Code
Avg.
NrQA Qusper MF es
HoqposiA
2WLRMQQA
Musique
Gov GovReport
QMSum
MuhiNNws
TREC
TriviQA S:MSum PCoust
Prc
Loc RB F
24.75
KV/Cach
3150
Surumingl
12.35
16,83
21.38
22.62
24.61
7.26
11:45
18.96
12.32
39.00
66.48
35.
0.00
12.50
42.05
36.00
25.71
StapKY
13.01
17:50
30.47
28.14
25.88
3.53
11.39
19.77
11.03
37.50
78. 13.59
37.62
0.00
54.50
41.42
36,92
28.48
PyramdKV
13:00
16:31
22.19
23.83
6.95
11:33
19:13
10:72
37.00
73.09
0.00
43.00
15:97
26.46
LODKABEADKV
19.12
21.73
44.25
37.38
22.53
19.73
36.50
55.56
41.72
36.90
K7 Cluce
Size
128
Stromingg LM
13.58
16:57
22.67
33.10 16
14.16 16
7.54
15001
19.36
14.80
44.50
70:06
37.72
0.00
13.50
46.07
25.24
StapKY
38.02
15.71
47.00
38.55
ParamidVV
LAQ
18.38
21.08
45.05
20.17
22.70
18.79
62.50
55.21
41.79
LODKABEADKV
19.46
23.27
44.59
37.81
33.82
17.97
25.71
23.12
21,70
65.50
$5.56
42.35
0.12
92.75
45.80
38.61
38.45
KV Caddy
Stoe
256
S.
13.41
17.66
22.58
7.65
16.15
19:34
17:07
47.50
37.64
26.47
Stapky
Psramoky
17.07
19:04
38.19
35.18 18
29 09
13.95
21.26
17.67
56.00
$1.47
39
0.00
91
45.19
96
LAO
18.74
21.71
45.85
37.93
33.55
16.16
22.63
23.05
21.51
70.00
55.21
42.75
0.17
55,00
41.83
41.63
38.1
LOOKABEARKY
19.60
24.30
45.69
38.51
34.02
17.91
25.51
23.11
23.15
70000
55.37
42.16
0.12
91.00
45.29
38.96
39066
Cacke
13:02
18.40
35.10
34.77
40
47.26
57
Stagky
19:32
22.22
41.07
36.25
30.04
22.15
21.73
70.50
$4.81
40.26
38.20
ParamidV
17.95
20.69
41.45
36.23
29.68
14.66
20.39
21.52
20.01
66.00
$4.65
40.15
0.14
93.50
45.61
38.83
36.98
LAQ
16.99
22.67
46.97
38.10 10
35.62
16.38
24.50 58
23.90
22.99
71.50
$4.71
42:25
0.00
94.00
40:85
40.30
38.11
LOOKABEABKY
19.04
35.66
26.46
24.17
75.00
85.37
39.60
SMrominggLLMM
15.32
15.63
26.90
21.38
26.44
3,43
25.56
20.34
23.82
65.30
$4.00
41.69
0.00
18.00
46.9
39,
20.45
VtapkV
18.08
24.05
46 15:25
18.57
30.72
1663
24
21.25
23.82
71.50
$5.26
80.45
0.50
95
46 39
39.00
3887
ParamuKy
18.76
23.44
43.47
36.36
29.08
16.21
23.08
23.21
28
72.50
54.76
40.19
0.50
95.50
4629
39.06
38.57
LAQ
16.76
22:60
45.45
26.88
24.11
73.00
$4.71
43.11
41.02
Carhe
Size
7046
Straming
16.32
22.12
29.96
31.34
28.13
9,37
26.32
20.96
24.58
68.50
$4.59
42.84
0.00
31.00
46.6
39.92
32.66
StagKY
19.44
24.89
45.18
28.97
32.55
17.32
27
22.00
24.51
72.50
55.21
42.37
0.17
95.00
46.50
40.43
39.75
ParamiKy
10.48
31.92
16.31
16
21.68
68
72.00
41.
05.00
10
16.22
22.65
16.70
$4.71
LOOKABEADKV
19.08
25,15
45.04
33.00
29.86
23,00
25.26
73.50
35.21
40.01
20

---

Published as a conference paper at ICLR 2026
Table 11; LongBench evaluation results for Llama3.2-33
Single Document QA
Multi Decussent QA
Summarizatios
Few shot Leaming
Synthetic
Code
Avg
NaQA
Qroper
MF-ci
HatporQA
2WAIMOA
Masique
GorReport
QMSum
MuliNNows
TREC
TaviaQA
SAMSum
PCcus
RE-P
Palky
25.35
40.77
50.31
53.79
40.05
25.62
35.11
24.38
25.95
72.00
$8.59
43.25
4,00
97.50
34.65
56.83
46003
Cache
Strzening IM
19.79
20.64
42.63
34.00
14.78
75.5
$1.19
1.50
43.76
StapKy
21 58
20.56
99.12
48.33
15.39
12.30
16.37
20.30
14.41
39.50
$2.17
35.05
4.50
21.50
47.90
44.30
12.11
PoramidVV
23.06
19.20
36.26
46:34
34.81
20.07
16.17
20.96
14.00
41.00
$1.93
35.0 25.05
4,00
23.00
46.82
43.94
31.67
LAQ
25:35
28.03
48.15
54.08
37.28
23:51
18.88
21:75
16.02
47.50
36.05
37:25
4.50
74.00
47.82
16.56
3K35
LOOKABARRK
25.22
33,43
$3.50
20.30
35.70
41.60
Smomingg
19.22
20.13
25.00
43.92
33.92
19.00
17.23
42.50
79.63
56.86
4.00
10.50
50.15
47.3
30.36
Stapky
20.59
27.04
45.17
51.15
37.77
21.49
19 64
21.00
18.22
49.50
56.00
37. 55
3.50
18.00
51.51
45.61
36.20
ParamoKV
22.19
25.25
45.65
$1.67
37.41
20.53
19.29
21.45
18.37
47.50
$4.95
37.59
5.50
62.00
50.54
19.46
37.43
LAQ
23:75
20.20
39-91
LOOKABEADKV
25:45
36.54
51.58
40.47
23.64
68.50
99.56
53,06
43.90
KV Cacher
256
Strominng L.M
19.22
22.27
28.
43.12
12.29
16
19
19.77
10.15
50.00
$4.00
99.15
4.00
11.50
52.76
40.5
12.06
StapKy
22.43
30.24
47:71
53.97
40.25
25.01
21.75
22.52
21.03
61.50
39.81
38.69
5.50
91 91.50
53.49
53.56
PsramdKV
23.27
21
22.61
20.57
$7.58
39-01
26.46
58.11
LOOKABEARKV
25.58
57.51
50.92
53.30
40.35
25.74
25.02
24.37
23.36
69.00
$9.51
42.54
4.50
94.50
53.02
55.3
44.76
KV Cache Size
5/2
Stroming
19:22
21.08
29.5 29.39
43.87
32.82
16.92
21.95
20.31
22.14
39.50
$5.12
41.04
4.00
12.50
55.06
51.91
33.76
StagKy
22.55
54.15
48.20
52.36
39.98
23.36
22.9%
23.35
68.50
58.41
39.90
86.50
LAQ
34.90
59.15
50.65
53.88
39.85
25.45
21,91
23.88
71.00
58.3
42.57
4.00
96.50
55.46
55.08
45
LOOKABEADKV
24.12
36.61
50.78
$3.84
40.06
25.78
57.81
24.15
24.83
60.00
89. 16
42.01
4.00
97.00
55.03
56.64
48.16
Carte
Sizr
27.49
24.45
Stagky
24.67
51.18
40.13
25.01
25.92
23.78
69.50
39.79
40.79
56.58
45.06
ParamilK
34.01
$7.98
51.43
$3.64
99.49
24.
35.4K
23.00
24. 50
68.5 50
58.
40.76
4.00
58.00
55.14
55.1
46.5
L'QQ
24.55
40.75
49.15
59.65
40.01
26.68
67.9
24.00
25.25
71.50
58.50
43.54
4.00
97.00
54.93
56.88
45.54
LOOKABRADKY
24:28
39.51
51.35
53.64
40.36
25.07
29.61
24.17
25.41
09.50
$3.99
42.90
4,00
98,00
55.59
56.72
45.57
Cache
SIzE
Sumsningl-LM
22.36
32.37
35.12
21.68
66.00
$7.56
41.68
27.50
37.48
Sungy
24.54
40.50
51.36
53.50
40.60
25.24
28.41
33.92
25.62
71.50
58. 56
42.39
4.00
58.00
55,0
56.97
45.67
PoramidV
24.05
39.80
51.31
53.88
40.03
28.55
27
24.20
25.85
70.00
58.5 50
42. 11
4.00
(8.00
86
55
LAQ
24:30
41.04
49.50
5410
39.86
28.13
30.09
24.44
80
71 11.30
58
11
4.00
97.00
LODKAREABKY
25:23
40.67
30.00
40.06
31.30
24.18
71.00
45.77
Table 12: LongBench evaluation results for Qwen3-4B
Documenz Qoeper
Mult Decument 2W3IMQQ Masiqqu
Summarizalion QMSum
Fee shor Leaming
Synihetic
Code
NeQA
HorporQA
GovReport
MuliNows
TriviQAA
SAMSum
Panky
27.45
43,30
$1.45
55.63
43.45
31.61
32.24
24.61
25,00
73.00
59.76
43.65
0.75
96,50
64.29
61.39
47.88
RV. Cacho
Siz0
Strmening...M
19:71
13.04
75.48
15.28
25.03
34.05
18.53
33.69
PsramidkV
15.50
24.84
36.35
40:30
35.07
19.39
13:46
19:85
13.00
41.50
76 69
33.
1.50
73.00
52.96
33.97
LAQ
16.55
30.24
46.21
40.38
18.10
18.35
14.90
20.14
14.48
43.50
71:25
$4.00
1,50
61.25
55.45
68
35.91
LODKABEADRV
20.49
37.99
51.37
54.71
42.30
30.50
22:10
22.05
18.71
54.50
$8.55
39.71
1.00
92.00
55.89
52.53
43.11
Cluche
13.69
23.56
26.02
38.00
32.38
15.20
45.50
78.32
37.32
63.50
51.76
33.68
Snapky
19.56
29.48
43.55
49.31
17.95
15
16:27
16.16
49.50
56. 31
38. 1.10
1.50
95.00
58.75
51.78
48
ParamoKV
15:88
22.33
38.51
40:35
13.50 56
70:45
72
20.85
14.85
45.00
76.
35. 05
1.50
80.50
58.56
40.40
36 19
LAO
21.44
37.82
$3.26
$4.98
42.75
33.08
20.44
23.09
18.86
60L50
$7.55
40.48
2.50
93.00
68.5
53.57
41.55
LOOKABEADKV
25.17
40.13
24,66
21.57
$9.35
41.37
45.74
Straminng LLM
15.66
25.74
23.99
37.34
32:47
19.02
6.65
20.1 13
18,00
49.50
$1.07
35.87
1.00
66.50
59
55.05
35.46
StapkV
26.54
35.80
47.67
54.45
40.78
29.60
19.85
21.75
19.8
60.00
$7.64
29. 19.45
1.00
06.00
62.57
58.34
43.6
PyramodVV
18:31
31.30
44.14
$1.08
36.87
25.14
18.88
21.04
17.78
36.30
$5.55
33.77
1.50
55
59.1
53.92
4100
LAQ
55.76
21.57
57.72
LOOKABEADKV
26.25
43.05
55.05
25.00
46385
KV Cache
Streoming L.
18:02
27.64
30.11
16.03
33.32
20.70
47
20.99
21.96
60.50
$5.45
21.27
0.50
56. 50.90
62.54
57.30
37.42
StapkV
15:27
50.10
51.45
54.33
4221
10
21.51
23.33
70.00
44.24
1.00
96.50
00.45
ParamiiK
40.38
21.77
20-77
59.24
LAQ
26.50
42.56
55.88
55.24
24,46
75000
42.96
LOOKABRARKE
76.50
41.97
$5.10
55.59
45.07
32.00
59.57
25.55 15
74.51
72.00
$8.76
42.85
1.50
05.50
63.83
60.06
47.47
KV Carhe Sier
1034
StreamingLLM
20.48
30.08
32.30
42.20
34.25
20.65
24.81
20.4
24.19
62.50
$7.39
30.95
1,00
47,00
64.74
99.17
38.1
StapkY
24,02
ParamnlKV
25.51
30.40
$1.89
53.36
42.26
70.00
LAQ
27.40
43.93
$4.30
55.05
45.62
31
28.18
25.16
24.64
75.60
$7.57
43.33
1.75
23.50
62.54
62.00
47.42
LOOKABEADKV
27.47
42.45
53.70
55.64
43:85
32.40
30,68
24.95
25.21
73.00
$8.96
42.95
1.00
96.50
66.61
61.89
47.82
KV Cache Size
2048
Siruainng
37.06
21.93
21.64
58.
40.33
Su
26.30
43.04
53.50
55.54
24.36
75,00
PatamalK
25.74
42.42
$3.91
$5.34
43.12
33.06
27.70
24.21
24.74
72.60
59.
41.54
1.25
86.50
63
61.78
47.24
LAQ
27:21
43.53
55.62
$5.67
45.80
31.75
42
24.01
25.00
73.00
$7: 77
45
1.50
01.25
62
81
47.11
27.48
42.85
55.71
55.31
45.82
31.79
24.75
25.33
75.00
58.33 76
1:25
96.90
18
67.23
47.92
21

---

Published as a conference paper at ICLR 2026
Table 13: LongBench evaluation results for Llama3.1-87
NRQA
Single Document QA
Qurper 46.06
MI
HoqporQA
Maia Document QA
2WIKIMQQ
Musque
GovRepor:
Summarizaion
QMisum
MuliNews
TRUC"
TANSSUA SAMSim
Pew shot Leorning
Synthetic
Pre
Code
RF-P
Arg
Palkky
31,62
36.93
54.10
45.50
31.57
34,46
25.25
26.98
72.50
91.65
45.79
6.64
99.5
99.50
05.12
35.75
49.88
Stig
Seruming!
74
21.75
31
42.11
21. 18
17.29
20.95
35,50
82.81
34.50
7,50
99.
01350
48.14
38.34
SaapKY
27.55 53
34.09
41.37
54.37
45.37
27. 53
16.75
21.74
16.32
59.00
86.33
36 16.58
7.50
08. 50
$5.57
48,07
40.47
Paraiidy
24:25
22:87
41.03
53,07
43.55
26, 96
16.46
21.53
15.61
38.50
81.95
15.68
7:50
09. 50
$4.40
47.20
19.40
LAO
27.62
33.71
52:35
19:74
23.19
18.00
46.00
100.00
51.49
43.57
SpeekV
23.50
LOOKABRADKY
30.02
41.46
55.77
48.56
30.30
23.50
24.05
21.25
91.62
42.56
7.50
99.53 19.50
58.34
46.4
KV Clache 5126
128
Scruming!
24.05
21.30
32.56
50.67
42.89
24.31
18,49
21.25
18.19
40.50
85.57
38.28
3.50
99.5 51
50.03
00.72
39.8
SaspKY
29:13
28.06
51.23
23.03
46.00
52.1
LAQ
30
18:31
55.75
52 1.50
13
20.00
24, 24.20
31
60.50
92.09
41.06
7.25
09.
50.54
55.5
46
Speeky
20:23
79:12
54.05
56.54
46 30
20 (0
2%.5%
21 11:25
$2.00
90002
42.14
8.81
09. 50
61:11
61
45
LOGKAHEADKY
31.32
42:85
56.78
$7.04
47.44
30.82
25.18
24.33
23.09
65.50
92.24
42.96
7,50
99.50
51.75
55.29
47.72
Seruning
25.96
24.08
33.75
23.
20.56
21,60
20.64
36,00
87.50
41.09
3.50
99. 00.50
61.39
51.53
4112
SnspKy
27.0
14.49
55.07
57
46.57
20.5 50
12.40 49
23.51
$2.42
$4.00
91.10
40.1
7.53
09.
50
63.48
55 56
4561
ParamilVV
26.00
36.64
55.86
$7.88
46.25
20.5 56
22.23
23.86
33.59
$6.50
01.56
41.35
7.33
09.
50
52.47
$1.02
J5.85
LAQ
31.01
43.07
55.03
57.78
65.00
92.20
42.61
10000
62.70
48.33
Speeky
30.85
91.15
LOOKABRADKY
31.95
43.01
36.30
$7.99
27.26
24.56
24.59
69.00
42.95
100.00
4861
KV Clcche
512
Scruning!
27.20
26.66
34.51
50.04
42.70
25.35
23.53
21.35
23.51
$7.50
87.65
41.87
1.50
97. 21
60.14
53.61
42.54
SmspKY
30.05
29.67
24.58
24.47
64.00
41.38
7.17
99
LAQ
31.64
45:55
55.11
52.73
89.60
30
26. 17
24.
24
71.00
92.35
4500
6.90
100000
58.65
4681
Specky
31.12
43.77
$7.32
$7.51
09.93
$1.05
26. 34
24.61
2450
65.00
02.13
45:92
7.00
100000 55.11
46.78
LOOKAHEADKY
31.39
46.92
$7.56
58.56
47.72
30.82
29.24
24.82
25.53
72.50
91.92
43.39
7,08
100.00
64.57
55.55
49.31
Site 1024
LL
27.23
30,30
36.64
50.59
43.26
23.45
23.73
21.67
25.49
63.50
88.84
42.56
1,50
93.50
55.77
43.35
SnspKy
20.64
44.60
57. 30
$7.60
48.51
51. 18
17.57
24.17
35.84
09.50
92.04
42.78
7.08
09. 50
64.57
58.66
4876
ParamilV
30.70
46.91
56.65
98.13
48.13 17
30.5
26.65
24.53
35 35.88
68.00
01.78
42.5m
6.81
09. 50
54.41
57.7
48.55
LAQ
31.63
45.63
55.02
78.
43.31
10000
50.04
Speeky
31,95
LOOKABEADKY
31.14
46.04
57.77
58.22
30,75
25.51
26,06
72.50
91.92
100.00
RV Cache
Sizw
2048
Seraming!
28.53
37.02
39.90
51.22
45.83
23.60
25.11
21.01
36.50
67.50
90.95
42.55
1.25
90.5 01
64.88
57.52
45.26
SuppY
31.23
46.14
36.94
31.74
30.34
2481
71.50
91.49
43.16
99
55.80
ParmidKy
LAQ
31.30
45.69
58
52.11
31.
10, 96
96-77
73.50
92.35
45.56
6.83
100.00
63-3
Speck
31.83
46.64
$7.30 39
$7.97
80
33. 73
10 10.96
25
36 36.92
71.00
91.48
45.65
5.88
99. 50
79
61
49.81
LOOKAHEADKY
31.01
46.37
57.24
58.15
48.31
31.12
32.56
25.22
27.07
72.50
91.45
43.56
6.38
99.50
64.96
59.13
49166
Table 14: LongBench evaluation results for Qwen3-8B
NiQAA Queer MF-en
Document
HoqpeQAA
2WiksMIJA
Dncement QA
Musqque GodRepprr
Summartzatio QMSuu MuhtiNwws
TRPC
ThivaKIA SAMSum
sbot Leorning
PCount
Syemenc
Pro
Lic
RB-P
Avg.
FalRy
26.04
47.56
53.33
59.23
43.37
36.11
33.56
24,115
34.79
21.30
90121
44.45
2.00
1000000
69.39
65.37
49.46
Cache Stzy -4
Seramingt
16.62
25.37
25.16
40.57
33.95
15.05
13.97
18.95
12.40
29.50
50.65
35.10
1.50
69
19.20
59.25
53.13
31.00
Stopky
ParamidKV
15:45
27.42
36.67
19.00
14,67
18.05
12.50
43.00
2.00
35.26
LAQ
16:22
32.11
45.02
42.35
37.53
21,07
15.71
19.56
13.47
44.00
76.64
35.15
3.50
$3.50)
59.06
$2.04
37.31
SpeoKV
15:30
27.35
46.29
32.60
36.55
12:94
16.55
19.59
14.67
$2.00
58.10
32.51
4.50
74. 50
06:22
54.37
13.1
LOOKAHRADKV
21:11
43113
51.85
59.01
42.50
34.54
22.56
21.61
18.49
64.50
85.75
39.48
1.50
67.75
62.78
56.0
43.8
Suruniool
17.65
26.69
28.40
41.05
20.02
19.15
43,00
82.57
1.50
70.00
62.86
35,82
Saapky
19:14
32.65
45.99
54.81
35.95
26.59
17.56
20,83
16.04
49.50
87.10
35.90
3.50
99.55 30.50
64.62
56.29
42.13
PyramidKv
15:57
30.19
41.84
46.01
35.73
19.57
16.51
19.07
14.36
47.00
83.51
35.56
2.50
92.00
62.14
53.07
38.46
LAO
42.15
59.55
64.50
3,00
100000
66.22
46
23,01
LOOKAHRADKV
26.05
44.30
$3.34
$8.78
42.70
35.80 $5.80
25
55.29
22.95
21.13
66.50
85.95
41.64
3.50
99. 50
65.05
62.55
68
47.46
KV Clucee
256
Seraming
18.15
28.15
28.52
42.81
33.58
21.54
19.53
19.20
17.16
38,00
85.58
40.08
1.00
69:20
55.50
59
Stopky
32,82
41.06
ParasidKV
15:47
34.57
47.44
20.43
20.01
$8.50
3.50
100.00
43.05
LAO
26.00
45.44
$3.34
$7.00
43,53
36.62
24.23
23.56
20.38
70.00
89.05
42.47
3.00
160,00
68.17
64.03
47.95
SpeoKV
22.58
41.09
53.89
59.85
42.42
$1.50
2A.53
25.64
21
68.00
88.15
45.12
3,00
100
58.19
64
47.42
LOOKAHRADKV
25.85
45:40
52.68
58.47
46.05
36, 13
27.51
25.71
23.88
00
89.05
45.32
2.00
100.00
61.83
64.77
48.31
S
15.94
30.36
30.21
43,59
22.51
19.62
21.16
55.50
87.45
2.00
57.00
67.06
38.59
SaupKY
24.62
43.72
$1.96
59.37
42.36
34.04
25.03
22.55
21.66
65.00
89.53
42.06
3,00
100,00
60.37
64.95
47.64
Paraiidy
23.12
40.52
$1.43
$7.57
40,89
32.85
23:85
21.92
19.70
65.50
89.55
40.29
3,00
100000
67.47
61.73
46.44
LAO
46.9%
53.70
37 64
76.0%
25.67
71.40
85.95
3,00
24.23
13.53
LOOKAHRADKV
25.31
46.49
$2.04
59.32
43.09
36.92
79.55 56
23,80
34.01
71.50
90.21
46:20
2.00
100.00
68.58
65.55
48.93
RV Cacle
Size
1034
SMraming
21.24
32:92
31,43
45.94
34.38
23.54
25:73
20.4
23.50
62.00
88.71
41.18
0.50
44.20
58.29
63.65
39.00 19
SuspKY
42.66
36-80
08.00
43.15
66.
ParasidyV
23.77
42.89
53.01
27.52
23,07
22.72
2.00
100
LAO
26.11
47.37
$3.45
$7.01
43.52
37.36
29.50
23.85
23.47
71.50
89.63
44.00
2.00
160,00
67.54
64.83
48.84
SpeoKV
21.05
46.36
56.07
59/04
83.37
31.1 12
29.3 32
24.18
23.08
71.00
90.11
46.55
3.00
100
00.09
05.3
48.8
LOOKAHRADKV
25.35
47:25
52.56
59.10
43.25
36,59
31 15
23
24.51
21.00
90.21
46.69
0.50
100.00
58.53
49.04
21.73
38.54
38.02
36.76
25.53
28,59
21.47
24.11
65.00
43.85
1.00
48
64.63
41.47
Suppy
25.55
47.52
$3.20
58.73
42.70
36,05
30.54
23.78
24.40
71.50
90.21
45.27
1.10
100,00
60.33
65.35
48.6
Paraiuy
25.47
40.69
55:21
59.11
42.90
36.61
29.11
25.61
24.18
71.50
90.05
42500
1.10
100000
69.28
65:21
48.78
LAO
24.04
34.01
71.40
44.54
3.00
100m0
47700
$3.00
31
LOOKAHRADEV
26.76
48.01
$2.92
59.43
43.20
36.21
$0.64
23.91
34.99
71.00
90.21
44.74
1.00
160000
60.33
65.01
49.33
22

---

Published as a conference paper at ICLR 2026
E.5 RESULTS ON RULER
We report the RULER results across all six models tested, with cache budget settings at 64 (Figure
and 128 (Figure TO
Llama3. 2-18-instruct
Llama3.2 3-38-instuuct
Llama3.1 1amma,1-8B-instre Inistuutt
73
BR.TR BULER
Aecrage RULER
50
Contest (x2
Contect 1x2
Conmect (x221)
Qwen3-1.77
Qwen3-4B
Qwen3-88
Accrage RULER Store
60
80
Content (x2"))
(x2t
Conmont 1x21
Ours
LAQ
SnapKV
PyramidKV
SpecKV
FullKV
Figure 9: Full RULER results across context lengths (budget 64)
Llama3.22 2- 118--tttt
Llama3.2 3.3-.3/nsttttrrucc
Llam3311 1-8-a31-88-nnerruct
RUIRR Sente
80
Ansrouee RUEE Scors
50
16
32
Contect (x2
Contect [x2
Cormect (x221)
Qwen3-1.78
Owen3-4B
Qwen3-8B
90
RUEER Goore
Avurage NULER Score
40
Contoxt(22")
Contoxt
Cansox tx2
Ours
LAQ
SnapKV
PyramidKV
SpecKV
FullKV
Figure 10: Full RULER results across context lengths (budget 128)
23

---

Published as a conference paper at ICLR 2026
E.6 ADDITIONAL EFFICIENCY ANALYSIS
We show the full results of the latency analysis that were omitted in the main paper due to space limitations in this section. Note that the empirical TTFT overheads for some methods can be larger than theoretical estimations. These are probably due to a combination of measurement noise and inefficient implementation of these methods KVCache-Factory or their official implementations. Better implementations may reduce these overheads significantly, more in line with the theoretical cost.
Table 15: Theoretical and Practical Analysis across various context lengths and methods.
Theoretical Cost
Empirical Cost
Context Length
Compute (TFLOPs)
Memory Traffic
TTFT (ms)
TTFT Overhead (ms)
TTFT
TTFT Overhead (ms)
Method
(GB)
(ms)
Forward Pass Only
60
13
113
N/A
130
N/A
LOOKAHEADKV
60
13
114
0.92
141
11.38
4K
SnapKV
60
13
113
0.01
143
13.14
SpecKV
70
165
52.10
223
92.42
LAQ
444
233.81
637
506.58
Forward Pass Only
136
13
257
N/A
291
N/A
LOOKAHEADKY
137
13
258
1.03
302
10.88
8K
SnapKV
136
13
257
0.01
311
20.17
SpecKV
159
81
337
79.53
411
120.51
LAQ
137
445
492
234.59
800
509.38
Forward Pass Only
336
635
N/A
658
N/A
LOOKAHEADKV
337
636
1.27
677
18.50
16K
SnapKV
336
13
635
0.01
695
37.12
SpecKV
398
89
792
157.05
866
207.31
LAQ
337
447
871
236.15
1182
523.54
Forward Only
928
1754
1760
LOOKAHEADKV
929
1755
1.74
1798
38.04
32K
SnapKV
928
1754
0.01
1838
77.67
SpecKV
1115
106
2156
402.80
2263
502.87
LAQ
930
451
1993
239.26
2314
553.68
24

---

Published as a conference paper at ICLR 2026
F HYPER-PARAMETERS
Training hyper-parameters. Learning rate was searched for Llama and Qwen model family among 10 1 x 10-4,2x 10 1 10-3), The final hyper-parameters for all experiments are shown in Table 16
Table 16: Training hyperparameters.
Parameters
Values
Optimizer
Adam
81.32
0.9, 0.95
Effective Batch Size
32
Drop-out (p)
0.0
Max Sequence Length
16384 (prompt length) + 512 (response length)
Train Iters
7600
Learning rate
1 10 (for Llama), 2 x 10 (for Qwen)
Schedule
Cosine
Warmup steps
2%
Min LR
0.0
Gradient clipping
1.0
Eviction hyper-parameters. We use the implementations in KVCache-Factory or their official implementations (SpecKV) for all baseline methods, except for LAQ which we re-implement our- selves due to the lack of an official release. Following prior works (Li et al. 2024 Cai et al. 2024 Galim et al., (026). we use standard configuration settings for all baseline methods, inclucing an observation window size of 32, maxpooling kernel size of 7, and mean reduction for GQA compat- ibility (Feng et al. 2024). For LOOKAHEADK window size, our method does not train with the suffix window for prediction. Further, since our lookahead size Moookhhead is 32, we set the maximum generation limit of LAQ and SpecKV to 32 tokens so that the methods can be compared using the same number of draft tokens.
we use the same settings, except we do not use
G DATASETS, BENCHMARKS, AND SOFTWARE
Software Our source code is available in the supplementary, and our implementation is built on KVCache-Factory
Training Dataset Our training dataset mixture consist of random samples from publicly available datasets: 50K long sft subset of SaaarQAA--LogSST------ 20K subset of samples from The Stack and 3K samples from MetaMathF-ewshtt HellaSwaggDDPO_Fewsaa and AAR.DDOO_Fessteee respectively.
Evaluation Benchmarks We used LongBench dataset as fetched and processed by KVCache- fot the official source. For RULER, we used RULER Github) For LongProc, we used LongProc Github
Factory, see
HF HH
H LLM USAGE
LLM assistants were used to refine the wording of selected sentences, while the majority of the text was written by human. All LLM-generated text was carefully inspected to ensure that it contained no harmful or controversial content. Additionally, we used LLMs to help in finding some of the related literature discussed in the paper.
25


Reviews:
[{"review_id": "hbGjTsSQ4h", "reviewer": "Reviewer_TVsC", "summary": "The paper introduces a learned KV cache management framework, LookaheadKV, that selectively evicts tokens during prefill stage by predicting future token importance using a small set of trainable tokens and LoRA adapters that are only active for these special tokens. The approach aims to achieve the accuracy benefits of similar draft-based approaches like LAQ and SpecKV, without their latency penalty from generating a draft response. LookaheadKV is trained with a normalized KL divergence loss and use the predicted importance to select top-$k$ KV cache entries. Experiments across various models show that LookaheadKV outperforms baselines and faster than draft-based approaches.", "questions": "1. Does inclusion of LoRA adapters enable using shorter length lookahead tokens? Could you achieve similar or better performance by simply increasing the number of lookahead tokens (e.g., n=64 or n=128) without LoRA adapters? What is the fundamental trade-off between lookahead token count and LoRA expressiveness in your framework?\n2. How does lookaheadKV's performance vary when using stochastic decoding strategies at inference time?\n3. The RULER evaluation (Figure 4, bottom) only reports results at budget=128, while LongBench shows multiple budgets (64, 256, 1024). How does LookaheadKV perform on RULER at these other budget settings, and are the performance advantages consistent across different budget constraints?\n4. Results show from Table 3 that at n=32, performance mostly saturates. What happens to both performance and latency overhead when number of lookahead tokens exceeds 32 (e.g., 64, 128, 256)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Presentation is overall clear and problem is motivated by the accuracy overhead trade-off from other draft-based methods. The latency problem is directly tackled through a learned approach. LookaheadKV provides an interesting application of parameter-efficient fine-tuning methodology for KV cache purposes.\n2. Experiments performed on various model families and sizes, and also long-context benchmarks. Inclusion of LongProc provides some support that LookaheadKV can preserve the model's reasoning ability beyond recalling isolated facts.\n3. Demonstrates efficiency by showing that method's impact on TTFT is almost negligible and competitive with SnapKV, while being significantly fast than both SpecKV and LAQ.", "weaknesses": "1. All ground-truth attention scores are generated using greedy decoding. As a result, lookaheadKV is effectively trained to predict attention patterns of a greedy future only, which may limit its applicability. In many practical applications, LLM inference is not deterministic and benefits immensely from stochastic sampling strategies such as temperature scaling to produce more diverse outputs. The attention patterns can differ substantially, which could lead to degradation in eviction quality when stochastic decoding is used such as in complex reasoning intensive tasks with longer thoughts.\n2. Training lookaheadKV is expensive, since it requires for each model trained, to first generate a number of decoded tokens to use as ground truth.\n3. The method defines ground-truth importance based on the attention of response tokens generated. However, attention scores do not always correspond to actual causal influence, so LookaheadKV may evict tokens whose attention is low but whose semantic contribution is high.\n4. The need for requiring both lookahead tokens and lookahead lora is not made clear. While the ablation studies show the benefits, some additional justification is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eciVem5KYB", "reviewer": "Reviewer_CrG9", "summary": "LookaheadKV proposes a lightweight, training-based framework for predicting token importance during prefill to enable accurate KV-cache eviction without draft generation or additional inference steps. The key idea is to introduce a small set of learnable “lookahead tokens”, activated only during prefill, whose attention interactions approximate the model’s future decoding behavior. These tokens are enhanced by LoRA-based low-rank adapters that learn to predict future attention distributions via a KL-divergence ranking objective, requiring no model modification or retraining of base weights. LookaheadKV achieves near-oracle token importance estimation, outperforming SnapKV, SpecKV, and StreamingLLM across benchmarks.", "questions": "1. Would LookaheadKV’s importance predictions remain accurate when decoding randomness changes token trajectories?\n2. How to select retained tokens with lookahead tokens. \n3. What is the breakdown scores for LongBench? How Lookahead performs on different tasks. \n4. Do different LLMs need different LoRA?\n5. How does this method differ from works such as “Learning to Compress Prompts with Gist Tokens”, which train models to summarize and compress prompt tokens to reduce KV-cache size, rather than to learn predictive lookahead behavior for future token importance?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Strong empirical results with broad coverage: Outperforms baselines on multiple benchmarks\n2. Comprehensive ablations and analysis: Includes detailed experiments on number of lookahead tokens, LoRA layer coverage, training context length, and budget scaling", "weaknesses": "1. Unclear generalization to stochastic decoding: The method is trained and evaluated primarily under greedy decoding, assuming deterministic next-token prediction. It remains untested under sampling-based decoding\n2. Absence of multi-turn or instruction-following benchmarks: Evaluations mostly use single-turn or synthetic long-context datasets. Multi-turn reasoning or conversational tasks, where future tokens are highly context-dependent, are missing\n3. Lack of Subtask-Level Analysis on LongBench: The paper evaluates on LongBench but reports only averaged or composite scores, omitting per-subtask breakdowns", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gDI5heSc8Q", "reviewer": "Reviewer_nWM2", "summary": "The paper proposes LookaheadKV, a training-time method that learns to predict token importance for KV cache eviction without running a draft generation. During prefilling, the model appends a small number of learnable lookahead tokens and activates a lightweight LoRA only on these tokens. The attention induced by these lookahead tokens is trained to match the attention that would be observed if the future response were known. At inference time, the method uses the learned mechanism to rank prompt tokens and evict low-importance items from the KV cache. Experiments on long-context benchmarks report accuracy that is competitive or better than recent baselines, with low overhead at prefilling and no added cost during decoding.", "questions": "1. How long does training take in time for the main models, and on what hardware budget. Do you believe the small TTFT improvements and decoding-time stability justify this training cost. How do you view this trade off in practical deployments.\n2. Have you tried combining the importance score from LookaheadKV with a SnapKV score. I am thinking of using both the lookahead tokens and the last few prompt tokens as the observation window, so that the score contains both prompt-side and generate-side information. Do you expect this to help, and can you share preliminary results if any.\n3. For extreme extrapolation, for example the 128k context window of Llama 3.2, can you report RULER results at 64k and 128k. Even a reduced subset would be helpful to show the scaling trend.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The central idea is interesting: use a pretrained module with a small LoRA applied on a short learnable window to substitute for running an explicit generate step. This reduces prefilling cost while still capturing signals about future attention. The method is simple to integrate, has low inference overhead, and shows consistent gains in low-budget regimes. The empirical results cover several model families and tasks, and the latency accounting is practical.", "weaknesses": "The presentation has gaps. I can understand the LoRA training and the alignment objective, but I do not understand precisely how the lookahead embeddings are obtained. It is not fully clear whether these lookahead tokens are new learned embeddings, adapted from existing vocabulary embeddings, or derived from another module. The paper would benefit from a precise definition of the parameterization, initialization, and update path of these embeddings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v3tTVJ5v3V", "reviewer": "Reviewer_4uAb", "summary": "This paper proposes to use LoRA and learnable special tokens during the attention step. These components assist the prefill phase in determining which parts of the KV cache should be retained and which can be evicted. The approach ensures accuracy and model performance without significantly increasing decode overhead.", "questions": "1. Could the authors elaborate on the detailed training cost for proposed LKV? Although this process is a one-time effort, it currently appears that training is required for each different model.\n2. The training process utilizes greedy decoding to generate ground-truth. But in practice, the most popular sampling strategy is not always greedy decoding. How does LKV's performance generalize to stochastic decoding strategies such as beam-search or top-p? Is there a risk that the learned modules overfit to the deterministic attention patterns of greedy decoding, leading to a performance drop when the generation path becomes less predictable?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The problem is well defined. The paper addresses a clearly defined and critical problem: how to efficiently predict the attention patterns of future tokens in the input context for KV Cache eviction. \n2. Concise Solution. The proposed solution is concise and seems to work well according to experiments.\n3. Comprehensive Experiments. The paper is supported by a thorough experimental evaluation. The accuracy tests cover multiple models and diverse scenarios, including long-input and long-output tasks. The efficiency analysis provides both theoretical cost models and empirical TTFT measurements.", "weaknesses": "1. Limited Practicality for Long-Form Generation: While LKV outperforms baselines on long-output tasks, its absolute performance still degrades significantly (as shown in Section 4.3, \"Long-Form output Evaluation\"). \n2. Prefill-Only Eviction Limits Applicability: The method's focus exclusively on prefill-stage eviction is a major limitation. Many SOTA reasoning models rely on a large number of decoding steps, during which the KV cache continues to grow. Ignoring decoding phase significantly reduces its overall utility in these increasingly important use cases.\n3. Performance Advantage is Concentrated in Extreme Low-Budget Scenarios: The most significant performance gains of LKV over strong baselines like LAQ are demonstrated in extremely low-budget settings (e.g., a cache budget of 64 or 128). These settings, while useful for stress-testing, are not always representative of practical deployments where budgets might be more generous. In higher-budget scenarios, the performance gap narrows considerably, questioning the method's marginal benefit when resources are less constrained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)