DefensiveKV: Taming the Fragility of KV Cache Eviction in LLM Inference
Authors: Yuan Feng
Haoyu Guo
Junlin Lv
S Kevin Zhou
Xike Xie
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient AI
Large Language Model; LLM Inference
Primary area: foundation or frontier models, including LLMs
TL;DR: This work exposes the inherent fragility of stability assumptions in KV cache eviction methods and introduces defensive aggregation to counter this issue,reducing quality loss by over 4x compared to leading methods.

Abstract: Large language models have revolutionized natural language processing, yet their deployment remains hampered by the substantial memory and runtime overhead of the transformer’s Key-Value cache. To mitigate this, recent methods employ a scoring-aggregation framework to evict unimportant cache entries, based on the "stability assumption"—that a fixed subset of entries remains consistently important during generation. However, prior work has largely focused on refining importance indicators for scoring, while defaulting to mean aggregation due to a faithful trust in the stability assumption. In this work, we argue that this underlying assumption is inherently fragile, making mean aggregation highly vulnerable in extreme cases. To counter this, we propose a simple yet elegant defensive aggregation strategy: a two-step, linear-time approach that controls worst-case risk, thereby defending against extreme cases with negligible computational overhead. Embodying this strategy, we propose a novel cache eviction method, DefensiveKV and its extension, Layer-DefensiveKV, which incorporates layer-wise budget allocation. Across seven task domains (18 datasets), our methods reduce generation quality loss by 2.3× and 4.3× respectively, versus the strongest baseline under a 20\% cache size. These results set new performance benchmarks and pioneer a promising direction for optimizing cache eviction against underlying fragility through worst-case risk management.Our code is available at https://github.com/FFY0/DefensiveKV .

Published as a conference paper at ICLR 2026
DEFENSIVEKV: TAMING THE FRAGILITY OF KV CACHE EVICTION IN LLM INFERENCE
Yuan Feng!.b,, Haoyu Guo Junlin Ly S. Kevin Zhou? Xike Xie
'School of Computer Science, University of Science and Technology of China
School of Biomedical Engineering, USTC DDta Darkness Lab, MIRACLE Center, Suzhou Institute for Advanced Research
ABSTRACT
Large language models have revolutionized natural language processing, yet their deployment remains hampered by the substantial memory and runtime overhead of the transformer's Key-Valus cache, To mitigate this, recent methods employ a Sooring-aggregatinnn framework to evict unimportant cache entries, based on the "stability assumption" " that a fixed subset of entries remains consistently important during generation. However, prior work has largely focused on refining importance indicators for scoring, while defaulting to mean aggregation due to a faithful trust in the stability assumption. In this work, we argue that this underlying assumption is inherently fragile, making mean aggregation highly vulnerable extreme cases. To counter this, we propose simple yet elegant defensive aggregation strategy: a two- step, linear-time approach that controls worst-case risk, thereby defending against extreme cases with negligible computational overhead. Embodying this strategy, we propose a novel cache eviction method, DefensiveKV and its extension, Layer- DefensiveKV, which incorporates layer-wies wise budget allocation. Across seven task domains (18 datasets), our methods reduce generation quality loss by 2.3x and 4.3x respectively, versus the strongest baseline under a 20% cache size. These results set new performance benchmarks and pioneer promising direction for optimizing cache eviction against underlying fragility through worst-case risk management. Our code is available at
com Dwtmm/PPooDeeesivekii
INTRODUCTION
Transformer- based Large Language Models (LLMs) have enabled wide range of applications Yi al. 2024 Gu 2023). Due to their autoregressive nature, LLMs maintain Key-Value (KV) cache to store intermediate representations of previously tokens, which supports efficient computation of future generation. However, as the input sequence length increases, the KV cache grows linearly, leading to substantial overhead. For example, 70B- -0--aramettr model with a batch size of 8 and sequence length of 128k may require up to 330GB of memory just for caching. This poses significant challenges regarding storage expenses and I/O bottlenecks for LLM deployment (Devoto al. 2025).
Early solutions like StreamingLLMM (Xiao et al. 2023) reduce cache size by keeping only recent cache entries, but this sacrifices long-range context. More recent solutions ÇZhang et al.) 2024b) Liu et al.) 2024c Li et al. 2024| Feng et al. 2025) on selective cache eviction operate under a key underlying assumption that a fixed subset of cache entries remain consistently important and contributes to future generation. Thus, by retaining the selected subset, the full KV cache can be approximated with much smaller memory footprint. Building on this, existing methods typically follow two-step scoring-aggregaln framework: In the scoring step, different historical token queries are used to observe multiple importance scores for each past KV cache entry. In the aggregation step, these multiple observed scores for each cache entry are aggregated-typicall by averaging to estimate its expected significance and guide the eviction strategy.
'quual Contribation:
fyfung, haoyuguo) 9mai 1 ustu. edu.en
"ooresponniig Author: xkxi e@uitc. edu.cn

---

Published as a conference paper at ICLR 2026
0.5
Stogle "oocerocl
0.2
Detershee (ocrbbaas
136
202
400
Sereratio
Figure 1: Defensive aggregation demonstrates robustness against fragile stability assumption (Llama- 3. 1-B8, 50% cache size, layer 14, summary task). Appendix provides additional visualizations.
Noatle to Baystack
Nertle Baysteck
Nestl Heystach
Mutii
Stdt d4
Stete Dac.
Stote Dwc 04
%
SyaBalte
Shit
Syntlutic
Shit
Syafhalic
Skirk
Stnamid.
Caookh (8-55)
Ca-aKY 21.001
Cauit) 25395)
(a) 1lama-3.1-8B. 20% Cache
(b) Mistral-7B. 7B, 20%
Cache
(c) Qwen-3222 32B, 20%
Cache
Figure 2: DefensiveKV and Layer-Defensiveks DefensiveKV achieve significantly lower losses of generation quality compared to all baselines across various domains and models.
Following this two-step framework, previous research has primarily focused improving the scoring step by exploring various importance indicators. Early studies often relied solely on naive attention weights (Zhang et 2024b Liu al. 2024c] Oren al. 2024). SnapKV (Li et al. (024) improved by introducing pooling mechanism, while more recent work like CriticalK (Feng et al. 2025) employed the norm of projected value states to offer more principled measure of importance. However, the second step agggegggggg these importance scores scores-remains largely underexplored.
Most existing methods default simple averaging strategy. While this may seem reasonable, averaging only effective if the underlying assumption holds that importance are stable-when it does, averaging helps reduce observation noise and capture the consistent significance of cache entries.
There raises critical question: if the stability assumption proves unreliable, is averaging still the best aggregation strategy, or might better alternatives exist?
In this work, we show that even when the assumption generally holds, it remains inherently fragile, importance scores can shift abruptly during generation. As demonstrated in Figure I performing cache compression based on the observed importance score single historical token often yields promising results, with most steps retaining over 0.8 correlation with full-cache importance. However, in certain intervals (e.g., steps 150-230), the stability assumption breaks down-consequently results based on single historical token fail, leading to sharp drops, with some outliers falling as low as 0.5. In these cases, the current standard practice of mean aggregation, simply averaging these single token predictions, inevitably results in similar outlier performance.
This reflects classic pitfall, flaw directly analogous to foundational lesson from finance: strategies that optimize only for the average case (expected returns) are fundamentally flawed because they ignore the risk rare but extreme negalive cases (worst-case risks).
Inspired by this insight, we abandon average-case optimization in favor of a worst-case risk man- agement framework for KV cache eviction, which we term defensive aggregation. Our strategy is actualized through elegant two-step process: worst-case estimation and adaptive prior-risk correc- tion. Remarkably, this approach requires only two linear-time operations, matching the computational efficiency of standard mean aggregation. As shown in Figure I| Defensive Aggregation demonstrates clear superiority, boosting the worst-case retained importance to 0.65 a substantial improvement over both mean aggregation (0.45) and single-token baselines (0.42, 0.47, 0.43).
2

---

Published as a conference paper at ICLR 2026
Building on the defensive aggregation strategy, introduce DefensiveKV, general cache eviction method, which we further develop into Layer-DefensivekY by leveraging a popular layer-wise budget allocation strategy. Figure summarizes that these two methods significantly outperform prior approaches across seven task domains, evaluated on 18 datasets from the LongBench and seedle-in-a-Haystaee benchmarks. With 20% cache budget, DefensiveKV and V V incur generation quality losses of only 4.8% and 2.6%, respectively, representing 2.3x and 4.3x reductions versus the best baseline, CriticalKV (11.1%).
2 RELATED WORKS
KV cache eviction is crucial for efficient long-sequence inference in LLMs. Current approaches largely fall into two paradigms based on their primary objective. Fixed-Patter Framework: This paradigm prioritizes operational efficiency, employing fixed rules for cache retention. Early methods like StreamingLLM (Xiao al. 2023) retained only recent cache entries, which is highly efficient but often loses valuable long-range information. More recent work, such as LaCache (Shi et al., 2025), uses a ladder-shaped structure. These methods are advantageous in scenarios where compression occur frequently, such as at every decoding step, where their low computational overhead is a core benefit. However, this efficiency often comes at the cost of compression quality; for instance, StreamingLLM can incur over 40% loss with 20% cache size as summarized in Figure 2 Scoring-Aggreeaatin Framework: The second and more dominant paradigm, which our work addresses, focuses on compression quality. These methods are better suited for common, less frequent compression scenarios. These methods, examplified by H2O (Zhang et al.) 2024b) and Scissorhands (Liu et al.) 2024c) introduced importance-baat based eviction, operating under the "cache importance stability" assumption that small set of entries remains consistently important. They follow scoring- aggregation framework: they observe importance multiple times and then aggregate these scores, often by averaging, to decide on eviction Ren & Zhu 2024); Oren et al. (2024).
Within the dominant acgring-aggregattion framework, prior research has almost exclusively focused on improving the importance observation (scoring) step. For example, SnapK (Li et al.] 2024) introduced pooling, and CriticalKV (Feng et al. (2025) proposed using projected value norms for a more principled score. However, the foundational stability assumption and the subsequent aggregation step have rarely been rigorously examined. This paper revisits and reveals the fragility of this assumption, further showing prevalent mean uggregation's 's vulnerability. Consequently, we the first underscore the necessity of risk-control defensive aggregation strategies to against fragile assumption. This pioneers a new research direction, entirely orthogonal to prior work focused on optimizing importance indicators. For demonstration, build our DefensiveK method upon CriticalKV, the current SOTA importance indicator.
Additionally, our contributions is also orthogonal various KV cache budget allocation strategies, including intra-layer (e.g., AdaKV (Feng et al.) 2024)), lnter-layer (e.g.. PyramidKV (Zhang et al. 2024a). LightTransfer (Zhang al. 2025), CAKE (Qin al. 2025 ). and also offline training- based allocation (c.g., HeadKV (Fu et al. 2024), DuoAttention (Xiao et al.) (025)). These strategies focus on optimizing budget allocation for cache eviction methods, and are thus inherently orthogonal to our investigation. Direct comparison is not essential for validating our contributions. However, to demonstrate our principles' adaptability, we introduce budget allocation for enhancement. Broader related methods like quantization, channel pruning, and Furthermore, we provide a case study in Appendix on integrating our D with quantization, showing minimal loss 10% cache footprint.
weweer-effnsivKK.. a variant using layer-wise
sparse attention are discussed in Appendix]] IF
3
METHODS
3.1 PRELIMINARY
LLM generation consists of two stages, prefilling and decoding. During prefilling, the KV states for all input tokens are computed and cached as:
HWK. V HWy, y. where H E Raxd denotes the hidden states for tokens, and WK, Wy E dddd are learned matrices. In decoding, the LLM 1,:
takes the most recent token, computes its query vector = H - Wo. and retrieves information from the cached KV entries using attention to produce the output oj and predict the next token:
3

---

Published as a conference paper at ICLR 2026
Single historical token 18th meen value-d 92, wont-case valuemb. 34, outlier countivelueeee 5
3
26
31
1.0
100
200
300
(a) 16th historical token: Vulnerable; importance drops to 0.34 (worst case).
Generation Stop
0.9
Mean aggregation: mean value-0 92, worst-case valta-o. 33, outher count (value<0.5)-65
0.8
layer
0.7
0.0
Generation
(b) Mean aggregation: Vulnerable; importance drops to 0.33 (worst case),
0.3
Defensive aggregation:
value=0.11 outter (v)luex0.5)_0
0.4
byer
0.3
100
Generation
(c) Defensive aggregation: Robust; maintains 0.61 importance (worst case).
Figure 3: Mean vs. Defensive Aggregation: Against Fragility in Importance Stability
0j = AVo where A, softmax (g,KT/Vdx))
To reduce the memory overhead of maintaining the full KV cache, cache eviction methods have been developed. These methods largely operate under stability assumption: a fixed subset of KV cache entries, denoted as (K, V), retains consistent importance throughout generation. Based on this assumption, the objective of cache eviction to identify this crucial subset (K, V) using historical queries (i.c., tokens from earlier generation process), and use it to replace the full KV cache (K, V) in subsequent steps. This process typically follows a two-step scoring scoring-aggresse aggregation framework, where the importance of each KV entry is first estimated (or scored) and then aggregated:
1. Scoring. Given m historical tokens, represented queries Q lg...... 9m). each of the KV cache entries K, V E Raxd, is scored. This results in an importance matrix I E R where each element I5.; measures the relevance of the i-th KV cache entry (kg, v;) for the j-th historical query Qj. In practice, the attention weight Aj.: serves as direct measure of importance I,. given the attention mechanism's 's inherent weighted-sum formulation.
2. Aggregation. Subsequent aggregation step converts the observed importance matrix I E IR' mm into a vector S E IR",, where each S: represents the importance the i-th KV cache entry. Existing works adopt aggregation, S₁ m ΣØ= Iz.,, to highlight entries with consistently high importance, in line with the stability assumption.
While subsequent studies have refined the scoring step-SnapKV employs pooling mechanism, and CriticalKV utilizes the norm projected value states vg Wo for more principled scoring aggregation step has received little attention. This is largely because mean aggregation appears to align closely with underlying importance stability assumption. However, this work, we show the fragility of that assumption and reveal the vulnerability of mean aggregation under this premise. This underscores the necessity of revisiting and improving the aggregation step.
3.2 FRAGILITY OF STABILITY ASSUMPTION AND VULNER ABILITY OF MEAN AGGREGATION
We examine this using Llama3.1 1-BB on the Government Report summarization task. Adopting the SOTA importance indicator, T5.a Aj.; x norm(r; Wo). we observe each cache entry importance with 32 recent historical tokens Qj- We then simulate a 50% cache eviction using two different criteria. The first criterion uses importance scores from single historical token observation, while the second uses scores averaged across all 32 historical tokens (mean aggregation). For each result, we track the proportion of total importance it retains during subsequent generation, relative to the full cache.
Fragility of the Stability Assumption. Figure 3a presents the results for the 16th historical token(see Appendix M for results from other tokens). The results reveal general, yet fragile, stability. On average, the 50% retained cache subset accounts for 0.92 of the full cache's total importance during generation. However, this high average belies the underlying fragility: the retained importance can
4

---

Published as a conference paper at ICLR 2026
Table 1: Defensive aggregation consistently improves the worst-case values across all task types.
Task type Dataset
Single- Doe. QA NrtvQA
Multi-Doc. QA HotpoiQA
Summary GovReport
Few-shot TREC
Synthetic PCount
Code
Lcc
Mean aggregation Defensive aggregation
0,44
0.39
0.28 0.52
0,47 0.61
0.47
0.30
0.62
0.60
0.61
0.50
Algorithm 1 Defensive Aggregation
1: Input: Importance scores I R="x,, where I,,, is the importance of entry i based on historical token
2: Output: Aggregated risk scores R E IR"
3: R. maxis)sm J,x,, Vi = I....,n
Worst-case Risk Estimation
4: R, max Rs, R) where R 2. E Ru, Vi vi=l, I,...,
Adaptive Prior-Risk Correction
5: return R
drop sharply, as seen in the interval between steps 150 and 320. these moments of instability, the worst-case retained importance drops to low as 0.34. Additionally, outliers-whers the retained 50% of the cache captures less than half of the total importance (value < < frequent, occurring in 89 instances this trial alone.
Vulnerability of Mean Aggregation. Current eviction methods commonly employ mean aggregation over multiple importance observations. The rationale to obtain an expected importance to guide eviction. However, by failing to account for worst cases, this strategy becomes vulnerable precisely under the fragile assumption, leading to outlier performance similar that of using single, unreliable token. As shown in Figure (bb significant drops persist at the same problematic steps observed in Figure 3al reaching a worst-case importance value of 0.33 and resulting in 65 outlier instances. This outcome is predictable. The observation score based on single historical token is inherently blind to the fragility of stability assumption; thus it cannot hedge against the worst-case risk. While simple averaging acts form of "reconciliation" among these individual observations to produce moderate result, it does not incorporate any mechanism to control for this underlying risk. Consequently. cannot aggregate prediction to consistently outperform every single-token observation. When most single token-based observations fail, the mean-aggregated result is inevitably dragged down with them, thus offering no meaningful improvement and producing similarly damaging outliers.
This underscores a critical point: rather than focusing solely on designing more accurate importance indicators, it is equally-if not more-immprttan develop new aggregation methods explicitly designed for worst-case risk control, which can provide reliable estimates even when most single- token observations fail.
3.3 DEFENSIVE AGGREGATION VIA WORST-CASE RISK CONTROL
Consider a cache entry may exhibits high importance in only few single-token observations while remaining low in most others duo to fragile stability. Mean aggregation would not recognize this as important and would erroneously evict it. When this entry becomes crucial again in future generation, the prior eviction results in substantial importance loss. Thus, relying mean aggregation fails to guard against these extreme cases. To address this, we introduce a novel defensive aggregation, novel strategy that eschews simple averaging in favor of worst-case risk control perspective as shown in Algorithm [
Worst-case Risk Estimation. From risk-control perspective, the penalty for evicting a KV cache entry is equivalent to the importance score it would have possessed at future moment. The "worst-case risk", R; therefore the peak importance score an entry could attain over the entire future generation process. If we denote the future generated sequence as L. then Rt max 4,,, As this future maximum unknowable eviction time, we instead approximate as the maximum importance score observed across all j historical tokens, e.g. R maxigism I5, Vi = 1..... n. This (()) procedure matches mean aggregation's 's runtime yet yields significantly better empirical performance, as it better captures the potential worst-case risk if the entry were removed.
For Grouped- Krouped-Query Attention, cache entry's 's worst-case risk estimate is the maximum importance score observed over historical tokens across all heads sharing its KV group
5

---

Published as a conference paper at ICLR 2026
Adaptive Prior-Risk Correction. Although the above estimator takes the maximum over observed history, it could still underestimate worst-case risk because eviction methods typically restrict the observation window (c.g.. 32 tokens) to limit overhead. [ Such restricted observations could miss rare but critical risks. Inspired by Laplace smoothing in Bayesian estimation, we introduce an adaptive prior-risk correction. For each head, define a head-level prior risk R = EE= R2, i.e., the average observed worst-case risk across entries for that head. If the observed i Rt falls below prior risk
R, we treat the shortfall as under-observation and substitute the prior: R
max R, R
Thereby, heads with higher overall risk receive larger priors, reducing reliance on limited historical observations. The effectiveness of correction and its adaptive design is validated in Section 4.4
By defending against risks of the fragile stability assumption, our defensive aggregation substantially improves worst-case performance compared to mean aggregation. As shown in Figure Bc it boosts the worst-case retained importance from 0.33 to 0.61 and completely eliminates the 65 outlier instances produced by mean aggregation. Table further confirms this advantage is consistent across six datasets with different task types. Therefore, this simple two-operation method provides a crucial defense against the fragility of the importance stability assumption.
3.4 IMPLEMENTING DEFENSIVEKV EVICTION METHOD WITH DEFENSIVE AGGREGATION
Building upon our proposed defensive aggregation strategy, we introduce two novel cache eviction methods: DefensiveKV and nayer-DefenssieeK.
D serves as the foundational variant. It directly integrates defensive aggregation into the traditional cache eviction workflow by replacing the conventional mean aggregation. Despite its simplicity, this modification alone leads to substantial performance improvements.
Layer-Defensivets further refines this by incorporating layer-wise budget allocation, inspired by existing strategies (Feng et al., 2024, Zhang et al. (2024a). It performs joint selection of risky entries across layers, enabling more budget to be allocated layers with more risky cache entries.
As shown in A the overall process of D adheres to the established practices: preserving the KV cache entries of several recent historical tokens (Line 3) and then utilizing the query states of these tokens for importance measurement. The importance calculation begins with basic attention weights (Line 4) and incorporates further refinements sppecfiicall pooling mechanisms from SnapKV (Line 5), and projected value norm scaling from CriticalKV (Line 7) H
The key innovation in D is the strategic replacement of conventional mean aggregation with our defensive aggregation (Line 6). This simple modification, requiring minimal changes, reduces over 2x in generation quality loss. The extension to Layer-Defensinn incorporates two additional refinements: first, projected value norms are normalized layer-wise to address their variance across layers (Line 11); second, risky entries are selected jointly across all layers (Line 12). This leads to even more impressive gain, with over 4x reduction in quality loss.
is also straightforward.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Models. We evaluate our approach on three open-source LLMs: 1aamaa-.---B-In-stee (la 2024
Touvron al.) 2 and Dwven..5-32B-Instrru (Team] 2024), supporting context lengths of up to
128K, and Distral-7B-Inssruett-v (Jiang et al. 2023 supporting up to 32K.
2023 is Baselines. We compare method against six baselines. StreamingI.LM (iioo et al.
early sliding window approach. SnapKV (Li et al. 2024). AdaKV (Feng et al. 2024). and CAKE QQn al. 2025) use attention weight-coupled pooling for importance indicators: CAKE also employs a cascaded architecture for layer-wise budget allocation. The SOTA Critical (Feng et al. 2025 introduces a more accurate importance indicator. DuoAttention (Xiao et al.) 2024b). a training- based
Explicitly computing attention weights for all tokens infeasible with FlashAttention optimization, and even storing all attention weights is prohibitively expensive (c.g.. R:64 GB for a 32k context Ilama-3.1 1-88). 'Although our method is based on current SOTA practice, defensive aggregation is widely applicable to other eviction methods. Appendix incluuees case study applying it different haseline methods for demonstration.

---

Published as a conference paper at ICLR 2026
Algorithm 2 (Layer)-Defensivek))
1: Input: Cache Entries K, V, Parameter Wo, queries of m recent historical tokens Q = [yn: ..... (mm
2: Output: Retained KV Cache K, V
Append the KV cache of recent historical tokens K[-m 1, V[-m ;,] K
4: é (ettmax(QK//Va
2TTx= +- Pooling(AL, dim -1)
Refined with pooling by SnapKV (Li et 2024
IRn +- Defensive Aggregation Algorithm (A) 7: R; < R2 x norm (n; Wo) Vn, E 1,. Refined with norm by CriticalKV (Feng et al. 2025
b Our modification
8: if without layer-wise budget allocation then
b Leading to Defensivek Select the cache entries with top worst-case risk It independently in each layer and append to K, V > Leading to Layer- Defee-DDffesiveKY
9:
10: else
11: H, H norm(e:Wo) Vou V L....,
12: Select the cache entries with top worst-case risk jointly across all layers and append to K,
13: end if
return K,
GtreamigLLM
AdsKV Cricee/VV
CAKE
DefensiveKY Layer-DefereiveRV
SnapKV
100%
Full Cocho
100%
Full Cache
100%
Full Coo
Score Ratio
96%
Score Ratlo
95%
Score Ratio
86%
90%
90%
80%
86%
85%
85%
80%
20%
40%
60%
80%
100%
20%
40%
80%
80%
100%
20%
40%
60%
80%
100%
Cache Budget
Cache Budget
Cache Budget
(a) Meta-Llama-3.1 -a..laaaaa..-------. 1-8-1-8--Issttce Instruct
(b)
(c) Qwen2.5-32B-Inss
Figure 4: Overview of averaged generation quality across 16 datasets on LongBench.
method, is included with official configurations for Llama-3. 1-BB and Mistral-7B--O.33 but marked N/A for Qwen2.5-32B due to unavailable configurations and high training costs. All methods use historical window size of 32 and are accelerated with Mashhttentionn2(Dee et al. 2022 Dao (023).
Settings. Following the settings in (Devoto et al. 2025 Feng et al. 2025). the context compressed independently before question is introduced. This better simulates practical scenarios (e.g., multi-turn QA or prefixed contexts) where multiple questions often pertain to the same context, or the question unavailable during context compression. Thus this setup more challenging and better reflects the real-world performance of cache eviction methods. (Feng et al. 2025
4.2 LONGBENCH EVALUATION
LongBench (Bai al. 2024) serves comprehensive benchmark, featuring 16 datasets structured into six task domains: single-document QA, multi-document QA, summary, few-shot learning, syn- thetic tasks, and code completion. Detailed information for each dataset can be found in Appendix N
Overall Analysis. Figure illustrates our methods' significant advantages in average quality loss across 16 datasets, As cache size drops from 100% 40%, all baselines degrade noticeably, while our DefensiveK and Layer-DefensiveK] V remain nearly lossless. For instance, with a 40% cache on Llama-3.1 1-8 (Figure Ha), CriticalKV (best baseline) loses 3. 1.15 % quality, whereas our DefensiveK the training-based DuoAttention (2.2% drop, despite offline
shows degradation, surpassing training costs). At a smaller 20% cache, CriticalKV's loss is 10.6%, while DefensiveKV limits it to 5.1 %over 2x reduction). Our Layer- Deffens-eins further cuts this loss 2.3%(over 4x reduction). Similar advantages hold for other models. For instance, on Mistral-7B with 20% cache, DefensiveKV and Defere--efessveKY achieve 4.0% and 1.3% loss, respectively, far below CriticalK 9.7%.
Task Analysis. Figure 5 displays Llama-3.1-8 1-BB average scores by task domain (Appendix A for more models). V and Layer-DefensivekY consistently excel. While simpler task
7

---

Published as a conference paper at ICLR 2026
Mahi Dec. QA
Mahi Dec. QA
100%
Slngle Doc- QA
Single Dec. OA
Jank Beat. ON
Code
Clste
Cutle
7554
2814
50%
Sumuttarry
Feu
StrcaningLL 30.3%
CratoakV--1155)
15
Legon
Figure 5: Analysis of the six task domains on LongBench for (c) 60% Cache Size
(a) 20% Cache Size
(b) 40% Cache Size
Table 2: Detailed scores of 16 datasets on LongBench.
Single Document QA
Multi- Ducumeni Q4
Summarization
Few--sho Learning
Synthetic
Code
Method
NavQ4
Qusper
MFui
Happll
Muniqua
GovRop
QMSam
MuttNNws
TREE
TriviQA
SAMSum
PCoun
PR--n
Lec
RBBP Arg.
Llamo-3.1 1-88- Instruct, 20% Cache Size
Full Cache
29,55
64.68
55.82
57.59
18,89
32.61
34.40
25.51
26.83
73.00
92.36
43.27
7.38
99.50
63.44
52
36
19,20
DusAtentio
23.28
21.22
34.03
42.89
28.14
20.57
25.32
19.48
23.12
56.00
86.54
10.67
7.50
78.50
65
94
59
19 19.52
SLreamingLLM
22,05
19.33
27.76
22:27
89.97
40.04
29.50
66
34.72
SeupKV
22.07
02.21
44.088
79.
40.21
CAKE
22.16
22.91
91.85
43.56
41.85
AduKV
27.07
28.69
30.85
89.64
30.89
21.57
26.70
21.85
22.67
55.50
91.30
43.89
7.30
80.50
66 66.44.. 55.
41.39
Chinealkyy
29.81
32.58
34.96
52.14
36.24
26.37
28.35
23.52
23.24
56.30
90.80
43.37
8.89
93:00
67
02 54.99
43.88
DofemsivelC
79.97
20.46
46.21
$2.30
18.40
28.06
29.95
23.89
24.11
68.00
91.58
47.17
8.28
100.00
67
17 55. 40 46.68
30,10
42.91
52.94
55.03
44,07
27.00
30.99
24.95
24.42
69.00
91.30
43.55
3.38
100.00
67.60
56.00 48.01
Misttral
Instruct +0.3, 20% Cache Sizee
Cache
50.75
37.41
34.45
DusAnemion
29.88
22.43
9.18
43.08
31.13
Streamingl.1M
18.30
28.00
20.73
80
40:57
78. 28:00
46.91
29.58
SnupKV
21.91
23.69
30.59
43.71
28.28
19.81
27.91
21.15
22.15
55.00
89:11
46.67
5.00
82.50
61.30 59.97
40.13
CAKE
23.08
25.42
35.31
44.10
28.96
18.50
26.27
21.18
22.61
60.00
90.36
46.40
4.00
77.00
64. 50 50
40.56
AdaKV
24.00
26.29
31.15
45:26
27.99
20.65
27.37
21.57
22.38
59.00
89.87
46.27
4.50
88.00
65 05 59.50
41.18
Chitialliv
24.14
20.56
58.91
45.42
32.05
21.36
28.59
22.71
23.11
65.50
91.13
46.65
4,11
90.00
65 42 58
42.88
DefensiveKV
21,05
26,09
901.1
21
Qwen2.5:3328 Listrrct, 20% Cache
Full Cache
30,88
46.13
52.87
63.39
59.75
38.78
32.59
24.35
24.95
72.00
88, 26
47.05
12.50
100.00
49
64
34.24
48.60
DusAnemion
N/A
N/A
N/A
N/A
N/A
NIA
N/A
N/A
NIA
NIA
N/A
N/A
N/A
N/A
N/A
N/A
N/A
SMromingLLLM
20.74
17.68
25:04
39:53
33.19
17.58
27.83
19.04
21.67
61.00
82.01
42.80
10.67
29.00
56.20
30,
14
13.37
StapkV
24.52
24.58
23.80
52.54
42.07
29.74
28.25
19.45
21.35
58.00
87.74
45.21
9.00
76.35
53.19
35.58 29.96
CAKE
22.25
26.19
31.33
89.3 19.20
12.48
28.11
27.86
16.92
22.02
58.00
67.77
17.22
11,00
81.25
51 12
35 25
10.06
AdaKV
29
19.33
48,02
74,00
40.20
CHicallV
32.51
34
10.50
43.72
DefensivKC
31.11
33.11
45.66
63.55
$7.68
40.55
30.30
22.06
22.08
21.00
NH.
9.50
36.98 47.10
Layer De
31.38
35.65
19.25
64.24
58.86
20.25
30.86
23.07
23.10
75,00
88.92
47.47
10.00
99.88
52 21
35.62
47.86
domains (Code, Synthetic) show high performance for most methods, challenging ones (Doc QA, Summarization) reveal significant baseline degradation under reduced cache. Our methods maintain their advantages. For instance, in Single-doc QA (20% cache), CriticalKV (strongest baseline) drops to 74.8% of full-cache quality; DefensiveKV achieves 89.6%, and Defees--Dffns reaches 96.7%. Table further reports detailed 20% cache scores (other results in Appendix K). On Llama-3. (20% cache), DefensiveK beats CriticalKV V on 13/16 datasets; Layer-Defensivek"s wins 15/16. Such a significant performance advantage, rarely observed between other baselines, highlights the effectiveness of our "worst-case risk" perspective against underlying fragility across diverse tasks.
4.3 NEEDLE-IN-AAHAASSICN EVALUATION
In the Baelle-in-a-Hasstad test, the key sentence is placed in a long context to evaluate retrieval ability. Following Ruler (Hsieh et al. 2024), we test two representative cases with a 32K context length: (1) Single-retrieval: one needle randomly inserted for retrieval. (2) Multi-retrieval: four needles are randomly inserted and all need to be retrieved. Further details, along with evaluations on more beeddl-in-a-haystack--ee tasks from Ruler, are provided in Appendix
8

---

Published as a conference paper at ICLR 2026
Meta waa.amaam.... 3.1-888
135 124
116
192
1AE
139
136
140
120
1:00
82
200
Ceche
Mistraal
Malti Noodle
Single Needie
161
13:1
120
80
40
20
180 199
139
1660
Single-Needle
157
124
113
125
80
73
40
Hia
Na
strnoring ton.
Snapty CAKS 1o%
Perfenlletty tou
do dtresningg zon S Snapky
CAKe s0n
crticatty 201 defensivety 201
St eaning Ap Snapey aor.
Adsky sot
Layar mer-oeeenivaak 40%
defenslven 4ttt.
40%
streniiing eox seapty Cake cpra
Asart 4o% Cottictity co%.
Due
Layer Defensivek
Buo
CAKE
Lay -po-Oaa theenniiee Mensteery
you
Figure 6: Evaluations on the needle-in haystack tasks.
Score 304
Gagi
Mete Liema Instnart, Need hayslck XDX
175
179
Mall-hesi
156
Prefiling
Critusi (Betch 11
Detensiveky (Batch
Crticatty (Batch 21
Defensivaky (Batch
Decoding
1.00
140
120
106
Labence
12
64328
64128
0.00
66328 54-226435
Abd itema Abrz- dama
Abr smistral Abd ! Durs-at maaammaaall
Abit Ab menn Ou Dura-pi swen
Length
Ab2
Pixed Pived Pived 18-5 ours
Figure 9: Efficiency(FlashAtnn))
Figure 7: Ablation 10% cache. Figure 8: Adaptive Correction
As shown in Figure 6 our D and Layer- Defensvefesiv achieve significantly higher scores across all settings. For instance, on long-context models like Llama-3.1 3.18BB and Qwen2.5-32B (both supporting 128K context length), our methods maintain near-lossless, with scores 194 and 193 for Llama-3.1-8D mere 10% cache size. contrast, even the strongest baseline, CriticalK V, drops to 140 under the same conditions, while others fall below -0-ddmmnnstrating substantial gap. On weaker long-contex: ability model, i..., Mistral-7B (maximum context length 32K), all baselines suffer severe performance degradation. At 10% cache size, most baselines score below 6, and CriticalKV V only reaches 28. However, our DefensiveKV and Layer-Defensivek achieve scores of 139 and 161, over 5x and 5.8x improvements, respectively.
4.4 ABLATION STUDIES
Component ablation. To analyze the roles of the two key operations in defensive aggregation-worte -aorst-csse risk estimation and adaptive prior-risk correction conduct ablation studies based on our DefensiveKV method. First, only remove adaptive prior-risk correction (denoted as Ab12). Then, we further ablate the worst-case risk estimation by replacing with common mean aggregation (denoted Abll). As shown in Figure 7 using only worst-case risk estimation (Abl2) already significantly outperforms that are with mean aggregation. For example, on Llama- 3. 3.1-8B, Abl2 improves the score from 103 (Abl1) 179. Adding our adaptive prior-risk correction provides further gains, with our full DefensiveK V method reaching a score of 194. These results confirm that both operations contribute meaningfully to the overall performance.
ddattiv correction analysis. To validate the adaptive design of our prior-risk correction, we ablated against fixed correction thresholds (1E-3, 1E-4, IE-5). The results in Figure show that fixed thresholds are ineffective. Most fail to outperform the no-correction baseline score of 179 (Abl2), with the IE-4 case providing only marginal gain to 182. Our adaptive correction, however, reaches a score of 194, confirming that tailoring the correction to each head's 's risk profile is crucial. Additionally, the hyperparameter-io free nature our adaptive design ensures consistently strong performance across two additional models (shown Abl2 ours in Figure 7

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study the historical window size.
Single Document QA
Multi Document QA
Sumaaurizzttinn
Few shot Learning
Synthetic
Code
Method
NirO4
Qeeper
Mf-ein -n-
Herpu
2WibiOA
Musique
GorRop
QMSuum
MuttiNws
TREE
TrividtA A
SAMSum
Piasmn
PR
RB-P Agg.
Liama 3.1 8D Instruct 2017
Cache Size
Full Cache
29:55
44.68
55.82
57.59
48.89
32.61
34.40
25.51
26,83
73.00
92.36
43.27
7,00
99,50
63.44
52.36
49.20
CriicalKV
29.81
32,58
34,96
52.34
30.24
25.37
28.35
23.52
23.24
$7.00
90.80
43.37
8.89
93.00
67.05 54,99 43.88
DefensivKY (36)
29.75
35,43
46.54
55.51
36,41
25. 25.64
29.30
23.68
23.34
66:00
91.69
44.02
7.31
100,00
67 00 55. 10 46,02
DefensöveK (33)
29.97
40.45
46.23
52:30
38.40
28.06
29.96
23.89
24.11
68.00
91.58
43.17
8.28
100.00
67.17 55, 46.68
DefensiveKV
28.98
36,09
45.49
27.50
24.12
25.31
68.00
43.35
6.77
100,00
66.72 55.04 46.41
DefensiveKV (26)
54.59
24.25
68.50
44,04
9,00
68.82 47.55
Tayes-Defnnivveky
42,91
55.03
44.07
8.38
48.01
Layer DefensiveKY (64)
43.40
51,42
55.36
40.03
24.29
70.00
91.61
100.00
47.75
Window size robustness. In our main experiments, we adopted the default historical window size 32, following the setting of the strongest baseline, CriticalK V Feng et al. (2025). To analyze the impact of this hyperparameter,
conduct an ablation study the Llama-3.1-88 1-8B-Inntrrct model (20% cache size) using historical window sizes of 16. 32, and 64. The results are presented in Table 3 Both D and Layer-Defensivekr consistently and substantially outperform the CriticalK baseline (43.88) across all tested window sizes. DefensiveKV achieves average scores ranging from 46.02 46.68, while Layer-Defensee V scores between 47.53 and 48.01. This study demonstrates that the effectiveness of our proposed methods is robust to variations in the historical window size.
4.5 EFFICIENCY TEST
We compare DefensiveKV V and CriticalKV, which differ only in their aggregation mechanisms, to demonstrate that defensive aggregation introduces negligible computational overhead. Our experi- ments, conducted on an 80GB A100 GPU with Llama-3. 1-BBB (20% cache), show in Figure that DefensiveKV and CriticalKV have nearly identical time-to-fiie tome-to-fistttoken (TTFT) and decoding latency. All KV cache eviction occurs during the prefilling stage and is included in TTFT, confirming that DefensiveKV adds negligible overhead. Additionally, cache eviction significantly reduces decoding latency versus Full Cache: e.g., for batch size and 128K context length, latency drops from 0.081s (Full Cache) to 0.028s with eviction based methods (a 2.9x speedup). Furthermore, cache eviction allows larger batch sizes; for example, eviction methods can handle batch size of 2, while Full Cache results in out-of-memory errors, leading to 4.2x decoding throughput boost. See Appendix for memory usage details.
5 CONCLUSION
This work challenges the fragile stability assumption underlying existing KV cache eviction methods. We show that widely used mean aggregation strategies are highly vulnerable under the fragile stability, resulting in poor worst-case performance. To address this, propose "defensive aggregation", novel strategy explicitly designed from a "worst-case risk" perspective with negligible computational overhead. Based this, we investigate D and its layer-aware variant, Layer- Defensivek both of which achieve significant improvements over state-of-the-e methods across comprehensive evaluations. Our work pioneers a new research direction by emphasizing the "worst-case risk"-aware aggregation to mitigate the often-overlooked fragility in cache eviction-a critical yet underexplored component of efficient LLM inference. We hope these contributions pave the way for more effective cache eviction methods, which are essential for advancing LLM inference.
ACKNOWLEDGE
We would like to thank the anonymous reviewers for their valuable feedback and constructive comments. This work was supported by the National Natural Science Foundation of China (NSFC) under Grants 62472400 and 62271466666 the National Key R&D Program of China under Grant 2025YFC3408300, and the Suzhou Basic Research Program under Grant SYG202338.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
-lama-3-bb-instueee h0a.:33-----ammm-aaaeee--------- 2024. URL htetp:://nugglggffccccco///rrrr
L.lama-33 -tn.-------nrrrrt-rrrraiiiieeee-
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv: 14508, 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding, 2024. URL/attps:/axxxi..oo /
2308 14508
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08N0 086611,,,.611.....0011 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances Neural Information Processing Systems, 35:16344-16359 2022.
Pradeep Dasigi. Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information--seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011, 2021.
Alessio Devoto, Maximilian Jeblick, and Simon Jégou. Expected attention: Kv cache compression by estimating attention from future queries distribution. arXiv preprint arXiv:2510.000 2025.
URL wtt.tttss///aii.....t wwtp//vi.rrr/ooooooo
00636
Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: large- scale multi-document summarization dataset and abstractive hierarchical model. arXiv preprint arXiv:1906.017499 2019.
Hugging Face. Unlocking longer generation with key-v cache quantization. Hugging Face Blog, 2024a. / htw.a////ggggggacccco.ooooo/oooe
cache wwt-tauaaaitiiiiiaeeee
Hugging Face. Quanto: pytorch quantiiition backend for optimum. Hugging Face Blog, 2024b.
URL
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient Ilm inference, 2024. URL
org/abs/240) 2407. 11550
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Identify critical kv cache in 1lm inference from output perturbation perspective, 2025. URL https://arxiv.orr abs//
1502.03800
Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning. arXiv preprint arXiv:2410.1 19258, 2024.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human- annotated dialogue dataset abstractive summarization. arXiv preprint arXiv: trXiv:1911.12237 2019.
Qiuhan Gu. Llm-based code generation method for golang compiler testing. In Proceedings of the s1 ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 20001, 2023.
Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. Longeoder: A long- pre- trained language model for code completion, 2023. URL htttts://aaatt orq/ abs 2306
14893
11

---

Published as a conference paper at ICLR 2026
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference Computa- tional Linguistics, pp. 6609-6625, Barcelona, Spain (Online), December 2020. International Committee Computational Linguistics. doi: 10.18653w12000.colinng maiaiinng-min.880.. URL
https://aclanthoet wttp://ww..rggo....oooooo waw.-eoiooinnnniininoooe
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270-1303, 2024.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 2024.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. arXiv preprint arXiv:210.002112, 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Cew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context iims via dynamic sparse attention. arXiv preprint arXiv:2407.02990 2024.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL https: h/tp://axxvvooooo/ abs/170s 03551
Gregory Kamradt. Needle In A Haystack pressure testing LLMs. Github, 2023, URL https
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002.
Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation. arXiv preprint arXiv:2404. 14469, 2024.
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Reza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models. Advances in Neural Information Processing Systems, 37:139997-140031 2024a.
Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chen- gruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattention: Accelcrating long-context 11m inference via vector retrieval. arXiv preprint arXiv:2409. 105 2024b.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository code auto-completion systems, 2023. URL http:p::/aaxvvv.orraab//////
03091
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang. Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for 1lm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024c.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.022 02750, 2024d.
12

---

Published as a conference paper at ICLR 2026
Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, and Edoardo M. Ponti. The sparse frontier: Sparse attention trade-offs in transformer 1lms, 2025. URL httph::://// aattps://arxiss
org/abs// 2tt/a/////50..777//..
Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi- state mas. arXiv preprint 0tfi::2440.6660000 2024.
Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. CAKE: Cascading and adaptive KV cache eviction with layer preferences. In The Thirteenth International Conference on Learning Representations, 2025, URL [tRL:ttt:///peeeeeee net /
Siyu Ren and Kenny Q. Zhu. On the efficacy of eviction policy for key-value constraiied generative language model inference, 2024. UtLhttps: URtpsttss://axiie http///axvvvvvaa orq/ a24p:///bb/442..... /2tp:/2222..0 06262
Dachuan Shi, Yonggan Fu, Xiangchi Yuan, Zhongzhi Yu, Haoran You, Sixu Li, Xin Dong, Jan Kautz, Pavlo Molchanov, et al. Lacache: Ladder- kv caching for efficient long-context modeling of large language models. arXiv preprint 1rXiv:2507.14204 2025.
Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404. 11912. 2024.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context Ilm inference. arXiv preprint 27X.v:2406.10774, 2024.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL
57]
Qwen Team. Qwen3 technical report, 2025. URL
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-553 5554, 2022.
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training- frree long-context extrapolation for Ilms with efficient context memory. arXiv preprint arXiv:2402.04617, 2024a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 17453 2023.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long- IIm inference with retrieval and streaming heads, 2024b. URL
10..10811 10819
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, junxian guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context LLM inference with retrieval and streaming heads. In The Thirteenth International Conference on Learning Representations, 2025. URL
https::/ httpss////peeeevee...co net/ www.mooidoooo.oo.
Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. Think: Thinner key cache by query- pruning. arXiv preprint arXiv:2407.21018, 2024.
Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du and Bo An. Longspec: Long-context speculative decoding with efficient drafting and verification, 2025. URL
https://a wwwpxxxwwwwxxpp/x/xxxxox
17421
13

---

Published as a conference paper at ICLR 2026
Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, and Zhi Chen. Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing. arXiv preprint arXiv:2410.18517, 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.0900 89900900000 2018.
Zihao Yi, Jiarui Ouyang. Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. survey on recent advances in Ilm-based multi-turn dialogue systems. arXiv preprint arXiv:2402.18013. 2024.
Xuan Zhang, Fengzhuo Zhang, Cunxiao Du, Chao Du Tianyu Pang, Wei Gao, and Min Lin. Lighttransfer: Your long-context Ilm is secretly a hybrid model with effortless adaptation. In Workshop Reasoning and Planning for Large Language Models, 2025.
Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong. Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406. 02069, 2024a.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024b.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz. Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based multi- domain meeting summarization. arXiv preprint arXiv:2104. 05938, 2021.
14

---

Published as a conference paper at ICLR 2026
Neatle Haprrack
Dec,
100
Stagl Boe Q4
Needl in Maystok
Multi 100%
Nerdle Haystack
Doc.
100
Steck Doc. 04
Dec. Q4
75%
50%
Emwnang
Samtmary
Syuthstjp
Stot
Synthetic
Fet Shol
Syathotic
Shot
StramilggL (2.5)
Struandigg Mi-36599
Cak-kY 63%
Dusi (ha5))
CalekYcsSinng
CrabatkVi dane)
MAKKV 4.00))
Craintkvv Jana
DefmutraaV 5.5%
DefamirdK
(a) Llama-3.1 3.1ma-3.1130 3.1-8B, 40% Cache
(b) Mistral 7B, 40% Cache
(c) Qwen-32B, 40% Cache
Neadle Htaywrk
Matti Dec. QA
Nestle Harsstck
Muläi Dac. DA
Nodis is Haysuck
Matti QA
Sunde Dec. QA
slack Dec. QA
Stapk Doc. Q4
50%
50%
%
Sommnn Commarr
100
Syntburie
Shot
Syothetic
Skot
Systhetic
Shot
StrumbeggL 36,56)
SnopKY/4.1 %0
Straningg
C3lakV--1.5%
CyaiaakVi dutt)
MakY(-11
dd Llama-3.1 1-8,, 60% Cache
(e) Mistral-P 7B, 60% Cache
(f) Qwen-32B, 60% Cache
Figure 10: Summarization quality losses.
A QUALITY LOSSES OF METHODS WITH 40% AND 60% CACHE SIZE
Figure 10 further summarizes the quality losses of different methods at 40% and 60% cache sizes. can observed that both DefensiveKV and Mayer-DefensiveKV maintain nearly lossless per- formance, in some cases even surpassing the original uncompressed results. In contrast, all other methods exhibit notable declines in quality. These results demonstrate the effectiveness our approach.
B DETAILED SETTINGS
The fundamental settings for SnapKV, CAKE, AdaKV, CriticalKV and our methods were kept as originally defined, with an average-pooling kernel size of 5 and a historical token size of 32 for observation. For StreamingLLMM (Xiao et al. 2023), follow standard settings, using sink tokens and retaining the most recent window's ' cache. For DuoAttention (Xiao et al. 2025), we follow the publicly released training settings. Following standard practices in prior studies (Li et al. (0244] Zhang et al. 2024a] Feng et al. 2024), we perform cache eviction immediately after the prefilling phase of each layer.
C THEORETICAL ANALYSIS OF DEFENSIVE AGGREGATION
To formalize our analysis of defensive aggregation versus mean aggregation, we first establish mathematical model that captures the risk of fragile stability. We then derive the statistical properties of the risk estimators resulting from each aggregation method. Therefore, to formally analyze this specific failure case, construct the following assumption:
15

---

Published as a conference paper at ICLR 2026
C.I A MATHEMATICAL MODEL FOR WORST-CASE RISK UNDER FRAGILE STABILITY
Definition (Risk Target) As introduced in Section 3.3, we define the True Risk Ra of evicting cache entry its peak importance over all fulure generation steps L. Thus, the objective of a risk estimator is to estimate this quantity:
sus Le, 1EL
An ideal eviction policy based worst-case risk control should retain the top cache entries with the highest True Risk R, Our analysis focuses modeling the performance of different aggregation strategies on worst-case risk control. Therefore, we must model the failure cases in estimating worst-case risk that arise from the "Fragile Stability" phenomenon. This fragility can manifest in two primary ways: (1) an entry that "stably high" in importance suddenly drops, or (2) an entry that "stably low" in importance suddenly spikes.
The first case (stably high) is not the critical failure mode for our analysis. Both Mean Aggregation and Defensive Aggregation will correctly identify such entries high-risk on average and retain them. The critical failure is the second case: an entry appears stable but unimportant for long durations, yet experiences rare, sudden spikes of high importance. In this case, Mean Aggregation fails in this scenario because its expectation is anchored to the common low iiwportteacce state, causing it to severely underestimate the true risk Rg. Our model is constructed specifically analyze this case.
Definition 2 (Critical Failure Case Model)
We model the "fragile" importance distribution F, For entry as a Bernoulli mixture model defined by two component distributions:
Stable (Low-Importance) Distribution (F Activated with probability 1 C,. Observa- tions are 13, Flow, with E - and sup(FFlow)
Fragile (High-Importance) Distribution (Fhigh)): Activated with probability G, Observa- tions are 33 phiigh with F 14, high and sup( (Fhishh R4.
The "fragility" is mathematically defined by two properties:
1. Sparsity: The risk event is rare, 6, 0.5.
2. Extremity: The risk event is severe, R; 2 p4g high
> plow
C.2 TIIBOREMS AND PROOFS
Theorem (Estimation Bias of Mean Aggregation) Under Definition 1 and Definition 2, the Mean Aggregation estimator, ST Smeon m "m Ij. is high-bias estimator of the true risk R., This bias is systematic and worsens the risk event's sparsity increases (c, 0).
Proof 1 By the definition of the mixture model (Definition 2), the expected value of a single observa- tion Tj is:
GEE E[4,,] = (1- +GE (1 es) low + 6,p;
high
By the linearity of expectation, the expected value of the Mean Aggregation estimator Syrean is:
m
= cc)ppt + csp. m
E Smeea
E
E
E
high
m
m
The bias of this estimator with respect to the true risk R; (from Definition 1) is:
Bias( ( Smmeanr R,
E[Smmean] R.
(1)
hight
Given Definition 2, R, 2 with and e, small, meaning the value of E E[Sumean dominated by ul In the limit as e + 0 (the "most fragile" case). ElStmean - low The bias thus approaches:
lim Bias(SSmeen R; 1m
+,++
16

---

Published as a conference paper at ICLR 2026
This is large negative value, indicating a severe underestimation of the true risk. The mean aggregation estimator is mathematically anchored to the mean of the low-importance mode, p not the peak risk R.-
Theorem (Lower Bias of Worst-case Risk Estimation Defensive Aggregation) Under Defini- tion and Definition 2, the core component of Defensive Aggregation (Algorithm 1), the worst- case risk estimator R2- maxisism Li is significantly lower-bias estimator of R& than Mean Aggregation.
Proof 2 The estimator R₁ aims to capture the high-importance mode, Ehiihh We first analyze its probability of success. R, fails to capture the risk signal if and only if all m independent observations are drawn from the low-importance mode Plow.
P(Failure) = P(R, S ") = P(l ~ E! for all == ......m)) = (1 ei)
The probability of successfully capturing at least one figh-importance signal is:
P(Success) = P(R; >> R( 1 (4) -4)
We now compare the bias by examining the exact expectations. From Theorem 1:
ElSt = (1-+e)) low tepl
high
For the Defensive Aggregation estimator Rú we establish a lower bound. The expectation E [s] is:
E E|R P(Failure) E[mmax(I] Fail] + P(Success) ElImax(])[Sucee))
We know E [mmax([] Fail] E[[ 1! and substituting these bounds:
> F H,
high
high then
E R. 2 (1 -1-16
+ (1 (1- (1) high
To compare this to E[STiron] let us define a function g(p) (1 p)4 " + PPp, high By Definition 2.
high wow so g(p) is strictly increasing function of p. We can write the expectations in terms
of g(p):
S g((,)
2(1
Since g is strictly increasing, this is equivalent to showing 1 - (1 () > Considering m 2 2 and 6; (0, 1), this clearly holds, which strictly proves that F.(R:] > F) E(Smmenn)
Both estimators are negatively biased (iasedeElssstimnn (Flestimator) S R.). thus
0 S Bias(Baa (R,) = R2 - E < R2 - E[SSteenn] Bias BBias(Smeenn (Steea) Therefore, the worst-case estimator RG in defensive aggregation has a smaller bias.
Corollary (Robustness from Adaptive Prior-Risk Correction) The Adaptive Prior-Risk Cor- rection step in Algorithm 1, R₁ max It), provides robusiness against the primary failure mode of R4: under-observation.
According to the Proof of Theorem 2, we can derive that:
Bias(R. )- = R, ER R₁ - P(Failure) E[max([]) [ail] - P(Success) E |
Bias (..) = R4 (1 (1 th Fl] [/) (1 (1 - r) E F[[max [
This highlights weakness of R.: if the observation m is too small relative the sparsity e; (i.e.. me; < 1), the failure probability (1 1--c ez) remains high. This corresponds to the "limited observation" problem (Section 3.3). this failure case, all observations are from Flow and the bias of estimator becomes Bias R&)) RG -E-Emmaa (mmax())[Fail)] & R₁ - Rllow which a severe underestimation.
Our Adaptive Prior-Riss Correction step. R; = max RR, R), replaces this underestimated value RS R Rlo with the adaptive prior R (the head-level average risk). This is mathematically equivalent Bayesian posterior estimation: when specific observation for entry is insufficient, distrust the observation R and revert to prior knowledge R.
17

---

Published as a conference paper at ICLR 2026
Table 4: Detailed scores of 16 datasets on LongBench -Llama33.188--Inttree 10% Cache Size). The best result highlighted in bold.
Single Document
Document Q4
Summarization
shot Learning
Syntbetic
Mathod
NivQA
Qooper
MF
Horper
ZWiAQAA
Mosignt
GovRep
QMSuu
MattiNeers
TRRC
IiviagA
SAMSum
PCiami
Pi-en
Lee
RB-P Arg.
Lama-3 I. Instncr 10% Cache Size
Full Cache
20.55
44.68
55.82
57.59
48,89
32.61
34.40
25.51
26.83
73,00
92.36
43:27
7.38
99 63,44
52.36
49.20
SnppKV
23.25
30.77
23.54
43.11
22.29
18.57
24.13
19.51
20.43
43,00
92.01
43.18
7.35
$1.50
64,80 54.17 (.2744)
CAKE
24.25
20.79
24
64.49
23.58
20.30
34.70
20.75
20.38
44.50
92.36
43.44
5.15
76.00
65
$2.95
57.87
AdsKV
34.54
23.69
25.47
41886
22.59
16.74
34 52
30.17
90.51
46.00
92.25
43.96
7.21
50 66.25
$4.84
36,24
ChiiailV
27.62
25.02
26.77 76
43.49
23.50
20.67
25.00
31.39
21.10
48,50
92.17
42.84
7.30
63.50
66.61
$4.09
$6.27
DefewiveKV
22.23
50
50
46.29
27.15
18.23
25.47
22.39
70.58
54,00
91.96
44,13
5.00
98.50
67.46 41.5541.5
Laye-Delomiekk
27.30
28,53
38,09
49.12
36,00
22.44
15.76
22.42
21.41
$2,00
92.01
43.94
8.68
98.00
67.66
56001 (2.76)133)))
Table 5: Detailed scores of LongBench on MoE LLMs (Qwen-3-30B-A3m) 20% Cache Size). The best result among compressed methods is highlighted in bold.
Stagle Decument
Document Q11
Summiaisuton
Shor Leariigg
Syndhetic
Mahod
NiveIA
Guyper
APP
Hespot
Gaviipp
Moltikeen
TREC
TrirusE
SAMSuM
RB- Aug.
Qwes-J-31B-A3B.
20
23,62
63.33
30:90
22.04
12.27
$1.34
Dincally (hongeet Raselinz)
23.00
51.32
38.51
54.35
41.18
26.92
20,70
20.49
30.64
68.00
90.38
45007
15:00
99
50
75.11
20.59 46 (1863))
DetencweKY
24.19
36
42.75
59.14
48.1
$1.95
29.56
21.13
2054
09.50
90.18
45.58
15,00)
109,00
73.16
70.55 (15,01)
Layen Devesneen
22.88
38.99
46.00
55.50
S0002
$1.69
29.17
21.54
21.18
74,00
90.29
45.32
16,00
109.00
72.75
71,10 (L38))
D EVALUATION UNDER MORE EXTREME COMPRESSION
In our main experiments, we initially defined a 20% cache budget as an extreme compression scenario in real-world benchmark. Under this condition, our proposed DefensiveKV and Layer-DefensiveK demonstrated minimal performance degradation on the Llama-3.1-88 model, with average quality losses of just 4.8% and 2.6%, respectively. In contrast, the strongest baseline, CriticalK suffers an 11.1% performance drop, while other baselines degrade even more significantly.
To push the limits further, we evaluated our method with the cache budget reduced 10% on LongBench. As presented in Table the results underscore the robustness of our approach. Layer- D maintained high average score of 42.76, modest 13.1% drop from the full-cache performance. This result significantly outperforms all baselines, including the most competitive one, CriticalKV, which scored 38.27 (a 22.2% drop). This provides further evidence that our defensive aggregation strategy remains highly effective even under severely constrained cache conditions.
E GENERALIZATION ANALYSIS ON MIXTURE-OF-EXPERT (MoE) ARCHITECTURES
To demonstrate the broad applicability of our approach, we extend our evaluation to the Mixture- of-Experts (MoE) architecture, which has become increasingly prevalent in recent state-of-the-art LLMs. While MoE architectures introduce sparsity within the Feed-Forward Networks (FFN) to scale parameters efficiently, the attention mechanism-wheen where the KV cache resides-typicalle retains the same dense structure as standard Transformer models. Since the cache eviction methods operates within the attention blocks, it is agnostic to the modification of MoE in FFN structure. Therefore, our defensive aggregation strategy inherently generalize to MoE models without requiring any architecture-specifi modifications.
We conducted experiments on the Qwen-3-301D 3-0---3--0-BBBB---- model Team (0025), a powerful open-source MoE LLMs. We evaluated the performance on the LongBench under strict 20% KV cache budget. As shown in Table 5] the results confirm the effectiveness of our method in the MoE setting: our methods consistently outperform the strong baseline, CriticalKV. Specifically, Jayer-DefensivekY limits the average performance degradation to only 3.8% compared to the Full Cache upper bound, whereas Critical V suffers a significantly larger drop of 8.6 %.
18

---

Published as a conference paper at ICLR 2026
Defensiveky (Batch 1)
eitialky (Batch 11 citiaakv (Batch
Full cacke (Barch 1)
DefencivekY (Batch 2)
Full Cadhe (Banch 2)
Peak Memory (GB)
60
40
20
0
16k
32k
64k
128k
16k
32k
64k
128k
Figure 11: Peak Memory usage(All with FlashAttention-2).
F ADDITIONAL RELATED WORKS
Beyond cache eviction methods, a broader range related work can reduce KV cache footprint. For example, Think (Xu et al. 2024) compresses the KV cache by reducing the number of channels in the key states, Other approaches. such as MiniCache (Liu et al. 2024a) and KVSharer (Yang al. [2044) exploit KV similarity between layers to achieve compression. These techniques are orthogonal to KV cache eviction methods and can be further combined with them. KV cache quantization (Hooper et al.. 2024; Liu et al. (024d), which reduces the precision of individual cache entries (e.g.. quantizing 16-bit entries to 4-bit or 2-bit), also offers footprint reduction. Because quantization methods typically retain all cache entries, they fundamentally orthogonal to the cache eviction methods explored in this paper and can also be applied to further enhance them. Furthermore, recent speculative decoding methods explore using reduced KV cache for draft generation in long-sequence generation (Sun et al., 2024; Yang et al. (025). Refining cache eviction to enhance speculative decoding is also a promising research direction.
Sparse attention methods are conceptually related to KV cache eviction (Xiao et al., 2024a Tang al. 2024, Jiang al., 2024 Liu et al. 2024b). The key difference that KV cache eviction retains only a subset of the KV cache, while sparse attention methods keep all entries but selectively utilize only critical subset during computation (Nawrot et al. (025).. Consequently, sparse attention methods do not reduce the memory footprint of the KV cache. The two technique lines are, in fact, orthogonal. Future research could explore firstly employing KV cache eviction to compress the cache to certain proportion (e.g., 40% cache size with minimal loss, as demonstrated in this paper) and then applying sparse attention for further acceleration. This represents a promising direction for future research.
G MEMORY USAGE DURING GENERATION
Following the efficiency evaluation in Section inference. The memory savings from cache eviction are primarily determined by the compressed cache size. Our introduced defensive aggregation method does not differ in memory usage from standard mean aggregation. As shown in Figure II] DefensiveKV and CriticalKV exhibit significantly lower peak memory usage than Full Cache. For example, with batch size of and 128K context length, DefensiveKV and C use only 36GB, far less than Full Cache's 's 61.5 GB. This allows them to support larger batch sizes, such as batch size further increasing decoding throughput, while Full Cache encounters oft-of-memory (OOM) errors. This advantage enables DefensiveK and CriticalKV to achieve up to 4.2x speedup in 128K decoding throughput compared to Full Cache.
also measured peak memory usage during
H INTEGRATION OF DEFENSIVEKV WITH ORTHOGONAL EFFICIENT INFERENCE TECHNIQUES
Both existing KV cache eviction methods and our enhanced method, DefensiveKV. can be seamlessly integrated with orthogonal efficient inference techniques to achieve even greater memory savings and computational efficiency. Such techniques include KV cache quantization (applying eviction then
19

---

Published as a conference paper at ICLR 2026
Table 6: Detailed scores of 13 datasets on Ruler.
nioh mkl
niah mik
niah mlk
niah. my
nish m
alah s
niah $2
nish kt
Method
ewe
Fwe
qa
qa.2
Avg.
Lluma- 3.1 38--nsttuctt 32K Ruler, 20%
Cache Size
FallCache
45.22 94,13 99:60
99.60
99.40
98.75
99.
100.00
100.00
100.00
79.00 80
54
80
24
89.97
Strongost Bascline CriticalKV
26.80 80 91.60
29.40
19.40
95.00
93.60
100.00
99.60
42.40
40.80
40.20
97.
76 66.57
DefensiveKV
22.94 90.00 99.80
86.80
97.00
98.65
97. 97.90
100.00
100.00
97.40
68.80
45.
76
84.91
Mayer-DefensiveK"
17.86 90.80 99.60
99.40
99.00
98.85
98.45
100.00
100.00
100.00
73.00
47.60
98.
86.39
Llama-3.1-8B-Insun
32K Ruler, 10% Cache Size
TuliCache
45.22 94,13 99.60
99.60
99.40
98.75
99.10
100.00
100.00
100.00
79.80
54,80
99.24
89.97
Strongest Baseline CriticalKV
49.08 ,93 99.60
94.00
54.00
98.75
98.
100.00
100.00
97.40
00
47.60
99.32
84.51
DefensiveKV
$1.12 92.87 99.60
99.80
98. 60
98.65
98.90
100.00
100.00
100.00
78.20
51.80
99.24
89 89.91
Layer-Defensee DefensiveKV
50.24 92.00
98.70
99.05
100.00
100.00
99.12
quantization to leverage both cache sparsity and precision reduction), KV cache retrieval (compressing the cache before retrieval to reduce overhead), and KV cache offloading (evicting before offloading to reduce storage and transfer costs in slower memory systems).
As case study, demonstrate DefensiveKV integration with KV cache quantization. We adopted the official HuggingFace- (024b). We first use DefensiveKV to retain 40% of cache entries, then apply 4-bit quantization, yielding 10% of the original memory usage. Moreover, we also present results for DefensiveKV retaining only 10% of cache entries without quantization.
int4 quantization (Face) (2024)) with Quanto backend support (Face)
As shown in Table 8 combining D with int4 quantization to reduce the cache to 10% of its original memory footprint achieves an average score of 48.55, incurring less than one-point drop compared to the full-cache baseline (49.20). This is not achievable with quantization alone, which would require approximately 1.6- quantization. also substantially outperforms using cache eviction alone to reduce the cache size to 10%. Thus, combining D and quantization substantially outperforms either technique individually, highlighting strong potential in real-world deployment.
I MORE HAATTA- STYLE EVALUATIONS ON RULER BENCHMARK
In the Needle-in- A-a--staatckk task, keyword, referred the "needle", is embedded within lengthy context known as the "haystack". The objective of this task is to extract the "needle" from the "haystack", which is composed of essays by Paul Graham (Kamradi) 2023). In main experiments, we adopt the respective prompt templates (see Table I3) used in the Ruler Benchmark (Hsich et al. 2024) (corresponding to NIAH-s2 and NIAH-MV in their formulation) to ensure consistency and reproducibility
The whole Ruler benchmark (Hsieh al. 2024) comprises 13 synthetic, aondll-inn--haystack-----
tasks designed to evaluate the long-context capabilities of models. A single evaluation on the full 32K RULER benchmark requires approximately 9 GPU hours. Consequently, a comprehensive assessment across all methods, compression rates, and models would demand an estimated 864 GPU hours, which is computationally prohibitive.
In this section, we further presents a more extensive analysis on the complete Ruler benchmark. We evaluated our proposed methods and the strongest baseline, CriticalKV, VV, at 20% and 40% cache sizes using Llama-3. 1--1--B--nnstue DefensiveKV V demonstrated significant advantages; for instance, at a 20% cache size, they achieved average scores of 84.91 and 86.39, respectively, substantially outperforming the CriticalKV baseline's score of 66.57. These findings underscore our method's ability to achieve strong compression performance with minimal loss in accuracy.
with the results detailed in Table 6 Both DefensiveKV and Layer-
20

---

Published as a conference paper at ICLR 2026
Table 7: Performance comparison with and without defensive aggregation on LongBench.
Single Document QA
Mulii- Document QA
Summarication
Few-shot Learning
Synthetic
Code
Method
NavQA
Qusper
Hespor MF-en
2WiksGA
Masigus
GovRop
QMSum
MultiNews
TREC
TriviaQA
SAMSum
PCoun
PR-sn
Lau
RB-P Aug.
Liama-3 3.1 1est-nst 20% Cache Size
Full Cache 29.55
44.68
55.82
57.59
48.89
32.61
34.40
25.51
26.83
73.00
92.36
43.27
7.38
99.50
63.44
52.36
49.20
AdaKV
Defensive 28.60
37.62
41.08
51.74
36.87
22.83
27.83
23.18
23.51
66.00
91.64
44.35
8.10
92.50
67.97
55.71 44.97
CmicaiKV 29.81
12.58
14
14.95
52.34
36.24
26,
28. 35
23.52
23.24
56.50
90.80
43,37
8.89
93.00
67.05
99 43 88
Defensive 29.97
40.46
52.20
38.40
28.06
29.96
23.89
24.11
68.00
91.58
43.17
8.28
100.00
67.17
55.40 46.68
Table 8: Performance of DefensiveKV combined with int4 cache quantization on LongBench
Single Decament Q4
Mult Docunerr QA
Summar
Few obor Learning
Syathetic
Code
Medhod
NovOA
Quoper
MP
Hodpes
JWOKIA
Mestqpe
Gailee
MoltiNeon
TiivleRA
SAMSiom
Ubma-31-AB--reiee Instnen 20% Carhe Size
Cache (102% momory)
46.09
92.36
momeey)
DefessicKV..i 10' Cadec (10% memses)
27.23
25.32
35.30
46.29
27.15
18.23
25.42
22.39
20.88
$4.00
91.96
44.13
5.00
96.50
67.46
54.15
41.52
DelensneV-ipP KV-0S Cxcheinnt (105 menory) MIAS
6462
54.44
56.14
42.
26.15
33.79
15.15
25.92
70.5
91.38
43.76
7,53
100
05
56.23 IKSS
J CASE STUDY: AUGMENTING EVICTION METHODS VIA DEFENSIVE AGGREGATION
Our proposed defensive aggregation is orthogonal strategy that is compatible with other cache eviction methods and can be seamlessly integrated in a plug-and and--nd--p -ud-play manner. In the main paper, we demonstrate its application to CriticalKV. Furthermore, we extend the defensive aggregation to AdaKV. As shown in the table 77 defensive aggregation effectively improved both AdaK and CriticalKV's V's performance, increasing the average score from 41.39 44.97 and from 43.88 to 46.68. 68, respectively, These results indicate that defensive aggregation can broadly enhance existing cache eviction methods.
K DETAILED SCORES OF LONGBENCH
We provide detailed and. [T] Our D and Layer-DefensiveK1 methods maintain nearly lossless generation quality across these settings, while other baselines fail achieve this level of performance.
individual datasets for 40%, 60% and 80% cache sizes in Tables 9 10
L THE EFFECTIVENESS OF DEFENSIVE AGGREGATION STRATEGY
To complement Figure I in the main text, Figure 12 provides additional visualizations demonstrating that defensive aggregation offers greater robustness than mean aggregation under 50% cache size. The results reveal that this fragility is prevalent across numerous layers. Both "Single Historical token" and "Mean aggregation" methods exhibit high sensitivity this fragility, leading poor worst- case performance. In contrast, defensive aggregation effectively mitigates this issue, consistently maintaining higher worst-case values.
M FURTHER ELABORATION OF THE FRAGILE STABILITY ASSUMPTION
Complementing Figure in the main text, Figure 13 provides more detailed illustration. It demonstrates how measurements from single historical tokens, which guide cache eviction, experience significant degradation at certain generation steps. The outlier cases occurs regardless of which specific historical token is used. Consequently, the failure such averaging approaches is an expected outcome.
21

---

Published as a conference paper at ICLR 2026
Table 9: Detailed scores of 16 datasets on LongBench (40% cache size).
Single-Dacumett
Molti-Dacameni
Summarinstion
Feu-shot Learning
Syntheric
Cade
Method
NArQA
Qusper
Mf
Hatpet
zwisiga
Masique
GovRap
QMSum
MattNNes
TREC
TriviaQA
SAMSum
PCwmal
PR-cn
Lec
RB-P
Aug
IJama-3.1 1.1-3..----sttnni 40% Cache
Pull Cache
29.55
44.68
55.82
57.59
48.89
32.61
34,40
25.51
25.83
73.00
92.36
43.27
7.38
99.50
63:44
52.36
49.20
DaxAnemion
28.83
42.51
53.35
55
45.37
30.
32.26
25.07
25.51
71.50
88 88.44
41.12
3.67
99.5
68.81
53.
85
48.
LL
24.32
28.77
28.74
43.75
30.71
18.55
30.26
21.86
24.83
65.50
92.24
41.65
2.92
46.00
66.
54
61.42
39,26
StupKV
27.72
36.42
38.31
54.92
40.02
26.95
30,60
23.33
24.31
56.00
92:31
43.92
7.62
96.50
65.95
53.77
44.92
CAKE
30.45
57.13
30.35
23.80
24.96
91.83
43.46
6.20
100.00
46.06
AddKV
38.36
54.80
24
63.50
91.75
43.57
64.55
04.01
54 75
Cittcalky
49.03
46.23
30,42
31,49
25.15
67.50
02.39
43.20
8.08
99.00
47.67
DefemsieKV
30.07
46.37
54.90
57.50
15.97
28.85
33.70
24.69
26.20
71,50
91.78
43.69
9.88
100. 00
66
255
55.97 19.21
L
30.94
53.34
55.01
50.36
49.14
29.55
34,09
25.71
26.64
72.00
91.49
42.96
9.56
99.50
07
30
57.98
49.46
Mistrol- Cosstal--B-Instnnnee Io.t--nt--u--.,...0
40% Cuche Size
Pull Cache
27.02
38.19
50.22
50.75
37.41
27:92
34.45
25.76
26.37
76.00
89.01
46,89
6.50
97.00
66.04
60.47
47,50
DusAmenion
20.37
26.96
19 19.68
48.99
34,95
20.16
29.14
21.74
24.86
73.50
87.39
44.06
3.00
93.00
63.95
58,
16 43.74
S.
24.05
83.25
45.
34
4..40
33.76
StapKY
25.32
39.68
22.38
22.39
24.22
89.37
47.17
44.05
CAKE
25.02
48.42
31.94
21.73
31,70
23.15
24.77
89
4.00
92.50
64
44.35
AdaK
24.76
$1.36
41.79
49.59
32.95
20.20
30.51
22.94
24.42
68.00
88.96
47.29
5.50
96.50
65.54
59.99
44.42
Crincally
26.97
34.17
47.50
45.00
38,07
24.64
31.51
24.70
25. 14
72.50
89.37
47.86
4.53
05. 35.50
65 50
50
68
46.00
DefensiveKV
25.
39:24
51.42
50.13
34,59
26.36
34.43
25.42
25.17
75.50
89.21
16.59
5.05
98,00
56.54
61.65
47.25
D
26.29
30.49
50.92
45.85
36.34
26.02
34.33
25.28
2666
76.00
89.36
46.71
3.86
27.00
66.65
62.12
47,30
Qwes2 5-328------00
Iestinesssssss 40%% Cache
Cache
52.87
38.78
32.59
24.35
24.95
88.25
47.05
12.50
DaxAnteiion
N/A
N1A
N/A
NIA
N/A
N/A
N1A
NJA
NIA
N/A
N/A
LLreamingLLM
23.95
25.49
28.08
69.59
$1.66
24.15
30.32
20.50
24.12
68.00
87.08
45.90
11.52
46.00
57.38
28.36
38.50
SuapKY
27.18
33.99
39.19
61.11
53.18
58.01
29.95
20.84
22.95
66:00
89.16
47.02
12.00
97.25
52. 19 34.75
45.27
CAKE
28.13
35.18
39:22
60.17
56.22
36.25
30.33
21.03
23.88
66.00
88:
46.68
10.50
98:75
52. 20
45.53
AdaKV
27.16
$1.83
37.07
60.81
53.92
57.66
20.84
70.79
23.07
68.50
88.91
47.40
12,00
97.00
53.05
34
56
45.22
ChiliaakV
31.65
35.23
42.70
59
59.26
40.00 00.10
31,25
22.67
24.02
71.00
88.3 80
16.91
11:00
99. 19.75
16.76
35,
16.94
DefensiveKV
30,977
43.38
42.20
32,82
23.50
24.67
88
47.50
10.00
24 49.13
Layer Defensivvn
31.61
44,02
64.25
61.30
40.88
31.94
24.26
24.62
88.75
46.76
12.00
34.64
48.89
Table 10: Detailed scores of 16 datasets on LongBench (60% cache size).
Single Document QA
Multi Ducumeni QA
Summarization
Few shot Learning
Synthetic
Code
Method
NovQA
Qusper
MF an
Happst
awasg4
Maniquaa
GorRop
QMSuu
MoltNNws
TREC
TrivisuA
SAMSum
PCoun
PRien
Lec
RB-P Aug.
Llama-3.1 3.B-BB-BBBBB--- Instrust 80% Cache Size
Cache
29.55
44.68
57.59
32.61
34,40
25.51
26,83
73.00
92.36
43.27
99.50
DusAmemiio
28.61
26.80
91.16
43.37
10.50
StreamingLLMM
25.21
49.72
22.62
31.51
23.12
25.91
02:27
42.75
57.50
66.27
61.75
42.79
SnupKV
28.92
40.35
48.00
56.79
48.60
30.12
32.54
24.20
25.58
50
91.64
44.53
8.85
99.00
47.65
CAKE
29.99
41.87
53.09
55.
42.83
32.17
32.12
24.87
25.61
66.50
92.50
43.37
7.96
99.50
64.11
52.65
47.7%
AdaKV
30.10
43.61
51.20
56.17
19.70
30.18
32.37
24.38
25.54
66.50
91.45
43.87
8.02
99.50
63.91
54.73
18.22
CincaalV
30.31
23.54
52.82
57.30
39.00
$1.77
35.48
25.18
26.00
72.50
91.80
43.95
7.47
99.50
64.05
54.92
48.98
DefeasiveKV
30,85
43.20
55.25
50.17
31.5
36,79
25.29
26.84
73,00
92.14
43.28
99.50
83
56.02
89.45
Leye--Defensiekky
81
49.52
Mistral
Instruct v0.3, 80%% Cache
Cache
27.02
38.19
50.75
37.41
27.93
34.45
25.76
26.37
76.00
89 .1
46.89
97.00
66
60
47.50
DuoAnemion
28.86
36.36
50.54
53.32
39.19
29.23
33.91
25.16
25.77
26.00
87.57
45.40
5.00
95.00
64.75
58.91
47.26
Streamingf.I
21.48
30.55
35.40
46.84
36.62
22.80
32.20
32.70
25.06
24.50
82.
41.87
3.00
57.00
34.43.5
47 38
16.51
ShapkY
25.44
33.77
45.63
52.52
34.30
25
25.04
32.61
24.59
25.37
68.00
89.41
47.12
5.00
05. 35.500
66.
22
59. 47 45.68
CAKE
27.17
36.69
48.77
50.91
38.21
23.
33.53
23.73
26.06
75.00
88
46.95
5.56
95.00
66
23
60.
76 46.64
A
48.61
25.90
32.38
24.24
25.54
70.50
88.91
47.00
96.00
60
20 45.84
ChiccalkV
50.28
50.41
25.90
47.54
97.00
47.06
DefensieKVV
27.82
51.50
38.39
27.57
34.29
25.41
26.78
76.00
46.89
98.00
66
Leyer-DefemsioeK
25.59
38.95
51.99
50.81
37.09
25.35
34,50
25.68
26.73
77.50
89.04
46.95
4.00
97.0
67.
61.50
Qwen2.5 Lnstruct, 50%% Cache Size
Full Cache
30,88
46.13
52.87
63.59
59.75
38.78
32.59
24.35
24.95
72.00
68.26
47.05
12.50
100.00
49
64
34.24
48.60
DanAnemion
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
NIA
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
LL
26.05
34.49
33.06
57.33
46.75
30.57
30.39
21.93
24.83
72.50
87.01
46.97
10.52
57.50
58.09
29. 86 41.75
StupKV
38.55
67.18
24:37
88
47.32
12.00
CAKE
51.15
31.64
22.70
24.80
89.11
47.06
10.50
100.00
35.24
47.48
A
29.58
45.63
61.
57.33
37.91
31,45
22.40
24.13
71.00
88
46.64
11.50
100.00
33.95 47.02
Chncaaky
33.01
41.97
43.07
02 12:55
62.58
39.79
32.45
25.69
24.59
72.00
88.85
46.95
12:00
99 88
52.03
34.02 48.39
DofemsieeV
12.10
46.38
51.50
64.50
63.07
13
32.77
24.41
24.96
73.50
88
47.07
11:00
09 88
51.
33.
49.13
Layer DefensveKKV
31.28
26.18
51.97
63
63.30
39,50
32.56
21.37
21.96
23.00
88.65
16.75
13.50
100.00
52.74
34.06
49.14
N DETAILS OF 16 DATASETS IN LONGBENCH
As a widely used long- contectt benchmark (Feng et al. 2024 Li et al. 2024 Zhang et al. 2024a).
LongBench consists of 16 datasets across six task domains: single- document question answering
(QA) (Kočiský et al. 2018: Dasigi et al.) (02T). multi-document QA (Yang et al. 2018; Ho et al.
2020
Trivedi al. 2022). summarization (Huang et al. 2021 Zhong et al.
2021
Fabbri al.
22

---

Published as a conference paper at ICLR 2026
Table 11: Detailed scores of 16 datasets on LongBench (80% cache size).
Single Dacamen QA
Mulli-Dnecmmnt Q4
Summarizztion
Feu-ahor Learning
Syntheric
NavDa
Qusper
MF on
Happor
TWAGGA
Mesiqua
GerRap
QMSum
MutoNews
TREE
TriiiQA
SAMSum
PC
PRocn
Lec
RELPP
Aag.
Llana 3.1-81 Instruct, 80% Cache Size
Full Cache
29.55
44.68
55.82
57.59
48,80
32.61
34.40
25.51
26.83
73.00
92.36
43.27
7.38
99.
63
44
52.
36
49.20
DuoAnemion
30,04
44.49
55.77
57.51
18.95
31.78
34.51
25.29
26.53
73.00
91.35
43.28
3.08
100.00
6.3
19
50.85
49.38
LLreamingLLM
28.57
43.96
52.57
46.05
26.50
32.88
24.23
26.58
70.50
90.52
13.33
4,03
83.50
65.34
60.57
15.94
SuapKV
44.58
57.31
50.85
24.39
91.47
43.21
09.50
48.60
CARE
02.75
42.82
AdıKV
34.13
23.00
10
8.65
09.
49.31
Chitialyy
29.73
$4.06
35.66
58.19
18.52
32.21
36,70
25.27
25.56
73.50
92:30
43.75
8,09
99.
63.
54.04
49.41
DefensiveK
29.63
44.49
56.70
57.41
39.49
51 18
34,97
25.23
27.25
75,00
92.05
43.06
8.077
09.
63.3
54.53 49.40
Layer- Defensiekyy 29.63
44.28
36.52
58.18
48.10
32.85
34,76
24.98
27.20
72.50
91.78
42.98
8,62
99.50
03.27
52. 49.26
Mistrrll 7BB Ins.muct--O0., 80% Cuche Size
Cache
27,02
37,41
34.45
26.37
97.000
666
47.50
DusAmemion
StreamingLLM
37.79
42.98
StupKV
49:00
28. 16.56
34.08
88.91
46.85
6.00
96.
46.82
CAKE
26.62
38.22
50.20
50.20
36.40
24.78
34.28
25.63
26.02
76.00
88.91
46.41
4.55
05.00
66.70
46.89
AdaKV
26.77
34.52
18.73
50.25
36.59
28.57
33:45
26,90
26.20
76.50
88.91
17.26
6.50
96.50
66
61.37
17.07
Cincealyy
27.34
56.72
49.04
51.66
36.94
27.13
33.85
25.52
25. 83.88
76.50
88.91
47.28
6.015
97.
65.
59.
47.19
DefemniveVV
27.70
38.29
14.14
50.86
37.84
27.63
34.24
25.87
26.25
75.50
89.21
47.29
6.00
07.00
66
14 60
47.58
Loye--Dfenssieky
89.04
Qwen2.5
2:222225:320 Instruct,
Cache
63.59
59.75
32.59
88,
47.05
12.50
DusAnemion
N/A
N/A
N/A
N/A
N/A
N/A
N6A
N6A
N/A
N/A
N/A
N/A
NIA
2/A
N/A
N/A N/A
I
27.44
41.24
35.78
50:73
51.76
33.23
31:32
22.84
25.40
74.50
86.83
48.18
11:00
82.00
55.99 32, 45.11
StupKV
30.97
42.95
19.08
63.92
59.10
39.73
32,00
23.33
24.75
71.00
88.45
47.48
12.00
100.00
51
33,
18.09
CAKE
30.32
45.68
53:31
63.96
61.16
39.91
32:17
23.67
25.06
70.50
88.77
47.39
11.00
100.00
51 02 34,84
48.60
AdaKV
30.42
+2.62
62.98
60.45
39.19
31.77
23.22
24.82
88.69
46.87
10.50
100.00
50.
33, 47.85
ChiiialVV
46.92
11.50
DefensiveKV
32,70
47.14
11:00
48.85
Lyye-DefensiveKV
31.13
46.01
52.46
63
60.22
8.
32,85
24.18
25.03
73.00
88
46.56
12.50
100.00
52.5 33.53 48.75
2019), few-shot learning (Joshi et al. 2017| Gliwa et al. 2019
Li & Roth (002). synthetic tasks (Bai 2023), and code generation (Guo al. [203]] Liu al. 2023). The average token length across
all 16 datasets is 6,711. Table 12 provides detailed information on the 16 datasets in LongBench.
Table 12: Details of 16 datasets in LongBench.
Task
Task Type
Eval metric
Avg len
Language
Sample Num
NarrativeQA
Single-Doc. QA
18,409
EN
Qasper
Single-Doc. QA
F1
3,619
EN
200
MuldFieldQA
Single-Doc. QA
F1
4,559
EN
150
HotpotQA
Multi-Doc. QA
Fl
9,151
EN
200
2 WiikiMulthhoQQA
Multi-Doc. QA
4,887
EN
200
MuSiQue
Multi-Doc. QA
11,214
EN
GovReport
Summarization
Rouge-L
8,734
EN
200
QMSum
Summarization
Rouge-L
10,614
EN
200
MultiNews
Summarization
Rouge-L
2,113
EN
200
TREC
Few-shot Learning
Accuracy
5,177
EN
200
TriviaQA
Few-shot Learning
Fl
8,209
EN
200
SAMSum
Few-shot Learning
Rouge-L
6,258
EN
200
PassageCount
Synthetic
Accuracy
11,141
EN
200
PassageRetrieval-en
Synthetic
Accuracy
9,289
EN
200
LCC
Code
Edit Sim
1,235
Python/C#/lava
RepoBenchhPP
Code
Edit Sim
4,206
Python/Java
500
O LIMITATIONS
In this paper, we reveal for the first time the fragility of KV cache eviction and propose a defensive aggregation strategy for robust optimization. However, our work serves starting point and does not provide an in-depth investigation of broader robust optimization techniques. Future research can explore these techniques to further improve cache eviction performance.
23

---

Published as a conference paper at ICLR 2026
Detera www.agaeettonnnnoo inrricccseeeee
Comerater layer
Singe Ltmratia Thian Zine vertiicus
Histrrica 130md
Appagation
100
(b) layer 14
Zega Hinorcc Toian (10es masso-oeanaaaa Moon 2013
layer
85-
Singi Hirrarical Deam 2nd verziiccas 621 restaiial 200m wersiccoen tD Diderri Didersiiedcppeeeeet meriicaneeeee
Caneradin Stp (d) layer 16
Sigge sustania Ioaa Lano, waitt casa valus 651
vestanial cossieseeeeeee 3
Mear Approeaiin menticcae veliet 72x
1in
layer 17
veliicel hear (144's canmatto kei
Siga vers-eass
saneraiion layer
Figure 12: Visualization across different layers using Llama-3.1 1-BBB with 50% cache size.
24

---

Published as a conference paper at ICLR 2026
Single historical token 2nd: mean valuem0 90, worst-case valuem0.31, outlier countivallu==0.3)113
Layer
16
31
100
200
300
400
500
Generation Step
Singlle historical token 6th: mean value=0.92, worst case value=0.2 34, outlier countvalue
Layer
16-
500
Stop
Single historical token 10th: mean value=0.
worst-case value=0 33, outlier (ount(value=005)1100
Layer
31
Generation
Single
43, or countivalue+D:5)-15
Layor
31
100
200
300
400
500
Generation
Single historical 18th:
valuem0. outiier countivalue=0,
Layer
31
100
200
300
400
500
Generation Step
Single historical token 22nd: mean value=0.2 91, werst-case value=0.33 outlerr countivaluecD..
Layer
16.
100
200
300
400
500
Generation on Stop
Single historical token 26th mean value=0..9 92 worst-case value=0 32, outlier countiyalue=00)=1100
Layer
16
100
200
300
400
500
Generation Step
Single historical taken 30th: mean value=0.90, worst case value=0.22 0.32, autlier countiyaluec0.5--117
Layer
16
100
200
300
400
500
Ganeration Stop
Figure 13: Breakdown of the stability assumption across different historical token measurements
25

---

Published as a conference paper at ICLR 2026
Table 13: Single retrieval and multi retrieval templates in Needle-in-A-Haystadd tests.
Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. will quiz you about the numhers afterwards.
Graham Essays.
Single retrieval
One special magic numbers for [word] is: [number), o.-.
....
What special magic number for [word] mentioned the provided text?
The special magic number for [word) mentioned in the provided text
Task Template:
Some special magic numhers are hidden within the following text. Make sure to memorize it. will quiz you about the numbers afterwards.
Paul Graham Essays.
One of the special magic numbers for [word) is:
[number-] .....
Multi retrieval
One of the special magic numbers for [word] is: [number-2] ]. .....
One of the special magic numbers for [word] is: [number-3] 3).
One of the special magic numbers for [word] is: [number-4),
What are all the special magic numbers for [word] mentioned in the provided text?
The special magic numbers for [word] mentioned in the provided text are
26


Reviews:
[{"review_id": "FQzCGzWBG9", "reviewer": "Reviewer_UXkx", "summary": "This paper identifies a key issue in KV-cache eviction: prior methods assume that importance scores derived from historical tokens remain stable. However, as generation progresses, these scores can shift significantly, making that assumption unreliable. To address this, the authors propose a worst-case aggregation strategy that accounts for score fluctuation and improves eviction effectiveness.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper identifies a new problem in KV cache eviction methods: the aggregation step ignores the worst-case risk.\n2. Comprehensive experiments evaluated the effectiveness of the proposed method.\n3. Simple but effective method for optimizing KV eviction methods further.\n4. Good writing, easy to follow, and well motivated.", "weaknesses": "1. Experiments:\n\n+ The proposed method is orthogonal to many existing KV-cache eviction strategies. To ensure fair comparison and demonstrate the generalizability of your approach, it would be beneficial to include an additional table showing how your method can be combined with representative prior techniques. For example, selecting one model and one benchmark to report results for “baseline methods” versus “baseline + your method” would more clearly highlight the benefits.\n\n      | Methods       | Task 1 | Task2 |\n      | ------------- | ------ | ----- |\n      | e.g. SnapKV   | ...    | ...   |\n      | + DefensiveKV | ...    | ...   |\n      | e.g. CAKE     | ...    | ...   |\n      | + DefensiveKV | ...    | ...   |\n\n2. Ablation Study:\n\n+ In the default setting, the historical window size is set to 32. According to the paper’s analysis, this hyperparameter plays a crucial role in evaluation, as it affects both scoring and aggregation behaviors. Therefore, it would be valuable to include an ablation study on the historical window size to better understand its impact.\n\n3. Typo:\n+ Figure 3 (a) 16st --> 16th", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AMVQYHYBta", "reviewer": "Reviewer_CK4Z", "summary": "This paper revisits the standard scoring–aggregation pipeline for KV cache eviction in LLM inference and argues that the usual mean aggregation is fragile because the importance of cache entries can vary abruptly during generation. Based on this, this paper introduces Defensive Aggregation, a linear-time, two-step rule that first estimates a worst-case risk per entry via a maximum over historical-token scores and then applies an adaptive prior-risk correction, yielding two methods, DefensiveKV and Layer-DefensiveKV with layer-wise budget allocation. Across LongBench and Needle-in-a-Haystack, the introduced methods achieve better performance versus baselines at various cache budgets.", "questions": "Will the code be open-sourced? I think it would be valuable for the community.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The introduced method is well-motivated by convincing observations about the fragility of importance stability.\n\n* Thorough experiments and analyses are presented in the paper.", "weaknesses": "* Could you elaborate on the rationale behind fixing the historical window size at 32 tokens? Are there any experiments to evaluate the sensitivity to this choice?\n\n* The cache budget settings could be more varied. 20% cache is not a strictly small budget. Would it be possible to provide experimental results under small budgets such as 1% and 2% to demonstrate how the method behaves under extreme compression conditions?\n\n* Missing discussion on non-scoring-aggregation pipelines in the paper such as “Lacache: ladder-shaped kv caching for efficient long-context modeling of large language models (ICML2025),\" which would help situate DefensiveKV more clearly within the broader space of KV cache compression methods.\n \n* Why on needle-in-a-haystack tasks, DefensiveKV outperforms Layer-DefensiveKV sometimes? Could you please provide any insight into under what circumstances DefensiveKV/Layer-DefensiveKV will perform better? A short discussion on this might help readers better understand their trade-offs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ltfm9oadUt", "reviewer": "Reviewer_WZVm", "summary": "This paper first argues that most of the existing methods (such as snapkv, criticalkv) are committed to improving the scoring step (i.e., looking for better importance indicators). They use simple average aggregation by default because they rely on a stability assumption: a small group of key cache entries will remain important throughout the generation process. However, the importance of cache entries may suddenly decline sharply at some time, resulting in extreme cases or outliers. The paper shows that the worst-case risk should be addressed. To do this,  this paper proposes a new defensive aggregation strategy to control the worst-case risk, which combines worst-case risk estimation and adaptive prior-risk correction. A DefensiveKV, which is a basic version, and a Layer-DefensiveKV are proposed, which integrate a layer-wise budget allocation strategy, respectively.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This new aggregation strategy has the same linear time complexity as the average aggregation strategy, and the additional computational overhead is negligible.\n\n2. Significant performance improvement\n\n3. Good writing and illustrations.", "weaknesses": "1. Is it possible that Adaptive Prior-Risk Correction will lead to: In a head with high risk, it may wrongly raise the score of really unimportant items; In a head with generally low risk, it cannot discover those rare key items that do not show high risk in the limited observation window.\n\n\n2. It seems that in the code dataset, most SOTA methods provide elegant results. Could you provide some discussion about this?\n\n3. In cirticalKV, the score is evaluated by using the attention weights and norm of V*W^O, which shows an excellent performance. This paper also adopts the same strategy. I also want to see some discussion about this: How many output differences does this paper's method reduce?\n\n4. Algorithm 1 is conducted in layer-wise or head-wise?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RSLwo8scLg", "reviewer": "Reviewer_dk2k", "summary": "The paper “Taming the Fragility of KV Cache Eviction in LLM Inference” argues that existing KV cache eviction methods rely on a fragile stability assumption—that a fixed subset of cache entries stays important throughout generation. This makes the common mean aggregation approach unreliable under instability. The authors propose Defensive Aggregation, a linear-time method that manages worst-case risk instead of average importance. Building on this, they introduce DefensiveKV and Layer-DefensiveKV, which achieve over 2×–4× lower quality loss than state-of-the-art baselines (e.g., CriticalKV) at small cache budgets, with negligible computational overhead. The work pioneers a “worst-case risk-aware” perspective for robust and efficient LLM cache eviction.", "questions": "Can we have some more formal guarantee or experiments to show the worst case scenario is prevented?\nCan we give some discussion on how this online process can be potentially integrated into inference?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. strong experiments comparing with many previous baselines. improvement is significant.\n2. New idea of adding online adapatation.\n3. Clearly identifies and formalizes a previously overlooked fragility in the “stability assumption” behind KV cache eviction and proposes a simple, elegant, and computationally cheap fix (Defensive Aggregation).", "weaknesses": "1. The “worst-case” heuristic lacks formal guarantees or ablation beyond empirical trends. Need to show some results on why this worst case is guaranteed besides it works empirically.\n2. No discussion on interaction with quantization, retrieval, or offloading beyond brief appendix notes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4hQhs0RtHX", "reviewer": "Reviewer_Gsr3", "summary": "This work identifies a critical fragility in the common stability assumption underlying KV cache eviction methods for LLMs. It introduces a defensive aggregation strategy that manages worst-case risk by estimating peak importance and applying adaptive correction, moving beyond vulnerable mean aggregation. The resulting methods, DefensiveKV and Layer-DefensiveKV, achieve substantially lower generation quality loss under strong cache size constraints. Comprehensive evaluations across 18 datasets show these methods outperform prior art, reducing quality loss compared with baseline.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* S1) The concept of Defensive Aggregation presents a novel approach to KV cache eviction, and its demonstrated effectiveness is supported by empirical evidence across benchmarks including LongBench and NIAH. \n* S2) The paper is written in a clear way, making it easy to follow the authors' ideas and understand their results.", "weaknesses": "* W1) The primary limitation of this work is its exclusive validation on dense and GQA-based model architectures. The proposed cache eviction method remains unverified for the increasingly prevalent Mixture-of-Experts (MoE) models. Since MoE architectures inherently reduce KV cache usage through selective parameter activation, the fundamental dynamics of token importance may differ. Consequently, the applicability and effectiveness of the defensive aggregation strategy in this critical and distinct setting are currently unknown. \n\n* W2) The authors mention in their setup that \"the context is compressed independently before the question is introduced\" is beneficial for multi-turn dialogue, but no experimental validation was conducted on real multi-turn dialogue datasets. \n\n* W3) While the paper demonstrates strong performance at cache retention rates of 20% and above, it does not systematically evaluate the proposed method under more extreme compression scenarios. The performance trajectory of DefensiveKV when cache size drops below 10% remains unclear. For example, HeadKV evaluates its performance when retaining an average of 128 to 1024 tokens on LongBench. \n\n* W4) The authors' experiments primarily focus on long-text datasets. I think it would be better to test the performance of the proposed method on mathematical and reasoning tasks.\n\n* W5) There is a lack of theoretical analysis or mathematical proof to demonstrate why defensive aggregation outperforms mean aggregation.\n\n* W6) It would be helpful if the authors could provide additional information to support reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)