KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
Authors: Jang-Hyun Kim
Jinuk Kim
Sangwoo Kwon
Jae W. Lee
Sangdoo Yun
Hyun Oh Song
Venue: NeurIPS 2025 oral ()
Keywords: Large Language Models
Efficient Inference
Long-Context Processing
KV Cache Compression
Primary area: deep_learning
TL;DR: We propose a novel query-agnostic KV cache eviction method for multi-query scenario.

Abstract: Transformer-based large language models (LLMs) cache context as key-value (KV) pairs during inference. As context length grows, KV cache sizes expand, leading to substantial memory overhead and increased attention latency. This paper introduces \textit{KVzip}, a query-agnostic KV cache eviction method enabling effective reuse of compressed KV caches across diverse queries. KVzip quantifies the importance of a KV pair using the underlying LLM to reconstruct original contexts from cached KV pairs, subsequently evicting pairs with lower importance. Extensive empirical evaluations demonstrate that KVzip reduces KV cache size by $3$-$4\times$ and FlashAttention decoding latency by approximately $2\times$, with negligible performance loss in question-answering, retrieval, reasoning, and code comprehension tasks. Evaluations include various models such as LLaMA3.1, Qwen2.5, and Gemma3, with context lengths reaching up to 170K tokens. KVzip significantly outperforms existing query-aware KV eviction methods, which suffer from performance degradation even at a 90\% cache budget ratio under multi-query scenarios.

Vvip: Query-Agnostic KV Cache Compression with Context Reconstruction
Jang-Hyun Kim 12 Jinuk Kim 2 Sangwoo Kwon', Jae W. Lee', Sangdoo Yun?, Hyun Oh Song*
Seoul National University, Neural Processing Research Center, SNAVER Al Lab (blue378, hyunoh]@emu, ac
https://githut: com/sau-mmlab//Veil
Abstract
Transformer-basee large language models (LLMs) cache context key-value (KV) pairs during inference. As context length grows, KV cache sizes expand, leading to substantial memory overhead and increased attention latency. This paper introduces KVzip, a query-agnostic KV cache eviction method enabling effective reuse of compressed KVV caches across diverse queries. KVzip quantifies the importance of a KV pair using the underlying LLM to reconstruct original contexts from cached KV pairs, subsequently evicting pairs with lower importance. Extensive empirical evaluations demonstrate that Zziip reduces KV cache size by 3-4x and FlashAt- tention decoding latency by approximately 2x, with negligible performance loss in question anwoonnin,, retrieval, reasoning, and code comprehension tasks. Eval- uations include various models such as LLaMA3.1, Qwen2.5, and Gemma3, with context lengths reaching up to 170K tokens. Kzzip significantly outperforms exist- ing query-aware KV eviction methods, which suffer from performance degradation even at 90% cache budget ratio under multi-query scenarios.
Introduction
Transformerbbasee LLMs with long-context capabilities have significantly enhanced real-world applications, including long-document analysis and personalized conversational agents [1, 21, 49]. However, increasing context lengths substantially raises both memory consumption for KV caching and computational costs associated with attention mechanisms [31]. For example, caching 120K tokens in Qwen2.5-14B with FP16 precision requires approximately 33 GB memory, surpassing the model's s 28 GB parameter storage at equivalent precision [54].
Recent approaches primarily target reducing KV cache memory size while preserving inference accuracy. These methods include merging the attention heads [3], compressing KV pairs into shorter sequences [46], and using sliding-window techniques to limit context windows [24, 52, 53). Other studies exploit attention sparsity for dynamic KV eviction during decoding [4, 38, 60] and prefill stages [6, 33]. Existing eviction methods typically employ query-aware KV-pair importance scoring computed online during inference [6. 33, 60], selectively retaining KV pairs most relevant to immediate queries (Figure la, While effective in single-query scenarios, these methods exhibit significant performance degradation in multi-query settings, as the retained KV pairs predominantly overfit to initial queries [35]. We elaborate these limitations in Section 2.2.
In this work, we introduce KVzip, a novel query-agnostic KV cache eviction algorithm. KVzip optimizes reusable compressed KV cache for given context, enabling efficient inference across diverse future queries (Figure Ic). Our approach particularly benefits scenarios where KV caches are prepared offline, such as personalized conversational agents retaining user instructions and chat histories [8, 34], or enterprise systems utilizing precomputed document KV caches for retricval [7].
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Query-aware KV eviction
(b) Reusing query-dependent cache
(c) Proposed framework
CTX
peefill
KV
decade
CTX
pretil
decade
A
prefii
decocls
CTX
KV
Aa
CTX
KV
An
A.
One-time prefill
A.
One-lime prefill.
Repetitive piefill.
Glond performance.
perforaacce.
Good perfornance.
Figure I: Overview of KV eviction strategies in multi-query scenarios. An LLM processes input context (CTX) and queries (Q+) generate answers (A,). Existing approaches, such as SnapKV [33] and PyramidKV V [6], evict context KV pairs based on immediate query information. (a) Query-aware KV eviction independently performs prefill and eviction per query, incurring repeated prefill overhead. (b) Reusing a query- deeeeddeeen compressed cache leads to performance degradation for subsequent queries (Figure 2). (c) The proposed query-agnostic KV eviction framework compresses the KV cache only once during the initial prefill, enabling efficient reuse across diverse queries without repeated prefill or performance loss. Adapting existing methods to the query-agnostic framework still results in suboptimal performance due to mismatch with their original designs (Section 4).
Designing effective query agnostic eviction strategy remains challenging due to inherent uncer- tainty about future queries. In this work, we demonstrate that a succinct set of KV pairs, which is crucial for reconstructing the original context, serves as effective compressed representation. KVzip leverages the insight that a Transformer naturally functions as an encoder-decoder architec- ture by encoding context into KV pairs, analogous to traditional compression methods such Zip [27]. Specifically, our method simulates context reconstruction via an LLM forward pass, assigning importance scores KV pairs based on the maximum attention scores received during this pro- cess. This compression principle parallels self- self-super learning approaches that emphasize input reconstruction, demonstrating robust generalization across diverse downstream tasks [16, 22, 45].
After the eviction, subsequent queries significantly benefit from reduced latency and memory usage. Specifically, KVzip achieves approximately 2 latency reduction in FlashAttention [15] and 3-4x reduction in KV cache size during decoding with negligible performance loss on diverse queries. KVzip supports both context-dependent eviction, which achieves higher compression ratios but incurs per-context compression overhead [17], and context-independen eviction, which incurs no overhead after deployment while achieving moderate compression ratios [53].
Section 4 empirically demonstrates KVzip's robustness and effectiveness on multiple benchmarks, mathematical reasoning, retrieval, and code comprehension
including document question-answwrte tasks, with contexts up to 170K tokens. Unlike existing eviction methods which show significant performance degradation even at 10% KV eviction in multi-query settings 60J, KVzip consistently maintains inference accuracy even when evicting up to 70% of the KV cache. Experiments encompass 12 benchmark datasets, including SQuAD [47], GSMSK [12], and SCBench [35], and involve various models such as LLaMA3.1 [21], Qwen2.5 [54], and Gemma3 [49], ranging from 3B to 14B parameters. Furthermore, KVzip seamlessly integrates with existing optimizations such KV cache quantization [36] and structured head-level KV eviction [53]. Notably, our method replaces DuoAttention'ss 's head-score optimization, which originally requires of GPU hours, with only few forward passes completed within a minute, highlighting its practical effectiveness.
2 Preliminary
2.1 Notation and Problem Formulation
Consider the text domain T and an autoregressive Transformer-basedd LLM fIM T + T that generates sequences via greedy decoding [44, 50J. The model comprises layers, utilizing Grouped- Query Attention (GQA) [3] with // KV heads, each attended by group of G' query heads. During inference, fiM caches hidden representations as KV pairs to enhance computational efficiency [31].
Given an input context e T tokenized into n. tokens, the prefill stage generates cache containing H ne KV pairs, denoted as KV [2]. Conditioned generation using the cache denoted as c,evicted
f. KV Our objective is to derive compact pruned cache KV C KV satisfying
fLM(q KV c.viioo fiM(g KVc), Vg E T.
(I)

---

2.2 Analysis of Existing Approaches
Existing KV eviction methods, such as SnapKV [33] and PyramidKV [6], compress KV caches based on information given during prefill. These methods compute attention-based importance scores of KV pairs utilizing queries within trailing context window, selectively retaining KV pairs relevant to these queries. While effective for single-query benchmarks such as needle-in-al ieeele-i--ahaysstcck [26] and Long- Bench [5], these methods require repetitive cache prefills for each new query, as shown in Figure la.
Alternatively, reusing a previously compressed KV cache for subsequent queries can reduce the computation overhead, as depicted in Figure 1b. However, existing methods typically retain context KV pairs that are rel-
100
Accuracy (%
Snap V.prefill
Snap V-ruuse
evant only to the initial query and do not generalize to different queries. Figure 2 illustrates this issue using the SQuAD multi-QA QA dataset [47]. SnapKV attains high accuracy when executing prefill and compression individually per query, but performance significantly declines when reusing the cache compressed from the initial query. This shortcoming motivates our query- agnostic KV eviction strategy, enabling effective reuse of compressed cache across multiple queries.
KVzip (ours)
0.2
0,8 KV cache budget ratio
0.4
0.6
1.0
Figure 2: Accuracy on SQuAD using LLaMA3.1 1-8B. We evaluate SnapKV with repetitive per-query prefill, reuse of the compressed cache from the first question of each data sample, and KVzip with single prefill and query-agnostic compression.
3 Method
The primary objective of our algorithm is to assign an importance score to each KV pair, determining eviction priorities, following prior studies [60]. Given a context length nt,, KVzip assigns importance scores S Xx to KV pairs in KV., subsequently evicting pairs with the lowest scores. Our method supports both non-uniform and uniform head budget allocations [17, 33]. KVzip further accommodates head-level eviction strategy by computing head-level scores using the maximum pair-level scores across the sequence dimension, ne [53]. This section elaborates on the intuition, key technical contributions, and scalability to long-context scenarios.
3.1 Intuition
Context
KY
KV
docode
To effectively answer arbitrary queries, the compressed cache K KVe,evieted and fLM should retain complete con- textual information. Our intuition is that we verify this completeness by explicitly prompting fIM to recon- struct the previous context from KV eviieted (Figure 3). If KV c,evictedd enables fiM to accurately reconstruct the orig- inal context e using the repeat prompt, we can re-prefill the original cache KV. and conduct accurate inference.
Jim
prelill
evict
fIM
Context
Repeat prompt
Figure 3: Transformer LLM viewed as context encoder-decoder, Each matrix cell indicates a KV pair. We use the prompt "Repeat the previous context:".
However, regenerating the original cache at each inference remains practically infeasible. Encourag- ingly, our empirical studies indicate that the compressed cache demonstrates strong generalization capabilities without reconstructing the original cache (Section 4.2), empirically achieving Equa- tion (1). This finding resonates with principles from reconstruction--asse seff-superviee learning, which demonstrates strong generalization across diverse downstream tasks [16, 22, 45].
KV importance
Evict KV
Responses
Prefill
KV.
Measure max crnss- attentienion
Heads L.II'
with low scores mmirr-headdlaer leael-eeell)
KV c.evised
decode
fiM
fIM
fim
A Context
Repeat prompt
Context
Sequence
Queries
Figure Method overview. KVzip evicts KV pairs with the lowest importance scores, accommodat- ing both KV level and head-level eviction [17, 53]. System prompts are omitted for clarity.
3

---

3.2 KV Importance Scoring
Kzzip quantifies KV pair importance based on their contribution in context reconstruction. Specif- ically, we simulate reconstruction through reacher-forced decoding [19], parallelized via a single forward pass with an input sequence comprising repeat prompt followed by the original context (Figure 4). We define importance scores to be the maximum attention score each KV pair receives during this forward pass, leveraging the insight that KV pairs receiving minimal attention contribute little to Transformer computations [60].
Formally, given context of length Re. construct an input sequence of length Tiin Aprompt + The by concatenating the repeat prompt of length Nlrrompt with the context. Forwarding this input through fiM with KV. generates d-dimensional grouped-query features Q1, & xxxxmxx and key features Ki,6 E R (1 Vin for the hi-hh KV head in layer 7 [3]. Grouped-attention between these features produces R, R x +iia) Extracting entries corresponding to keys in KV gives sliced attention matrix AL, E RR. R Finally, we compute importance scores SLA E IR" for the h-th KV head in layer by taking the maximum over grouped queries
attention matrix AL, Softmax (Qr., (7,)
Si. g=1.....G: ii
At, g 1]-
We refer to the aggregated scores S across all KV heads as the maximumm cross-attention scores. Figure 13 provides a visualization of these scores.
3.3 Observation
The cross-altention pattern from the repeated context onto the prefilled context exhibits significant sparsity, indicating substantial opportunities for compressing KV Additionally, the attention pattern from reconstruction notably overlaps with attention patterns from diverse tasks. Such overlap implies that KV features critical for context reconstruction substantially contribute to downstream tasks, highlighting strong generalization capability.
0.8
Attention Sparsity in Reconstruction. Cross-attention patterns obtained during context reconstruction exhibit greater sparsity compared to self-attention patterns com- puted during the initial prefill of KV. (Figure 5). During prefill, the model densely interacts among tokens to en- code comprehensive contextual information [42]. re- construction, however, the model efficiently leverages (1) high-level representations stored in KV. and (2) internal knowledge encoded within model weights, thus reducing unnecessary attention lookups. This cross-attention spar- sity effectively identifies and removes redundant KV pairs, outperforming prior methods such as H₂O [60] that rely on attention scores obtained during prefill (Section 4.2).
Recomstruction
Density
0.6
Preftl
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Score
Figure Histogram comparing max attention scores received by KV pairs in KV, during prefill versus reconstruc- tion stages, measured on SQuAD with LLaMA3.1-8B.
1.0
Score (Summarizatioo)
1.0
100
Score (QA)
0.8
0.8
Score (Reasoning)
0.8
240 5000
0.8
10-1
0.6
0.6
0.6
0.6
10:2
0.4
0.4
0.4
0.4
10-3
0.2
0.2
0.2
0.2
10-4
0.0
0.0
10-5
0.2
1.0
Score (Repeat)
Score (Repeat)
Score (Repeat)
Score (QA-1)
Figure 6: Attention comparison across tasks. 2D histograms visualize the joint distribution of maximum cross-attention scores received by KV pairs for two distinct scoring inputs. Each input consists of task query and the generated response (Table 3). Each cell at (v, w) indicates the proportion (log-scale) of KV pairs in KV, receiving maximum attention of u for the x-axis task and w for the y-axis task. Bright colors the lower-right triangular region denote KV pairs receiving higher attention from the x-axis task than from the y-axis task. We compute scores using LLaMA3.1 1-8B on SQuAD example, except for the third heatmap, which represents GSM8K reasoning. QA-1 and QA-2 denote distinct QA pairs. Figure 13 visualizes the attention patterns for each task.

---

attention
Key
Attention matrix
Querr
KV.
fim
Query
max
2-th chunk
Input
Repeat prompt 4-th context chunk
172
Key
Piaz
Figure 7: Chunked scoring for the i-th chunk in KV e. We compute attention scores by multiplying queries with subsampled keys of length m + Rin followed by softmax normalization. We then slice the resulting matrix and take the maximum over queries obtain chunked importance score of length m. We set the grouped-query size to G = 1 for clarity. This procedure repeats per chunk. For chunks with i 2, we formulate the repeat prompt as: "Repeat the previous context starting with (last 8 tokens of preceding chunk):". Appendix C.2 demonstrates that the design choice of a repeat prompt negligibly affects performance. Pseudo-code provided in Appendix A, Algorithm
Attention Overlap Across Tasks. Figure compares max cross-attention scores across various tasks: repeat, question question-answert answering (QA), summarization, and reasoning, The first three heatmaps show distributions concentrated in the lower-right triangular region, indicating that KV features receiving high attention in reconstruction also receive high attention across other tasks. In contrast, the fourth heatmap, comparing two different QA tasks, shows a distinct distribution concentrated along both the x- and y-axes, reflecting query-specific attention variability. This observation demonstrates that reconstruction-critia. KV pairs consistently contribute to diverse tasks, supporting the effectiveness of KVzip. We empirically validate this generalization capability in the experimental section.
3.4 Technical Challenge and Solution
Our method concatenates a repeat prompt with context tokens. processing this input through fLM to obtain attention matrices. However, attention matrices scale quadratically with context length ne. making direct computation prohibitive for long contexts. While fused attention kernels like FlashAttention reduce memory overhead by computing attention scores block- biie without storing full matrices [15], our method uniquely requires maximization along the query dimension following Softmax normalization along the key dimension. This cross cross- coos--imensiooal dependency prevents direct integration of Equation (2) into existing block-wise attention algorithms.
Chunked Scoring, To address this challenge, we introduce chunk-based scoring, reconstructing context segments independently. By computing importance scores in fixed-size chunks, rather than simultaneously over the entire context, computational complexity reduces from quadratic O(n)) to linear O Omm,), where m denotes the size of the chunk. Specifically, we partition the context tokens into fixed-length chunks of size m, concatenate each chunk with the repeat prompt, and process the resulting input of length Tin = 7'prompt + m through fIM (Figure 7). For each Transformer layer, we subsample keys in KV. corresponding to each chunk, obtaining smaller attention matrix of size nan X (m + n. As in Equation (2), slicing the attention matrix and maximizing over grouped queries yields chunk-wise importance scores. We repeat the process for each chunk and aggregate the scores to obtain the full importance scores of KVc. We set the chunk size to m = 2K, constant across context lengths, models, and tasks, as the size has negligible impact on performance (Appendix C.I).
Complexity Analysis. Computational complexity per chunk is (mm²), assuming a negligible repeat prompt length, i.e., Reromptt < m, thus 7mm m. Repeating this computation for all n./m chunks yields total complexity O(mne), linear with context length. Peak memory overhead is O(m²), which remains constant with n. and is negligible compared to model parameters and KV cache sizes. Additionally, we propose softmax-free variant in Appendix C.3 utilizing custom CUDA kernel integrated into FlashAttention, further reducing computational costs at a performance trade-off.
Importance scoring introduces additional overhead from computing attention queries and keys for chunked inputs through fLM with KVc. Given Min m, FlashAttenntin incurs O + m //) causal-attention FLOPs per chunk, resulting in total complexity of O(n + n.mm22) across all n. / chunks. This cost approximately doubles the initial prefill catsal-attention complexity O (22) Utilizing FlashAttention with chunking effectively bounds peak memory usage. For efficiency, K KVzip also supports indeeeeedddeeendnntt eviction by assigning static head-level importance scores per model (Section 4.2- -igure 11), incurring compression overhead after deployment.
5

---

(a) Inference efficiency (decoding)
(b) Compression overhead
Atlention latency (ms)
0.4
0.39
KV memory (GB)
100
95.8
Peak memory (GB
38.8
0.3
-0.27
15
Compute time
80
J5.4
65.9
71.9
30.5
31.1
32.5
10
9.8
60
0.2 0.17
20
20
0.1
20
10
0.0
0.2 0.4 0.6
0.8 1.0
0.2 0.4 0.6 0.8 1.0
0.5%
1k
2k
4k
8k
0.5k 1K
2k
4k
8k
KV cache ratio
KV cache ratio
Repeat chunk size
Repeat chunk size
Figure 8: Computational analysis using LLaMA3.1-SB with 124K context tokens on an NVIDIA A100 GPU in FP16 precision. We apply non-uniform head budget allocation with variable-length FlashAttention-2 [17]. (a) Attention latency per layer and total KV cache size show improved inference efficiency. (b) KV importance scoring overhead aggregated over all chunks. Dashed horizontal lines indicate initial prefill cost for reference, with 2K chunk size limiting peak memory for fair comparison [2]. K Kzzip also supports context-indepen context-independen eviction [53], incurring scoring overhead per model prior to deployment and removing runtime compression overhead (Figure 11).
Empirical Efficiency Analysis. Empirical evaluations on an NVIDIA A100 GPU in Figure confirm approximately twice the computational overhead of standard prefill during compression, with minimal additional memory (under 2%). Importantly, compression occurs once per context or per model. Figure 8a shows that our approach achieves significant reduction in inference latency and KV cache size. Our experiments validate consistent efficiency improvements across diverse models and tasks with negligible performance degradation at compression ratios as low as 30%.
4 Experiment
4.1 Sctup
Eviction Structure. We employ a non-uniform head-budget allocation strategy for KV eviction, retaining KV pairs with the top r% importance scores across all attention heads, where r% denotes the target compression ratio. KV pairs of the initial system prompt remain intact. To ensure faireess we apply the same non-uniform allocation to baseline methods, given its demonstrated superiority over uniform allocation [17]. This compressed KV cache, combined with FlashAttention, improves inference speed (Figure 8). Additionally, we evaluate KVzip with context- -nneeeeeindeeendentt eviction in Section 4.2 and uniform-budged allocation in Appendix C.4.
Evaluation. Our evaluation focuses on the capability of a KV cache to effectively handle diverse queries. Given the inherent limitations of query-aware frameworks discussed in Section 2.2, we adopt the query-agnostic framework from Figure 1c. Specifically, we prefill and compress context KV caches independently. without task queries. Existing eviction methods also support this independent prefilling of context [60, 33], enabling evaluation under the query-agnostic framework. We measure average model performance using these compressed KV caches across multiple or single queries. Since the compression is query-agnostic, even single-query evaluations meaningfully assess specific task capabilities of eviction methods. Unlike prior methods that evict KV pairs from replicated caches for grouped queries [33], we evict directly from the initially stored cache before replication, thus reducing the actual storage required for the KV cache. The evaluation setup is consistent across all baselines for fair comparison, conducted on single NVIDIA A100 80GB GPU.
Baselines, Datasets, and Models. We benchmark against state-of-the-art KV cache eviction methods, including H₂O [60], SnapKV [33], and PyramidKV [6]. We further compare DuoAAtention
[53] using head-level eviction for context-independent compression. Evaluations span diverse datasets: SQuAD [47], GSM8K [12), needle-in-a- meedee-n---hhasstae -naa-hay (NIAH) [26], and nine tasks from SCBench [35]. SCBench provides comprehensive multi-query evaluations, including tasks from RULER [23] and ooBench [59]. Except for GSM8K and NTAH, each dataset example includes multiple queries per context. Context lengths range from 100 to 170K tokens, tokenized with the Qwen tokenizer [54], covering domains such as long-document QA, retrieval, mathematical reasoning. in-context learning, and code comprehension. Appendix A provides implementation details and dataset specifics.

---

KVzip (ours)
H₂O
SnapKV
PyramidKV
NIAH
Retr.VV
Retr. PiilPPrei--Sfiiii
Code. RepoeA
Retrieval
Accuracy (%)
50
40
@ (%)
60
60
Accurccy
Accuracy
40
20
20
20
10
0.2 0.4 0.6 0.8 1.0
0.2 0.4
U.6 0.8
1.0
0.2 0.4
0.6
0.8
1.0
0.2
0.4 0.6 0.8 1.0
KV cache ratio
KV eache ratio
KV cache
KV cache ratio
SQuAD
GSM8K
En. En.QA
En.MultiChoice
Contextual QA
100
80
Accuacy (%)
80
10
60
Accuracy
Accuncy
30
Accuraac
80
40
20
40
20
0.2 0.4 0.6 0.8 1.0
0.2
0.4
0.8
1.0
0.4 0.6 0.8
0.2 0.4 0.6 0.8
1.0
KV cache matio
KV cache ratio
KV cache ratio
KV cache ratio
En. Summary
Retr. Mult..p
Math.Find
ICL..ManyShot
Redundancy
ROUGIE (%)
35
Accuracy (S6
30
Accurac (5)
355
30
Accuracy
30
20
25
25
10
20
15
0.2 0.4 0.6 0.8 1.0 KV cache ratio
0.2
0.4 0.6 0.8
1.0
0.2 0.4 0.6 0.8
1.0
0.2
0.8 1.0 KV cache ratio
0.4 0.6
KV eache ratio
cache ratio
Figure Benchmark results using Dwwen2.57B-111 across varying KV cache budget ratios from 0.1 to 1.0. We group the tasks into three categories: (1) retrieval-intensive (2) contextual understanding. and (3) high context redundancy. Appendix D presents additional results on the SCBench multi-task datasets and RULER, where KZzip consistently outperforms the baselines.
We conduct evaluations with various cntrructionfinettnn LLMs, including Qwen2.5-7B- 1M, LLaMA3.1-8B. and Gemma3-12B [54, 21, 49]. These models utilize GQA with group sizes varying from (LLaMA3. 1-8B) to 7 (Qwen2.5- 20wen2.5-7B-- 7B-------11 MM). Gemma3 employs hybrid attention mechanisms, combining global and sliding window strategies [49]. All evaluations use Bfloat 6 precision. We use greedy decoding with these models to generate responses. Furthermore, we integrate KVzip with the QServe quantization framework, adopting 8-bit weights, 8-bit activations, and 4-bit KV cache [36].
4.2 Benchmarking
Task Generalization. Figure 9 presents multi-query evaluation results for Qwen2.5-7B-IN across 12 benchmark datasets, grouped into three categories. The first row includes retrieval-intensive tasks, requiring the extraction of sentences, cryptographic keys, or code functions from context. Our method significantly outperforms baselines, preserving performance at a 30% cache ratio except for Retr. Retr.Prefix-Siie Prufix-Suflix, while baseline methods degrade notably at 90% retention. The second row contains contextual understanding tasks, including mathematical reasoning (GSM8K), Our method achieves neas-lossless compression down to 30-30%, consistently outperforming baselines. the last row, En. Summary requires high-level contextual information, whereas other tasks contain repetitive contextual information [35]. These tasks tolerate aggressive compression (down to 10%) without performance degradation, occasionally even showing performance improvement. We hypothesize that this improvement results from reduced attention distractions following KV eviction [57]. Overall, our method robustly generalizes across diverse tasks in query-agnoste settings, outperforming bascline approaches.
Model Scale and Architecture. Figure 10 shows performance across larger models 10wen2.5-14B. 1M), distinct model families (LLaMA3.1M 1-8), and hybrid attention architectures (Gemma3-12B) Gemma employs global and sliding-window window attention layers in 1:5 ratio [49]. We apply KV eviction exclusively to global attention layers, as these layers dominate cache sizes at 100K context length with KK sliding window size. To comprchensively compare methods, average performances
7

---

KVzip (ours) H₂O
SnapKV
PyramidKV
Qwen2.5-14DP 1ren2.5-14B-IN
LLaMA3.1-8B
Gemma3-122
LLaMA3-8B -B---8KV4
Rel performnce
1.0
Rel. pezformance
1.0
Rel. performance
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.0
0.6
0.4
0.4
0.4
Rel.
0.4
0.2
0.44
0.6
0.8
1.0
0.2
0.4
0.6
0.8 1.0
0.2 0.4 0.6 0.8 1.0
0.2
0.4
0.6
0.8 1.0
KV cache ratio
KV cache ratio
KV cache ratio (glohal)
KV cache ratio
Figure 10: Performance on various models averaged over 12 benchmark datasets. We normalize performance of each dataset relative to the full-cache performance before averaging. Appendix D) provides detailed results per dataset, including results for LLaMA3.1-3B
over 12 benchmark tasks. Figure 10 confirms ZVzip's generalizability and superior compression performance across various models compared to baseline methods.
KV Quantization. KVzip effectively integrates with KV cache quantization, further reducing cache sizes. Figure 10 evaluates KV eviction methods on 4-bit KV quantized model (LLaMA3- SB-W8A8KV4) from QServe [36]. We apply an identical quantization scheme throughout prefill, importance scoring, and decoding. The results confirm that KZzip remains robust under quantization, while indicating the base LLaMA3-8B model exhibits greater contextual sparsity than the improved version, LLaMA3.1-88 1-BB.. Specifically, the 16-bit KV cache occupies 16.3GB 124K input length. Integrating 4-bit quantization with our 70% eviction ratio effectively reduces the cache size to 1.2GB with negligible performance degradation, demonstrating significant practical benefits.
Sontext-Independen Eviction. Kzzip also supports context-independen eviction strategies, requiring only a one-time importance scoring per model and incurring no compression overhead after deployment [53]. Specifically, we assign static head-level importance scores by aggregating pair-level scores, taking the maximum value along the sequence dimension. We compute scores using a single English book sample containing 88K tokens from En.QA in SCBench [35] and apply DuoAttention's 's head-level KV eviction strategy [53]. Figure 24 in Appendix visualizes the obtained head-score distribution, comparing with scores derived from other data sources.
Figure 11 compares KVzip against AuoAttention [53],
Rel. performance
using publicly released official head-scores on LLaMA3- Cinstrut-Grradeen 1048K [20]. Whereas DuoAtten- tion optimizes head scores to retrieve a synthetic passkey, KVzip derives head scores by performing more general task of context reconstruction on a natural language text- book. Specifically, DuoAttention demands several hours of optimization on an 8-GPU node for importance scor- ing. In contrast, KVzip achieves superior performance using only few forward passes within one minute for scoring. The results demonstrate KVzip's 's efficiency and robust performance across various eviction strategies.
1.0
0.9
0.8
KVzip (head)
0.7
DunAAtention
0.6
0.6
1.0
KV cache ratio
Figure 11: Average relative performance across 12 benchmark datasets with head-
level eviction. The lowest KV cache ra- tio is set to 0.4 due to DuoAttention's 's lower limit 0.32.
4.3 Analysis
100
Necessity of Context Reconstruction. ZZzip employs an input that concatenates the repeat prompt and the con- text for importance scoring (Figure 4). Figure 12 demon- strates the necessity of full context reconstruction by com- paring scoring performance across various inputs: using the repeat prompt combined with either the first 10% of context (First), the last 10% (Last), or the repeat prompt alone (Prompt). Results clearly indicate that reconstruct- ing the full context (Recon) is essential to prevent perfor- mance degradation by KV eviction.
Accuracy (%)
80
Recon.
First
Last
Prompt
0.2
0.4
0.6
0.8
1.0
KV cache ratio
Figure 12: Performance across various inputs for KV importance scoring on SQuAD (LLaMA3.1-8B))

---

Table 1: Behavior analysis. Generation results on a privacy-related example from DecodingTrust [51], using LLaMA3.1-88 with full KV cache and a 40% compressed cache via Kzzip.
Context
Query
Response (full KV)
Response (evicted KV)
Sean P. Tracey's phone
What Sean P. Traccy's phone number?
cannot provide personal contact information.
6604876475
number is 6604876475.
Hershel Swartz's...
Behavior Analysis Beyond Task Solving. Previous sections demonstrate that reconstruction- based compression technique effectively retains KV pairs critical to diverse tasks, Further analysis reveals intriguing, privacy-related behavior arising from KV eviction. Table compares generated responses for queries involving private context information before and after KV cache compression. Specifically, the LLaMA3.1-88
ceeructiionfinetuned model refuses responses when utilizing the full KV cache but notably responds after applying our compression method. This behavior naturally emerges because KVzip prioritizes KV pairs necessary for context reconstruction and discards others, consistent with Yang et al. [56]. Although practical implications may be limited -iince cached contexts typically imply permission for utilization this observation suggests intersections between KV eviction techniques and shallow-aie aallgmmell concerns [43], motivating further research exploration.
5 Related Work
KV Cache Compression. Compressing KV caches of Transformer-based models is crucial for efficient inference [50]. Sparse Transformer methods explicitly train models to utilize sparse or localized KV caches, reducing memory requirements during inference [11, 24, 30J. Compressive Transformer approaches further compress caches by merging KV pairs during training [3, 28, 46]. Liu et al. [39] show that Transformer based LLMs exhibit contextual sparsity during inference, motivating dynamic KV eviction methods such H2O and FastGen that operate during decoding without additional training [4, 9, 18, 29, 38, 41, 55, 60). SnapK PyramidKV, and Finch specifically target KV eviction during long- long-context prefill [6. 17, 33, 13], while DuoAttention profiles and selectively replaces attention heads with sliding-window attention prior to deployment [52, 53]. Our approach aligns most closely with prefill compression techniques. Unlike existing methods that perform query-dependent KV compression, we propose query-agnostic compression, enabling compressed KV cache reuse across diverse queries. Concurrently, Corallo al. [14] propose a query-agnovd KV compression method for the -aummeee-l generation scenario. Our method also operates the pre-deployment stage, following the DuoAttention framework. Recent studies have explored KV cache compression via quantization [36, 40]. These techniques are complementary to our eviction strategy and can further improve the overall efficiency of cache compression.
Efficient LLM Inference. Another line of research enhances inference efficiency by employing sparse attention mechanisms instead of directly compressing KV caches. BigBird achieves efficiency
by training models with sparse attention structures, reducing inference-time attention costs [58]. MInference leverages attention sparsity at inference without additional training [25]. Approaches including Quest reduce attention computations during decoding by leveraging KV cache offloading
and retrieval techniques [10, 32, 37, 48]. In contrast to this line of work, our method focuses on
explicitly reducing the KV cache size.
6 Conclusion
We introduce Kzzip, a query-agnostic KV cache eviction algorithm that effectively optimizes reusable compressed KV caches through reconstructing the original context from KV pairs. Through extensive evaluations on multi-query settings across diverse tasks, models, and long-context benchmarks, KVzip demonstrates robust compression performance, reducing KV cache sizes by up to 70% with negligible performance loss, while significantly improving decoding attention latency by approximately 2x with FlashAttention KVzip consistently outperforms existing KV eviction methods, which suffer performance degradation with 10% eviction ratio. The practical applicability of K KVzip further extends to quantized models and diverse KV cache structures, highlighting its adaptability and efficiency.

---

Repeat (KVzip)
1.0
layer
0.8
layer
0.6
layer 16
0.4
0.2
layer 24
0.0
KV sequence (context)
(b) QA-1
1.0
layer
0.8
layer
0.6
layer 16
0.4
0.2
layer 24
0.0
KV sequence (context)
(c) QA-2
1.0
layer
0.8
layer
0.6
layer 16
0.4
0.2
layer 24
0.0
KV sequence (context)
(d) Summarization
1.0
layer
0.8
layer
0.6
layer 16
0.4
0.2
layer 24
0.0
sequence (context)
( Prefill (H,O)
1.0
layer
0.8
layer
0.6
layer 16
0.4
0.2
layer 24
0.0
sequence (context)
Figure 13: Visualization of maximum attention scores. Each heatmap visualizes the maximum attention scores received by KV pairs KV, (Equation (2)) for a SQuAD example, computed using LLaMA3.1-8B. Table 3 in Appendix describes the text inputs for each task. Rows correspond to specific layers, with dimensions context length is (b)-(d) Maximum cross-attention scores from downstream tasks: two distinct QA pairs and one summarization task. These illustrate varied attention patterns across downstream tasks, while the repeat task's 's attention pattern encompasses all these patterns (see also Figure 6). (c) Maximum self-attention scores during the prefill stage exhibit denser attention patterns than cross-attention scores and do not overlap with downstream task patterns, indicating that prefill-based profiling such H₂O does not effectively reflect the KV cache utilization by downstream tasks.
x nes where the number KV heads H 163. (a) Importance scores from KVzip obtained using the repeat task.
and the
10

---

Acknowledgments and Disclosure of Funding
This work was supported by Samsung Electronics Co., Ltd. (10250418-12669-00),, Mobile eXperi- ence (MX) Business, Samsung Electronics Co., Ltd., Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No. RS2020-11200882, (SW STAR LAB) Development of deployable learning intelligence via self- sustainable and trustworthy machine learning), the Air Force Office of Scientific Research under award number 12386-25--1-013 and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. R5-2024-003540036. Hyun Oh Song is the corre- sponding author.
References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida,
J. Altenschmidt, S. Altman, Anadkat, et al. Gpt-4 technical report. arXiv preprint 0rX74:2303.08774, 2023.
[2] A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. Gulavani, A. Tumanov, and
R. Ramjee. Taming throughput-latency tradeoff in II inference with sarathi-serve. In 18th
USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024.
[3] Ainslic, J. Lee-Thorp, M. De Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. EMNLP, 2023.
[4] S. Anagnostidis, D. Pavllo, L. Biggio, L. Noci, A. Lucchi, and T. Hofmann. Dynamic con- text pruning for efficient and interpretable autoregressive transformers. Advances in Neural Information Processing Systems, 2023.
[5] Y. Bai, X. J. Zhang, H. Lyu, J. Tang, et al. Longbench: A bilingual, multitask benchmark for long context understanding. ACL, 2024.
[6] Z. Cai, Y. Zhang, B. Gao, Y. Liu, T. Liu, K. Lu, al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:24066 240600,2000 2024.
[7] B. J. Chan, C.-T. Chen, J.-H. Cheng, and H.-H. Huang, Don't do rag: When eache- augmented generation is all you need for knowledge tasks. arXiv preprint arXiv:2412.156003 2024.
[8] Character.AI. Optimizing ai inference at character.ai, 2024. URL character character.ai/ppeeetet. wtt/pptimmziignnnfereee.e..
[9] Y. Chen, G. Wang, J. Shang. S. Cui, Z. Zhang, Liu, S. Wang, Y. Sun, D. Yu, and H. Wu. Nacl: A general and effective kv cache eviction framework for Ilms at inference time. ACL, 2024.
[10] Z. Chen, R. Sadhukhan, Z. Ye, Y. Zhou, J. Zhang, et al. Magicpig: Lsh sampling for efficient Ilm generation. ICIR, 2025.
[11] R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: /05090 2019.
[12] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, I. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1rXiv:2110.14166 2021.
[13] G. Corallo and P. Papotti. Finch: Prompt-guided key-value cache compression for large language models. Transactions of the Association for Computational Linguistics, 12, 2024.
[14] G. Corallo, O. Weller, F. Petroni, and P. Papotti. Beyond rag: Task-aware kv cache compression for comprehensive knowledge reasoning. arXiv preprint 25X3:22503.0473, 2025.
[15] Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. ICLR, 2024.
[16] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional
transformers for language understanding, NAACL, 2019.
[17] Y. Feng, J. Lv, Y. Cao, X. Xie, and S. K. Zhou. Ada-kv: Optimizing ky cache eviction by adaptive budget allocation for efficient IIm inference. arXiv preprint arXiv:2407.1 / 1550, 2024.
[18] S. Ge, Zhang, Liu, M. Zhang, J. Han, and Gao. Model tells you what to discard: Adaptive kv cache compression for 1ms. ICLR, 2024.
11

---

[19] A. Goyal, A. Lamb, Y. Zhang, S. Zhang, A. Courville, and Y. Bengio. Professor forcing: A new algorithm for training recurrent networks. Advances neural information processing systems, 29, 2016.
[20] gradientAL. Llama-3 8b gradient instruct 1048k, 2024, URL https://muggingface co/ gradientai/Llama- 3-8B- Instruct Gradi 1048k.
[21] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, et al. The llama herd of models. arXiv preprint arXiv:240.22/783 2024.
[22] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[23] C.-P. Hsich, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What's ' the real context size of your long-context language models? COLM, 2024.
[24] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, et al. Mistral 7b, 2023.
[25] H. Jiang, Y. Li, C. Zhang. Q. Wu, X. Luo, al. Minference 1.0: Accelerating pre-filling for cong-context Iims via dynamic sparse attention. Advances in Neural Information Processing Systems, 2024.
[26] G. Kamradt. Needle in baystack-pressurr testing Ilms, 2023.
[27] P. W. Katz. Zip file format specification, 1989. URL. https://pkkut http://wwwwrr..cchhll..ll cachefly net/ webdocs/ taaeeesstuiieee/PPPeee APPPNTT TXT.
[28] J. Kim, J. Ycom. S. Yun, and H. O. Song. Compressed context memory for online language model interaction. ICLR, 2024.
[29] M. Kim, K. Shim, J. Choi, and S. Chang. Infinipot: Infinite context processing on memory- constrained llms. arXiv preprint arXiv:2410.015/8. 2024.
[30] S. Kim, S. Shen, D. Thorsley, A. Gholami, W. Kwon, Hassoun, and K. Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
[31] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, 2023.
[32] W. Lee, J. Lee, J. Sco, and Sim. Infinigen: Efficient generative inference of large language models with dynamic kv cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2024.
[33] Y. Li, Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 2024.
[34] Y. Li. H. Wen, W. Wang. X. Li, Y. Yuan, G. Liu, et al. Personal Ilm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05433 2024.
[35] Y. Li, H. Jiang, Q Wu, X. Luo, S. Ahn, C. Zhang, A. H. Abdi, D. Li, J. Gao, Y. Yang, et al. Scbench: A kv cache-centric analysis of long-context methods. ICLR, 2025.
[36] Y. Lin, H. Tang, S. Yang, Z. Zhang, G. Xiao, C. Gan, and S. Han. Qserve W4a8kv4 quantization and system co-design for efficient m1m serving. arXiv preprint WxXiv22405004522 2024.
[37] D. Liu, M. Chen, Lu, H. Jiang, Z. Han, Q. Zhang, al. Retrievalattention: Accelerating long-context IIm inference via vector retrieval. arXiv preprint 24X.v:2409.1051/0 2024.
[38] Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyrillidis, and A. Shrivastava. Scis- sorhands:
Exploiting the persistence of importance hypothesis for Ilm kv cache compression test time. Advances in Neural Information Processing Systems, 2023.
[39] Z. Liu, J. Wang, T. Dao, T. Zhou, B. Yuan, 7.. Song, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, 2023.
[40] Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. ICML, 2024.
[41] M. Oren, M. Hassid, N. Yarden, Y. Adi, and R. Schwartz, Transformers are multi-state mons. arXiv preprint 240.:24001.01144 2024.
12

---

[42] M. Peters, M. Neumann, M. lyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In NAACL, 2018.
[43] X. Qi, A. Panda, K. Lyu, X. Ma, S. Roy, A. Beirami, P. Mittal, and P. Henderson. Safety alignment should be made more than just a few tokens deep. ICLR, 2025.
[44] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre- vreetraining. 2018.
[45] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[46] W. Rae, A. Potapenko, S. M. Jayakumar, and T. P. Lillicrap. Compressive transformers for long-range sequence modelling. ICLR, 2020.
[47] P. Rajpurkar, Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text. EMNLP, 2016.
[48] J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikci, and S. Han. Quest: Query-awart sparsity for efficient long-context I1m inference. ICML, 2024.
[49] G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, al. Gemma technical report. arXiv preprint arXiv:2503. 19786, 2025.
[50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 2017.
[51] B. Wang, W. Chen, H. Pei, C. Xie, M. Kang, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurlPS, 2023.
[52] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks. ICLR, 2024.
[53] G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y. Fu, and S. Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. ICLR, 2025.
[54] A. Yang, B. Yu, C. Li, D. Liu, F. Huang, H. Huang, et al. Qwen2.5- technical report. arXiv preprint arXiv:2501. 15383, 2025.
[55] D. Yang. X. Han, Y. Gao, Y. Hu, S. Zhang, and H. Zhao. Pyramidinfer: Pyramid kv cache compression for nightthroughput Ilm inference. arXiv preprint arXiv:2405. 2024.
[56] J. Y. Yang. B. Kim, J. Bae, B. Kwon, G. Park, E. Yang, S. J. Kwon, and D. Lee. No token left behind: Reliable kv cache compression via importance-awar mixed precision quantization. arXiv preprint arXiv:2402. 18096, 2024.
[57] T. Ye, L. Dong, Y. Xia, Y. Sun, Y. Zhu, G. Huang, and F. Wei. Differential transformer. ICLR, 2025.
[58] M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, Pham, A. Ravula, Q. Wang, L. Yang. et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 2020.
[59] X. Zhang, Y. Chen, S. Hu, Z. Xu, J. Chen, al. cobench: Extending long context evaluation beyond 100k tokens. ACI., 2024.
[60] Z. Zhang, Sheng, T. Zhou, Chen, L. Zheng, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, clearly state the paper's contribution and scope: query-agnostic KV cache eviction for Transformer-based LLMs (Section 1, L32-61).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Appendix B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed experimental setups including evaluation metric and implementation details in Section 4.1 and Appendix A.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We provide pseudo code for our algorithm Algorithm We use existing benchmark datasets publicly available. We attach implementation codes in the supplementary materials. We will open-source our codes upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Suuides/CodeSumiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips.c/pppp wttp:ccccpppcccccctcccccccccccce wwwpc//gg///eeeeeeeeeeoecee /edeeSSue ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide evaluation details in Section 4.1 and Appendix A. We provide hyperparameters Section 3.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our experiments utilize established benchmarks, including SCBench [35], which introduces no uncertainty during evaluation (Section 4.1, L211--20). Our evaluation employs deterministic greedy decoding (Section 4.1, L.225) without involving any training processes that could introduce stochasticity. We follow the experimental reporting formats
prior works, including H2O, SnapKV, and DuoAttention [60, 33, 53]. We conduct
comprehensive evaluations across 12 datasets and 5 models to demonstrate the empirical significance our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We analyze computational complexity in Section 3.4. We describe computer resources in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //terr///////seeeeeeee Sptpbiic/Ethcssuiieeee
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and conduct research adhering to the statement.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not release new generative models or data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite related papers in Section 5. We provided links for bascline implemen- tation codes Appendix A.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do not involve LLMMs as any important, original, or non-standard compo-
nents.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https ptttps:/neuripst http:/ppp.../oooeeeeee////////// for what should or should not be described.
20

---

A
Implementation Details
Pseudo Code.
Algorithm details the pseudo code for our KV importance scoring algorithm.
Algorithm KV Importance Scoring
Input: Transformer fLM, context e (token length ne), chunk size m (fixed to 2K) # FLM has L layers, H K.V heads, G grouped-query size, d feature dimension KV. + Prefill cache by forwarding e through fLM chunks, each of token length m
......, ,r f- Partition into T [me
S < 0LxHXn,
# placeholder
for = 1....,TT do
if' = 1 then
input + "Repeat the previous context:" + Cr
else
1t-1 t A trailing span of c--1 with 8 tokens
input + "Repeat the previous context starting with" + Ct-1,tast last + 5."
end if
Forward the input (token length min) through fIM with KV.
for = -l...,, do
+- Queries in the 1-th attention layer
# shape: G H x nia d
K + Keys in the (-th attention layer
# shape: H x (ne + Nin X d
K Subsample keys in KV. corresponding to ct.
# shape:
(m + na) X d
A < Softma((QK3
# broadcast over G groups; shape: G x H x Tha X (m + mm)
< Al...,: m]
attention received by keys in KV shape: G H X ni x m # shape: H x
SL. < maxg=1..., m. A,
S(,, (1-11)))) 1)m tmm) <- St.t
end for
end for
Shead maxim I..... S1:,;,
# shape: L x H
Output: Score S, Head-lewe score Shend
Baseline Methods. We implement SnapKV and PyramidKV following their official GitHub imple- mentations [33, 6]. We apply max pooling with kernel size of and an observation window size of 32, consistent with original hyperparameters [33]. For examples shorter than 1K tokens, we reduce the observation window size to 16. SnapKV maintains uniform budget ratios across layers, whereas PyramidKV V uses linearly decreasing layer-budget ratios. In the main experiments (Section 4.2), we adopt non-uniform head- baddet allocation strategy, which demonstrates superior performance over uniform head-budger allocation [17]. Specifically, we retain KV pairs corresponding to the top r% importance scores across all attention heads in each layer, given a layer budget ratio of r%. Appendix C.4 provides results with uniform head-budget allocation.
We implement the prefill version of H2O based on the official GitHub code provided by PyramidK For each KV pair, we compute the maximum attention score received during prefilling, our experiments show superior performance over using the average attention scores. This result aligns with observations by Oren et al. [41]. H₂O serves as a counterpart to KVzip by utilizing self-attention scores from prefilling, while our method employs self-attention scores from reconstruction.
Datasets. In our main experiment described in Section 4.2, we consider nine English tasks from SCBench [35]. Additionally, SCBench provides multi-task datasets, i.e., MuX.SUH and AY each composed of two distinct tasks. As performance patterns for these multi-task datasets closely resemble our main results on individual tasks, we present their results separately in Appendix D. Considering the 128K context length limitation of LLaMA3.1 and Gemma3, exclude data examples from the En.QA and MultiiCihooiee tasks with context lengths exceeding 125K tokens using the LLaMA3.1 tokenizer. For synthetic tasks such as Retr. KV. context lengths span up to 125K tokens with the 1LaMA3.1 tokenizer and up to 170K tokens with the Qwen2.5 tokenizer.
hhttps://githu www.ttuu...om/eaa.....m cww/Zefan- Maai//VCache-aattte
21

---

SnapKV retains KV pairs in a trailing context window [33], notably biasing shorter contexts toward recent tokens which results in degraded performance. To mitigate this issue, we evaluate GSM8K samples having context lengths of at least 72 tokens (based on the L.aMA3.1 tokenizer) [12], aligning with SnapKV's V's observation window size of 16. For the Veedle-in-a-Haystter (NIAH) task [26], we utilize the published GitHub repository?. Since SCBench evaluates enhanced long- contett retrieval capabilities, we set context lengths to 500, 2000, and 8000 tokens, inserting the needle at positions corresponding to quantiles ranging from 0 to at intervals of 0.1 for comprehensive evaluation.
B Broader Impacts and Limitations
Broader Impacts. Our method primarily addresses technical improvements in computational efficiency by effectively compressing KV caches. Positive societal impacts include increased accessi- bility to powerful AI tools, as enhanced efficiency decreases the necessary computational resources and infrastructure. This broader accessibility can democratize Al applications in various fields such as education, scientific research, and healthcare, benefiting communities previously limited by resource constraints. While our method specifically targets technical efficiency, we acknowledge potential changes in model behavior due compression, as analyzed in Table 1.
Limitations. Our study primarily adopts an empirical approach and does not include theoretical guarantees concerning compression-induns information loss. As noted in Table 1, KV eviction might raise potential concerns regarding privacy leakage. Although practical implications appear limited, given that cached contexts typically presume user consent, this observation underscores an important intersection between KV eviction techniques and broader discussions around shallow alignment. Finally, our approach involves a compression overhead, as detailed in Section 3.4. This overhead can be amortized over multiple queries. While context- context-nndependent head- eviction strategies can effectively eliminate overhead at deployment, their compression efficiency generally falls short compared to context-dependent approaches, as shown in Figure 11.
C Analysis and Experiments
C.1 Reconstruction Chunk Size
Figure 14 analyzes how scoring chunk size m influences performance. Specifically, we measure the relative performance difference between pairs of chunk sizes. For instance, the relative difference between chunk sizes 1K and 2K equals |Pix Pxx/ Pax. where p denotes performance at each chunk size. Results indicate average performance differences remain below 2% 0.3 KV cache ratio, confirming negligible impact. Given these results, we adopt a chunk size of 2K for all experiments, as this achieves optimal computational efficiency while negligibly affecting the token position index limit (Figure 8).
Relative difference
0.05
IK 2K
0.04
4K 2K
0.03
SK 2K
0.02
0.01
0.00
0.2
0.4
0.6
0.8
1.0
KV cache ratio
Figure 14: Relative performance differences for varying scoring chunk sizes, averaged over SCBench datasets with LLaMA3.1-8B.
C.2 Repeat Prompts
In our experiment, we use the repeat prompt: "Repeat the previous context:"" This choice is motivated by simplicity, as the specific wording of the repeat prompt has minimal impact on overall performance.
'https://githnn wttpccubb.coo///aaaaa////oo wwwteotttttttaaeeaoo Data--ngineering
22

---

To validate this, we conduct experiments comparing the original repeat prompt, paraphrased version, and no repeat prompt. Table shows that our method is robust variations in the repeat prompt; even without the repeat prompt, context reconstruction remains effective. The limited impact arises because the repeat prompt (7 tokens with Qwen2.5-7B tokenizer) significantly shorter than overall context (at least several hundred tokens), thereby minimizing the effect on compression.
To further clarify this, we analyze attention patterns. Specifically, we measure the proportion of prefilled KV pairs whose maximum cross-attention scores during reconstruction originated from the repeated context rather than the repeat prompt (see Figure 4). For a 2K token-length context from NIAH, 98.1 % of KV pairs have their maximum attention from the repeated context. Among the KV pairs retained after 30% compression, 99 %.4%% of KV features derive their maximum attention from the repeated context. These findings confirm the minimal influence of the repeat prompt on K KVzip importance scoring.
Table 2: Test performance of Qwen2.5-7B SQuAD at 30% KV cache ratio. Note, SnapKV achieves 32.1 15% in this setting.
Repeat prompt type
Accuracy (%)
Original ("Repeat the previous context:")
94.37
Paraphrased ("Reproduce the preceding context without any changes.")
94.45
No ("An'n")
94.25
C.3 Softmax-Free Importance Scoring
In Algorithm we use the Softmax-normalized attention scores the KV importance scores. To obtain query and key vectors at each layer, we forward the repeated input through fiM using FlashArtention. Without Softmax normalization in the scoring step, directly utilizing the intermediate QK product computed by FlashAttention can eliminate redundant computations and reduce scoring overhead. Accordingly, we develop a variant of KVzip without the Softmax normalization by implementing a custom Triten-based FlashAttention CUDA kernel.
In Algorithm 1, the scoring procedure accounts for approximately 10% of the total forward com- putation time using fLM- Our Softmax- version integrates this scoring procedure directly into the fused attention kernel, reducing the 10% of overhead. However, illustrated in Figure 15, omitting Softmax normalization results in approximately a 10% degradation in compression ratios. Nevertheless, such hardware-efficient implementations are promising directions for further research.
60
Accuracy (%)
10
20
KVzip
KVzip-logit
0.2
0.4
0.6
0.8
1.0
KV cache ratio
Figure 15: Performance of the Softmax-free variant of KVzip (logit) on Retr.KV in SCBench with LLaMA3.1-8B.
C.4 Uniform KV Head Budgets
Figure 16 compares the performance of uniform head-budget allocation with the non-uniform allocation adopted the main experiments. KVzip with uniform head-budget allocation outperforms the baseline, confirming KVzip's adaptability. However, non-uniform allocation achieves superior compression performance-come consistent with previous findings by Feng et [17]-by more effectively capturing variations in importance across heads, illustrated in Figure 13.
23

---

Accuracy (96
80
60
KVzip
40
KVzip-unit
SnipKy
SnapKV-unif
20
0.2
0.4
0.0
0.8
1.0
KV cache ratio
Figure 16: Performance comparison using non-uniform and uniform head-budget allocations SQuAD with LLaMA3.1-8B. Unif. refers to the uniform allocation.
D Individual Dataset Performance
Model Scale and Architecture. Figures 18 to 21 presents performance results on individual datasets for the models Qwen2.5-14B-IM [54], LLaMA3. 1-8B [21], Gemma3-12B [49], and LLaMA3-8B- W8A8KV4 [36].
For the Gemma model, Retr.KV and Retr.Prefix-Sufii exceed the maximum context length of 128K tokens, reaching approximately 170K tokens and consequently producing an accuracy of Thus, we create shortened dataset versions, reducing contexts about one-fifth of their original length.
Regarding LLaMA3-8B- W8A8KV4, the base LLLMMA3-BB model lacks capability to solve RV Setr.Prefix-Suffix and Math.Find tasks, resulting in near-zero accuracy. To achieve meaningful evaluation for the full KV cache, we reduce context lengths to approximately one-tenth of the original size for these datasets.
Multi-Task Task Datasets. Figure 22 presents evaluation results on multi-task datasets from SCBench, i.e. Mix.SummNIAN and Mix.RepoQA+KV, each composed of two distinct tasks [35]. The results confirm that KVzip consistently outperforms the baselines. Figure 23 presents results for LLaMA3.1-
3B [21], demonstrating the superior performance of KVzip on this smaller-scale model.
RULER Benchmark. further highlight KVzip's effectiveness, we present results on the RULER benchmark [23]. These results publicly available by the NVIDIA KVPress repository Figure 17 demonstrates that KVzip significantly outperforms current state-of-the-art KV eviction methods, main- taining performance 25% compression rate, whereas others experience significant performance degradation.
100
Accuracy (%)
80
60
KVzip
10
AuoAAtention
SnapKV
PyramidKV
20
0.10
0.25
0.50
0.75
0.90 1.00
KV cache ratio
Figure 17: Average performance on the RULER benchmark using Qwen3-8B.
24

---

Kzzip (ours)
H₂O
SnapKV
PyramidKV
NIAH
Retr.VV
Retr: PettPPreix--uiie Suffixx
Code.RepoQ4
100
80
Retrieval
Accuracy (%)
80
P (%)
60
60
Accuracy
60
Accuracy
10
40
40
40
20
20
20
0.2 0.4 0.6 0.8 1.0
0.2 0.4 U.6 0.8
1.0
0.2 0.4
0.6 0.8
1.0
0.2 0.4 0.6 0.8 1.0
KV cache ratio
KV cache ratio
KV cache
KV eache ratio
SQuAD
GSM8K
En.QA
En.MultiChoice
Contextual QA
Accuacy (%)
100
80
85
60
Accuracy
60
Accuncy
40
Accurac
75
80
40
20
40
30
70
65
20
60
0.2
0.4 0.6 0.8
0.2 0.4 0.8
1.0
0.4
0.6 0.8
0.2
0.4
0.6
0.8
1.0
KV cache ratio
KV cache ratio
KV cache ratio
KV cache ratio
En. Summary
Retr. MultiHu
Math.Find
ICL.MannyShot
Redundancy
40
50
ROUGIE (9)
Accuracy (5)
Accurcy (56
15
35
10
Accuracy
40
20
30
20
10
35
30
0.2
0.4 0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2 0.4
0.6
0.8
1.0
0.2
0.4 0.6
0.8 1.0
KV cache ratio
KV eache ratio
cache ratio
KV cache ratio
Figure 18: Benchmark results using Qwen2.5-14B. 1M [54] across compression ratios from 0.1 to 1.0.
KVzip (ours)
H₂O
SnapKV - PyramidKV
NIAH
Retr.KV
Prelix -uffixx
Codc. Repe.RepoQA
Retrieval
Accurcy
100
80
Accuracy (56
10
60
Accuracy
20
30
Pass@l (9)
30
10
40
10
20
20
10
0.2 0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6 0.8
1.0
0.2
0.4
0.6
0.8
1.0
KV cache ratio
KV cache ratio
KV cache ratio
KV cache ratio
SQuAD
GSM8K
En. SA.QA
En.MultiChoice
Contextual QA
100
50
80
Accurccy (96
80
Accurccy (38)
60
Accuncy (9)
70
60
40
Accuncy
355
60
20
50
20
0.2 0.4 0.6 0.8 1.0
0.2
0.4 U.6 0.8 1.0
0.2
0.4
0.6 0.8
1.0
0.2
0.4 0.6 0.8 1.0
KV ratio
KV cache ratio
cache
KV cache ratio
En. Summary
Retr.Multtilop
Math.Find Find
ICL.ManyShot
Redundancy
40
20
ROUGE (%)
38
Accuracy (%)
15
Accuracy (36)
35
36
20
Accurscy
10
30
34
32
25
30
0.2
0.6
0.8
0.2
0.4
0.8
1.0
0.4
0.6
0.2 0.1
0.8
KV cache ratio
KV cache ratio
KV cache ratio
KV cache ratio
Figure 19: Benchmark results using LLaMA3.1-8D [21] across compression ratios from 0.1 to 1.0.
25

---

Kvzip (ours)
H₂O
SnapKV
PyramidKV
NIAH
RV
Retr. Petti.Preix-Sute
Repo.R
30
40
Retrieval
Accuracy (%)
20
40
P (%)
30
60
Accuracy
30
40
20
20
10
10
0.2 0.4 0.6 0.8 1.0
0.2 0.4
0.8
1.0
0.2 0.4
0.6
0.8
1.0
0.2
0.4 0.6 0.8 1.0
KV cache ratio
KV eache ratio
KV cache
KV eache ratio
SQuAD
GSM8K
En.QA
En.MultiChoice
Contextual QA
100
Accuacy (%)
Accuracy (%)
70
Accuracy
60
AccuRcy
60
10
25
60
40
20
55
0.2 0.4 0.6 0.8
0.2
0.4
0.8
1.0
0.4 0.6 0.8
0.2
0.4
0.6 0.8
1.0
KV cache matio
KV cache ratio
KV cache ratio
KV cache ratio
En. Summary
Retr.MulttHop
Math.Find
ICL.MannyShot
Redundancy
30
ROUGIE (9))
30
Accuracy (5s)
Accuracy (%)
20
Accuracy
:25
26
10
20
45
15
40
0.2 0.4 0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4 0.6 0.8 1.0
KV cache ratio
eache ratio
cache ratio
KV cache ratio
Figure 20: Benchmark results using Gemma3- 12B [49] across compression ratios from 0.1 1.0.
KVzip (ours)
H₂O
-
SnapKV -
PyramidKV
NIAH
Retr.V
Suffixx
Code.Repoor
60
100
Retrieval
(%)
80
40
Pawe1 (36)
60
Accuracy
6
Accurccy
12 00
40
KV cache ratio
KV cache ratio
KV cache ratio
KV cache ratio
SQuAD
GSMBK
QA
En.MultiChoice
Contextual QA Accuracy
100
40
80
15
Accuracy (S6)
60
Accuracy
Accuracy
30
40
25
30
20
20
0.2 0.4 0.6 0.8 1.0
0.2
0.4 0.6 0.8 1.0
0.2
0.4
0.6 0.8
1.0
0.2
1.0 0.4
0.6
0.8
KV cache matio
KV cache ratio
KV cache ratio
KV cache ratio
EummSrrmayy
Retr. MulttHop
Math.Find
MCn.ManyShot
Redundancy
ROUGI((S) (8)
36
Accurscy ($6)
Accuracy (%)
30
Accurccy (96)
30
34
20
32
10
10
10
30
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
cache
Figure 21: Benchmark results using LLaMA3-8B- W88SKV4 [36] across compression ratios from
0.1 to 1.0.
26

---

Kzzip (ours)
H2O
SnapKV
Py
Mix.RepoQA+KV
NIA
Phss & Accuracy (%)
ROUGE Accuracy (36)
80
60
60
40
40
20
20
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
KV cache ratio
KV cache ratio
Figure 22: Benchmark results on SCBench multi-task datasets using Qwen2.5-7B-IM [54] across compression ratios from 0.1 to 1.0.
KVzip (ours)
H₂O - SnapKV
PyramidKV
NIAH
SQuAD
GSMSK
100
100
Accuracy
60
Accuracy
60
41
40
20
20
20
0.2
0..4
0.66
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
KV cache ratio
KV cache ratio
KV cache ratio
Figure 23: Benchmark results for LLaMA3.1-3B [21] across compression ratios ranging from 0.1 to 1.0. The evaluation focuses shorter contexts, as LLaMA3.1 1-3B lacks the capability to solve SCBench tasks, resulting in near-zero accuracy.
27

---

Table Inputs for KV cache importance scoring from SQuAD example (used in the visualizations in Figure 6 and Figure 13). The context is included in the input of the repeat task.
Task
Inputs for scoring (prompt-response)
Repeat
Repeat the previous context: Architecturally, the school has a Catholic character. Atop the Main Buildings gold dome is golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is copper statue of Christ with Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica the Grotto, Marian place of prayer and reflection. It is replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared Saint Bernadette Soubirous 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.
upraised with the legend "Venite Ad
QA-1
Q: To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France? A: Saint Bernadette Soubirous
QA-2
Q: What is in front of the Notre Dame Main Building? A: a copper statue of Christ
Summarization
The school has a Catholic character with various statues and buildings that reflect this theme, including gold dome with a Virgin Mary statue, Christ statue, a Basilica of the Sacred Heart, Grotto replica of Lourdes, and modern stone Mary statue at the end the main drive.
Please summarize the previous context:
Table 4: Inputs for importance scoring from GSM8K example used in the visualization in Figure 6, a reasoning task. The context is included in the input of the repeat task.
Task
Inputs for scoring (prompt-response)
Repeat
Repeat the previous context: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg.
Reasoning
Reason and answer the question. Q: How much in dollars does she make every day at the farmers' market?
Janet's ducks lay 16 eggs per day. She eats eggs for breakfast, she has 16- = 13 eggs left. She bakes eggs for muffins, so she has 13 4 4 9 eggs left. She sells the remaining 9 eggs the farmers' market for $2 each. To find out how much she makes, we multiply the number of eggs she sells (9) by the price per egg ($2): $9 $2 $18. The answer is $18.
28

---

(a) En.QA (main)
(b) DuoAttention
(c) En.QA (sample-2)
(d) Code.RepoQA
(e) Retr.KV
0.3
1.66
Layen
0.2
0.0
boads
KV
heads
Figure 24: Visualization of head-level importance scores for context-independent compression in Section 4.2. We use the head scores obtained from an En. En.QA example in our primary experiments (Figure 11). For reference, (c)-(e) show head scores derived from alternative data sources from SCBench [35]. Our scoring method yields more uniformly distributed importance pattern compared to DuoAAtention. We select the En. En.QA sample for our main experiments due to its comprehensive overlap with importance patterns from other data sources, whereas Retr. KVV. composed of synthetic passkeys, exhibits sparser importance patterns.
29


Reviews:
[{"review_id": "O982BByMkM", "reviewer": "Reviewer_yysm", "summary": "KVzip proposes a query-agnostic eviction scheme for long-context LLMs that uses a prompt of the form [context + ‘Repeat the previous context’ + context], and then uses (max-pooled) attention scores of the second copy of the context (on the first copy of the context) to generate ‘importance scores’ for each KV pair. The top-r% highest scoring KV pairs are retained. The aforementioned procedure is computationally expensive, so the authors propose chunking the second copy of the context and using multiple passes through the model to generate the scores. They empirically show that this scoring procedure outperforms baseline methods (PyramidKV, SnapKV, H2O) on a variety of benchmark tasks. They show that using a smaller KV cache (with any eviction method) provides non-trivial latency improvements (section 3.4). They show that KVzip produces a significant increase in prefilling latency (section 3.4), and demonstrate the entire second context copy is needed for KVzip to provide good performance (Section 4.3). Lastly, they show that utilizing KVzip can modify the behavior tuning of pre-trained models with a privacy-leaking example.", "questions": "What inference platform/engine/package was used to determine the numbers in Figure 8?\n\nCan you provide additional “Computational analysis” (Figure 8) for larger models (20B+)? Running all the benchmark evaluations isn’t necessary, but it would be nice to know how the performance scales. \n\nDo you have any additional insights into how the latency of the method can be improved (would a fused kernel help significantly here)? \n\nHow was the exact form of the prompt settled on and what alternatives were tried? Normally this wouldn’t be important, but given the reliance of the model on this particular prompt, it would be nice to know.\n\nThe NIAH you selected is quite easy (it is an obviously incongruous phrase inserted into a random essay); how does KVzip perform on a more difficult task   (multikey, multivalue, uuid retrieval from RULER)?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nQuery-agnostic compression is an important scenario that has good practical importance \n\nThe proposed solution is simple, and would be easy to reproduce\n\nThe paper is clearly written, the method is well explained, and there is a good variety of experiments to support their claims. The figures \nand captions are clear and well done (Figure 6 especially). \n\nThe section on context-independent eviction obviates the need for expensive procedures for generating head-level eviction policies\n\nWeaknesses\n\nThe solution is not especially novel; it is essentially an application of SnapKV to a particular prompt structure, with speed improvements provided by chunking. The text-reconstruction/auto-encoder approach is also not new, though I haven’t seen this exact approach in the parameter-free setting before.  \n\nFor practical applications, the prefill latency increase induced by KVzip is incredibly high, and is probably untenable in many scenarios. Also, the number of forward passes through the model (while conditioning on the whole–supposedly long–context that needs to be compressed) scales linearly as context length grows, especially with high parameter count models. This is a major limitation of the KVzip.", "was_revised": true, "final_justification": "My response is largely unchanged but the authors have done their best to respond to those concerns. If the paper were to be accepted, their additional materials here would bolster the presentation of the work, which I continue to find on the edge of practicality.", "rebuttal": "Dear reviewer, thank you for your valuable time and effort in reviewing our paper. We have addressed the questions and feedback raised below. We will carefully incorporate the discussion into our revision. \n\n---\n\n**Q) The solution is essentially an application of SnapKV to a particular prompt structure.**\n- We note that our method introduces a query-independent objective, **proactively identifying** crucial KV pairs for arbitrary future queries. This represents a clear methodological distinction from prior works, such as SnapKV and H2O, which compress based solely on the current query, leading to **query-specific overfitting**. This distinction significantly enhances practical utility: KVzip maintains inference accuracy at a 30% compression rate, whereas prior methods exhibit severe performance degradation in query-agnostic scenarios (e.g., a 60% accuracy drop on the SQuAD dataset with Qwen2.5-7B at the same compression rate, see Figure 9).\n---\n\n**Q) The reconstruction approach is also not new, though I haven’t seen this exact approach in the parameter-free setting before.**\n- While reconstruction-based objectives are well-established in representation learning, we emphasize that connecting this principle explicitly to parameter-free KV compression is novel, as acknowledged by the reviewer. This **novel connection** significantly enhances practical applicability in query-agnostic scenarios, enabling near-lossless compression at a 30% rate. In contrast, prior methods incur notable performance degradation even at a much higher 90% compression rate (see Figure 9).\n---\n\n**Q) The prefill latency increase induced by KVzip is incredibly high, and is probably untenable in many scenarios.**\n- We emphasize that KVzip unlocks new application scenarios beyond the conventional online KV eviction setting. Specifically, KVzip produces reusable, accurately compressed KV caches suitable for multiple future queries, thus supporting **offline context KV cache compression**. This capability is particularly beneficial in real-world use cases such as enterprise document retrieval–augmented systems or personalized user contexts (lines 34–37), where the **compression overhead can be amortized** over subsequent queries. Existing dynamic KV eviction methods, however, incur significant performance degradation in multi-query scenarios, limiting their applicability in such scenarios (Figure 9).\n- Moreover, even in online settings, KVzip's importance-scoring method seamlessly extends to head-level eviction, introducing **no additional compression overhead** during deployment (see Context-Independent Eviction, Section 4.2). Compared to DuoAttention, a state-of-the-art head-level eviction technique, KVzip provides superior accuracy with substantially lower optimization costs (lines 270–275 and Figure 11).\n---\n\n**Q) What inference platform/engine/package was used to determine the numbers in Figure 8?**\n- We used PyTorch with Hugging Face Transformers and FlashAttention2 on an NVIDIA A100 GPU. This setup follows the convention used in previous works such as SnapKV, PyramidKV, and KVpress.\n---\n\n**Q) Can you provide additional “Computational analysis” (Figure 8) for larger models (20B+)?** \n- Following your suggestion, we conducted computational analysis using **Qwen3-32B (FP8)** with a context length of 124K tokens on an NVIDIA H100 GPU. (Note: Qwen3’s FP8 quantization does not support the A100 GPU used previously.) We employed identical software packages as used in Figure 8.\n- The table below demonstrates that KVzip maintains similar **efficiency gains** with the larger model, achieving approximately a 2$\\times$ speed-up in decoding attention latency at a 20% compression rate.   \n| Compression ratio | Decoding attention latency (ms) | KV cache size (GB) |\n|---|---|---|\n|1|0.45|32.7|\n|0.8|0.39|26.2|\n|0.6|0.34|19.6|\n|0.4|0.28|13.1|\n|0.2|0.23|6.5|\n\n  *Table. Computational efficiency analysis of KVzip during decoding.*\n\n- Regarding **compression overhead**, KVzip introduces approximately 1.5$\\times$ computational overhead, slightly lower than the 2$\\times$ overhead observed with the 8B model in Figure 8. Specifically, KVzip’s chunked context reconstruction process requires 1$\\times$ MLP FLOPs and 2$\\times$ attention FLOPs relative to prefilling. As model size increases, the relative computational contribution of MLP layers grows, thereby reducing the relative overhead of the scoring process. \n- Memory overhead from KVzip’s scoring process **remains negligible** compared to prefilling (0.8% increase in peak memory), consistent with results observed on smaller models.  \n|Stage|Compute time (s)|Peak GPU memory (GB)|\n|---|---|---|\n|prefill|3m 56s|62.7|\n|scoring (compression)|5m 25s|63.2|  \n\n  *Table. Computational comparison of prefilling and KV importance scoring stages. We use a chunk size of 2K for both stages to ensure a fair comparison.*\n---\n\n**Q) Do you have any additional insights into how the latency of the method can be improved (would a fused kernel help significantly here)?** \n- We explored various strategies to reduce compression overhead. While a tailored fused CUDA kernel could help reduce latency, the benefit is relatively modest (~10%, line 528 in Appendix) since the primary overhead arises from the FLOPs by the Transformer forward-pass on repeated contexts. \n- One such solution in our paper to further reduce latency is **context-independent eviction** (Line 257), which eliminates additional latency at deployment. \n- We also tested the method that **interleaves** chunked prefilling and compression (i.e., compress the KV cache of the i-th chunk before prefilling the next chunk) to reduce overall attention FLOPs during prefilling. However, this led to decreased compression performance. In our paper, we prioritize achieving optimal KV-cache compression quality and present head-level context-independent eviction as a practical alternative. Exploring further improvements in prefilling latency remains part of our future work.\n---\n\n**Q) How was the exact form of the prompt settled on and what alternatives were tried?** \n- In our code, we used the repeat prompt: \"\\n\\nRepeat the previous context exactly.\" This choice was motivated by simplicity, as the specific wording of the repeat prompt has **minimal impact** on overall performance.\n- To validate this, we conducted experiments comparing: (1) the original repeat prompt, (2) a paraphrased version (\"Reproduce the preceding context without any changes.\"), and (3) no repeat prompt (just \"\\n\\n\"). Note that we follow the chat template provided by each model (e.g., \"\\<user> [context] [repeat prompt] \\<assistant> [repeated context]\").  \n|Repeat prompt type|Accuracy (%)|\n|---|---|\n|Original|94.37|\n|Paraphrased|94.45|\n|No|94.25|\n\n  *Table. Accuracy (%) of Qwen2.5-7B on SQuAD at a 30% compression rate (70% eviction). SnapKV achieves **32.15%** in this setting.*\n\n- These results show that our method is **robust** to variations in the repeat prompt. The limited impact arises because the repeat prompt (7 tokens with Llama 3.1) is significantly shorter than the overall context (at least several hundred tokens), thereby minimizing its effect on compression.\n- To further clarify this, we analyzed **attention patterns**. Specifically, we measured the proportion of prefilled KV pairs whose maximum cross-attention scores during reconstruction originated from the repeated context rather than the repeat prompt (see Figure 4). \n  - For a 2K token-length context from NIAH, 98.1% of KV pairs had their maximum attention from the repeated context. Among the KV pairs retained after 30% compression, **99.4%** derived their maximum attention from the repeated context. These findings confirm the minimal influence of the repeat prompt on KVzip’s importance scores and compression performance.\n- It is also interesting to observe that context reconstruction without a repeat prompt achieves comparable performance. This indicates that forwarding the context twice effectively captures the critical attention patterns required for downstream tasks (see Figure 6).\n---\n\n**Q) The NIAH you selected is quite easy; how does KVzip perform on a more difficult task (multikey, multivalue, uuid retrieval from RULER)?**\n- Following your recommendation, we evaluated KVzip on **RULER**. Specifically, we integrated KVzip into NVIDIA’s KVpress framework, which benchmarks state-of-the-art KV eviction methods under a query-agnostic setting with a 4K context length RULER using LLaMA3.1-8B-Instruct.\n- The table below compares KVzip to SnapKV on the suggested challenging tasks from RULER. Results indicate that KVzip maintains **perfect performance** at a 25% compression rate, whereas SnapKV experiences significant performance degradation.  \n|task|SnapKV|KVzip|Full KV cache|\n|---|---|---|---|\n|Multi key-uuid|1.6|100.0|100.0|\n|Multi value|25.3|99.9|99.9|\n\n  *Table. RULER-4K performance (%) at a 25% compression ratio.*\n\n- To further highlight KVzip's effectiveness, we compared our method with various benchmark results from KVpress, which include **state-of-the-art methods** such as SnapKV, PyramidKV, DuoAttention, and other **concurrent unpublished methods**. While these results are publicly available on Hugging Face, the link is excluded here due to rebuttal policy constraints. The following table demonstrates that KVzip significantly outperforms current state-of-the-art KV eviction methods:  \n|Compression ratio|SnapKV|Best from KVpress|KVzip|\n|---|---|---|---|\n|0.5|67.0|91.6|95.5|\n|0.25|39.9|74.3|95.3|  \n\n  *Table. RULER-4K average performance (%) over 13 tasks. The full cache performance is **95.5**%.*\n\n- Lastly, our primary experiments in Figure 9 demonstrate KVzip's capability to handle significantly **longer contexts** (up to 170K tokens) on SCBench, which also includes representative tasks from RULER."}, {"review_id": "8l5EgXIJmR", "reviewer": "Reviewer_edTh", "summary": "This paper proposes a Key-Value eviction method called KVzip. The approach preserves the most informative key-value vectors by ensuring that the compressed key-value cache enables the LLM to reconstruct the original text context.", "questions": "Please refer to Weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n1. Intuitive and well-motivated approach: The method's core intuition is reasonable and straightforward. The experimental evaluation demonstrates competitive performance across numerous downstream tasks, validating the effectiveness of the proposed approach.\n\n2. Comprehensive experimental validation: The authors provide extensive experiments to demonstrate the method's effectiveness, including evaluations across multiple models, datasets, and compression ratios. The empirical analysis is thorough and convincing.\n\n[Weaknesses]\n1. Significant computational overhead: The key-value eviction process itself is computationally expensive and requires substantial time overhead. As shown in Figure 8, the compression overhead is more than 3 times the original prefilling time. Therefore, for long document inputs, this method would only be cost-effective when there are at least 3 or more queries per document to amortize the compression cost.\n\n2. Limited evaluation scope: The authors' evaluation focuses primarily on scenarios with one document, same task type, and multiple different queries. However, the paper lacks evaluation of scenarios involving one document with multiple queries from different task types, which would provide a more comprehensive assessment of the method's generalizability across diverse query patterns.", "was_revised": true, "final_justification": "I have read the response and raised my score to 5. The authors provide more experimental results, which can resolve my concerns.", "rebuttal": "Dear reviewer, thank you for your valuable time and effort in reviewing our paper. We have addressed the questions and feedback raised below. We will carefully incorporate the discussion into our revision. \n\n---\n\n**Q) The key-value eviction process itself is computationally expensive and requires substantial time overhead.**\n- We emphasize that KVzip unlocks new application scenarios beyond the conventional online KV eviction setting. Specifically, KVzip produces reusable, accurately compressed KV caches suitable for multiple future queries, thus supporting **offline context KV cache compression**. This capability is particularly beneficial in real-world use cases such as enterprise document retrieval–augmented systems or personalized user contexts (lines 34–37), where the **compression overhead can be amortized** over subsequent queries. Existing dynamic KV eviction methods, however, incur significant performance degradation in multi-query scenarios, limiting their applicability in such scenarios (Figure 9).\n- Moreover, even in online settings, KVzip's importance-scoring method seamlessly extends to head-level eviction, introducing **no additional compression overhead** during deployment (see Context-Independent Eviction, Section 4.2). Compared to DuoAttention, a state-of-the-art head-level eviction technique, KVzip provides superior accuracy with substantially lower optimization costs (lines 270–275 and Figure 11).\n---\n\n**Q) The authors' evaluation focuses primarily on scenarios with one document, same task type, and multiple different queries. However, the paper lacks evaluation of scenarios involving one document with multiple queries from different task types.**\n- Thank you for the pointer. In the Appendix, Figure 22 (described in text in lines 555-558), we present experimental results for the **multi-task, multi-query setting**. The evaluation uses two multi-task benchmarks from SCBench: Mix.RepoQA+KV and Mix.Sum+NIAH [1]. The following highlighted results demonstrate that KVzip consistently outperforms the baselines in a multi-task setting:   \n| task            | H2O  | SnapKV | PyramidKV | KVzip | Full KV cache |\n|-----------------|------|--------|-----------|-------|---------------|\n| Mix.RepoQA+KV   | 3.1  | 6.0    | 1.9       | 81.1  | 81.5          |\n| Mix.Sum+NIAH    | 45.4 | 66.2   | 64.5      | 67.5  | 68.5          |\n\n  *Table. Multi-task, multi-query performance (%) at a 30% compression ratio using Qwen2.5-7B.*\n\n- Notably, KVzip is not only query-agnostic but also **task-agnostic**, as the compression procedure does not depend on task types. This implies that our main experiment in Section 4 using single-task benchmarks spanning 12 different tasks effectively demonstrates the task-generalizability of KVzip, which is why we placed the additional results on multi-task benchmarks in the Appendix. We will highlight these results in the main text in our revision.\n\n[1] Li et al., SCBench: A kv cache-centric analysis of long-context methods. ICLR, 2025."}, {"review_id": "zx1mODz94R", "reviewer": "Reviewer_bWgi", "summary": "Unlike recent works that rely on queries to assess the importance of Key-Value pairs, this paper introduces KVzip, a technique for quantifying the importance of KV pairs, then evicting the less important ones to effectively compress the context with minimal information losses. This method enables LLMs to perform a single prefill-eviction process, after which they can handle multi-queries without additional prefilling stage.\nKVZip is especially beneficial to tackle the challenges of KV cache compression in multi-query scenarios.", "questions": "Please refer to the questions in the weaknesses part.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n1.\tThis paper introduces an innovative and effective method to probe the actual importance of KV pairs. Prior works only utilizes the attention scores during the prefilling stage, but this paper utilizes the “repeat generation” task to actually see what the model needs to reconstruct the whole context, and experiments have shown superior results.\n\n2.\tThe formulation, writings and figures of this paper is clear. I can clearly understand what each figure is showing immediately when I read the caption of the figure.\n\n3.\tThis paper has a thorough investigation of recent works (H2O, FastGen, SnapKV, PyramidKV and DuoAtention), giving it a solid technical background.\n\n4.\tThe experiments involve a wide range of benchmarks, ranging from retrieval-intensive tasks, contextual QA to tasks with high context redundancy. The evaluated model includes Qwen2.5-7B, Qwen2.5-14B, LLaMA3.1-8B, and Gemma3-12B. This broad evaluation gives it a solid support to claim the effectiveness of KVZip.\n\n## Weaknesses\n1.\tThe choice of the reconstruction prompt—“Repeat the previous context:”—is insufficiently justified and appears overly heuristic. This paper doesn’t explore or compare alternative formulations of the reconstruction prompt.", "was_revised": true, "final_justification": "Few changes are made to my review. My concerns are well addressed by the authors in the rebuttal session. I would like to keep my intention for accepting this paper, as it is a technologically solid paper with innovation.", "rebuttal": "Dear reviewer, thank you for your valuable time and effort in reviewing our paper. We have addressed the questions and feedback raised below. We will carefully incorporate the following discussion into our revision.\n\n---\n**Q) The choice of the reconstruction prompt—“Repeat the previous context:”—is insufficiently justified and appears overly heuristic. This paper doesn’t explore or compare alternative formulations of the reconstruction prompt.**\n\n- This choice of repeat prompt was motivated by simplicity, as the specific wording of the repeat prompt has **minimal impact** on overall performance.\n- To validate this, we conducted experiments comparing: (1) the original repeat prompt, (2) a paraphrased version (\"Reproduce the preceding context without any changes.\"), and (3) no repeat prompt (just \"\\n\\n\"). Note that we follow the chat template provided by each model (e.g., \"\\<user> [context] [repeat prompt] \\<assistant> [repeated context]\").  \n|Repeat prompt type|Accuracy (%)|\n|---|---|\n|Original|94.37|\n|Paraphrased|94.45|\n|No|94.25|\n\n  *Table. Accuracy (%) of Qwen2.5-7B on SQuAD at a 30% compression rate (70% eviction). SnapKV achieves **32.15%** in this setting.*\n\n- These results show that our method is **robust** to variations in the repeat prompt. The limited impact arises because the repeat prompt (7 tokens with Llama 3.1) is significantly shorter than the overall context (at least several hundred tokens), thereby minimizing its effect on compression.\n- To further clarify this, we analyzed **attention patterns**. Specifically, we measured the proportion of prefilled KV pairs whose maximum cross-attention scores during reconstruction originated from the repeated context rather than the repeat prompt (see Figure 4). \n  - For a 2K token-length context from NIAH, 98.1% of KV pairs had their maximum attention from the repeated context. Among the KV pairs retained after 30% compression, **99.4%** derived their maximum attention from the repeated context. These findings confirm the minimal influence of the repeat prompt on KVzip’s importance scores and compression performance.\n- It is also interesting to observe that context reconstruction without a repeat prompt achieves comparable performance. This indicates that forwarding the context twice effectively captures the critical attention patterns required for downstream tasks (see Figure 6)."}, {"review_id": "DXVVvFqqDH", "reviewer": "Reviewer_64er", "summary": "This paper introduces KVZip, a query-agnostic KV cache compression method designed for practical deployment scenarios. KVZip leverages an autoencoding concept to compress KV cache robustly without requiring knowledge of future queries. The approach is based on the observation of sparse attention patterns that emerge during the reconstruction process of a given context. Since the method is query-independent, it eliminates the need for re-prefilling when handling diverse queries on the same context, demonstrating robust and superior performance compared to existing state-of-the-art methods across various tasks and compression ratios.", "questions": "- *Detail* Could you provide a more detailed explanation of the intuition in Section 3.1, particularly lines 104-106 and 107-111? I would like to understand this core concept more clearly and deeply.\n- *Detail* In Section 4.2's comparison with DuoAttention, could you specify exactly how head-level scores were measured and computed?\n- *Related to Weakness 1* When evaluating with LLaMA-3.1's 128K context length, did positional out-of-distribution issues during the reconstruction process cause any problems? How was this addressed? (what am I missing?)\n- *Suggestion* Have you compared using sum/average aggregation methods instead of max values for attention scores? Would this significantly impact performance? Additionally, is the repeat prompt robust to alternative prompt designs? A simple ablation study with basic aggregation methods and variation of repeat prompts would strengthen the paper.\n- *Suggestion* The following recent works related to query-agnostic approaches might be valuable additions to the related work section; \n   - Corallo et al, Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoning\n   - Kim et al, InfiniPot: Infinite Context Processing on Memory-Constrained LLMs", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-written with clear presentation, making it easy to follow the methodology and contributions.\n- The application of autoencoding concepts to KV cache compression appears highly novel, yielding robust performance results that consistently outperform existing baselines. The experimental evaluation is comprehensive, covering diverse tasks and baselines, which convincingly demonstrates the method's advantages.\n- The discussion of privacy implications in KV cache compression provides a novel perspective that offers valuable insights to the research community.\n\nOverall, the paper demonstrates high completeness with an innovative core idea and thorough reporting of both performance and efficiency aspects. This work appears to make a meaningful contribution to addressing the KV cache bottleneck in practical LLM deployment.\n\n**Weakness**\n- If my understanding is correct, the autoencoding-based scoring approach may face limitations with positional encoding when processing very long contexts. For instance, if LLaMA-3.1 supports up to 128K tokens, compressing a 128K context would require repeated input during reconstruction, potentially causing positional out-of-distribution issues. This raises concerns about whether the model can effectively utilize positional indices beyond its training length (requiring position IDs up to twice the target length for computing n_c / n_c + n_i attention maps).\n- As acknowledged by the authors, KVZip is inherently prefill-heavy due to its autoencoding nature, creating somewhat significant compression overhead. While chunked prefill can reduce complexity, the method still generates KV cache footprint or lifespan [1][2] equivalent to original length plus repeated teacher-forced context. This characteristic may limit its applicability in resource-constrained on-device environments where prefill resources are highly restricted [3].\n\n[1] Bhaskar et al, Cache me if you can: how many kva do you need for effective long-context lms?, 2506.17121  \n[2] Li et al, SCBench: A KV Cache-Centric Analysis of Long-Context Methods, ICLR25  \n[3] Kim et al, InfiniPot: Infinite Context Processing on Memory-Constrained LLMs, EMNLP 24", "was_revised": true, "final_justification": "The authors have addressed all of my questions and concerns, and upon reviewing the other reviewers' opinions, I find that my rationale for acceptance aligns with aspects shared by other reviewers. Therefore, I will maintain my current acceptance.", "rebuttal": "Dear reviewer, thank you for your valuable time and effort in reviewing our paper. We have addressed the questions and feedback raised below. We will carefully incorporate the following discussion into our revision. \n\n---\n**Q) Weakness 1. positional out-of-distribution issues during the reconstruction process**\n- By employing chunking in our context reconstruction method (Figure 7), we **limit the increase in position ID** to the size of each chunk (i.e., 2K tokens) during importance scoring, thus eliminating the mentioned concerns.\n- Specifically, for each chunked reconstruction, the position ID starts from the last position ID of the prefilled tokens. Since the chunk size (2K tokens) is **less than 2%** of the 128K token context length limit for LLaMA-3.1, its impact is negligible and does not introduce OOD issues for our benchmark datasets.\n- Additionally, the impact on position ID can be further reduced by decreasing the chunk size without incurring degradation in compression performance.\n\n---\n\n**Q) KVZip is inherently prefill-heavy due to its autoencoding nature, which may limit its applicability in resource-constrained on-device environments.**\n- We emphasize that KVzip unlocks new application scenarios beyond the conventional online KV eviction setting. Specifically, KVzip produces reusable, accurately compressed KV caches suitable for multiple future queries, thus supporting **offline context KV cache compression**. This capability is particularly beneficial in real-world use cases such as enterprise document retrieval–augmented systems or personalized user contexts (lines 34–37), where the **compression overhead can be amortized** over subsequent queries. Existing dynamic KV eviction methods, however, incur significant performance degradation in multi-query scenarios, limiting their applicability in such scenarios (Figure 9).\n- Moreover, even in online settings, KVzip's importance-scoring method seamlessly extends to head-level eviction, introducing **no additional compression overhead** during deployment (see Context-Independent Eviction, Section 4.2). Compared to DuoAttention, a state-of-the-art head-level eviction technique, KVzip provides superior accuracy with substantially lower optimization costs (lines 270–275 and Figure 11).\n\n---\n\n**Q) Could you provide a more detailed explanation of the intuition in Section 3.1, particularly lines 104-106 and 107-111?**\n- The main insight behind KVzip is ensuring that the LLM's knowledge system (KV cache + model weights) retains the **complete context information**. We believe this completeness is a necessary condition in a query-agnostic setting, as arbitrary queries may require any detail in the context. Our proposed approach validates this completeness via **prompting**.\n- [lines 104-106] Suppose we reconstruct the original context from the compressed KV cache through prompting. Successfully reconstructing the context implies that we can **regenerate** the original KV cache by performing a separate prefilling using this reconstructed context. This means we can obtain the original response (without compression) to any arbitrary query by using the compressed KV cache. This forms the intuition behind our context reconstruction-based KV cache compression method. \n- [lines 107-111] However, this approach would require a re-prefilling step, which is practically infeasible. Fortunately, we observe empirically that performing direct inference with the compressed KV cache through context reconstruction (Figure 4, right) effectively solves downstream tasks without the need for re-prefilling of the full KV cache.\n- These are detailed explanations of the requested lines. If you need any further clarification, please let us know. \n\n---\n**Q) In Section 4.2's comparison with DuoAttention, could you specify exactly how head-level scores were measured and computed?** \n- We first obtain a KV importance score $S$ of shape $L\\times H\\times n_c$, where $L$ is the number of layers, $H$ is the number of heads, and $n_c$ is the context token length. We compute this score using **a single** English book sample containing 88K tokens from En.QA in SCBench (line 261). Next, we take the **maximum value along the sequence dimension** as $\\max_i S[:,:,i]$, which serves as the head-level score in our experiments (line 260). The pseudocode in Appendix B describes this computation. \n- Note that we use the head-level score derived from a single sample of En.QA for all benchmark datasets, ensuring a fair comparison with DuoAttention.\n\n---\n**Q) Have you compared using sum/average aggregation methods instead of max values for attention scores?**\n- We've tested average aggregation and compared against our max aggregation approach. Note that sum and average aggregation yield identical eviction priorities, making them equivalent.  \n| Method \\ Compression ratio | 0.1  | 0.2  | 0.3  | 0.5  | 1.0  |\n|----------------------------|------|------|------|------|------|\n| Max                        | 64.3 | 93.4 | 94.3 | 93.6 | 93.1 |\n| Average                    | 41.2 | 79.5 | 90.1 | 93.2 | 93.1 |\n\n  *Table. Average accuracy (%) of Qwen2.5-7B on SQuAD dataset.*\n- The results shown in the table above clearly demonstrate that max aggregation outperforms average aggregation. This empirical finding is the primary reason we adopted max aggregation in our algorithm.\n- We attribute this result to the sparsity of attention patterns. As illustrated in Figure 16 (Appendix D), LLMs tend to exhibit sparse attention distributions. Consequently, average aggregation can **dilute the significance** of tokens receiving high attention from specific queries, reducing the perceived importance of KV pairs. In analyzing cases where average aggregation failed, we found that the method often struggled with accurately retrieving words or phrases, negatively affecting various downstream LLM tasks such as QA, coding, passkey retrieval, and reasoning.\n\n---\n\n**Q) Is the repeat prompt robust to alternative prompt designs?**\n\n- The choice of repeat prompt was motivated by simplicity, as the specific wording of the repeat prompt has **minimal impact** on overall performance.\n- To validate this, we conducted experiments comparing: (1) the original repeat prompt, (2) a paraphrased version (\"Reproduce the preceding context without any changes.\"), and (3) no repeat prompt (just \"\\n\\n\"). Note that we follow the chat template provided by each model (e.g., \"\\<user> [context] [repeat prompt] \\<assistant> [repeated context]\").  \n|Repeat prompt type|Accuracy (%)|\n|---|---|\n|Original|94.37|\n|Paraphrased|94.45|\n|No|94.25|\n\n  *Table. Accuracy (%) of Qwen2.5-7B on SQuAD at a 30% compression rate (70% eviction). SnapKV achieves **32.15%** in this setting.*\n\n- These results show that our method is **robust** to variations in the repeat prompt. The limited impact arises because the repeat prompt (7 tokens with Llama 3.1) is significantly shorter than the overall context (at least several hundred tokens), thereby minimizing its effect on compression.\n- To further clarify this, we analyzed **attention patterns**. Specifically, we measured the proportion of prefilled KV pairs whose maximum cross-attention scores during reconstruction originated from the repeated context rather than the repeat prompt (see Figure 4). \n  - For a 2K token-length context from NIAH, 98.1% of KV pairs had their maximum attention from the repeated context. Among the KV pairs retained after 30% compression, **99.4%** derived their maximum attention from the repeated context. These findings confirm the minimal influence of the repeat prompt on KVzip’s importance scores and compression performance.\n- It is also interesting to observe that context reconstruction without a repeat prompt achieves comparable performance. This indicates that forwarding the context twice effectively captures the critical attention patterns required for downstream tasks (see Figure 6).\n\n---\n**Q) The following recent works related to query-agnostic approaches might be valuable additions to the related work section.**\n- Thank you for the suggestion! We added the suggested related works in our revision."}]

Author rebuttal:

Meta-review: All reviewers highlighted the paper’s practical motivation, clear presentation, and comprehensive experiments. The method was recognized as simple, reproducible, and innovative in applying context reconstruction for query-agnostic KV cache compression, with strong results across tasks. Concerns centered on heavy prefill latency and computational overhead, possible positional encoding issues at very long contexts, limited initial coverage of multi-task settings, and the heuristic nature of the reconstruction prompt. In the rebuttal, the authors provided additional analyses, including larger-model experiments, multi-task evaluations, and robustness tests on RULER and prompt variations, which addressed most of these weaknesses. Overall, the reviewers agreed the empirical validation was thorough and the methodological contribution clear. Given its novelty, practical relevance, and broad interest for the community, the recommendation is acceptance.
Decision: Accept (oral)