Value-Guided KV Compression for LLMs via Approximated CUR Decomposition
Authors: Ayan Sengupta
Siddhant Chaudhary
Tanmoy Chakraborty
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
KV Compression
Attention Compression
Needle-in-a-haystack
Primary area: deep_learning

Abstract: Key-value (KV) cache compression has emerged as a critical technique for reducing the memory and latency overhead of autoregressive language models during inference. Prior approaches predominantly rely on query-key attention scores to rank and evict cached tokens, assuming that attention intensity correlates with semantic importance. However, this heuristic overlooks the contribution of value vectors, which directly influence the attention output. In this paper, we propose CurDKV, a novel, value-centric KV compression method that selects keys and values based on leverage scores computed from CUR matrix decomposition. Our approach approximates the dominant subspace of the attention output $\mathrm{softmax}(QK^\top)V$, ensuring that the retained tokens best preserve the model’s predictive behavior. Theoretically, we show that attention score approximation does not guarantee output preservation, and demonstrate that CUR-based selection minimizes end-to-end attention reconstruction loss. Empirically, CurDKV achieves up to $9.6$\% higher accuracy than state-of-the-art methods like SnapKV and ChunkKV under aggressive compression budgets on LLaMA and Mistral, while maintaining compatibility with FlashAttention and Grouped Query Attention. In addition to improved accuracy, CurDKV reduces generation latency by up to 40\% at high compression, offering a practical speed-accuracy tradeoff.

Value-Guided KV Compression for LLMs via
Approximated CUR Decomposition
Ayan Sengupta", Siddhant Chaudhary" Tanmoy Chakraborty
Department of Electrical Engineering
Indian Institute of Technology Delhi, India
ayan. swwgupta@egeeeeeeeeeeeee witeaaeeeeeiieaaaee........
urssiddegmail.com com, tanchak@litd.en ac.
Abstract
Key-value (KV) cache compression has emerged as critical technique for reducing the memory and latency overhead autoregressive language models during infer- ence. Prior approaches predominantly rely on query-key attention scores to rank and eviet cached tokens, assuming that attention intensity correlates with semantic importance. However, this heuristic overlooks the contribution of value vectors, which directly influence the attention output. In this paper, we propose CurDKV, a novel, value-centric KV compression method that selects keys and values based on leverage scores computed from CUR matrix decomposition. Our approach approximates the dominant subspace of the attention output Doftmax(KK" WV ensuring that the retained tokens best preserve the model's predictive behavior. Theoretically, show that attention score approximation does not guarantec out- put preservation, and demonstrate that CUR-based selection minimizes end-to-end attention reconstruction loss. Empirically, CurDKV achieves up 9.6% higher curacy than state-of-the -f-theartt methods like SnapKV and ChunkKV under aggressive compression budgets LLaMA and Mistral, while maintaining compatibility with FlashAttenticn and Grouped Query Attention. In addition to improved accuracy, CurDKV reduces generation latency by up 40% high compression, offering a practical speed-accuracy tradeoff.
Introduction
Transformer based large language models (LLMs) have achieved remarkable performance across wide range of natural language understanding and generation tasks (Yang et al., 2024 Grattafioni et al., 2024] Jiang et al. 2023]. However, their inference-time memory consumption remains a significant challenge, particularly for application requiring long-context information. A major source of this overhead is the Key- Value (KV) cache, which stores the key and value vectors corresponding to previously generated tokens. These vectors are used compute attention outputs without recomputing intermediate states, but their memory footprint grows linearly with sequence length. For instance, as highlighted by Feng al. (2025). LLM-8B, which generats sequence of 2M tokens, consume up to 256GB of GPU memory while storing the KV cache.
Typically, KV cache is populated in two stages: during the prefill phase, where long long-context input encoded parallel and all token-level KV vectors are stored; and during the generation phase, where tokens are produced autoregressively and KV entries appended step at time. In particular, the prefill phase dominates memory usage, especially when handling large input contexts.
These authors contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

This has motivated extensive research (Li et al., 2024a into KV compression techniques that reduce the number of cached entries while preserving output quality. Existing KV compression methods prioritize tokens based on attention scores. For instance, SnapKV (Li et al. 2024b] and H2O (Zhang) al. 2023) estimate token importance by accumulating attention weights across heads and layers. While such heuristics are computationally cheap, they only capture query-key alignment and neglect the downstream contribution of the value vectors. Moreover, these methods mostly overlook the fact that tokens with low attention scores can still carry rich semantic information through their value vectors. Recent work formalizes this mismatch through the concept of eviction loss (Feng et al.) 2025], which quantifies the performance degradation from evicting specific KV entries. Critically, eviction loss is not always correlated with attention score (c.f. Figure D. highlighting the need for value-aware selection strategies.
In this work, we introduce CurDKV (CUR Decomposition for KV Compression), princi- pled and efficient KV compression method that circumvents these limitations by leveraging the structure of the original value and key matrices
softmax(OK") |V eviction loss
8-
6-
rather than relying on attention scores. CurDKV assigns importance to each key and value based on its leverage score, computed as the squared lg norm of the corresponding left-singular vec- tor of the matrix in consideration. These scores
4-
2-
reflect each token's 's contribution to the atten- tion output and naturally align with the goal
of minimizing eviction loss. To improve scal- ability, we also introduce a fast approximation based on random Gaussian projections, where the value and key matrices are projected into lower- diveeddiensional subspace before computing leverage scores. This maintains relative impor- tance across tokens while significantly reducing computational cost. We further cast the KV se- lection problem as a CUR decomposition task on the attention matrix product, selecting key and value vectors that best reconstruct either the attention matrix QK or the final output softmax (attmax(KT]] (QKT) VV Unlike prior work, our formulation directly targets the fidelity of the attention output, leading to more semantically consistent compression strategy. As shown in Figure 2 CurDKV achieves signifi- cantly lower reconstruction loss on both the QK" matrix and the final soft demonstrating improved preservation of contextual semantics post-compression
-4
Average attention score of keys log10 scale)
Figure I: tention value do not necessarily have high post- eviction so roftmax(11 } reconstruction (evic- tion) loss.
Keys associated with high average at-
V product,
We evaluate CurDKV on two popular long- congecontext benchmarks - LongBench [Bai et al. 2023 and Ruler (Hsieh et al. 2024). spanning 24 tasks with LLaMA-3. -nn--rrrtut [Grattafiori et al. 2024] and Mistral-7B - Jiiag et al. 2023]. Our experiments demonstrate that CurDKV consistently outperforms existing ineention-based KV compression baselines such as SnapKV [Li et al. 2024b),
ChunkKV (Liu et al. 2023), and Streaming LLM (Xiao et al. 2024]. In particular, CurDKV achieves significantly lower degradation in generation accuracy at aggressive compression ratios (e.g.. 90% cache reduction). With LLaMA, CurDKV surpasses SnapKV by up to 9.6% in average task score, and achieves higher fidelity (similarity as the full cache model) across all types of long-context- dependent tasks. Similar gains are observed with Mistral, where CurDKV maintains over 95% average fidelity even under constrained memory budgets. These results validate that our value-centrie, CUR- based strategy more effectively preserves semantic attention outputs than methods relying solely on query-key attention scores
2 Related Work
KV Cache compression via attention heuristies and top-k eviction. A widely used class of
KV compression methods rank cached tokens by heuristics and retain the top-k scoring entries. H2O [Zhang et al. 2023] uses observed attention scores from specific query-key interactions to
We have uploaded the source code and datasets as supplementary; we are committed to release them upon acceptance of the paper.

---

1.00-
Key Norm
70-
Eviction (ccaled) of ofr
SnapKV Heuristic
Eviction loss an softmax(QX
SnapKV Heuristic
0.75-
H2O Heuristic
H2O Heuristic
CurDKV (Ours)
50-
CurDKV (Ours)
0.50
0.25
0.00
0.
1000
2000
3000
4000
1000
2000
3000
4000
Number of evicted takens
Num evicted tokens
(a) Reconstruction of QK"
(b) Reconstruction of fo (Q (Q
Figure 2: Preservation of the QK and softmax(QK")) matrices under different KV compression strategies. CurDKV achieves the lowest eviction loss, indicating more faithful reconstruction of intermediate attention. We compute the reconstruction loss for LLaMA-3. 1-8--IBstut, averaged over all layers and attention heads for sample text obtained from https: //en. http::://en.wiiiieeiiaa.. wwkiwedia.orr.. viki/Attention Al_ Alw.ALL_T_l.. Need
estimate per-token importance and prunes less relevant tokens online. However, it requires access to attention weights, which breaks compatibility with efficient kernels like FlashAttention (Dao et al. 2022]. SnapkV eL al. [2024b] computes cumulative attention scores and selects the highest-ranked tokens, assuming that frequently attended tokens are most informative. Scissorhands [Liu al.
2023c further introduces temporal windows to detect consistently attended tokens. PyramidK V Caai al. 2024] proposes hierarchical scoring scheme across layers and heads, combining multi-level lop-k selection with scaling-aware normalization. ChunkKV [Liu et al. [0255 compresses the cache by segmenting into chunks and retaining only top-ranked entries per chunk to ensure temporal coverage. While these approaches are efficient and easy to deploy, they focus on approximating QK and neglect the downstream contribution of value vectors softmax VV, the final attention output propagated to the subsequent layers. Beyond static heuristics, adaptive methods attempt to allocate compression budgets more intelligently across heads or layers. AdaK (Feng et al. 2025) computes attention statistics such as entropy and average weights to estimate the relative importance of each head and distributes the token retention budget accordingly. While more flexible, these approaches still rely on attention-deriver metrics and do not explicitly account for the downstream impact compression on the attention output.
Compatibility with efficient attention implementations. A critical practical consideration is compatibility with modern inference backends such as FlashAttention [Dao et al.] [2022], which avoids materializing the full attention matrix by fusing softmax and matrix multiplications. Methods such as H2O [Zhang et al. 2023), which rely on observed attention weights, are incompatible with FlashAttentinn and similar kernels that trade explicit attention computation for efficiency. Another major limitation with the majority of the existing KV compression methods is their inability to work with Group Query Attention (GQA), GQA [Ainslie et al. 2023] has been widely adopted in recent LLM architectures such as LLaMA [Grattafiori et al. 2024 and Mistral [Jiang et al.) 2023) due its ability to reduce the memory footprint of the KV cache by sharing keys and values across multiple heads. However, most existing KV cache compression techniques, including SnapK V and PyramidKV V. do not support GQA natively. These methods redundantly replicate the compressed KV cache across heads within a group, thus forfeiting GQA's inherent efficiency. To address this limitation, Feng et al. 2025 proposed ECQA-compatible eviction framework that computes average attention weights within each group identify important tokens. This enhancement enables state- of-the-art KV compression methods function effectively GQA-enabled models and achieve significant cache size reductions while preserving generation quality.

---

3 Methodology
3.1 Preliminaries
KV Cache. The generation process of an LLM is autoregressive in nature, i.e., each generation step relies on the outputs of all previous steps. Suppose X E IR"xd be the matrix containing the hidden states of a layer in an LLM upto a time step. Further, suppose x E R xd be the hidden state of the input for the current generation step. Suppose, there are h attention heads. For head i [1, M.l the query, key and value matrices, W.O W.K and W.Y (all in Bdxds) map hidden states X query, key and value states; specifically, the following are computed:
last generated token, which is used
Q: = =x= K2 = XW V, = XWW
(1)
For the current time step, the states K; and Vi constitute the KV cache elements for head i. KV caching refers to caching K; and V; in memory to avoid recomputing them at every generation step. After computing the output for the current time step, the KV cache is updated accordingly:
K; = Cat [K; xW V = Cal(V; : xWM]
KV cache compression techniques aim to retain only the most relevant key and value states from K; and Vi, without significant loss in generation quality. Particularly, show that the loss the attention output is upper bounded by the Frobenius distance between the original value matrix and it's compression:
Lemma 3.1. Let Q, K, E Raxd be the query, key and value matrices in the attention computation. Further, let K', V E Raxd be sub-matrices obtained by zeroing-ond a subset of rows of K and V respectively. Then.
|softmax [ssottax(QK*)) softmax(QK" VIll S vRllV +
Note that for higher compression levels, the second term ([V) becomes negligible; in that case, the role of the approximation V V becomes significant. On the other hand, the contribution of Q, K and K" is absorbed by the softmax operator. This motivates us to design value- sulue--uided compression technique.
Low-rank decompositions. Let A E Raxd be matrix. The problem of low-rank decomposition of A seeks to factor A a product of low-rank matrices, and a well-studied problem with wide applications in mathematical modeling and data compression (Halko al., 2010). The optimal solution to this problem is the well-known singular value decomposition (SVD), which represents A in the form A = UEV*, with U and V being semi-unitary matrices, and Σ square diagonal matrix of dimension equal to the rank of A. While the SVD is widely adopted in data analysis problems, when applied to the context of the KV cache compression problem, it presents an inherent challenge; namely, the column/row vectors of U and V are not representative of the column/row vectors of A itself. Another suboptimal low-rank decomposition, called the CUR decomposition, resolves this limitation: is an approximation of the form A Ra CUR, R. where C is a subset of columns of A, R is a subset of rows of A and U is low-rank matrix. Here, while the matrices C and R are representative of the underlying data, the matrix U optimizes the approximation to A. The CUR decomposition is known be good low-rank approximation the original matrix [Woodrufi] 2014 Halko et al. 2010 Mahoney and Drineas 2009].
Computation of CUR and Leverage Scores. We utilize leverage scores, a popular technique used to compute CUR factorizations based on identifying key elements of the singular vectors of a matrix Mahoney and Drineas 2009 As before, let U be the SVD of Rnxd where U E R Σ E श and V E प with r rank(A) The idea of leverage scores is to compute importance scores for each row and column of A, which are then used to sample the rows and columns that then constitute the matrices R and C' of the CUR decomposition respectively. Particularly, to rank the importance of rows of A, we define the row leverage scores Lr.j (for the jth row of A) as
4. = || 1))
(4)
See section A of the Appendix for proof.
4

---

Similarly. to rank the importance of columns of A, the column leverage scores le,5 (for the jth column of A) are defined analogously by
l.. : IIV (t. 5)1l2 - [VV()
(5)
Since U and V contain the left and right singular vectors of A, it easily follows that Σラ= P. = n
and Σラー 1 - d. In turn, these scores lead the distributions { and (_ from which the matrices R and C' can be sampled. Alternatively, static methods the top-k scores from either distribution to sample the corresponding matrices.
3.2 CurDKV: Algorithm Design
We now introduce CurDKV, our proposed KV cache compression method, which uses combined leverage scores of key and value matrices to compress the KV matrix. CurDKV can be applied during the pre-filling as well as the token generation phase, is easy implement, and seamlessly integrates with inference acceleration methods such as FlashAttention (Dao et al. [022] and GQA [Ainslie et al., 2023]. Algorithm| | provides the pseudocode of CurDKV. In the following discussion,
assume the
attention setup as in GQA [Ainslic et al. 2023]. wherein a group of attention heads uses common key and value matrix. Particularly, we assume that there are g groups, with the key and value matrices of group i, denoted by K; E IRd and V; E IRxxd respectively. As usual, n represents the number of keys (or values) in consideration, and d is the dimension of the space of vectors. The combined key and value matrices across all groups is denoted by K E Raxnxd and V E Raxxxd
As shown in Algorithm 1 CurDKV takes input the key matrix K E Roxaxd across all attention groups, the corresponding value matrix V E RRxxxxd the group budget k E [m] representing the level of compression, projection dimension and the number of initial attention sinks to be preserved. It outputs compressed key and value matrices K" E Roxxxxd and V' E Roxxxd, which are then used compute the output of the attention module. At a high-level, CurDKV computes approximate leverage scores for each row of the key and value matrices K, E g and V2 E jn within a single group (lines 3-5). The leverage scores for corresponding keys and values are then combined, followed by normalization (lines 6-7). Finally, the computed leverage scores are then used to retrieve the top-k keys and values from K; and Vú (lines 9-11). In addition, CurDKV also preserves the initial s tokens (lines 8, 10) due to the prevalence of attention sinks (Xiao et al. 2024].
Approximate leverage scores and Gaussian projections. The computation of exact leverage scores as in Equations and requirees the computation of the SVD of the matrices in consideration, which turns out to be bottleneck for the KV compression problem. To that end, CurDKV projects K; and V/2 to matrices K,G and V,G, respectively, where G E Rdd is random matrix, each of whose entries are sampled from a normal (0, +) distribution; our implementation, we use r = 20 After this projection step, the norms of the rows of K, and V,G are used as proxies for the actual leverage scores (lines 3-5 of Algorithm D.
Adaptive budget allocation for CurDKv. Along with CurDKV, we also implement AdaCurDKV, an adaptive variation of CurDKV. AdaCurDKV adaptively achieves head-specific compression by choosing the top-k key and value vectors across all heads in layer, where the selection is based on the computed leverage scores. As in the standard adaptive implementation introduced by Feng al. 20255). J, safeguard parameter is used ensure minimum fraction of key/value vectors preserved for each head (a default value of a = 0.20 is used in our implementation)
4 Experiments and Results
4.1
Experimental Settings
For the empirical analyses, following the contemporary studies, we use two instruction-tumm tuned LLMs
LLaMA-3.1-8D 1-B-..118--nntrreec [Grattafiori et al. 2024 and Mistral-7B-IInsruutt00.333 [Jiang et al. 2023
'The Gaussian projection motivated by the construction Theorem 19 Woodruff 2014
'Refer to for an implementation of the adaptive compression logic.
http://www....eeee/s//////////// ada prwssv.pppespppppp
5

---

Algorithm 1: CurDKV: CUR-based KV Compression with GQA Support
Input: Key matrix K E Roxaxd, Value matrix V E Rexaxd, group budget k Gaussian
projection dim r, num sink
Output: Compressed keys K' E Rexkxd, values V' E IRoxkxd
for each group gi in I,..., do
Sample Gaussian matrix G E Raxe with Giy ~ (0,,
Project K; + K,G. Vi f VGG;
Compute key leverage scores: p(K)
|K1 value leverage: eg ||ViD>
Combine scores: (( ((K) (V)
Normalize: E, = - e
(KV)
7 Preserve first s tokens as sink indices: Ssink = [0,...,,--1.
Select top-(k- s) indices from E|s :]: Soo TopK (e(s :1, k 8) +
Combine: S Siink U Swp
KJ 4 [S])]] V V.[S.]
11 return K', V
All the evaluations are done iet al. 2023] and Ruler [Hsich et al. 2024]. LongBench benchmark contains a total 16 tasks, covering various task domains including single-document QA, multiple-document QA, summarization, few- shot learning, synthetic tasks and code completion. From Ruler benchmark, we consider a total eight needle-in- geedle-in-a-haysee an-a-haystack tasks with maximum context length of 16K. Details of the LongBench and Ruler tasks and the prompt templates can be found in the appendix. For all these 24 tasks, we evaluate only on the more challenging omitted during compression and only the context is compressed. As argued by Feng et al. [2025]. this setup mimics more challenging scenarios, where the compression method is unaware of the questions being passed to the model during inference.
widely popular long-context benchmarks LongBench [Bai
prrestion-agnosiie setting [NVIDIA] 2024], where the questions are
We compare CurDKV with various competitive KV compression base- lines, including SnapKV [Li et al.
Table 1: Results of LLaMA-8B and Mistral-7B with full KV cache (0% compression) LongBench.
2024b). ChunkKV [Liu et al. 2025].
LLM Streamline [Xiao al.
2024
Task Type
Task
LLaMA-SB
Mistral-7B
and V [Feng et al. 2025
NιtQQ
30.7
28.4
Single-Doc QA
Qusper
47.2
40.3
We consider an additional baseline.
MF--nn
55.6
51.9
KNorm [NVIDIA (2024) that uses key norm as proxy to determine which keys to evict during compression. H2O Zhang et al., 2023] is purpose- fully omitted from the evaluation as throws out-of-memory error when single NVIDIA A100-80GB GPU card (as it does not naively sup- port flashattention, therefore requiring 4x more cache memory on long se- quences). All these baselines for different compression (eviction) ratios (30%, 50%, 70%, 90%). Fol-
HotpotQA
39.5
-48.9
Multi-Doe QA
2WikiMQA
$1.8
37.4
Musique
32.6
28.0
Gov Repori
35.0
"34.7
Summarization
QMSum
25.1
25.5
Multi News
26.8
26.8
TREC
29.5
55.8
Few-shot Learning
TriviaQA
85.7
SAMSum
38.7
20.8
Pcount
10.7
5.1
Synthetic
Pre
100.0
98.0
Lee
53.9
49.8
Code
RB-P
47.6
56.1
Average
45.7
43.3
lowing Xiao et al. [024], we use attention sinks of size s = for all the baselines (all the baseline numbers are produced in-house).
4.2 Results on LongBench
We report the results obtained with full cache (0% compression) for LLaMA-8B and Mistral-7B in Table Table summarizes the results obtained with LLaMA-8B and Mistral-7B with KV cache compressed at 30% and 90% with different baselines.

---

Table 2: KV compression methods on LongBench with LLaMA-3.1-8 -88--ttrutt and Mistral-7B. Instruct compression ratio). Compression ratio is the complement of the cache budget, e.g. for cache budget of 30% the corresponding compression ratio is 70%. Friedman statisties of 45.2/35.7 (for 30% compression) and 21.1/8.5 (90% compression) with p-values 0 indicate the statistical significance of the result obtained by CurDKV over the baselines. Bold highlights the best adaptive and non-adaptive baselines for each model and task, with blue highlighting the cases where CurDKV or AdaCurDKV are the best baselines. Results with other compression ratios are reported in Tables [4 Each result cell "x/y" indicate accuracies with LLaMA/Misttal..
and I5 in Appendix C
Task
ChunkKV
Knorm
Streaming LLM
SnapKV
CurDKV
AdaSnupKY
AdaCurDKV
Non-Adaptive Methods
Adaptive Methods
NrtQQA
25.
30.6/ 23.8
26.8/ 24,4
30.2 26.4
31.7 28.8
29.7/ 25,4
31.9 27.6
Qasper
45.7 33.3
44.4 35.3
43.4/ 36.1
46.5/ 35,5
48.2/ 39.4
45.8/ 35.7
48.6 41.4
MF-en
50.2/48.7
48.2/48.7
36.01 34,4
53.8/49.0
56.3/
48,4
56.9/51.0
HotpotQA
58.0/49.0
48.7
52.1
58.4
17.8/49.2
59.9 59.9/47.3
59.71
2WikiMQA
49.6/37.3
49.4/6..6
425/31.6
49.3/37.2
513/41.5
50.1
39.6
Musique
30.0/25,4
33.6/24.5
28.9/20.7
35,0/254
29,4
32.5/28.1
32.9 28.9
Gov Report
33.9/34.1
34.0/34.0
31.7/32.9
33.6/33.6
34,5/35.9
34.1 34.1
QMSum
24.0/24.6
24.2/24.2
235/2388
24.2/24.2
25.0/25.1
24.6// 24.8
25.2 25.3
30%
Multi News
26.8/26.2
26.5/26.3
26.3/26.3
26.6/26.0
27.2/ 26.9
26.7/26.2
27.1 27.3
TREC
25.0/50.9
66.0/
32.5/59.5
31.0/1.5
67.0/60.0 60.0
30.5/52.0
63.5 63.5/73.0
TriviaQA
85.5/84.8
87.8/85.6
92.2/19..0
SAMSum
39.6/20.8
36.5
38.7/19.1
40.1/21.0
41./41.1
/
41.5 46.6
Pcount
11.1/5.6
13.1/4.1
831/2
12.2/5.2
13.7
12.2/5.1
12.1
Pre
99,5/97.5
98.0/80.8
68.5/68.0
100.0/97.0
100,0
190.0/98.0
/9.5
Loc
52.5/516
36.7/37.2
47.1/511
51.0
52.1/51.1
$2.7 50.7
518/51.5
RB-P
56.3/56.2
54.7
474/549
/
57.0
57.4
/
Average
44.4742.2
46.47
40.3/33.88
A5.3/42.1
48.9
44.5
46.4
NrtvQA
18.7
13.0
18.6/18.4
17.7
27.7/14.2 14.2
25.3/ 18.3
27.0/ 12.1
Qusper
20.0 12.7
15.9/6.0
18.//3.11
1
28.5 26.2
21.6/ 15.2
MF-en
23.6/26.2
29.2/25.0
23.0/23.1
23./30.3
32.6/35.0
26.2/32.0
35.2
HotpotQA
43.1/37.3
40.1/30.9
37.2/3122
44.5/
47.4/30.7
46,9 38.1
46.1 /6.1/34.8
2WikiMQA
22.4/2199
20.1/
22.4/20.1
24.6/23.1
/
24.0/24.0
33.6 31.7
Musique
16.9/16.9
14.0/399
20.0/
Gov Report
25.0/25.5
26.0/26.0
24.8/
25.3/
26.9//3.3
25.3/
26.9 26.2
QMSum
19.5
21.0/18.9
/9.1
19.7/20.2
22.6/21.8
20.9/
22.2
Multi News
17.1/17.9
19.7/
20.1
20.8/20.8
23.2/2166
20.6/
23.0/
TREC
11.0/15.8
26.5
28.0/35.0
33.5/
22.5
33,0/
19.5 33.0
TriviaQA
88.8
684/85.4
90.7/524
829/87.9
93.5/
/
/2.3/ 76.3
SAMSum
31.6/29.7
28.7/38.9
34./3388
/
37.5/40.9
39.3/
33.7 38.7
Penunt
5.0/3..6
12.0/26
4.0/4.0
6.0
5.5/3.3
8.0/3.9
7.91244
Pre
49.5/42.0
10.0
16.0/155
55.0/
61.5/15.5
56.5/
25.5
Lee
50.6/51.6
23.1/22.2
52.3/
51.0/
39.4/39.7
49.2/53.0
39.//4.1
RB-P
49.3/54.9
49.7/55.1
52.9/5333
49,2/54.8
$2.2153.6
47.3/55.0
$4.2/54.6
AAeriige
308773..
28387
25.5/26.7
35.7/32.
38.7
342/3332
35.0/31
On LLaMA-SB, CurDKV outperforms SnapKV by +3.6% at 30% compression (48.9% vs. 45.3%) and by +2.0% points at 90% (35.7% vs. 33.7%). ). On Mistral-7B, CurDKV shows similar trend, surpassing SnapKV by +2.9% at 30% (45.6% vs. 42.7%) and by +0.5% at 90% (33.2% vs. 32.7%). These gains are robust across task types, including multi-hop QA, summarization, and code completion. Norm-based heuristics such Knorm and ChunkKV fail consistently preserve semantic fidelity, often underperforming even under moderate compression. This is most evident in Mistral-7B few-shot learning tasks (e.g., TriviaQA, SAMSum), where CurDKV maintains over 89% performance, other baselines trail by 5-10%.
We compare AdaCurDKV, our adaptive compression strategy with its static variant (CurDKV) and the attention-based adaptive baseline, AdaSnapK V. Across models and retention levels, AdaCurDKV con- sistently outperforms AdaSnapKV, while performing competitively with CurDKV. With LLaMA-8B under 30% KV compression, AdaCurDKV achieves the highest average score (49.1%) outperforming V (45.5% by +3.6% and CurDKV (48.9%) by +0.02%. At 90% retention, AdV main- tains a strong average score of 35.1 %, closely trailing CurDKV (35.7%) and surpassing AdaSnapKV (34.2%). For Mistral-7B, AdaCurDKV achieves 45.2% performance at 30% retention, outperforming AdaSnapKV (42.1 %), and remaining close to CurDKV (45.6%). Although AdaCurDKV's average score drops to 29.1% 1% at 90% retention, it remains competitive and stable across all task types. While CurDKV typically attains the best or second-best overall score, AdaCurDKV provides additional ro- bustness by allocating head-wir wise budgets proportional to leverage score mass, leading to improved performance on head-sensitive tasks, especially in summarization and multi-hop QA.
7

---

We conduct ablations (c.f. Figure 3 to assess two core design choices in CurDKV: the source of leverage scores and the use of random projections. At 30% compression, different leverage modules (key, value, key-value product) perform similarly, but at 90%, value-centric and combined variants show better robustness than key-only. Random projections offer marginal gains overall, with slightly improved stability at high compression (more elaborated discussion in Appendix (
CurDKV con- sistently demonstrates superior robustness across compression levels and model scales. For the Qwen-143) model (Yang et al.] 2024), it performs competitively with SnapKV at 30% compression and achieves a clear margin at 90% compression, out- performing all baselines by over 4 4-6% on average. The trend becomes more pro- nounced in the larger Qwen-32B model, where CurDKV surpasses other adaptive and non-adaptive methods by significant mar- gin under both 30% and 90% compres- sion. These results highlight that value- aware KV compression scales favorably with model size and remains effective even under extreme memory constraints.
As shown in Table 3a- 3b
Table 3: LongBench evaluation results for Qwen models at 30% and 90% compression.
(a) Qwen-14B
Task
Chunk
KNorm
Snap
Streaming
CurDKV
2WikMMQA
28.71
28.76
25.06
27.97
HetpoiQA
33.53
31.44
34.56
26.67
31.58
30%
MF-en
30:11
29.27
32.11
23.33
31.58
Qusper
24.84
21.89
22.08
27.04
Average
29.37
27.34
30.12
22.91
29.63
ZWiKIMQA
12.39
10.04
13.37
9.56
20,88
HetpoiQA
14.51
21.54
90%
MF-en
17.03
7.20
18.25
14.66
22.08
Qasper
9.25
7.20
9.56
10.22
16.23
Average
1494
10.00
13363
12.24
20:18
(b) Qwen-32B
Task
Chunk
KNorm
Snup
Streaming
CurbKV
2WikiMQA
51.64
40.14
34.10
58.07
HegoiQA
58.64
46.94
57.52
47.15
62.60
30%
MF-en
45.41
43.39
48.56
32.61
50.42
4.3 Results on Ruler
Qusper
Averagee
3y533
42.46
442.0"
37.73
"5025
41.86
"9.24
43.11
5577
47.75
2WikIMQA
24.24
6.14
9.81
18.54
40.59
We further evaluate CurDKV and AdaCurDKV on the Ruler benchmark, focusing on needle-in---hhasstack (NIAH) subtasks, which measure model's ability to recover rare, high-salience facts embedded in long distractor contexts. These tasks are particularly sensitive to the retention of semantically critical key-value tokens. Full cache results are highlighted in Table
HerpoiQA
37.85
3.80
36.29
30.38
43.51
90%
MF-en
24.31
13.12
23.04
22.03
31.24
Queper
13,62
7.02
14.29
14.56
16.10
Average
25000
7.55
"29.0
221.88"
32236
As shown in Table 5 CurDKV and AdaCurDKV achieve the highest average score (98.7% and 97.7%) under 30% compression, outperforming all baselines. CurDKV substantially outperform based (ChunkK V: 87.0%, Knorm: 71.8%) and attention based approaches (SnapK 80.4%, StreamingLL.M 68.2%) On particularly hard retrieval subtasks like MK-3 and MQ, AdaCurDKV scores 93.6% and 100.0% respectively, indicating its strong capacity to retain tokens that encode key factual content. In S-3 a shallow, multi-hop setting, AdaCurDKV scores 88.2% compared 82.4% (AdaSnapKV) and 94.1% (CurDKV), reflecting robustness in both adaptive and static modes.
At 90% compression, performance drops sharply for most baselines, with norm-based methods like Knorm (13.9%) and StreamingLLM (10.1%) failing to preserve fidelity. In contrast, CurDKv (34.7%) and AdaCurDKV (39.1%) main- tain significantly higher performance, with AdaCurDKV again achieving the best overall average. Notably, on MQ and MV, AdaCurDKV reaches 56.7% and 52.0% re- spectively, far ahead of all other methods, highlighting its advantage tasks requiring deeper contextual aggrega- tion. On S-2, AdaCurDKV outperforms both AdaSnapk and CurDKV (63.2% vs. 36.8% and 57.9%), showcasing the benefit of adaptive head-wise allocation in early-token regimes. Albeit the strong performance across different subtasks, we observe marginal underperformance on a sub- set of NIAH tasks with Mistral 90% compression ratio. This can be attributed to Mistral's use of sliding-window attention which reduces head specialization and makes head-local token selection less effective. In such cases, heuristies like ChunkK V. which retain tokens based on uniform position or locality, can incidentally preserve critical information. Nonetheless, CurDKV continues to show superior performance on more challenging retrieval subtasks, indicating its robustness under complex attention patterns.
Table 4: Results of LLaMA-8B and Mistral-7B with full KV cache on Ruler (Needle saalll-i----hyasee a-ayssstcc subtasks).
Sub LLaMA-8B
Mistral-7B
S-1
100.0
95.2
S-2
100.0
98.3
S-3
100.0
100.0
MK-1
100.0
97.9
MK-2
100.0
97.8
MK-3
97.9
80.9
MQ
99.5
85.6
MV
100.0
88.8
Average
99.6
92.8
8

---

Table 5: KV compression methods on -aalle--n---hystackk tasks for Llama-3. 3.1-8B- SB-Instruct and Mistral-7B Batral-7B-Instreee 1--Intrruct compression ratio). Friedman statistic of 25.6/21.7 (for 30% compression) and 12.3 (for 90% compression) with p-values < 0.05 indicate the statistical significance of the result
obtained by CurDKV over the baselines. Full results for other compression ratios are highlighted in Tables 16 and 17 Appendix C. Each result cell "x/y" indicate accuracies with L.aMA/Mistral.
Task
ChunkKV
Knorm
Streaming LLM
SnapKV
CuroKV
AdaSnapKV AdaCuroKV
Non-Adaptive Methods
Adaptive Methods
100.0/95
100.0
100.0/
S-2
100.0/91 /912
100.0 12.3
64.9 64,9
100.0/ 63.2
100.0/91.2
100.0 80.7
100.0/ 96.5
S-3
96.1 36.9
51.0/0.0
78.4 78,4
25.5/2.0
94.1/72.6
82.4/ 13.7
88.2 60.8
MK-1
100.0/ 69.6
90.7
70.8
100.0/41.7
100.0/91.7
100.0/ 77.1
100.0/
30%
MK-2
57,8/
20.0/0.0
711/68.9
71.1 /44.4
100.0/93.3
97.8 84,4
100,0/ 84,4
MK-3
44.7 46.8
17.0/211
55.3/31.9
23.2/
95.7 23.4
89.4 68.1
93.6 33.6/44.7
MQ
100.0 / 85.1
99
74.0/70.7
99.5/31.7
92.2
100,0/ 58.7
100,0/ 83,7
MV
97.4 88.8
98.7/1.3
68.4
94.1/316
100.0 93.4
100.0 64.5
100,0 95.4
Average
87.0/76.2
14.3
68.2/63.6
80.4737.4
98,7177.88
96.2/6440
97.77 80.6
S-1
88.7
100,074638
6.576.5
72.6735.5
30.7
85.5746.8
96.8716.1
S-2
38.6/3.5
L8/0.0
14.0/12.3
38.613.5
$
3.5
63.2 5.3
17.7
0.0/0.0
2.0/000
0.0/0.0
2.0/2.0
0.0/0.0
MK-1
2
0.0/0.0
8..4/8.3
14.614.2
3.36/
20.8 6.3
3.36/6.3
90%
MK-2
6,9
0.0/0.0
8.9/8.9
8,9
4.4
11.1 11.1/22
4.4
1..8000
2.1/0.0
MQ
28.4
4.3/0.0
9.6/6.7
15.4/1155
/
1
56.7 1.4
MV
/
33/00.0
7.217.2
21.7
45.4/ 12.5
19.7 1558
$2,0/999
Average
32.8/194
13.9/5.8
10.1/7.7
22.0/9.1
34.7 /7.8
21.4/113
39.1/3.9
Accuracy LongBench tasks
Key
Accuracy LongBench tasks
Random Projection Random Projection
80
Vay-Value Product
80
Value
60
60
40
40
20
20
30
90
30
90
Compression ratio (in
Compression %)
(a) Leverage computation type
(b) Ablation with and without random projection
Figure 3: Accuracy of 1--.MMA--.1-BB-Isttttee on LongBench tasks for different ablations of CurDKV.
Full results reported in Table 18 in Appendix
4.4 Computational Efficiency of CurDKV
To evaluate the practical benefits KV compression beyond accuracy, we analyze CurDKV in terms of memory usage, prefill latency, and generation latency across varying sequence lengths and compression ratios. As expected, CurDKV yields (c.f. Figure 44) near-linear reduction KV cache size with increasing compression ratio. For example, at 80% compression, memory usage drops from 15.6 GB to under 3 GB for a 128K-token context. This linear trend holds consistently across all tested sequence lengths. Since CurDKV compresses the cache on per-layer and per-head basis, these
Prefilling time (seconds)
(b)
Genrraation time (seconds)
20
(c)
JK tokers
..... 64K tokers
1K tokens
....
64k takens
1K Lakens
..... 64K Lokens
Cache size (GB)
128K tokens
25
128K
128K
15
16K tokens
16K tokeens
16K tokens
10
10:
20
40
60
80
20
40
60
20
Compression ratio (in %))
Compression ratio (in %)
Compression ratio (in %)
Figure 4: Prefilling and generation statistics of CurDKV for different text lengths with LLaMA-8B,

---

700
Srart Carteed Hanzem Durbiic
50
Ooneration
Generation Time (secands)
600
Chundoy
Drupey
Runtime (seconds)
40
500
400
300
20
200
bactCuber
10
100
DhunkeV
30%
90%
Compression Ratio
Compression Ratio
(a) Breakdown analysis of runtime of LLaMA 8B(b) Comparative analysis of generation runtime with model with generation length of 100.
4K generation length.
Figure 5: Comparative analysis of different compression methods with 128K context length.
reductions are directly proportional to the retained key-value tokens, enabling predictable memory scaling long-context scenarios. Figure 4b shows slight upward trend in prefill latency at higher compression ratios, especially for longer sequences. This is because CurDKV computes leverage scores and performs token selection during the prefill phase, introducing modest computational overhead. For instance, with 128K tokens, prefill time increases from approximately 10 seconds (no compression) around 14-15 seconds at high compression levels. However, the added cost remains constant beyond 40-60% compression, indicating amortization of overhead due to batch-efficient scoring. Interestingly, generation latency (Figure44c) decreases with compression for all sequence lengths, particularly for longer inputs. This is attributed to the smaller number of cached tokens being read during autoregressive decoding. With 128K-token contexts, generation time reduces from 10 seconds (no compression) to under 6 seconds at 80% compression. The effect is more muted for shorter sequences, but consistently observable. These properties make CurDKV a practical and scalable solution for deployment timettmmetttteeeeeee KV cache optimization in LLMs.
As shown in the runtime and generation analyses in Figure 5 CurDKV achieves a favorable balance between latency and accuracy across compression levels. At 30% compression, CurDKV with random projection yields the lowest overall runtime due to reduced prefill cost, while maintaining comparable generation latency to other baselines. Under 90% compression, it continues to outperform in total inference time, benefiting from efficient subspace projection of key-value pairs. The longer- sequence generation study further confirms these trends, where CurDKV consistently achieves faster decoding than SnapKV and ChunkKV, with up to 15% reduction in generation time. At higher compression, exact CurDKV (without random projection) yields higher generation time, due to iterative CUR decomposition; however, CurDKVwith random projection achieves the least generation time, highlighting the trade-offs between exact and randomized CurDKV.
5
Conclusion
In this paper we proposed CurDKV, a value-centric KV cache compression method based on CUR decomposition, and AdaCurDKV, its adaptive variant. Unlike prior approaches that rely on attention scores, our method selects key-value tokens using leverage scores derived from the value matrix, with efficient approximations via random projections. Experiments on LongBench and Ruler benchmarks showed that CurDKV consistently outperformed existing methods across tasks and compression ratios, while remaining compatible with Flash and Grouped Query Attention.
Limitations and future work. While CurDKV offers principled and effective approach to KV cache compression, it relies on static token selection during the prefill phase, which may limit its responsiveness to dynamically emerging information needs during generation. Future work could explore query-aware token selection, hybrid token-chunk strategies, or lightweight learned components to improve performance in semantically sparse settings. Extending CurDKV to support dynamic compression during generation is another promising direction to better adapt to evolving query demands.
10

---

Acknowledgments
T. Chakraborty acknowledges the support of the IBM-IITD AI Horizons network and Rajiv Khemani Young Faculty Chair Professorship in Artificial Intelligence. He acknowledges the support of Google GCP Grant for providing the necessary computational resources.
References
J. Ainslie, J. Lee-Thorp, M. De Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai. Gqa: Train- ing generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305. 13245, 2023.
Y. Bai, X. Lv, Zhang, H. Lyu, J. Tang. Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, al. Longhench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Z. Cai, Y. Zhang, B. Gao, Y. Liu, Liu, K. Lu, W. Xiong, Dong, B. Chang. J. Hu, and W. Xiao. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling, Oct. 2024. URL httt:::/aaxxvv.or//bbbb.b
02069, arXiv:2406.0206 [cs].
T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, June 2022. URL http://aaxiv.orr/////200..00 1AX1: [cs].
P. Dasigi, K. Lo, I. Beltagy, A. Cohan, N. A. Smith, and M. Gardner. A dataset of inftrrttcoko questions and answers anchored in research papers, 2021. URL/nttps///arxii.on//////220 03011
Y. Feng, Lv, Y. Cao, X. Xie, and S. K. Zhou. AV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference, Jan. 2025. URL 11550 arXiv:2407.1 1550 [es].
B. Gliwa, I. Mochol, M. Biesek, and A. Wawer. SAMSum corpus: A human-annotate dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China, Nov. 2019. Association for Computational Lin- guistics. doi: 10. (0.18653/11/D19-54000
https://://// http:///wwwwwwwwbbco wwwwww.cwwwbb.c... wwwp:/o.rg///neeeeo/eeo www.eeborrg//nooooo////3333 5409
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sra- vankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak. Bi, C. Marra, C. McComnel, C. Keller, C. Touret, C. Wu, C. Wong. C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song. D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olana D. Perino. D. Hup- kes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cu- curell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang. J. Copet, J. Lee, Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. v. d. Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala- Rantala-Ycary, L. v. d. Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, Madaan, L. Malo, L. Blecher, L. Landzaat, L. d. Oliveira, M. Muzzi, M. Pasupuleti. M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov. N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, Alrassy, Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, Dubal, Kr- ishnan, P. S. Koura, P. Xu, Q. He. Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang. S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende,
J. Spisak, Park, Rocca, J. Johnstun,
S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler,
11

---

T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami,
V. Gupta, Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti. V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang. X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos. A. Singh, A. Srívastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Fein- stein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe. A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C.-H. Chu, C. Cai, C. Tindal, C. Feichtenhoferr C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang. D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery. E. Presani, E. Hahn, E. Wood, E.-T. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang. H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, Damlaj. I. Molybog, Tufanov, Leontiadis, I.-E. Veliche, Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J.-B. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, Zhong, J. Jin, J. Yang, J. Cummings. J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg. J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg. I.. A,
L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang. M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. San- thanam, N. Parks, N. White. N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar. P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto,
S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng. S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang. S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng. S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang. W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang. X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi. Y. Li, Y. Zhang. Y. Zhang, Y. Adi, Y. Nam, Yu, Wang. Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. Veto,,, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma. The Llama 3 Herd of Models, Nov. 21783, arXiv:2407.21788 [cs].
2024. URL http:://///a org
D. Guo, C. Xu, N. Duan, J. Yin, and J. McAuley, Longcoder: A long-range pre-trained language model for code completion, 2023. URL http::t/aax.l..orrrrr
14889
N. Halko, P.-G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions, 2010. https///t:// org/
X. Ho, A.-K. Duong Nguyen, S. Sugawara, and A. Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1609-6625. Barcelona, Spain (Online), Dec. 2020. Interna- tional Committee on Computational Linguistics. URL httpp:://ww..ccluee http:://www..lllebbbgggggtttoooo
2020. 1020.coling coling- main.580
C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint 20l.:22044006544, 2024.
12

---

L. Huang, S. Cao, N. Parulian, H. Ji. and L. Wang. Efficient attentions for long document summariza- tion. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436. Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/0222 maacl-main.11 112. URL.
https://aclantthologr/ wwtp:y...rrg//2// ottp naacl main. 112
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, F. Bressand, G. Lengyel. G. Lample, L. Saulnier, L. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril.
T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7B, Oct. 2023. URL http:: http://arxiv.org/abb..
2310 06825, arXiv:2310.06825 [es].
M. Joshi, E. Choi, D. Weld, and Zettlemoyer. triviaqa: Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art.
2017.
T. Kočiský, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette. The narrativeqa reading comprehension challenge, 2017. URL (RLLLttts////rxx..rrr/////////// 07040
H. Li, Y. Li, A. Tian, T. Tang, Z. Xu, X. Chen, N. Hu, W. Dong, Q. Li, and L.. Chen. A survey on large language model acceleration based kv cache management, arXiv preprint 47444:2412.1444. 2024a.
X. Li and D. Roth. Learning question classifiers. In COLING The 19th International Conference on Computational Linguistics, 2002. URL http::/wwwwwwwceeebb..ccrc/tttts
Y. Li, Y. Huang, B. Yang. B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen. SnapKV: LLM Knows What You are Looking for Before Generation, June 2024b. URL http://arxiv org/ /abs/2404. 14469 arXiv:2404.14466 14469 [cs].
N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts, 2023a. https//////r http:///ggabb////////// /0s 03172
Liu, C. Xu, and J. McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2023b. URL http:::/arr....rggbbb/////////// 03096
X. Liu, Z. Tang, P. Dong, Z. Li, B. Li, X. Hu, and X. Chu. Chunkkv: Semantic-preserving kv cache compression for efficient long-context I1m inference. arXiv preprint arXiv:2502.00299, 2025.
Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyrillidis, and A. Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for I1m kv cache compression at test time. Advances in Neural Information Processing Systems, 36:52342-523 -23664 2023c.
M. W. Mahoney and Drineas. Cur matrix decompositions for improved data a of the National Academy of Sciences, 106:697 - 702, 2009. URL https://ppp..emmmtticcchh/////
NVIDIA. LLM KV Cache Compression Made Easy. 2024. Accessed: 2025-05-06.
http::///gthuub.cc///////////
H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal. Musique Multihop questions via single-hop question composition, 2022. URL
00573
D. P. Woodruff. Computational advertising: Techniques for targeting relevant ads. Foundations and Trends@ in Theoretical Computer Science, 10(1-2):1-157, 2014. ISSN 1551-3068. doi: 10.1561 /04000000000000000.00 URL http:://xx.../// doi. ortoorgg////// 10. 1561/4499900066
G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient Streaming Language Models with Attention Sinks, Apr. 2024. URL
17453 ItXiv:2309.17453 [cs].
A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://aaxxv 1b///
org/abs 1809. 09600
13

---

Z. Zhang. Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, Z. Wang. and Chen. HS 2SO: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, Dec. 2023. http:///://rrx org/bbs/2366 14048 arXiv:2306.144 14048 [es].
M. Zhong, D. Yin, T. Yu, A. Zaidi, M. Mutuma, R. Jha, A. H. Awadallah, Celikyilmaz, Y. Liu, X. Qiu, and D. Radev. Qmsum: A new benchmark for query-based multi-domain meeting summarization, 2021.
05938
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the claims made in the abstract and introduction empirically validated in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings. is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
much the results
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have highlighted the limitations of work in conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
14

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Theoretical results are described in Section 3 with proofs being provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
All the experimental details are provided in Section 4.1 for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The source code used in the study will be provided in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https / //nipscc/pubb li httcc/guieseeeeee des wondddSSumissiooPoee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the experimental details are provided in Section 4.1 for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The statistical significance results of the experiments are provided in Section
4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experimental settings are described Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: There no societal impact of the work performed. The work conforms all the aspects of NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the assets used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode com/ donaattsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components. /ttpr///s.ccc//ooeeeeee///////// for
Please refer to our LLM policy what should or should not be deseribed.
20

---

A Proof of Lemma 3.1
Proof. Let A = roftmax(QK")) and let A' = softmax(QK" )v. In this proof, we will upper bound || - A'IIF. Throughout the proof, we also assume that K and K' have the same dimensions, which is achieved by adding zero rows to K' as necessary (and the same holds for V and V").
First, we have
A A' softmax (aaaax(QQQ) (QK"VV softmax((K")) softmax (o softmax(QK" y
softmax (atmaxx(K=22 (QK (V - V') (softmax (QK") - softmax
and hence using the triangle inequality for norms, we get
||A A'I ||softma((Q (V V'Ill= + I[softmax(QKk)) softmax -00tma(((""x"
Next, if || -lllap is the operator norm, then we have that ||AB|| ||Al| B|I Applying this to the two terms on the RHS above, we get
||A A < ||softma((Q"") lop - V'Il + (onfitmax(Q]] softmax(Q 1opp I |softmax(QK") [op V V'Il+ [|] (ccoftma(QK")) |lop ||softmax (QK") (oop)))
where in the second step above, we have again used the triangle inequality for norms. Now, we know that both softmax (QK2) and softmax(Q are row-stochastic matrices; hence, the I1-Ilop norm of both these matrices is exactly vn. So, we get that
| 1A-AA1 A'Il S VnllV V' + 22/nllVV
completing the proof of the claim.
B Datasets, Task Descriptions and Task Templates
B.1 The LongBench benchmark
LongBench [Bai et al.) 2023) is set of multitasks designed assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning. Tasks include single-document question-answering mutli-document question answering, summarization, few-shot learning, synthetic tasks and code generation. Below we present a detailed overview of all tasks:
Single-Doe QA: Models are required to obtain the answer from a single source document. Test samples are derived from variety of datasets, including NarrativeQA (Kočiský et al.)
2017], Qasper Dasigi et al. [021], and MultiFieldQA [iu et al. 2023a Templates for these tasks can be found in Table 6
Multi-Doc QA: Models are required to extract and combine information from multiple
source documents to obtain the answer. Data samples are derived from three multi-hop QA
datasets: HotpotQA YYngg et al. 2018). 2WikiMultihopQA [Ho et al. 20200) and MuSiQue
[Trivedi al. 2022 Templates for these tasks can be found in Table 7
Summarization: These tasks require a comprehensive understanding of the context. The data samples are obtained from the GovReport (Huang et al. 2021 and QMSum (Zhong
et al. 2021] datasets. Templates for these tasks can be found in Table
Few-shot Learning: These tasks have classification, summarization and reading- comprehension tasks integrated within them to maintain task diversity. The TREC dataset and Roth 2002) is used for classification problems. The SAMSum dataset [Gliwa et al.) 2019] is used for summarization tasks. Finally. TriviaQA (Joshi et al.) (017] is utilized for comprehension tasks. Templates for these tasks can be found in Table
Synthetic Tasks: These tasks assess a model's ability to handle specific scenarios and patterns. In LongBench, the PassageRetrieval-een and PassageCount datasets are used. Templates for these tasks can be found in Table TT
Code Completion Tasks: These tasks are meant to assist users by completing code based user's input and context. The LCC dataset from the original Long Code Completion dataset [Guo et al. 2023] as well as the RepoBench-2 -P dataset [Liu et al. 2023b is used.
Templates for these tasks can be found in Table
21

---

Table 6: LongBench templates for Single-Doc QA tasks.
Task Template:
You are given story, which can be either a novel or movie script, and question. Answer the question as concisely you can, using a single phrase if possible. Do not provide any explanation.
NarrativeQA
Story: (context)
Now, answer the question based on the story as concisely as you can, using single phrase if possible. Do not provide any explanation.
Question: {question]
Task Template:
You are given scientific article and question. Answer the question concisely as you can, using single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable", If the question is a yes/no question, answer "yes", "no" or "unanswerable". Do provide any explanation.
Article: [context]
Qasper
Answer the question based on the above article as concisely as you can, using single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no" or "unanswerable". Do not provide any explanation
Question: (question)
Task Template: Read the following text and answer briefly.
(context)
MultificlQQA EN
Now, answer the following question based the above text, only give me the answer and do not output any other words.
Question: [question]
B.2 The Ruler benchmark
Ruler Hsich et al. 2024 is a collection of synthetic examples to evaluate long-context language models, containing four task categories to behaviours beyond simple retrieval from context. Here is a detailed description of each task category:
Singe Weedle-In-A-Haystee (S-): is embedded within lengthy text, called the "haystack". The objective of the task is to retrieve the needle from the context.
In these tasks, a keyword sentence, called the "needle".
Multi-keys NIAH (MK-): These tasks are similar to S- tasks, with the difference being the presence of multiple "needles" inserted into the "haystack". However, the task retrieve only one of them.
Multi-values NIAH (MV-): Here, multiple "needles" the form of key-value pairs are hidden in the "haystack". The task is to retrieve all values associated to a single key. Multi-queries NIAH (MQ-): Here, multiple "needles" in the form of key-value pairs are inserted into the "haystack", All "needles" corresponding to the keys specified in the queries are to retrieved.
Templates for these tasks can be found in Tables 12 and respectively.
22

---

Table LongBench templates for Multi-Doo QA tasks.
Task Template:
Answer the question based on the given passages. Only give me the answer and do not output any other words.
The following are given passages. [context]
HotpotQA
Answer the question based on the given passages. Only give me the answer and do not output any other words.
Question: {question)
Task Template:
Answer the question based on the given passages. Only give the answer and do not output any other words.
The following are given passages. [context]
2WikimQA
Answer the question based on the given passages. Only give me the answer and do not output any other words.
Question: (question)
Task Template:
Answer the question based on the given passages. Only give me the answer and do not output any other words.
The following are given passages. [context)
Musique
Answer the question based the given passages. Only give the answer and do not output any other words.
Question: [question]
C Additional Results
C.1 Results LongBench
We evaluate the performance of CurDKV and AdaCurDKV against baseline compression methods intermediate compression ratios of 50% and 70%, using both the L.L.MA-31-880 and Mistral-7B in Table 14 and Table [5] respectively.
On LLaMA, CurDKV achieves the highest average accuracy (48.3), outperforming all non-adaptive baselines such as SnapKV (44.3) and Knorm (43.1), ), as well as Streaming L.M (37.7). The adaptive variant AdaCurDKV also performs competitively, with an average score of 47.9, surpassing AdaS- napKV (44.7). At 70% compression, CUR-based approaches continue to perform favorably. On LLaMA, CurDKV again achieves the highest average (45.4), ahead of SnapKV (41.9) and Knorm (36.4), confirming its robustness across moderate compression levels. AdaCurDKV further improves upon AdaSnapKV (42.4) with an average of 46.1, marking consistent gains across adaptive strategies.
On Mistral, CurDKV again delivers the best average score (43.8), outperforming SnapKV (41.2) and ChunkKY (41.0). AdaCurDKV (43.7) also surpasses AdaSnapKV (41.9), particularly benefiting from adaptive budget reallocation heterogeneous tasks such as TREC (72.0) and SAMSum (45.4). Across both models, CUR-based methods show superior stability, particularly under structured reasoning and information dense settings.
23

---

Table 8: LongBench templates for Summarization tasks.
Task Template:
You are given a report by a government agency. Write a one-page summary of the report.
Gov Report
Report: [context)
Now, write a one-page summary of the report.
Task Template:
You are given meeting transcript and query containing question or instruction. Answer the query in one or more sentences.
Transcript [context]
QMSum
Now, answer the query based on the above meeting transcript in one or more sentences.
Query: [question]
Task Template:
You are given several news passages. Write a one-page summary of all news.
Multi News
News
[context]
Now, write one- one-page summary all the news.
Table 9: LongBench templates for Few-shot learning tasks.
Task Template:
Please determine the type of the question below. Here are some examples of questions.
TREC
[context)
(question)
Task Template:
Answer the question based on the given passage. Only give me the answer and do not output any other words. The following are some examples.
TriviaQA
(context)
(question)
Task Template:
Summarize the dialogue into few short sentences. The following some examples.
SAMSum
(context)
(question)
24

---

Table 10: LongBench templates for Synthetic tasks.
Task Template:
There are some paragraphs below sourced from Wikipedia. Some of them may be duplicates. Please carefully read these paragraphs and determine how many unique paragraphs there are after removing duplicates. In other words, how many non-repeating paragraphs are there in total?
Passage Count
[ontext]]
Please enter the final count of unique paragraphs after removing du- plicates. The output format should only contain the number, such 2, 3, and on.
Task Template:
Here are 30 paragraphs from Wikipedia, along with abstract. Please determine which paragraph the abstract from.
[context]
Passage Retrieval EN
The following is an abstract.
[question]
Please enter the number of paragraph that the abstract from. The answer format must be like "Paragraph 1", "Paragraph 2", etc.
Table 11: LongBench templates for Code tasks.
Task Template:
Please complete the code given below. [context)
Lcc
Next line code:
Task Template:
Please complete the code given below. [context)
Repobench-P
{question] Next line code:
C.2
Results on Ruler
Table T6 and 17 report the results of different KV compression models with LLaMA and Mistral models, respectively, at 50% and 70% compression ratios.
With LLaMA, 50% compression, CurDKV achieves strong average of 83.1, outperforming all non- adaptive baselines, including ChunkKV (79.2) and SnapKV (67.0). The adaptive variant, AdaCurDKV further improves performance to 89.6, benefiting from its dynamic budget allocation across heads. Notably, AdaCurDKVachieves perfect accuracy (100.0) on 6 out subtasks, demonstrating its ability to preserve high-salience tokens even moderate compression. At 70% compression, the relative advantage of CUR-based methods persists. CurDKV scores an average of 62.0, substantially ahead of V (48.0) and Knorm (32.7), however, falling short from the best non-adaptive baseline ChunkKV (64.1) 1). AdaCurDKV continues to perform best overall with an average of 69.8, highlighting its robustness in token- token-starved regimes. These results reinforce the effectiveness of CUR-base compression for preserving semantically critical tokens required for pinpoint retrieval.
On Mistral at 50% compression, CurDKV achieves an average score of 54.2, considerably higher than SnapKV (23.9), Knorm (12.0), and Streaming LLM (41.3). AdaCurDKV pushes this further to 56.1, showing consistent gains across nearly all subtasks.
As shown Tables 19 even for a longer context length of 128K, CurDKV achieves the strongest overall performance on the NIAH benchmark, particularly under higher compression. At 30% compression, attains the highest average accuracy (79.95%), marginally surpassing SnapKV V while maintaining consistent gains across multi-key and multi-value retrieval tasks. Under 50% compression, the
25

---

Table 12: Ruler templates for S and MK tasks.
Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards. The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.
S-1
One of the special magic numbers [word] [number). ...... What is the special magic number for (word) mentioned in the provided text?
The special magic number for (word) mentioned in the provided text is
Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards. Paul Graham Essays.
S-2
One the special magic numbers [word] [number). What is the special magic number for (word) mentioned in the provided text?
The special magic number for (word) mentioned in the provided text is
Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. I will quiz you about the numbers afterwards.
Paul Graham Essays
S-3
One of the special magic numbers for [word) is: [number). What is special magic number for (word) mentioned in the provided text?
The special magic number for (word) mentioned in the provided text is
Task Template:
Some special magic numbers are hidden withia the following text. Make sure to memorize it. will quiz you about the numbers afterwards.
Paul Graham Essays.
One of the special magic numbers for [word-1] } is: [number-]] I ].
One of the special magic numbers for [word-2] is: (number-2)
MK-1
One of the special magic numbers for [word-3) is: [number-3),
One of the special magic numbers for [word-4) is: [number-4).
What is the special magic number for (word-4) mentioned in the provided text?
The special magic number for [word-4) mentioned in the provided is
Task Template:
Some special magic numbers are hidden within the following text. Make sure memorize will quiz you about the numbers afterwards.
Paul Graham Essays.
One of the special magic numbers for (word-1) is: (number-) 1].
One of the special magic numbers for [word-2] is: [number-2)
MK-2
One of the special magic numbers for [word-x] is: [number-x]
One of the special magic numbers for [word-n-1] is: [number-n-l]
One of the special magic numbers for (word-n) is: (numher-n)
What the special magic number for (word-x) mentioned the provided text?
The special magic number for (word-x) mentioned in the provided
text is
Task Template:
Some special uuids are hidden within the following text. Make sure to memorize it. I will quiz you about the uuids afterwards.
Paul Graham Essays.
One of the special magic uuids for [uuid-1] is: [uuid-1]. ......
One of the special magic uuid (uuid-2) is: (uuid-2) ......
MK-3
One of the special magic uuid (uuid-x) is: [uuid-x) .....
One of the special magic uuid for (uuid-n-1) 1) is: [uuid-n-1).
One of the special magic uuid for [uuid-n] is: [uuid-n).
What is the special magic number for (uuid-x) mentioned in the provided text?
The special magic number (uuid-x) mentioned in the provided
text is

---

Table 13: Ruler templates for MV and MQ tasks.
Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. will quiz you about the numbers afterwards.
Paul Graham Essays.
One of the special magic numbers for [word] is: [number-I
One of the special magic numbers for [word] is: (number-2)-
MV
One of the special magic numbers for [word) is: (number-3)
One of the special magic numbers for [word] is: (number-4).
What are all the special magic numbers for (word) mentioned in the provided text?
The special magic numbers for [word) mentioned in the provided text are
Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it. will quiz you about the numbers afterwards.
Paul Graham Essays.
One of the special magic numbers for (word=1) is: (number-1e 1]
One of the special magic numbers for [word-2) is: (number-2) ).
MQ
One of the special magic numbers for (word-3) is: (number-3)
One of the special magic numbers for [word-4) is: (number-4)
What are all the special magic numbers for (word-1), (word-2), (word-3), and [word-4] mentioned in the provided text?
The special magic numbers for (word-1), [word-2), [word-3), and [word-4) mentioned in the provided text are
advantage becomes more pronounced, where CurDKV yields the top accuracy on most long-context string-
string-matching matching tasks and sustains a clear lead in overall average (68.69 (8.). These results indicate that value-aware KV compression enhances retrieval fidelity even when memory budgets are severely constrained, preserving precise information access across longer contexts.
Albeit the minor performance drops, these experiments confirm that CUR-based selection, especially when adaptively guided, provides a robust and effective mechanism for maintaining retrieval fidelity under tight memory constraints, outperforming both norm- and attention-based KV selection across two architectures and multiple compression levels.
C.3 Ablation Study
We perform two ablation studies to investigate the key design decisions behind CurDKV (1) the module on which leverage scores are computed, and (2) the effect of using randomized projections for score approximation. While in our algorithm we combined leverage scores computed using both key and value vectors (line of Algorithm []. in our ablations we study the affect of using only the key value vectors to compute leverage scores. We also experiments wherein leverage scores are computed on the original key/value matrices without the use of the Gaussian projection (i.e., we drop lines 3-4 of Algorithm
We first ablate the module on which leverage scores are computed. Figure 3 shows the average performance (task-level scores reported in Table (8) when using key-only, value-only, and the default key-value product (elementwise product key and value leverage scores). All three variants perform similarly at 30% compression. However, at 90% compression, the differences become more pronounced: key-only scores degrade significantly. while value and key-value product maintain higher median accuracy and robustness. This confirms that value information is more predictive of output quality, and that combining key and value structure yields the most reliable token selection strategy.
Next, we evaluate the role of randomized leverage score approximation. This determines whether the key and value matrices are projected to low-rank space via Gaussian random matrices before
27

---

Table 14: Results of KV compression methods on LongBench tasks with LLaMA-8B at 50% and
70% compression ratios compression ratio).
Task
ChunkKV
Knorm
Streaming LLM
SnapKV
CurDKV
AdaSnapKV
AdaCurDKY
Non-Adaptive Methods
Adaptive Methods
NrtQQA
28.8
30.4
29.2
Qasper
39.6
37.1
49.0
43.4
48.1
Mf--nn
43.7
49.3
32.0
45,4
54.9
51.5
52.4
HotpotQA
55.8
55.3
50.1
57.3
57.2
56.0
55.6
2WikiMQA
49.6
44.7
38.0
50.6
49.1
51.1
49.3
Musique
27.9
26.4
23.7
30.7
34.7
30.0
32.0
Gov Report
32.7
33.4
32.0
32.9
QMSum
23.0
24.3
21.9
23.8
24.7
24.3
25.2
50%
Multi News
25.6
25.4
25.6
25.4
26.5
25.9
26.6
TREC
23.0
60.8
31.0
36.5
68.0
36.0
67.5
TriviuQA
84.9
88.4
92.0
85.5
92.3
86.1
92.9
SAMSam
39.8
40.7
40.5
41.2
Pcount
10.6
12.8
11.1
11.1
Pre
97.0
86.5
54.0
100,0
98.5
100.0
99.0
Lee
50.9
32,6
48.8
53.3
49.9
51.1
50.4
RB-P
49.2
49.3
48.5
47.7
50.4
47.4
52.3
Average
42.6
43.1
37.7
44.3
48.3
44.7
47.9
NrivQA
30.9
30.4
Qasper
3166
32.5
27.8
30.8
43.2
34.9
44.2
MF-en
34.6
42.7
26.7
37.0
45.1
41.9
46.5
HotpoiQA
53.6
49.1
43.5
54.8
$1.7
55.6
54.6
2WikiMQA
38.5
33,3
30.9
46,9
45.2
47.5
45.5
Musique
25.8
19.7
20.8
27.5
29.3
29.9
30.7
Gov Report
30.6
29.4
31,2
29.3
30.7
QMSum
22.3
23.5
21.1
22.6
24.4
22.9
24.5
70%
Multi News
23,5
23,8
24.2
23.8
26.0
24,4
25.9
TREC
15.0
54.5
31.5
39.0
58.0
34.0
63.7
TriviaQA
84.0
74.9
91.5
84,5
92.0
85.3
91.9
SAMSum
41.5
41.7
41.2
Pcount
7.6
10.1
10.8
10.1
9.7
Pre
92.5
58.5
36.5
93.0
98.5
96.0
98.5
Lee
51.0
28.5
50.9
53.2
46.7
50.6
46.4
RB-P
49.4
48.6
49.9
47.8
51.6
45.9
52.8
Average
"39.2
33..4
41.9
75.4
46.1
computing leverage scores. We observe marginal differences between the two variants. At 30% compression, both versions perform similarly median accuracy, though random projection slightly improves the worst-case performance. At 90% compression, however, random projections result in modest improvements in median and lower-quartile accuracy. These gains suggest that random projections can help stabilize leverage score estimation under extreme compression, but their effect is secondary the choice of leverage module.
We analyze the robustness of AdaCurDKV with respect to the safeguard parameter a and the Gaus- sian projection dimension т. As shown in Tables 20al and 20b] performance remains remarkably stable across a E {0.1, 0.5, 0.9), indicating that the adaptive safeguard mechanism is largely self- regularizing. The average scores vary by less than one point even under aggressive 90% compression, demonstrating strong resilience. Increasing consistently improves performance, saturating around r 100, suggesting that moderate projection ranks suffice for accurate subspace estimation without incurring high overhead. At the same time, increasing r converges to the exact CUR UUR-decomposition of key-value matrices.
C.4 Runtime Analysis
We analyze the runtime behavior of CurDKi and SnapKV V under varying compression ratios and input lengths, as illustrated Figures 4 and 6 We focus on three key metrics: KV cache size, prefill time, and generation time.
Cache Size. Both methods exhibit nearly linear reduction in KV cache size as compression increases. For example, with 128K token input, the cache size reduces from approximately 16 GB at 0% compression to under 2 GB at 80% compression for both CurDKV and SnapKV (Figures 4 and
28

---

Table 15: Results KV compression methods on LongBench tasks with Mistral-7B at 50% and 70%
compression ratios compression ratio).
Task
ChunkKV
Knorm
Streaming
SnapKV
CurDKY
AdaSnapKY
AdaCurDKY
Non-Adaptive Methods
Adaptive Methods
NrtvQA
24.6
20.5
23.5
24.3
25.6
24.4
25.1
Qasper
30.4
29.0
30.8
32.6
39.4
30.3
39.2
MF-en
47.1
51.3
49.4
HotporQA
47.0
45.5
47.2
42.7
48.5
44.8
2WikiMQA
35.0
31.3
30,5
36.5
39.5
36.8
37.7
Musique
25.1
22.5
18.6
23.0
27.7
26.0
26.9
Gov Report
32.4
30.0
31.7
31.6
33.2
31.7
31.4
50%
QMSum
23.7
24,9
24.0
24.9
Multi News
25.2
25.1
25.2
26.6
25.5
26.7
TREC
48.8
40.7
56.0
50.0
69.0
51.0
72.0
TriviaQA
86.8
86.9
69.9
86.5
87.4
87.7
85.2
SAMSum
22.3
36.6
21.3
21.4
45.7
22.7
45.4
Peount
1.5
4.7
5.9
5.8
6.6
Pre
96.0
78.0
97.5
68.5
52.4
32.8
52.2
47.8
52.1
57.4
RB-P
55.5
54.3
53.9
56.8
55.3
57.3
58.7
Average
41.0
37.0
35.2
41.2
43.8
41.9
43.7
NπTVQA
21.3
16.5
20.4
23.0
21.02
23,2
19.1
Qasper
21.4
21.6
24.2
35.1
25.1
35.0
MF-en
38.2
46.8
42.1
HotpotQA
44.1
38.9
38.1
43.9
38.2
45.5
34.3
2WikiMQA
33.6
26.0
26.9
29.2
32.3
31.7
33,9
Musique
20.9
18.9
16.7
21.2
18.0
21.9
15.3
Gov Report
29.9
27.1
29.6
29.7
27.0
29.2
23.3
QMSum
21.9
23.6
23.1
70%
Multi News
23.5
23.7
25.4
24.3
25.1
TREC
38.3
36.0
46.0
46.0
65.5
48.8
66.0
TriviaQA
87.6
86.3
62,8
86.8
59.4
87.0
52.4
SAMSum
25.4
39.6
25.7
22.2
42.4
23.0
42.1
Peount
5.6
5.9
5.1
4.2
5.7
25.5
Lec
54.1
29.5
$2.2
53.54
43.6
53.0
55.3
RB-P
55.3
55.0
53.1
56.0
52.2
56.5
57.0
Average
38.22
33200"
31.6
38.9
35.1
39.5
34.0
Table 16: Results of all KV compression baselines needle-in-a needle-in-a-haystacck subtasks with LLaMA-8B at 50% and 70% compression ratios compression ratio).
SubTask ChunkKV Knorm Streaming LLM
SnapKV
CurDKV
AdaSnapKV
AdaCurDKV
Non-Adaptive Methods
Adaptive Methods
S-1
100.0
100.0
38.7
100.0
100.0
100.0
S-2
100.0
96.5
40.4
98.3
100.0
100.0
100.0
S-3
88.2
9.8
51.0
11.8
25.5
60.8
54.9
56.3
100.0
100.0
50%
MK-2
40.0
6.7
48.9
40.0
82.2
93.3
95.6
MK-3
29.8
12.8
36.2
19.2
61.7
46.8
68.1
MV
84.2
80,3
48.0
80.9
96.7
100.0
100.0
Average
770,2
56.1
46.7
67.0
35.
87.4
$9.6
100.0
100.0
16.1
100,0
100.0
100.0
S-2
89.5
50,9
26.3
86.0
98.3
100.0
98.3
S-3
70.6
2.0
31.4
3,9
15.7
13.7
MK-1
79.2
33.3
35.4
75.0
87.5
97.9
97.9
70%
MK-2
20.0
0.0
22.2
24.4
44.4
53.3
57.8
MK-3
17.0
19.2
4.3
MQ
79.8
26.4
33.7
70.2
94.7
94.7
MV
56.6
40.8
25.0
42.8
62.5
92.8
91.5
Average
32.7
69.3
29

---

Table 17: Results of all KV compression baselines on beedle-in-a-haystace subtasks with Mistral-7B at 50% and 70% compression ratios
compression ratio).
SubTask
ChunkKV
Knorm
Streaming LLM
SnapKV
CurDKV
AdaSnapKV
AdaCurDKY
Non-Aduptive Methods
Adaptive Methods
S-1
87.1
95.2
37.1
50.0
69.4
69.4
71.0
S-2
86.0
40.4
5-3
33.3
0.1
51.0
21.6
2,0
15.7
MK-1
77.1
0.0
50.0
20.8
79.2
43.8
77.1
50%
MK-3
25.5
0.0
14.9
12.8
2.1
38.3
10.6
MQ
79.8
0,5
40.9
16.4
70.7
33.7
70.2
MV
79.0
0.7
47.4
82.2
32.9
81.6
Average
62.9
12.0
41.3
23.5
54.2
41.2
56.1
S-1
88.7
80.7
14.5
45.2
19.4
69.4
12.9
S-2
56.1
0.0
26.3
68.4
40.4
52.6
S-3
11.8
0.01
31,4
2.0
0.0
2.0
0.0
MK-1
35.4
27.1
20.8
37.5
70%
MK-2
13.3
17.8
24.4
2.2
MK-3
17.0
0.0
10.6
8.5
0.0
19.2
0.0
MQ
26.0
12.0
13,9
MV
44.7
0.0
25.0
15.1
63,8
17.8
43.4
Average
39.5
10.1
22.3
16.3
314
26.0
"1..""
Table 18: Ablation of CurDKV LongBench datasets. We calculated leverage attention modules (key, value and key-value both) with and without random Gaussian projections
different
compression ratio).
Key
Key+random
Key-value
Key-value-random
Value+random
NriQQ
24.0
24.9
31.6
31.9
30.6
30.1
Qasper
24.8
27.5
49.5
48.1
48
48.5
MF-en
40,9
43.2
57.1
55.6
56.3
HorpotQA
51.6
54.5
58.0
56.9
57.0
58.0
2WikiMQA
38.7
44.9
48.6
4844
50.3
50.7
Musique
23.4
33.2
32.7
Gov Report
30,9
32.0
33.7
33.6
34.6
34.3
30%
QMSum
24.3
24.4
25.2
25.1
24.7
Multi News
10.6
26.6
26.8
26.7
26.9
TREC
33.0
25.0
65.5
54.5
66.5
62.5
TriviaQA
91.0
90.9
92.1
92.4
91.7
SAMSum
36.8
38.8
41.6
39.9
40.2
40.0
Pcount
2.5
7.0
8.0
9.2
11.4
12.1
Lec
52.9
53.5
51.1
51.6
50.7
51.4
RB-P
51.7
50.4
50.1
50.0
49.9
49.4
Average
48.3
NriQA
20.4
21.7
28.5
20.9
27.8
25.4
Qusper
15.3
15.6
28.5
25.8
29.7
24.9
MF-en
26.7
34.7
HorpotQA
25.1
28.0
38.4
39.7
47.2
46.9
2W1kiMQA
20.9
21.7
30.4
30.7
31.1
Musique
9.5
20.0
19.7
21.9
Gov Report
14.7
19.1
24.1
24.0)
26.8
25.9
QMSum
18.9
22.3
22.3
90%
Multi News
8.6
15.2
22.1
21.9
23.1
22.8
TREC
1.5
6.5
7.0
15.3
22.5
16.0
TriviaQA
87.0
90.7
91.7
SAMSum
32.5
32.2
30.4
27.9
37.2
33.2
Pcount
0.0
0.0
1.0
1.0
5.7
5.6
Loe
33,5
32.0
39.7
38.2
34.9
35.3
RB-P
55.3
55.1
54.0
53.6
49.6
51.9
Average
23.3
25.0
32.3
31.8
35.2
33.7
This confirms that CUR-based token selection inherits the cache sparsity advantages of heuristic methods like SnapKV.
Prefill Time. SnapKV maintains flat prefill latency regardless of compression. For instance, at 128K tokens, prefill time remains consistently around 10 seconds across all compression levels (Figure (p). CurDKV by contrast, incurs higher latency due to leverage score computation, rising from 10 seconds at 0% compression to about 14-15 seconds at 60-80% compression for 128K tokens
30

---

Table 19: Results the geedle-in-a-haystee benchmark (128K context length) at 30% and 50%
compression with LLaMA-8B model.
Task
ChunkKY
KNorm
SnapKV
Streaming
CurDKV
S-1
100,00
100,00
100.00
55.56
88.89
S-2
87.50
87.50
100.00
62.50
87.50
S-3
90.91
81.82
63.64
54.55
81.82
MK-1
92.31
69.23
100.00
69.23
84.62
30%
MK-2
50.00
0.00
50.00
78.57
92.86
MK-3
20.00
10.00
10.00
20.00
10.00
MQ
95.00
82.50
100.00
70.00
97.50
MV
92.86
82.14
92.86
67.86
96.43
Average
78.57
64.15
77.06
59.78
79.95
S-1
100.00
100,00
100.00
44,44
100.00
S-2
62.50
62.50
100.00
50.00
75.00
S-3
72.73
36.36
9.09
36.36
36.36
MK-1
84.62
23.08
92.31
53.85
84.62
50%
MK-2
35.71
0.00
42.86
50.00
57.14
MK-3
20.00
0.00
10.00
10.00
10.00
MQ
85.00
45.00
95.00
40.00
90.00
82.14
46.43
92.86
39.29
96.43
Average
67.84
39.17
67.77
40.49
68.69
25
(a)
Prefilling time Iseconds
30
(b)
Generation time (secands)
20
(c)
1K tokens
64K tolkens
1K tokens
64K takens
1K takens
64K tokens
Cache size (GB)
25
tokens
15
16K tokens
20
16K tokens
16K tokens
15.
10
10
5
20
40
60
80
20
40
60
80
20
8D
Compression ratio (in %)
Compression ratio (in %)
Compression ratio (in %)
Figure 6: Prefilling and generation statistics SnapKV for different sequence lengths with LLaMA- 8B.
(Figure 4p). For shorter sequences like 4K or 16K, the difference is marginal, under I second for both methods.
Generation Time. CurDKV demonstrates a stronger reduction in generation time, particularly for long sequences. For 128K-token imputs, generation latency drops from 1~2 seconds at 0% compression to just ~4.5 seconds at 80% compression (Figure 4p). SnapKV also sees drop from 15 seconds to ~6 seconds, but exhibits irregular behavior for mid-length sequences (e.g., 64K shows a flat profile between 80% compression in Figure Gb).
While SnapKV is faster to prefill, CurDKV yields larger reductions in generation time, especially for long sequences (e.g., 128K tokens) and high compression levels. Its additional prefill cost roughly 3-5 seconds at worst is offset by its smoother and more aggressive runtime scaling, making it more efficient choice for long-con contecontext inference when accuracy and latency matter.
31

---

Table 20: Sensitivity and ablation analysis for CurDKV on selected LongBench tasks.
(a) Sensitivity safeguard parameter
2WikiMQA
HotpotQA
MF-en
Qasper
Avg
0.1
49.13
58.74
55.82
48.36
53.01
30%
0.5
49.13
58.74
55.82
48.36
53.01
0.9
48.99
58.67
56.33
48.46
53.11
0.1
34.50
45.91
30.80
28.75
34.99
90%
0.5
32.73
46.61
30.15
29.83
34.83
0.9
32.27
46.44
31.37
28.99
34.77
(b) Ablation on Gaussian projection dimension
2WikiMQA
HotpotQA
MF-en
Qasper
Avg
50.99
58.77
55.48
47.97
53.30
30%
50.71
59.21
55.71
47.33
53.24
49.31
57.85
54.78
47.83
52.44
100
52.07
58.23
56.41
49.52
54.06
28.45
41.80
29.29
25.50
31.26
90%
10
35.04
41.98
29.94
26.02
33.24
50
34.25
45.48
32.30
28.64
35.17
100
35.30
46.52
33.15
28.13
35.77
32


Reviews:
[{"review_id": "tLevHR3J2x", "reviewer": "Reviewer_kZ3Z", "summary": "This paper introduce CurdKV, that rethinks kv cache compression by consider not only keys but also values via a leverage score metric. Specifically, Curdkv uses cur decomposition to identify more important individual keys and values via leverage score, where a importance is measures for rows and cols of the key and value matrices. In practice, this process is done on a group level to compatible with GQA. And to avoid svd it use a random project for the decomposition to reduce overhead. At last, an adaptive budget version of curdkv is proposed.", "questions": "No", "limitations": "No", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The proposed kv cache include values in the kv cache compression process, that were overlooked previously. And this is well motivated by fig 1 and introduction.\n\n2. The proposed method to use cur decomposition is solid when identifying the important rows and cols.\n\n3. To use a random projection for leverage score computing seems a good trick to avoid svd. And the extra cost involved is acceptable. Plus this importance evaluation process can be fit inside flash-attn, which is important in real applications.\n\n4. Solid overall performance on longbench and the NIAH test.\n\nWeakness\n\n1. Algo 1 is not very clear to me. How's the leverage score is used for the decomposition?\n\n2. The max length in the experiments is rather limited. I would expect to test upto 128k on RULER.\n\n3. Why not include a latency comparision with and without curd kv? to show the exact overhead used in computing leverage score and the ranking process.", "was_revised": true, "final_justification": "I'll keep my score.", "rebuttal": "We thank the reviewer for giving us a comprehensive analysis of our work. We are grateful that the reviewer found our approach of using values to compress KV caches well-motivated. Below, we would like to address all concerns of the reviewer and hope that they answer all their questions.\n\n**A clearer description of Algorithm 1.** Here we give a clearer description of the algorithm, with a particular focus on how leverage scores are used to compute the decomposition.\n\n1. Consider a set of key and value matrices $K\\in\\mathbb{R}^{k\\times d}$ and $V\\in\\mathbb{R}^{k\\times d}$, i.e $k$ key-value pairs, each of dimension $d$.\n2. Project key-value vectors do a vector of dimension $r$ (we use $r = 20$) via a Gaussian projection matrix $G\\in\\mathbb{R}^{d\\times r}$ (lines 2-3 of the algorithm). The significance of this step is to reduce the overall run-time complexity (computing exact leverage scores requires SVD).\n3. After this, we compute *leverage scores* using the new key and value vectors (line 4 of the algorithm). In simple words, these scores are just row norms of the key-value matrices.\n4. The key-value leverage scores are then combined (line 5) and normalized (line 6).\n5. Finally, the indices with the top-$(k - s)$ scores are computed, and the keys and values for these indices are retained in the KV cache (lines 8-10 of the algorithm).\n6. Note that we preserve the first $s$ tokens in the context due to the prevalence of *attention sinks*. [1]\n\n***Note***: We've further provided an ablation study involving different values of $r$ for reviewer hDao. TL;DR: we observe only marginal differences when using different values of $r$, and the Gaussian approximation improves as $r$ increases (which is intuitive).\n\n**Regarding the maximum length in experiments for the RULER benchmark.** As per the reviewer's request, we ran NIAH experiments on the **RULER** benchmark with a context length of 128K tokens. We've reported the results in the table given below. As in the main paper, we compare `CurDKV` with the `ChunkKV`, `KNorm`, `SnapKV` and `StreamingLLM` baselines. We carry out the experiments with compression ratios of $30$%.\n\n**Table 1:** Results on the **RULER** benchmark with 128K context length with a compression ratio of $30$%.\n\nTask/Method| `ChunkKV` | `KNorm` | `SnapKV` | `StreamingLLM` | `CurDKV` |\n| -------- | -------- | -------- | -------- | -------- |  -------- |\n| `Accuracy.niah_multikey_1.string_match` | 92.31     | 69.23     | **100**     |  69.23 |  84.62 |\n| `Accuracy.niah_multikey_2.string_match` | 50     | 0     | 50     |  78.57 |  **92.86** |\n| `Accuracy.niah_multikey_3.string_match` | **20**     | 10     | 10     |  **20** |  10 |\n| `Accuracy.niah_multiquery.string_match` | 95     | 82.5     | **100**     |  70 |  97.5 |\n| `Accuracy.niah_multivalue.string_match` | 92.86     | 82.14     | 92.86     |  67.86 |  **96.43** |\n| `Accuracy.niah_single_1.string_match` | **100**     | **100**     | **100**     |  55.56 |  88.89 |\n| `Accuracy.niah_single_2.string_match` | 87.5     | 87.5     | **100**     |  62.5 |  87.5 |\n| `Accuracy.niah_single_3.string_match` | **90.91**     | 81.82     | 63.64     |  54.55 |  81.82 |\n| `Average` | 78.572    | 64.15     | 77.06    |  59.78 |  **79.95** |\n\n**Table 2:** Results on the **RULER** benchmark with 128K context length with a compression ratio of $50$%.\n\nTask/Method| `ChunkKV` | `KNorm` | `SnapKV` | `StreamingLLM` | `CurDKV` |\n| -------- | -------- | -------- | -------- | -------- |  -------- |\n| `Accuracy.niah_multikey_1.string_match`       | 84.62  | 23.08 | **92.31**  | 53.85        | 84.62 |\n| `Accuracy.niah_multikey_2.string_match`       | 35.71  | 0     | 42.86  | 50.00        | **57.14** |\n| `Accuracy.niah_multikey_3.string_match`       | **20.00**  | 0     | 10.00  | 10.00        | 10.00 |\n| `Accuracy.niah_multiquery.string_match`       | 85.00  | 45.00 | **95.00**  | 40.00        | 90.00 |\n| `Accuracy.niah_multivalue.string_match`       | 82.14  | 46.43 | 92.86  | 39.29        | **96.43** |\n| `Accuracy.niah_single_1.string_match`         | **100.00** | **100.0** | **100.00** | 44.44        | **100.00** |\n| `Accuracy.niah_single_2.string_match`         | 62.50  | 62.50 | **100.00** | 50.00        | 75.00 |\n| `Accuracy.niah_single_3.string_match`         | **72.73**  | 36.36 | 9.09   | 36.36        | 36.36 |\n| `Average`                                 | 67.84  | 39.17 | 67.77  | 40.49        | **68.69** |\n\t\t\t\t\t\t\nAs we see from the above results, `CurDKV` consistently either outperforms other methods or performs in a way sufficiently comparable to the best baseline for a given task, leading to the overall best average score.\n\n**Note:** For NIAH 128K experiments, we chose 30% and 50% compression ratios for comparing the KV compression methods.  At higher compression ratios (>50%), most of the methods' performance drops poorly (almost to 0 for most tasks) due to a lack of the appropriate context length. This behavior is more prevalent in the 128K context than in shorter context NIAH tasks (reported in the paper).\n\n**A latency comparison with and without `CurDKV`.** Per request of the reviewer, here we provide a latency comparison of other baselines against `CurDKV`. We compare four methods: exact `CurDKV` (where exact leverage scores are computed), random `CurDKV` (where the random Gaussian projection is used to project the key-value vectors, `SnapKV` and `ChunkKV`. We do comparisons on a context length of 128K tokens and generated length of 100 tokens, and we use compression ratios of $30$% and $90$%. We report the total runtime, prefill time, and the generation time for all methods. Here are the detailed results:\n\n**Table 3:** Runtime results for a context length of 128K tokens and a compression ratio of $30$%.\n\n| Method          | Length | Total time  (in sec)     | Prefill time  (in sec)   | Generation time (in sec) |\n| --------------- | ------ | ---------------- | ---------------- | ---------------- |\n| Exact `CurDKV`  | 128K | 55.39 | 46.51 | 8.88 |\n| Random `CurDKV` | 128K | **44.76**  | 35.17 | 9.59 |\n| `SnapKV`        | 128K | 44.97  | **33.37** | 11.59 |\n| `ChunkKV`       | 128K | 54.36 | 50.53 | **3.83** |\n\nAs we see from the above table, `CurDKV` with a random Gaussian projection outperforms the baselines for a compression ratio of $30$%. This is also intuitive, since the Gaussian projection reduces the effective dimension of the key-value pairs. This, combined with the computationally efficient leverage scores, leads to the overall smallest total time (prefill + generation) for random `CurDKV`.\n\n**Table 4:** Runtime results for a context length of 128K tokens and a compression ratio of $90$%.\n| Method          | Length | Total time       | Prefill time     | Generation time  |\n| --------------- | ------ | ---------------- | ---------------- | ---------------- |\n| Exact `CurDKV`  | 128K | 51.36 | 46.90 | **4.46** |\n| Random `CurDKV` | 128K | 42.82 | 35.41 | 7.41 |\n| `SnapKV`        | 128K | 44.50 | **33.59** | 10.91 |\n| `ChunkKV`       | 128K | **42.74** | 35.83 | 6.91 |\n\nFor a high compression ratio of $90$%, random `CurDKV` still performs approximately the same as the best baseline (`ChunkKV` for this case). This, combined with the performance gains of `CurDKV`, makes it a competitive choice.\n\n**References**\n\n[1] Efficient Streaming Language Models with Attention Sinks, Xiao et al., 2024."}, {"review_id": "muemYMeGT3", "reviewer": "Reviewer_1huD", "summary": "The paper propose CurDKV for compressing the key-value (KV) cache, a value-centric strategy. They estimate each token’s leverage score via a (fast, projected) CUR matrix decomposition of the attention output and keep only the most influential keys and values.", "questions": "see weaknesses", "limitations": "Yes, the paper discusses the pertinent limitations and outlines directions for future work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. CUR-based leverage scores + Gaussian projection yield an O(n d r) lightweight scoring pass; algorithm integrates with FlashAttention & Grouped-Query Attention without kernel changes.\n\n2. Memory scales predictably; although pre-fill cost rises by ~3–5 s for 128 K contexts, generation is faster for long sequences\n\n\nWeaknesses\n\n1. Static Compression Only. CurDKV freezes the cache after prefill; it cannot react if later tokens shift relevance, and it adds noticeable prefill overhead for very long inputs\n\n2. Limited Model Coverage. It is better to provide more results on bigger models. The results use 7 B / 8 B LLaMA/Mistral; scalability to 34 B–70 B or MoE models is untested.\n\n3. Hyper-Parameter Sensitivity. Adaptive safeguard α (0.20) is fixed; sensitivity analysis would help.", "was_revised": true, "final_justification": "After reading the authors’ rebuttal and the comments from other reviewers, most of my concerns, including some experimental justifications involving larger models, have been addressed.\n\n Therefore, I am raising my score by one point.", "rebuttal": "We thank the reviewer for giving us valuable insights to the potential weaknesses of our work, and we aim to address some of those concerns below. We hope the following discussion clarifies some concerns raised by the reviewer.\n\n**`CurDKV` as a static compression method.** It is true that `CurDKV` is a static compression method, in the sense that once a key-value pair of the context is evicted from the cache, `CurDKV` never recovers it (even if future tokens shift relevance). However, we'd like to point out that some of the best-performing baselines which we compare against are also static in this sense: `H2o`, `SnapKV`, `StreamingLLM` and `AdaKV` to name a few. Infact, any reasonable KV cache compression technique must be static in this sense, because the recovery of an evicted key-value pair from the cache must inevitably require either 1) storing the key-value pair in GPU memory or 2) recomputing the key-value pair, both of which may have significant overhead for larger context lengths. \n\n**Results on bigger models.** Per request of the reviewer, we tested `CurDKV` on larger models; particularly, we chose the Qwen3 14B and Qwen2.5 32B models, and tested the efficacy of `CurDKV` on these models against other strong baselines. As in the paper, we report results on tasks from the **LongBench** benchmark. We report results for compression ratios of $30$% and $90$% respectively.\n\n**Table 1:** **LongBench** evaluation results for the Qwen 14B model for a compression ratio of $30$%.\n\n| Method | LongBench/2wikimqa | LongBench/hotpotqa | LongBench/multifieldqa_en | LongBench/qasper | Average |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n| `ChunkKV`     | **28.71**     | 33.53     | 30.41     | 24.84     | 29.37     |\n| `KNorm`     | 26.78     | 31.44     | 29.27     | 21.89     | 27.34     |\n| `SnapKV`     | 28.45     | **34.56**     | **32.11**     | 25.38     | **30.12**     |\n| `StreamingLLM`     | 20.56     | 26.67     | 22.33     | 22.08     | 22.91     |\n| `CurDKV`     | 27.97     | 31.58     | 31.95     | **27.04**     | 29.63     |\n\nAs we see from the above table, `SnapKV` seems to perform the best on average for the **LongBench** tasks. However, `CurDKV` is not far, and consistently performs close to the best baseline.\n\n**Table 2:** **LongBench** evaluation results for the Qwen 14B model for a compression ratio of $90$%.\n\n| Method | LongBench/2wikimqa | LongBench/hotpotqa | LongBench/multifieldqa_en | LongBench/qasper | Average |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n| `ChunkKV`     | 12.39     | 21.09     | 17.03     | 9.25     | 14.94     |\n| `KNorm`      | 10.04     | 9.73     | 13.06     | 7.2     | 10.00     |\n| `SnapKV`       | 13.37     | 21.34     | 18.25     | 9.56     | 15.63     |\n| `StreamingLLM`     | 9.56     | 14.51     | 14.66     | 10.22     | 12.24     |\n| `CurDKV`     | **20.88**     | **21.54**     | **22.08**     | **16.23**     | **20.18**     |\n\nThe effect of `CurDKV` seems to be more prominent for the regime of high compression ratios. For instance, in the above table ($90$% compression), `CurDKV` significantly outperforms other baselines by a significant margin. Below, we see a more prominent effect for an even larger model (a 34B quantized model).\n\n**Table 3:** **LongBench** evaluation results for the Qwen 32B model for a compression ratio of $30$%.\n\n| Method | LongBench/2wikimqa | LongBench/hotpotqa | LongBench/multifieldqa_en | LongBench/qasper | Average |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n| `ChunkKV`     | 51.64     | 58.64     | 45.41     | 42.46     | 49.54     |\n| `KNorm`     | 40.14     | 46.94     | 43.39     | 37.73     | 42.05     |\n| `SnapKV`     | 53.85     | 57.52     | 48.56     | 41.86     | 50.45     |\n| `StreamingLLM`     | 34.1     | 47.15     | 32.61     | 43.11     | 39.24     |\n| `CurDKV`     | **58.07**     | **62.6**     | **50.42**     | **47.75**     | **54.71**     |\n\nHere again, we see that `CurDKV` consistently outperforms other baselines by a significant margin.\n\n**Table 4:** **LongBench** evaluation results for the Qwen 32B model for a compression ratio of $90$%.\n\n| Method | LongBench/2wikimqa | LongBench/hotpotqa | LongBench/multifieldqa_en | LongBench/qasper | Average |\n| -------- | -------- | -------- | -------- | -------- | -------- |\n| `ChunkKV`     | 24.24     | 37.85     | 24.31     | 13.62     | 25.00     |\n| `KNorm`     | 6.14     | 3.8     | 13.12     | 7.02     | 7.52     |\n| `SnapKV`     | 22.8     | 36.29     | 23.04     | 14.29     | 24.10     |\n| `StreamingLLM`     | 18.54     | 30.38     | 22.03     | 14.56     | 21.38     |\n| `CurDKV`     | **40.59**     | **43.51**     | **31.24**     | **16.1**     | **32.86**     |\n\nAnd yet again, we see that for the 32B model quantized model, the gains for `CurDKV` in a high compression ratio regime are even more prominent. This raises a further interesting question: *do value-guided KV caching methods perform better for quantized models?* We leave this question for a future work.\n\n**On the sensitivity of the safeguard parameter ($\\alpha$).** As requested by the reviewer, here we do experiments to run a sensitivity analysis on the safeguard parameter $\\alpha$. To recall, for KV compression methods that distribute the per-head budget adaptively (aka the `AdaKV` technique [1]), the parameter $\\alpha$ controls the proportion of KV pairs to preserve in each head (see line 28 of the `KVPress` implementation at https://github.com/NVIDIA/kvpress/blob/main/kvpress/presses/adakv_press.py). The default value of $\\alpha$ as proposed by the authors of `AdaKV` is $0.20$, which we also use in our implementation of `AdaCurDKV`. Tables 5 and 6 below contain the results of our ablations.\n\n\n**Table 5:** Sensitivity analysis of the safeguard parameter $\\alpha$ for **LongBench** tasks at a compression ratio of $30$%. We compare the two adaptive techniques: `AdaSnapKV` and `AdaCurDKV`. \n\n| $\\alpha$ | Method | LongBench/2wikimqa | LongBench/hotpotqa | LongBench/multifieldqa_en | LongBench/qasper | Average |\n| -------- | -------- | -------- |  -------- |  -------- |  -------- |  -------- |\n| 0.1     | `AdaCurDKV`     | 49.13     | 58.74     | 55.82     | 48.36     | 53.01     |\n| 0.1     | `AdaSnapKV`     | 50.58     | 59.93     | 55.28     | 45.74     | 52.88     |\n| 0.5     | `AdaCurDKV`     | 49.13     | 58.74     | 55.82     | 48.36     | 53.01     |\n| 0.5     | `AdaSnapKV`     | 51.34     | 59.55     | 55.61     | 45.84     | 53.08     |\n| 0.9     | `AdaCurDKV`     | 48.99     | 58.67     | 56.33     | 48.46     | 53.11     |\n| 0.9     | `AdaSnapKV`     | 51.06     | 57.9     | 54     | 45.72     | 52.17     |\n\n**Table 6:** Sensitivity analysis of the safeguard parameter $\\alpha$ for **LongBench** tasks at a compression ratio of $90$%. We compare the two adaptive techniques: `AdaSnapKV` and `AdaCurDKV`.\n\n| $\\alpha$ | Method | LongBench/2wikimqa | LongBench/hotpotqa | LongBench/multifieldqa_en | LongBench/qasper | Average |\n| -------- | -------- | -------- |  -------- |  -------- |  -------- |  -------- |\n| 0.1     | `AdaCurDKV`     | 34.5     | 45.91     | 30.8     | 28.75     | 34.99     |\n| 0.1     | `AdaSnapKV`     | 26.7     | 46.18     | 26.86     | 21.53     | 30.32    |\n| 0.5     | `AdaCurDKV`     | 32.73     | 46.61     | 30.15     | 29.83     | 34.83     |\n| 0.5     | `AdaSnapKV`     | 27.15     | 46.07     | 26.18     | 20.39     | 29.95     |\n| 0.9     | `AdaCurDKV`     | 32.27     | 46.44     | 31.37     | 28.99     | 34.77     |\n| 0.9     | `AdaSnapKV`     | 23.48     | 45.27     | 26.46     | 21.65     | 29.21     |\n\nFor both 30% and 90% compression ratios, we observe only a marginal performance difference for different values of $\\alpha$, indicating that the adaptive KV compression methods (including `AdaCurDKV`) are very robust to the selection of this parameter. \n\n**References**\n\n[1] Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference, Feng et al., 2025."}, {"review_id": "1fFZfhkXCX", "reviewer": "Reviewer_NWDK", "summary": "The authors propose CurDKV, a value-centric KV compression method that selects keys and values based on leverage scores computed from CUR matrix decomposition. CurDKV achieves up to 9.6% higher accuracy than state-of-the-art methods like SnapKV and ChunkKV under aggressive compression budgets on LLaMA and Mistral, while maintaining compatibility with FlashAttention and Grouped Query Attention. In addition to improved accuracy, CurDKV reduces generation latency by up to 40% at high compression, offering a practical speed-accuracy tradeoff.", "questions": "How does the method process the KV of generated tokens? \nDoes the method support vllm?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The experiments are solid. The proposed method is better than strong baselines.\n2. CurDKV is simple and effective without further training.\n3. The paper is well-written and easy to read.\n\nWeaknesses:\n1. The proposed method is only tested on the tasks with longer context. It is not clear how the method works on the tasks that need to generate longer sequence, such as story writing, code generation etc. How does the method process the KV of generated tokens?\n2. The paper claims the value guided method, while both key and value are used to compute scores in Algorithm 1. Need clarification on it. And it would be better to have an ablation study on which is important to guide topk indices.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for raising important questions about our claim to develop \"value-guided\" KV compression methods. We are glad to know that the reviewer found our work strong, easy to understand, and practically effective. We address the concerns raised by the reviewer below and hope that this discussion resolves their queries.\n\n**How does CurDKV work in inference-time for longer generation tasks** \n\nBelow, we detail how our method processes the Key-Value (KV) cache for tokens generated during inference -\n\n1. At each generation step, the model generates a token and computes the corresponding query, key, and value vectors.\n2. The newly generated token's key and value vectors are appended to the KV cache of the prefilled (context) tokens.\n3. After appending the new token, the method periodically recomputes leverage scores for the cached keys and values (optionally, using the approximation via random Gaussian projections to ensure computational efficiency)\n4. Tokens with the lowest combined leverage scores are evicted from the cache to maintain the predefined memory budget. The most informative tokens, identified via high leverage scores, are retained.\n5. CurDKV explicitly preserves the first few tokens (attention sinks) as they typically carry critical global context relevant for subsequent generation steps.\n\nWe further compare four KV compression methods: exact `CurDKV` (where exact leverage scores are computed), random `CurDKV` (where the random Gaussian projection is used to project the key-value vectors, `SnapKV` and `ChunkKV`. We do comparisons on a context length of 128K tokens and generated length of 100 tokens, and we use compression ratios of $30$% and $90$%. We report the total runtime, prefill time and the generation time for all methods. Here are the detailed results:\n\n**Table 1:** Runtime results for a context length of 128K tokens and a compression ratio of $30$%.\n\n| Method          | Length | Total time  (in sec)     | Prefill time  (in sec)   | Generation time (in sec) |\n| --------------- | ------ | ---------------- | ---------------- | ---------------- |\n| Exact `CurDKV`  | 128K | 55.39 | 46.51 | 8.88 |\n| Random `CurDKV` | 128K | **44.76**  | 35.17 | 9.59 |\n| `SnapKV`        | 128K | 44.97  | **33.37** | 11.59 |\n| `ChunkKV`       | 128K | 54.36 | 50.53 | **3.83** |\n\nAs we see from the above table, `CurDKV` with a random Gaussian projection outperforms the baselines for a compression ratio of $30$%. This is also intuitive, since the Gaussian projection reduces the effective dimension of the key-value pairs. This, combined with the computationally efficient leverage scores, leads to the overall smallest total time (prefill + generation) for random `CurDKV`.\n\n**Table 2:** Runtime results for a context length of 128K tokens and a compression ratio of $90$%.\n| Method          | Length | Total time (in sec)      | Prefill time (in sec)    | Generation time (in sec) |\n| --------------- | ------ | ---------------- | ---------------- | ---------------- |\n| Exact `CurDKV`  | 128K | 51.36 | 46.90 | **4.46** |\n| Random `CurDKV` | 128K | 42.82 | 35.41 | 7.41 |\n| `SnapKV`        | 128K | 44.50 | **33.59** | 10.91 |\n| `ChunkKV`       | 128K | **42.74** | 35.83 | 6.91 |\n\nFor a high compression ratio of $90$%, random `CurDKV` still performs approximately the same as the best baseline (`ChunkKV` for this case). This, combined with the performance gains of `CurDKV`, makes it a competitive choice.\n\nWhen we increase the generation length to 4K, the generation time increases for all compression methods due to periodic eviction/retention of key-value pairs. \n\n**Table 3:** Generation time results for a context length of 128K and generated 4K tokens.\n| Method          |  Compression ratio  | Generation time (in sec)  |\n| --------------- | ------ | ---------------- | \n| Exact `CurDKV`  | 30% | **464.9** |\n| Random `CurDKV` | 30% | 516.9 |\n| `SnapKV`        | 30% | 553.2 |\n| `ChunkKV`       | 30% | 539.3 |\n| Exact `CurDKV`  | 90% | 656.8 |\n| Random `CurDKV` | 90% | **536.9** |\n| `SnapKV`        | 90% | 553.4 |\n| `ChunkKV`       | 90% | 545.6 |\n\nAt a 30\\% compression ratio, exact CurDKV exhibits the fastest generation, whereas, for a 90\\% compression ratio, random CurDKV achieves the fastest generation. At moderate compression ratios, the Exact CurDKV precisely identifies and retains highly informative keys and values through exact CUR decomposition; therefore, reducing the need for repeatedly decomposing the key-value pairs on generated tokens. At a higher compression rate, more repeated computation of key-value eviction is needed; therefore, under those circumstances, the effectiveness of low-rank projections increases, leading to lower generation time. \n\n**On the usage of both the key and value matrices, and the claim of the paper.**\nWe appreciate the reviewer for seeking clarification for the main claim of the paper, as we believe it is the main contribution of this work. Most of the existing SOTA literature on KV cache compression methods (`SnapKV`, `H20`, `AdaKV`, to name a few) focus on estimating the importance of a cache entry by using _attention weights_, which rely only on the corresponding query and key vectors. However, as we show in Lemma 3.1 of our paper, the true Frobenius distance between the exact attention output and the compressed attention output also depends on how well the compressed value matrix approximates the original value matrix; intuitively, this happens because the query and key vectors are absorbed by the softmax operator. This motivates the design of eviction policies which also incorporate value vectors. At the same time, we point out that this analysis _does not undermine_ the importance of the corresponding key vectors. Infact, we provide ablation studies to empirically show that the combined scores of both the key and value produces the best results (Figure 3 and Table 17 of our paper). As our ablations also show, the next best choice is to use the leverage scores of only the value vectors. Overall, through our paper, we aim to establish the importance of incorporating information gained from value vectors in developing key-value eviction policies.\n\n**`vllm` support** \n\nWe thank the reviewer for raising this important point regarding practical compatibility and integration. CurDKV is fully supported by NVIDIA’s KVPress library, which is specifically designed to implement and benchmark KV cache pruning strategies within Hugging Face Transformers-based pipelines. However, at present, KVPress (and thus CurDKV) does not natively support vLLM. The vLLM inference backend manages KV caches differently and currently has its own separate mechanisms (e.g., KV-Compress, LeanKV) for cache compression. While direct compatibility between CurDKV and vLLM would require additional engineering effort, methodologically, CurDKV can be adapted to work with vLLM by modifying vLLM’s cache eviction and retention mechanisms accordingly."}, {"review_id": "1YP1QL1SOr", "reviewer": "Reviewer_hDao", "summary": "This paper introduces CurDKV, a method which compresses KV-cache to reduce the inference-time memory comsumption for transformer-based models. CurDKV takes into account the importance of both K and V simultaneously and introduces a novel KV compression method based on CUR decomposition, in order to retain the most important KV-cache. In addition, CurDKV introduces Gaussian projections to compute approximate scores, further improving the efficiency of the KV-cache compresses process.", "questions": "* In Algorithm 1, CurDKV uses the product of K and V leverage scores, which assigns equal importance to K and V, which seems inconsistent with the earlier motivation that V is more significant. Can authors explain more about this point?\n* In Figure 4(b) and fig 5(b), where is the data line for 128K tokens?", "limitations": "My questions are listed above and there is no other major limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* This paper leverages CUR decomposition to approximate the principal subspace of the attention output, which theoretically better preserves the LLM capacity. It is a good idea to use CUR decomposition to select KV cache based on the leverage score, considering both key and value.\n* CurDKV is compatible with existing techniques such as FlashAttention and GQA.\n\n**Weaknesses**\n\n* In Section 3.1, the motivation for value-guided compression is not clear enough, as key can be equally important to the attention output. Moreover, the sentence \"for higher compression levels, the second term becomes negligible\" is a bit hard to follow.\n* In Section 4.4, the paper analyze computational efficiency of CurDKV and lacks a comprehensive comparison with baseline methods. Since CurDKV introduces extra computation such as Gaussian projection and score combination, it is better to compare the end-to-end efficiency with other KV cache compression baselines.\n* This paper uses Gaussian projections with $r = 20$, but lacks more analysis around Gaussian projections like how different r can impact the accuracy-efficiency trade-off. This ablation study will be very useful and this can be a minor weakness.", "was_revised": true, "final_justification": "The authors solved most of my concerns in detail during the rebuttal period. Hence, I will keep my positive score as \"borderline accept\".", "rebuttal": "We thank the reviewer for giving us a comprehensive analysis of our work and highlighting key weaknesses of our manuscript. We address some of the concerns below and hope that the discussion resolves all of them.\n\n**Clarification of the claims in Section 3.1 and a motivation for value-guided compression.** \n\nHere, we'll try to give a simpler motivation behind our approach of \"value-guided compression\" of KV caches. Recall that the standard attention output computes $\\text{softmax}(QK^T/C)V$, where $C$ is a suitable constant involving the dimension of the vector spaces in question. Most of the current SOTA KV-cache compression methods (`SnapKV`, `H20`, `AdaKV`, to name a few) focus on estimating the importance of a cache entry by only using *attention weights*, which are computed using corresponding query and key vectors. However, as the attention equation suggests, value vectors may also contribute significantly to the final attention output. Lemma 3.1 of the paper bounds the true Frobenius distance between the attention outputs of the compressed and uncompressed KV caches; as we see, the upper bound depends on how well the value matrix is approximated. This intuition suggests that a good KV cache eviction policy should also incorporate the use of value vectors (and not just the attention weights). Having said this, we want to emphasize on the fact that this _does not_ undermine the importance of key vectors. Infact, we provide ablations to show that a combination of leverage scores of both the key and value vectors provides the best generation results (Figure 3 and Table 17 of our paper). Our ablation results also show that the next best choice is to only use the leverage scores of value vectors, which further highlight the importance of value vectors in designing key-value pair eviction policies.\n\nRegarding the sentence, \"*for higher compression levels, the second term becomes negligible*\": note that the matrix $V'$ is the compressed version of $V$, where dropped value vectors are simply zeroed out. At higher compression levels, say above 70%, most of the rows of $V'$ will just be zeros. Inevitably, the Frobenius norm of the resultant matrix will decrease, and will slowly become insignificant compared to the first term of the upper bound of Lemma 3.1.\n\n**End-to-end computational efficiency comparison with other baselines.** As requested by the reviewer, we run experiments to compare the end-to-end runtimes of `CurDKV` against other baselines. \n\nWe compare four methods: exact `CurDKV` (where exact leverage scores are computed), random `CurDKV` (where the random Gaussian projection is used to project the key-value vectors, `SnapKV` and `ChunkKV`. We do comparisons on a context length of 128K tokens and generated length of 100 tokens, and we use compression ratios of $30$% and $90$%. We report the total runtime, prefill time and the generation time for all methods. Here are the detailed results:\n\n**Table 1:** Runtime results for a context length of 128K tokens and a compression ratio of $30$%.\n\n| Method          | Length | Total time  (in sec)     | Prefill time  (in sec)   | Generation time (in sec) |\n| --------------- | ------ | ---------------- | ---------------- | ---------------- |\n| Exact `CurDKV`  | 128K | 55.39 | 46.51 | 8.88 |\n| Random `CurDKV` | 128K | **44.76**  | 35.17 | 9.59 |\n| `SnapKV`        | 128K | 44.97  | **33.37** | 11.59 |\n| `ChunkKV`       | 128K | 54.36 | 50.53 | **3.83** |\n\nAs we see from the above table, `CurDKV` with a random Gaussian projection outperforms the baselines for a compression ratio of $30$%. This is also intuitive, since the Gaussian projection reduces the effective dimension of the key-value pairs. This, combined with the computationally efficient leverage scores, leads to the overall smallest total time (prefill + generation) for random `CurDKV`.\n\n**Table 2:** Runtime results for a context length of 128K tokens and a compression ratio of $90$%.\n| Method          | Length | Total time       | Prefill time     | Generation time  |\n| --------------- | ------ | ---------------- | ---------------- | ---------------- |\n| Exact `CurDKV`  | 128K | 51.36 | 46.90 | **4.46** |\n| Random `CurDKV` | 128K | 42.82 | 35.41 | 7.41 |\n| `SnapKV`        | 128K | 44.50 | **33.59** | 10.91 |\n| `ChunkKV`       | 128K | **42.74** | 35.83 | 6.91 |\n\nFor a high compression ratio of $90$%, random `CurDKV` still performs approximately the same as the best baseline (`ChunkKV` for this case). This, combined with the performance gains of `CurDKV`, makes it a competitive choice.\n\n**An ablation study on the dimension of the Gaussian projection.** As rightfully requested by the reviewer, we run an ablation study to determine the effect of the output dimension $r$ of the Gaussian projection matrix. Intuitively, we expect an overall increase in the generation quality as the dimension $r$ increases; this is due to the simple fact that as the dimension increases, the leverage scores of the projected vectors will better approximate the leverage scores of the original vectors. However, we still need to empirically verify this claim, and we do so below. We carry out these ablation studies for compression ratios of $30$% and $90$%. We run evaluations on the tasks in the **LongBench** benchmarks.\n\n**Table 3:** Ablation study on the output dimension $r$ of the Gaussian projection matrix for a compression ratio of $30\\%$.\n\n| $r$ | LongBench/2wikimqa | LongBench/hotpotqa | LongBench/multifieldqa_en | LongBench/qasper | Average |\n| -------- | -------- |  -------- |  -------- |  -------- |  -------- |\n| 5     | 50.99     | 58.77     | 55.48     | 47.97     | 53.30     |\n| 10     | 50.71     | 59.21     | 55.71     | 47.33     | 53.24     |\n| 50     | 49.31     | 57.85     | 54.78     | 47.83     | 52.44    |\n| 100     | 52.07     | 58.23     | 56.41     | 49.52     | 54.06     |\n\n**Table 4:** Ablation study on the output dimension $r$ of the Gaussian projection matrix for a compression ratio of $90$%.\n\n| $r$ | LongBench/2wikimqa | LongBench/hotpotqa | LongBench/multifieldqa_en | LongBench/qasper | Average |\n| -------- | -------- |  -------- |  -------- |  -------- |  -------- |\n| 5     | 28.45     | 41.8     | 29.29     | 25.5     | 31.26   |\n| 10     | 35.04     | 41.98     | 29.94     | 26.02     | 33.24     |\n| 50     | 34.25     | 45.48     | 32.3     | 28.64     | 35.17     |\n| 100     | 35.3     | 46.52     | 33.15     | 28.13     | 35.77     |\n\nFor both compression ratios, we observe that increasing $r$ improves the performance.  In Figure 3 in Section 4.2, we have discussed that CurDKV without random projection (Exact CUR) usually performs better than CurDKV with random projection. Having a large $r$ in the approximated leverage score computation approximates the exact leverage score computation. Therefore, the performance increases to that of the exact CUR. Moreover, this behavior is more prominent for larger compression ratios.\n\n**128K data line for Figures 4(b) and 5(b).** \n\nWe thank the reviewer for pointing this out. Due to a limit in the y-axis (the maximum y value for prefill time is taken as 30 sec in the plots, but for 128K, it exceeds 40 sec), the plots for 128K were omitted from both plots. We will correct this in the revised version of the paper."}]

Author rebuttal:

Meta-review: The paper proposes CurDKV, a value-guided KV-cache compression method for LLMs that uses approximated CUR decomposition. Major concerns such as evaluation on longer sequences and latencies are successfully addressed by the authors. All reviewers are positive toward accepting this paper. Hence I vote for acceptance for this paper.
Decision: Accept (poster)