Title: Value-Guided KV Compression for LLMs via Approximated CUR Decomposition
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

KV Compression

Attention Compression

Needle-in-a-haystack
Abstract: Key-value (KV) cache compression has emerged as a critical technique for reducing the memory and latency overhead of autoregressive language models during inference. Prior approaches predominantly rely on query-key attention scores to rank and evict cached tokens, assuming that attention intensity correlates with semantic importance. However, this heuristic overlooks the contribution of value vectors, which directly influence the attention output. In this paper, we propose CurDKV, a novel, value-centric KV compression method that selects keys and values based on leverage scores computed from CUR matrix decomposition. Our approach approximates the dominant subspace of the attention output $\mathrm{softmax}(QK^\top)V$, ensuring that the retained tokens best preserve the model’s predictive behavior. Theoretically, we show that attention score approximation does not guarantee output preservation, and demonstrate that CUR-based selection minimizes end-to-end attention reconstruction loss. Empirically, CurDKV achieves up to $9.6$\% higher accuracy than state-of-the-art methods like SnapKV and ChunkKV under aggressive compression budgets on LLaMA and Mistral, while maintaining compatibility with FlashAttention and Grouped Query Attention. In addition to improved accuracy, CurDKV reduces generation latency by up to 40\% at high compression, offering a practical speed-accuracy tradeoff.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CurDKV, a method which compresses KV-cache to reduce the inference-time memory comsumption for transformer-based models. CurDKV takes into account the importance of both K and V simultaneously and introduces a novel KV compression method based on CUR decomposition, in order to retain the most important KV-cache. In addition, CurDKV introduces Gaussian projections to compute approximate scores, further improving the efficiency of the KV-cache compresses process.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* This paper leverages CUR decomposition to approximate the principal subspace of the attention output, which theoretically better preserves the LLM capacity. It is a good idea to use CUR decomposition to select KV cache based on the leverage score, considering both key and value.\n* CurDKV is compatible with existing techniques such as FlashAttention and GQA.\n\n**Weaknesses**\n\n* In Section 3.1, the motivation for value-guided compression is not clear enough, as key can be equally important to the attention output. Moreover, the sentence \"for higher compression levels, the second term becomes negligible\" is a bit hard to follow.\n* In Section 4.4, the paper analyze computational efficiency of CurDKV and lacks a comprehensive comparison with baseline methods. Since CurDKV introduces extra computation such as Gaussian projection and score combination, it is better to compare the end-to-end efficiency with other KV cache compression baselines.\n* This paper uses Gaussian projections with $r = 20$, but lacks more analysis around Gaussian projections like how different r can impact the accuracy-efficiency trade-off. This ablation study will be very useful and this can be a minor weakness.\n\nQUESTIONS:\n* In Algorithm 1, CurDKV uses the product of K and V leverage scores, which assigns equal importance to K and V, which seems inconsistent with the earlier motivation that V is more significant. Can authors explain more about this point?\n* In Figure 4(b) and fig 5(b), where is the data line for 128K tokens?\n\nLIMITATIONS:\nMy questions are listed above and there is no other major limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
The authors propose CurDKV, a value-centric KV compression method that selects keys and values based on leverage scores computed from CUR matrix decomposition. CurDKV achieves up to 9.6% higher accuracy than state-of-the-art methods like SnapKV and ChunkKV under aggressive compression budgets on LLaMA and Mistral, while maintaining compatibility with FlashAttention and Grouped Query Attention. In addition to improved accuracy, CurDKV reduces generation latency by up to 40% at high compression, offering a practical speed-accuracy tradeoff.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The experiments are solid. The proposed method is better than strong baselines.
2. CurDKV is simple and effective without further training.
3. The paper is well-written and easy to read.

Weaknesses:
1. The proposed method is only tested on the tasks with longer context. It is not clear how the method works on the tasks that need to generate longer sequence, such as story writing, code generation etc. How does the method process the KV of generated tokens?
2. The paper claims the value guided method, while both key and value are used to compute scores in Algorithm 1. Need clarification on it. And it would be better to have an ablation study on which is important to guide topk indices.

QUESTIONS:
How does the method process the KV of generated tokens? 
Does the method support vllm?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04

comments: SUMMARY:
The paper propose CurDKV for compressing the key-value (KV) cache, a value-centric strategy. They estimate each token’s leverage score via a (fast, projected) CUR matrix decomposition of the attention output and keep only the most influential keys and values.

STRENGTHS AND WEAKNESSES:
Strengths
1. CUR-based leverage scores + Gaussian projection yield an O(n d r) lightweight scoring pass; algorithm integrates with FlashAttention & Grouped-Query Attention without kernel changes.

2. Memory scales predictably; although pre-fill cost rises by ~3–5 s for 128 K contexts, generation is faster for long sequences


Weaknesses

1. Static Compression Only. CurDKV freezes the cache after prefill; it cannot react if later tokens shift relevance, and it adds noticeable prefill overhead for very long inputs

2. Limited Model Coverage. It is better to provide more results on bigger models. The results use 7 B / 8 B LLaMA/Mistral; scalability to 34 B–70 B or MoE models is untested.

3. Hyper-Parameter Sensitivity. Adaptive safeguard α (0.20) is fixed; sensitivity analysis would help.

QUESTIONS:
see weaknesses

LIMITATIONS:
Yes, the paper discusses the pertinent limitations and outlines directions for future work.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-05

comments: SUMMARY:
This paper introduce CurdKV, that rethinks kv cache compression by consider not only keys but also values via a leverage score metric. Specifically, Curdkv uses cur decomposition to identify more important individual keys and values via leverage score, where a importance is measures for rows and cols of the key and value matrices. In practice, this process is done on a group level to compatible with GQA. And to avoid svd it use a random project for the decomposition to reduce overhead. At last, an adaptive budget version of curdkv is proposed.

STRENGTHS AND WEAKNESSES:
Strengths

1. The proposed kv cache include values in the kv cache compression process, that were overlooked previously. And this is well motivated by fig 1 and introduction.

2. The proposed method to use cur decomposition is solid when identifying the important rows and cols.

3. To use a random projection for leverage score computing seems a good trick to avoid svd. And the extra cost involved is acceptable. Plus this importance evaluation process can be fit inside flash-attn, which is important in real applications.

4. Solid overall performance on longbench and the NIAH test.

Weakness

1. Algo 1 is not very clear to me. How's the leverage score is used for the decomposition?

2. The max length in the experiments is rather limited. I would expect to test upto 128k on RULER.

3. Why not include a latency comparision with and without curd kv? to show the exact overhead used in computing leverage score and the ranking process.

QUESTIONS:
No

LIMITATIONS:
No
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-07