Title: Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language models

long context
Abstract: Recent advances in Large Language Models (LLMs) have highlighted the critical importance of extending context length, yet the quadratic complexity of attention mechanisms poses significant challenges for efficient long-context modeling. KV cache compression has emerged as a key approach to address this challenge. Through extensive empirical analysis, we reveal a fundamental yet previously overlooked asymmetry in KV caches: while adjacent keys receive similar attention weights ({\it local homogeneity}), adjacent values demonstrate distinct {\it heterogeneous} distributions. This key-value asymmetry reveals a critical limitation in existing compression methods that treat keys and values uniformly. To address the limitation, we propose a training-free compression framework (AsymKV) that combines homogeneity-based key merging with a mathematically proven lossless value compression. Extensive experiments demonstrate that AsymKV consistently outperforms existing long-context methods across various tasks and base models. For example, on LLaMA3.1-8B, AsymKV achieves an average score of 43.95 on LongBench, surpassing SOTA methods like H$_2$O (38.89) by a large margin.Our code can be found in this \href{https://github.com/the-scale-lab/Asymkv}{link}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AsymKV, a training-free KV cache compression method for large language models (LLMs) that enables efficient long-context processing by leveraging a previously overlooked asymmetry in attention: keys are locally homogeneous (similar across adjacent positions), while values are heterogeneous. Unlike prior methods that treat keys and values uniformly, AsymKV merges adjacent keys using a second-order optimization technique and preserves values through a mathematically lossless, cardinality-normalized representation. This design minimizes information loss and maintains attention output equivalence. Experiments across various models (e.g., LLaMA3, Mistral) and benchmarks demonstrate that AsymKV provides better accuracy than existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n1. The paper empirically validates a key-value asymmetry in attention — keys are homogeneous, values are heterogeneous, and correspondingly proposes a merging method to compress the KV cache.\n2. The accuracy of the proposed method is validated across different models and benchmarks, surpassing the existing KV cache compression techniques such as H2O and CaM.\n\n\n[Weaknesses]\n1. Citations and comparisons to some important related works are missing, such as TOVA[1], DuoAttention[2], FastGen[3], and etc.\n2. The proposed method, although preserves accuracy better than existing methods such as CaM, does not provide better inference speed than existing solutions.\n\n[1] Matanel Oren, et al. Transformers are multi-state rnns, 2024.\n\n[2] Guangxuan Xiao, et al. Duoattention: Efficient long-context llm inference with retrieval and streaming heads, 2024.\n\n[3] Suyu Ge, et al. Model tells you what to discard: Adaptive KV cache compression for LLMs, 2024.\n\nQUESTIONS:\n1. How is the proposed method compared with some other KV compression techniques such as TOVA, DuoAttention, and FastGen?\n2. What is the compression ratio in Figure 3?\n3. Why is the x-axis in Figure 4 denoted as compression ratio? What does it mean when the compression ratio is 100%? Is this a typo?\n4. For Table 2, could you explain why almost all KV compression methods seem to have a very good accuracy, compared to the full context baseline, especially on long sequences?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-10"}

comments: SUMMARY:
This paper addresses the phenomenon of asymmetry in KV caches –– adjacent values have distinct heterogeneous distributions, even though adjacent keys are similar in attention weights. Existing methods overlook this asymmetry by merging keys and values identically, while AsymKV instead merges only homogeneous keys using a mathematically derived method that minimizes information loss, and combines the corresponding values via a lossless cardinality-aware normalization to preserve attention outputs exactly. Experiments on LongBench and LongBenchV2 across multiple models (e.g., LLaMA3.1-8B, Mistral-7B) demonstrate that AsymKV outperforms prior state-of-the-art methods like H2O and CaM in both performance and compression efficiency, achieving robust gains even under aggressive compression without retraining.

STRENGTHS AND WEAKNESSES:
## Strengths

1.	The discovery of the heterogeneous distributions of key values is innovative. 

2.	The derivation of the best key merging strategy is mathematically rigorous, and ablation study shows its effectiveness.

3.	AsymKV does not require model retraining or fine-tuning, making it practical and easy to adopt across existing LLMs.

4.	AsymKV excels in tasks requiring retention of early context (e.g., Topic Retrieval), where many other methods fail, highlighting its advantage in maintaining semantic continuity.

## Weaknesses

1.	The experiments shows marginal superiority to other compression methods in some tasks, for example “Sum” and “Few-shot”, in table 1. And in some tasks this method is beaten by other baselines.

QUESTIONS:
1.	In Figure 3, why the result of StreamingLLM is not showing?

2.	According to table 2 and figure 4, why after compression, the results may be better?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AsymKV, a training-free KV cache compression method that uses some qualitative insights from the authors on the distribution of keys and values in attention mechanisms. The authors observe that the keys of nearby tokens exhibit local homogeneity (i.e., similar attention patterns), while values of nearby tokens can have very different attention distributions. Based on this, they propose two separate KV cache compression strategies: a homogeneity-based merging approach for keys, and a cardinality-aware normalisation for values that preserves attention distributions. AsymKV shows significant improvements compared to baselines like H2O on long-context benchmarks like LongBench.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors did a great job at identifying and characterising the key-value asymmetry in attention distributions; they show on several datasets that attention distributions of the keys of nearby tokens correlate strongly, while this is often not true for values. The framework for deriving the key merging strategy is sound and well-motivated, and LMA also makes sense. The evaluation on several model families (LLaMA, Mistral, Qwen) and benchmarks (LongBenchV2) is comprehensive (although it would be nice to include some straightforward KV cache compression approach like L2 from https://arxiv.org/abs/2406.11430 as a baseline).\n\nI have some concerns about the use of backward passes for gradient computation -- how significant is the overhead it introduces? Does it limit the applicability of the method in memory-constrained settings (which is where KV cache compression makes the most sense)? Can you please expand on this?\n\nSince the whole paper is based on the observation from the authors of key-value asymmetries, it would be nice to have a bit of analysis on those (when/how do they emerge? How do they look like?) -- at the moment this is largely lacking.\n\nQUESTIONS:\nHow/when do the key-value asymmetries emerge during inference?\nHow big is the overhead introduced by the backward passes for gradient computations, in terms of time and memory?\nA big issue with many KV cache compression methods is the integration with FlashAttention, since it makes materialising attention scores very computationally expensive. Is this a limitation for AsymKV as well?\n\nLIMITATIONS:\nPlease see my previous comments", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AsymKV, a novel training-free KV cache compression method tailored for long-context LLM inference. The key insight lies in the empirical discovery of a local asymmetry in attention KV caches: adjacent keys tend to be similar (homogeneous), while adjacent values are highly diverse (heterogeneous). Based on this observation, the authors propose a compression technique that merges adjacent homogeneous keys while preserving the values using cardinality-aware normalization. AsymKV demonstrates consistent improvements over prior methods such as H2O and StreamingLLM across a range of tasks and base models, especially on long-context understanding and early information retrieval.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1) The discovery of the homogeneous key vs. heterogeneous value asymmetry in KV caches is both original and significant. It challenges prior uniform treatment of keys and values and lays a foundation for more nuanced cache compression strategies in LLMs. It might leads to more exploration of KV Cache compression.\n2) The adjacent key merging combined with value addition is a simple yet elegant solution that leverages attention linearity. \n3) The authors conduct extensive experiments across multiple base models (LLaMA3.1, Mistral, Qwen2, LLaMA2) and tasks, demonstrating consistent improvements\n\nWeakness:\n1) The comparison lacks several state-of-the-art methods such as SnapKV, PyramidKV, and Ada-KV, which have shown strong performance in recent studies. Moreover, in the cache merging category, only CaM is used as a baseline, while other recent methods (e.g., D2O, KVMerger) are omitted, weakening the empirical claims.\n2) Some key evaluations, notably compression rate analysis (§4.4) and merge strategy ablations (§4.6), are conducted only on a single dataset or task, which may not be fully representative. This reduces the generalizability of the ablation findings.\n\n[1] https://arxiv.org/abs/2404.14469\n[2] https://arxiv.org/abs/2406.02069 \n[3] https://arxiv.org/abs/2407.11550\n\nQUESTIONS:\n1. The paper claims that the method is efficient due to selective gradient computation. However, it would be helpful to include a detailed overhead analysis\n2. Can the method be extended to the decoding stage? Given that the outputs of current reasoning models are becoming increasingly long, applying the method only during the prefill stage may be insufficient. While I understand this is beyond the current scope, it would be valuable to hear the authors’ perspective or insights on potential extensions in this direction.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}