Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs
Authors: Wanyun Cui
Mingwei Xu
Venue: NeurIPS 2025 poster ()
Keywords: large language models
long context
Primary area: deep_learning
TL;DR: We propose a novel KV cache compression method that exploits the asymmetry between locally homogeneous keys and heterogeneous values in attention mechanisms, enabling more efficient long-context processing.

Abstract: Recent advances in Large Language Models (LLMs) have highlighted the critical importance of extending context length, yet the quadratic complexity of attention mechanisms poses significant challenges for efficient long-context modeling. KV cache compression has emerged as a key approach to address this challenge. Through extensive empirical analysis, we reveal a fundamental yet previously overlooked asymmetry in KV caches: while adjacent keys receive similar attention weights ({\it local homogeneity}), adjacent values demonstrate distinct {\it heterogeneous} distributions. This key-value asymmetry reveals a critical limitation in existing compression methods that treat keys and values uniformly. To address the limitation, we propose a training-free compression framework (AsymKV) that combines homogeneity-based key merging with a mathematically proven lossless value compression. Extensive experiments demonstrate that AsymKV consistently outperforms existing long-context methods across various tasks and base models. For example, on LLaMA3.1-8B, AsymKV achieves an average score of 43.95 on LongBench, surpassing SOTA methods like H$_2$O (38.89) by a large margin.Our code can be found in this \href{https://github.com/the-scale-lab/Asymkv}{link}.

Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs
Wanyun Cui and Mingwei Xu'
Shanghai University of Finance and Economics
*MoE Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics
cui wanyun suf..edu..ee........ mingweixu@stuu.see sww..uue.eduucc
Abstract
Recent advances in Large Language Models (LLMs) have highlighted the critical importance of extending context length, yet the quadratic complexity of atten- tion mechanisms poses significant challenges for efficient long-context modeling. KV cache compression has emerged as a key approach to address this challenge. Through extensive empirical analysis, we reveal fundamental yet previously overlooked asymmetry in KV caches: while adjacent keys receive similar attention weights (local homogeneity), adjacent values demonstrate distinct heterogeneous distributions. This key-value asymmetry reveals critical limitation in existing compression methods that treat keys and values uniformly. To address the limita- tion, we propose training-free compression framework (AsymKV) that combines homogeneity- key merging with a mathematically proven lossless value compression. Extensive experiments demonstrate that AsymKV consistently out- performs existing long-context methods across various tasks and base models. For example, on LLaMA3.1 1-8B, AsymKV achieves an average score of 43.95 on LongBench, surpassing SOTA methods like H₂O (38.89) by large margin. code can be found in this link
Introduction
The ability to process long contexts is crucial for Large Language Models (LLMs) [3]23] However, processing such long contexts poses significant challenges: pre-trained LLMs face both architectural and computational constraints in handling extended contexts. In particular, as the context length increases, the complexity of attention mechanisms increases quadratically (O(n)), while storage overhead increases linearly (O(n)) 77
Various approaches have been proposed address this challenge, with KV cache compression emerging as a promising direction [15). These methods aim to compress the KV cache while preserving essential information for maintaining model performance. A straightforward strategy
to keep tokens with high historical importance (e.g., attention scores [16] 19 20 33).. This
approach leverages the observation that attention weights exhibit significant variation across different tokens. Another line of work attempts to identify more general token importance rather than the history information [28 27 5 However, these approaches share a fundamental limitation: they fail to capture certain tokens that are less important in the histroy but suddenly become critical for subsequent predictions.
To address the information loss caused by directly discarding tokens, cache merging methods have been proposed to merge multiple tokens into fewer representations rather than hard pruning, thereby preserving more information 32 26 27 These merging approaches implicitly assume that certain
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Layer Head (Spearman
Average Spearman cor::
OSpearman
Layer 5 Head
Average Spearman con:: 0.392
1.0
0.0
0.00
8.5
9.4
0.2
0.8:
Atmili
Head
Head
(a) Scatter plot of key sim-(h) Spearman correlation Scatter plot of value (d) Spearman correlation ilarity (attention) weight map of adjacent attention similarity percentile ranks map of adjacent value percentile ranks between weights adjacent positions
between adjacent posi-similarity tions
Contrasting distributions local homogeneity attentions (keys) versus local hetero- geneity in values. Statistics from Llama-2-7b-clee
Figure I:
the ShareGPT dataset. (a-b) demonstrate strong positive correlations between adjacent attention percentile ranks (normalized to [0,1], where indicates highest attention) across all layers and heads, supporting the local homogeneity hypothesis for keys. (c-d) reveal weak or negative correlations between adjacent value similarity percentile ranks, computed from sim (val, valy), indicating distinct heterogeneity in values. The similarity is measured by cosine. This fundamental difference between keys and values suggests the need for separate compression strategies.
redundancies or patterns exist in the KV cache. This raises new fundamental question: what specific characteristics of L.MMs lead to these redundancies and make cache merging feasible? We answer this question by identifying the key-value asymmetry in LLM attention mechanisms.
Local Key- Key-Value Asymmetry Through extensive empirical analysis, we reveal a fundamental pattern in attention distributions: the homogeneity of local keys. Specifically, observe that adjacent tokens consistently receive similar attention weights when a query assigns high attention to position j. the neighboring position (j 1) typically receives comparable attention weight (Fig. [ap. This pattern shows remarkable consistency across all layers and attention heads, with an average Spearman correlation coefficient of 0.727 (Fig. lb. This consistent local attention pattern, arising from query-key interactions, suggests an underlying homogeneity of local keys adjacent keys must share certain structural properties to produce such stable attention patterns. Such key homogeneity naturally emerges from language structure, where adjacent words form coherent semantic units and contribute collectively to meaning representation.
The observed homogeneity of adjacent keys provides evidence for merging neighboring tokens, offer- ing principled explanation recent token merging approaches. Specifically, when multiple adjacent keys exhibit high homogeneity, computing and storing only one key for the merged representation effectively approximate the original attention output, leading to both computational and memory efficiency.
However, our analysis reveals a striking asymmetry: while keys exhibit strong local homogeneity. adjacent values demonstrate markedly distinct heterogeneous distributions. As shown in Fig. Ic and Fig. [d. when switching from keys to value similarities, adjacent value vectors (v, and V often show much lower or even negative correlations in some layers.
This local key-value asymmetry reveals critical limitations in existing methods: cache merging methods [32] 26 apply identical merging strategies to both keys and values. overlooking their fundamentally different distributional characteristics. More studies of this phenomenon will be discussed in
Training-free Asymmetry Modeling Based on the above analysis, the key challenge in cache merging lies in modeling heterogeneous values. Fortunately, through careful examine of attention mechanisms" mathematical structure, we discover an elegant solution to this value heterogeneity. We develop a mathematically proven value representation scheme that guarantees lossless attention computation after merging adjacent keys. Notably, our method remains distribution-agnostiic, making it inherently robust to value heterogeneity.

---

Building on the key-value asymmetry and the property for values, we propose AsymKV, novel training-free cache merging method for efficient long-context modeling. Our key insight to shift the information loss from heterogeneous values to homogeneous keys during merging, thereby minimizing overall loss. Extensive experiments demonstrate that our method consistently outperforms existing long-context methods across various tasks and base models. On LLaMA3. achieves an average score of 43.95 on LongBench, surpassing H₂O [3 (38.89) by a significant margin. These results demonstrate AsymKV's V's effectiveness in extending LLMs context handling capabilities without additional training.
AsymK V
The key contributions of this work are threefold. First, we reveal contrasting, yet previously overlooked asymmetry of local keys and values in LLM attention mechanisms. Second, based on this asymmetric property, we propose a novel training-free compression framework that combines homogeneity-bant
Our Contributions:
key merging with a mathematically proven lossless value representation. Third, we demonstrate through extensive experiments that our method consistently outperforms existing long-context methods across various tasks and base models.
2 Related Work
KV Cache Pruning Recent research focuses on compressing the KV cache through selective token retention and importance-bast based pruning. H₂O [3] introduces the concept of "Heavy Hitters" tokens that contribute significantly to attention scores and develops
preoretically-groundee
eviction policy. Building on this idea, RoCo [22] improves the robustness of cache compression
by considering both temporal attention scores and stability measures. More recent works like SnapkV [66 and Scissorhands [19] leverage the persistence of token importance across generation steps, while (9] demonstrates that L2 norm-based compression can achieve competitive results with simpler implementation. However, these compression methods face a fundamental challenge: they rely heavily on token-centric measures (e.g., attention scores or norm values) to determine which tokens to retain, potentially discarding tokens that suddenly become crucial for future predictions.
KV Cache Merging Another line of works have explored merging similar KV cache positions to reduce memory footprint during inference. CaM [32] proposes an adaptive merging strategy guided by attention scores, while D [26] introduces two-level discriminative approach considering both layer-wise patterns and token similarities. KVMerger [ adaptively constructs the KV cache by analyzing the intrinsic structure of attention modules. However, a fundamental limitation of these approaches is their uniform treatment of keys and values during merging despite their distinct distributional characteristics. As discovered in the introduction, this oversight is particularly problematic given the inherent heterogeneity of value vectors, which, unlike keys, often exhibit significant variations even between adjacent positions.
Context Segmentation and Sliding One polular variant of KV cache compression leverages context segmentation and sliding. These approaches stem from StreamingLLM (28), which discovered that initial tokens and recent tokens are more important than other middle tokens in the attention. Therefore, it only keeps such tokens. LongCacke (18] expands StreamingL.LM by keep critic middle tokens. These tokens are identified via the historical attention weights. SirLLM [31] uses token entropy to identify and keeps critic middle tokens. These segmentation-basse methods require minimal KV cache operations, making them computationally efficient. However, these compression strategies essentially involve directly discarding tokens with lower weights, which results in significant information loss when these tokens suddenly become critical for future predictions.
3 Proposed Method
Building on the key-value asymmetry, we propose AsymKV. Our main idea shift the information loss from heterogeneous values to homogeneous keys during merging, thereby minimizing the loss. We show the key intuition and framework of AsymKV in Figure 2 (Left) Previous approaches that apply identical merging strategies to both keys and values suffer from significant information loss, especially considering the heterogeneous nature of values. contrast, we leverage the asym- metry between keys and values in the attention mechanism. Our method compresses keys with minimal information loss (.D) due to their local homogeneity nature, while preserving the distinct characteristics of heterogeneous values through cardinality awaar normalization ($ 3.2
3

---

Key vectors
Value vedtors
Conventional Strategy Identical Merging - Loss from Value Heterogeneity
Our Key Strategy
Our Value Preservation Strategy
Merge Adjacent Keys with Minimal Loss
Via Cardinality-awaree Representation
K
K
K
minimal loss
+
K
Information loss when merge identically
Find optimal merged key (Sec 3.1)
Lossless attention output preserved
Locally Merged Attention (LMA)
Adjacent V: low similarity
Adjacent K: high similarity
IK]
exp(q
ex k,)
Illustration of our A mechanism. Left:: Conventional approaches that uniformly merge both keys and values lead to information loss. Middle: We merge adjacent homogeneous keys for minimal loss. Right: We preserve their heterogeneous values through cardinality normalization.
Figure 2:
3.1 Homogeneous Key Merging
Our primary goal of adjacent token compression to convert the original n tokens in the KV cache into n 1 by merging pair of adjacent positions m
First, consider the key merging. Based on our observation adjacent key homogeneity, we can merge adjacent keys without significantly affecting model performance. Given key vectors K = [ki, .g..... k..] from the KV cache and pair of adjacent positions m, m + be merged, let L(K) denote the language modeling loss. After compressing km, km+1 into one embedding k, we denote the new loss as:
C((Kem k, K>m-1))
(I)
where Kei denotes the sequence [kı,... k;-1). 1). and K denotes [k;+1:-.. kn). Our objective of optimal key compression to find that minimizes the information loss.
Due to the dimensional mismatch between the original K and [K k, Kym+1] in Eq. I (n x + (n (n-1) x d), cache merging takes two steps: Find pair of identical embeddings (k* k*) to replace (km, km+1) while preserving dimensionality, which is mathematically tractable. 2. Leverage attention properties merge the two tokens.
We first find optimal embeddings k that minimize L[[K<m k, k, K m))+ while keeping the dimensionality. For simplification, denote it as L(k, k), and the optimal k k* k" arg ming L(k, k).
We approach this optimization problem using Newton-like method. By applying second-order Taylor expansion of L(x, y) around (km km+1):
x km km+1
1 x km
x km
C(x,y) v) = L(km, km+1) + WLkkm km+1)
+
H
2
y
km+11
y
km+1
where is the Hessian matrix at (km; km+1). ). We denote the Hessian matrix H as:
1227
(3)
H
22

---

Each submatrix Hnt is a d x d matrix. To minimize L(k, k), we set x = k, y k and substitute into our quadratic approximation:
k km+11 )
k km
1
k km k km+1
k - km km+1
C(k, k)) C(kms km+1) + V (mmm))kmt1) km+1)
H
2
Following the Newton method, we find the critical point by setting the gradient of this quadratic approximation zero. This yields the optimal solution (details in Appendix
k- -HH + 2H H "H"',,,
km HH²(k,,, + km+1)=++ ) 2 km=2 (g.. gm++)]
(5)
where 8m kk L and gm++1 Vkm+1f. L.
To efficiently compute the Hessian matrix H, we use the Fisher information matrix as an approxima- tion, which is common technique in second-order optimization methods. Following the approach neural network pruning [1), we assume that the interactions between parameters are negligible and approximate the Fisher information matrix diagonal matrix. The diagonal elements can be efficiently computed using their gradients:
H Fa = VL(km km+1)
(6)
In our empirical analysis, we discover that the magnitude of gradient terms gm and gm++1 can exceed that of H by six orders of magnitude in Eq. (5). This a known issue in Newton-liks methods when the function is far from its minimum or when the curvature is very small. To stabilize the optimization
and maintain k* as a valid key, we adopt a modified Newton approach by dropping the gradient terms from Eq. 5) effectively using only the curvature information to guide our solution.
Compression Position Selection To minimize information loss during merging, we select positions m, m + 1 with the lowest sum of attention scores, where positions receiving minimal attention have the least impact on the model's attention mechanism.
3.2 Cardinality Normalization for Lossless Value Merging
After replacing adjacent keys with identical embeddings, two challenges remain: (I) how to reduce input tokens (k' k") k" to improve computing efficiency; (2) how merge their corresponding values. We elaborate how to extend the attention mechanism to address both challenges while maintaining output equivalence.
In the original attention mechanism, the output for query q is:
K
exp q kj)
Attention(q. K, V)
K
exp(q k,)
After key compression where km = km+1 k" in $ 3.1
we have:
exp(a k;)
exp(q-k'
Attention K, V) =
Σ
v, +
(vm + vm+1)
k
exp(q kj)
j=1 explq: kj)
attention one merged KV pair (key=k", value=v,, + V
Examining Eq. observe key insight: after converting both k,n km+1 to k" the attention output for two original tokens m and m + is mathematically equivalent to the attention output for a single compressed token with key k* and value (vm + Vm++1).
(Locally Merged Attention) The insight above naturally suggests an alternated attention mechanism for merged tokens, which we denote as Locally Merged Attention (LMA)::
exp(q ki)
LMA(q. K, V.C
= exp(q k;)c; V.
(9)
5

---

where indicates the number of original tokens represented by the i-th compressed token. The cardinality vector с is designed to maintain the denominator in Eq. ( ensuring mathematical equivalence between the original and compressed attention mechanisms. Initially, tokens. After merging positions m and m 1, we update C as [C<m; C. + Cm+11 12 ensuring the denominator in our attention calculation remains equivalent to the original uncompressed attention.
for all
(Equivalence) Using LMA, the following equivalence holds:
Attention(q. K, V) = LLMA K' K'KVV V C)
(10)
where K, V are KV caches for n tokens while K', V are for 72 - 1 tokens:
K' K <m, k" K K>m+1]
(11)
V' [V <m: Vm + Vm++1 V2m+1]
This equivalence reveals an elegant characteristic of attention mechanisms: they permit lossless compression of values through simple vector addition. The only cost to store C with n integers. This property is particularly powerful as it enables preservation of attention outputs while reducing sequence length, effectively solving the quadratic complexity challenge of long- onn-sequence processing.
3.3 Efficient Implementations for Long- Generation
3.3.1 Time-Efficiency by Chunk-wise Parallel Compression
We propose a chunk-based parallel compression method for efficient long-text generation. The process predicts the next token using HLA in place of the original attention, without requiring any model fine-tuning. More specifically, when the context length reaches max max_length, after every chunk_ size new tokens, we compress max max_length + chunk tokens into max_ length tokens in parallel by: 1. Identifying chunk pairs of adjacent tokens with lowest attention scores. 2. Computing optimal compression according to Eq. (5) 3. Merging keys, values and cardinalities.
Since the merge operation is performed only once every chunk_s size tokens (c.g., 512 tokens), its computational overhead minimal relative to the overall inference process. The compression step requires only a single backward pass to compute the Hessian matrices for all candidate compression positions, followed by parallel execution of the optimal compression operations. This design ensures that AsymKV V maintains inference efficiency. More experimental results are shown in $ 4.5
3.3.2 Memory-Efficcincy by Selective Gradient Computation
Our method requires gradient computation (Eq. 5 which might raise concerns about increased memory usage. However, AsymKV V still maintains memory efficiency compared to other approaches. Unlike typical backpropagation that computes gradients for all parameters, we only compute gradients for key embeddings within the current chunk, not for all model parameters.
To elaborate further from a quantitative perspective, this selective gradient computation yields gradient tensor of size approximately c x d, where e denotes the chunk size and is the dimension per token. In contrast, standard forward pass (and its associated gradient computation) requires storing the full model parameters (with size (pp). where is the total number of parameters) along with the KV cache states, sized Thus, the additional memory overhead from our selective gradients is on the order of O(ed), which significantly smaller than the baseline's s O(2ld p) when e < (a typical scenario in chunked processing). This avoids unnecessary gradient computations across the entire sequence, ensuring that both memory and computational overhead are greatly reduced. Detailed memory statistics are provided in $ 4.5
[ (where 1 represents the maximum sequence length).
4 Experiments
4.1
Experimental Setup
Baselines We compare AsymKV against several categories of approaches: KV cache compres-
sion: H₂O [33), KV cache merge: CaM 332 prompt compression: I..MIingua-2.0 21], context
segmentation: StreamingLLM (28 and LongCache [18.

---

Table 1: Performance LongBench. AsymKV outperforms its baselines most settings.
Single-Doc
Multi-Doc
Sum
Few-shot
Synthetic
Code
Avg.
Llama2-75 B-achat
Full Context
25.80
21.47
24.62
62.86
4.96
48.99
32.00
StreamingI.IMM
19.29
21.05
23.15
60.85
1.81
48.58
29.61
LongCache
19.73
20.06
23.19
61.26
2.24
49.05
29.71
H2O
19.92
25.64
23.85
61.37
4.27
50.28
31.34
I. Linguu222
21.47
23.29
23.53
33.23
6.17
35.12
24.20
CaM
19.53
20.64
22.67
61.81
4.18
48.53
30.15
AsymKV
24.63
24.15
24.22
62.11
10.18
52.16
33.12
Full Context
43.73
44.49
29.12
69.36
53.56
52.94
60.21
StreamingLLM
28.15
27.19
25.15
63.17
16.33
54.02
35.73
LongCache
28.98
27.84
25.35
64.73
19.68
53.60
36.70
H₂O
33.30
34.43
26.60
66.23
14.75
55.56
38.89
LLMLingua-2
32.02
32.24
24.99
27.87
17.67
52.63
30.75
CaM
32.14
32.63
24.91
63.09
16.77
54.03
37.49
AsymKV
39.42
38.93
27.30
65.66
39.39
55.24
43,95
Cestral-7B-Instrufff
Full Context
38.74
38.29
29.04
70.70
51.00
55.06
46.40
StreamingI.IMM
24.80
22.14
25.18
66.49
15.14
53.51
34.57
LongCache
26.05
22.31
25.44
66.21
14.93
53.43
34.80
H₂O
29.66
28.22
26.32
67.78
14.83
53.95
37.09
LLMLingua-2
28.12
28.62
25.75
45.85
16.00
48.81
32.17
CaM
26.15
29.06
26.81
66.16
20.96
53.76
37.12
AsymKV
33.71
32.81
27.04
67.21
34.56
54.93
41.33
Qwen2-7BB Owen2-7B-Instruct
Full Context
37.07
41.77
28.27
68.61
36.25
50.67
43.81
StreamingLLM
27.73
28.14
24.32
66.85
7.50
49.55
34.70
LongCache
27.98
28.98
24.80
66.38
9.00
48.34
34.94
H₂O
29.91
28.49
25.21
68.08
12.25
53.63
36.68
LL.Mingua-2
30.04
31.71
24.63
46.32
6.50
50.09
31.96
CaM
29.14
28.59
25.87
66.33
9.25
48.88
35.38
AsymKV
33.72
34.46
26.24
69.88
14.25
49.33
38.76
Base Models To demonstrate the generality of AsymKV, evaluate across diverse model ar- chitectures: Jlama2-7B-cha (24), 1lama3.1-8B-Instrue
10]. Misttrl-78-Innstrutt-0.333 [1] and
Qwen2-7B-Instrucc [29]
Implementation Details Unless otherwise specified. we set the compression context max_ max_length to 2048 tokens and chunk size to 512. All baselines use the same settings for fair comparison. For H2O, we set recent budget to 2048 and heavy budget to 512. Following attention sink (28), we always preserve the initial 32 tokens. All experiments are conducted on NVIDIA A100 80GB.
4.2 Long Context Performance Evaluation
We evaluate AsymKV's effectiveness on LongBench (2] a comprehensive benchmark for long- context understanding, LongBench contains 16 English tasks from wide range of categories.
Results As shown in Table I AsymKV consistently outperforms existing long-conters methods across different models and tasks. On LLaMA3.1-800 AsymKV achieves a 43.95 average score, surpassing H₂O (38.89) and other baselines by significant margin. The improvement is particularly pronounced in challenging tasks like Synthetic reasoning, where AsymK scores 39.39 compared to H₂O's 14.75. For all base models, AsymKV maintains its advantage with average scores, showing substantial improvements over baselines.

---

Table 2: Performance on LongBench
Model
Overall
Easy
Hard
Short
Medium
Long
Full Context
30.02
30.73
29.58
35.00
27.91
25.93
LL
27.04
27.60
26.69
32.78
23.26
25.00
LongCache
28.43
28.13
28.62
32.78
25.58
26.85
H2O
28.23
28.12
28.29
31,67
26.98
25,00
CaM
28.23
28.64
27.97
31.67
26.98
25.00
AsymKV
30.02
30.23
29.90
32.78
27.44
28.85
Table 3: Performance on early topic retrieval.
5000
Origin
Streaming
Model
Qwen2-7B
Llama3.1-8B
Mistral-7B
Time Costis)
4000
LongCache
H,O
Full Context
47.33
80.00
42.67
2000
CaM
LIM
0.00
0.00
LongCache
0.00
0.00
8.67
CaM
12.67
24.67
15.33
1000
HgO
21.33
63.33
38.67
01
LLMLingua2
0.00
0.00
0.00
20
30
50
60
Length
AsymKV
36.00
75.33
40.67
Figure 3: Inference efficiency.
Precise Information Retrieval In Single-Doc and Multi-Doc QA tasks, which require precise information retention, AsymKV consistently outperforms other methods by significant margins (5-10 points). This suggests that our homogeneity-base -ased compression effectively preserves key information needed for accurate question answering.
Extreme Long-Context Compression We evaluate AsymKV on LongBench 3 benchmark with contexts ranging from 8,000 to 2 million tokens across six task categories (multi-documeent QA, code comprehension, temporal reasoning. mathematical derivation, cross-lingual understanding. and hierarchical information synthesis). Using Llama3.1 --BB-Instucc with cache sache_size=8199 Table shows that AsymKV V matches full-context methods in short contexts while significantly outperforming baselines in medium long contexts (up millions of tokens), demonstrating its effectiveness in extreme long-come context senarios.
Regularization Effect of AsymKV. AsymKV often outperforms the full-context model across various tasks, suggesting that it acts as a form of regularization in long-context settings. Due to the inherent limitations of LLMMs in handling extended contexts, full KV caches tend accumulate redundant tokens with low attention scores, diluting focus on relevant information. By selectively merging these low-attention tokens, AsymKV effectively suppresses contextual noise, leading to more focused and efficient inference. A similar regularization phenomenon has also been observed in related K V-cache optimization methods 33
4.3
Comprehensive Information Retention
KV cache compression methods face multiple challenges in information retention: they must preserve not only early context details but also maintain the ability to capture document-level semantic structure and sequential relationships. capabilities, we conduct experiments using TopicRet [14] from L-Eval [I] This benchmark is particularly challenging as requires models to answer questions about the second or third topic in multi-topic documents, testing their ability to retain early context information.
evaluate models' comprehensive information retention
The results in Table 3 reveal several key findings. First, context segmentation methods (StreamingL_LM, LongCache and prompt compression approaches (LLMLingua2) completely fail this task, scoring zero or near-zero across all models. This dramatic performance drop confirms our hypothesis that discarding or imprecisely compressing early tokens severely impairs models' ability to access historical information. Although methods like CaM and H₂O show some capability in retaining early information, their performance significantly lags behind full-context processing. In contrast, AsymKV demonstrates remarkable effectiveness in preserving early context information, achieving scores close to full-context processing (75.33 vs 80.00 on LLaMA3.1-8B)

---

50
200
Score
Asymey
Asymky
AsymkV
Stramingu
StraemingLi
StreamigLLl
26.0
Lonatarhe
LongCache
Longfarhe
27.3
20
L 2
Hyo
LUMiingua 2
Hj0
IMiLingu
150
CAM
CsM
CaM
12.0
10%
2016
32%
40% 100%
10% 20% 2056
50%
130%
32
24
10%
25%
30%
100%
12%
20%
70%
107%
Compression Ratioi%)
Compression Ratiol%)
Compression Ratio(
Compression Ratio(9)
(a) Llama-3.1-8B-Innsee
(b) Mistral-7B-Inst-v0..3
(c) Qwent-7b-Instruut
(d) Llama-2-7b--hat-4k
Figure 4: Effect of different compression ratios.
Table 4: Peak GPU Memory (MB)
Figure 5: Ablation study different merge strategies.
Method
L.2-7B
L.2-13B
Value Merge Strategy
StreamingLLM
19,592
39,911
Key Merge Strategy
Same as Key
Asymmetric
LongCache
24,968
49,236
Mean Merge
18.93
21.86
H2O
22,479
47,310
Weighted hy Cardinality
20.90
23.91
CaM
22,548
47,476
Weighted by Attention
14.99
23.83
AsymKV
24,923
48,671
Optimal k" (Eq. (5)
21.21
24.43
4.4 Compression Rate Analysis
To systematically evaluate AsymKV's context compression capabilities, we analyze its performance across different compression rates on the long-context HotpotQA (30) task from LongBench. Here, the compression rate is defined as the ratio between the compressed and original token counts.
As shown in Fig. AsymKV demonstrates superior compression capabilities across all compression rates. Most notably, with only 20% of the original context length, AsymKV achieves performance comparable to full-context processing, significantly outperforming all baseline methods. This robust performance highlights AsymKV's effectiveness in preserving crucial contextual information even under aggressive compression.
4.5 Inference Efficiency
We evaluated the computational efficiency different approaches during text generation. To do this, we had the models generate text using greedy sampling on Fi-tral----Instuctte and measured the time required to generate different numbers of tokens.
Inference Speed Fig. 3 reveals that among the evaluated methods, context segmentation approaches MireaminggL.MM and LongCache) achieve the highest computational efficiency due to their minimal KV cache operations. However, this efficiency comes at the cost of performance. AsymKV strikes better balance, achieving the highest efficiency among compression- based methods.
Memory Consumption We measure peak GPU memory usage on LLaMA3. L.MA3.1-8B 1-B.--BBIimn Instruct with cache size of 2048 and chunk size of 128. As shown in Table AsymK V's 's memory consumption is comparable to other baselines. This validates that both generation and compression costs are practical and scalable.
4.6 Ablations on Merge Strategies
We conduct ablation study to compare different strategies for merging keys and values during compression. For key merging, we compare four approaches: simple mean pooling, cardinality- weighted averaging, attention score-war weighted averaging, and optimal strategy derived from Eq. (5). For each key merge strategy, we experiment with two value merge strategies: either using the identical strategy keys, using our proposed asymmetric cardinality-nommalin method.
Results in Table are scores on the multi-hop QA task MuSiQue (25 from LongBench. The results demonstrate two key findings. First, theoretically-derte derived optimal key merge strategy consistently

---

outperforms other approaches. This empirically validates the foundation of our optimal merging strategy presented in [3.1 Second, the results show that using distinct strategies for keys and values is beneficial.
5 Conclusion
In this paper, through extensive empirical analysis, we reveal a fundamental yet previously overlooked pattern: local KV cache Asymmetry. This property motivates key technical innovation training- free merging framework that combines homogeneity- baasbgeeety-bbeed key merging with mathematically proven lossless value representation. We present AsymK V, novel approach address the computational challenges long- Long-context modeling in LLMs.. Our comprehensive experiments demonstrate that AsymKV outperforms existing cong-context methods across various tasks and base models.
Limitations. Further applications of AsymKV need to consider compatibility with methods like FlashAttention and vLL.M. We view this as engineering problem and actively working to address it.
10

---

Acknowledgments and Disclosure of Funding Science Foundation (252R1402137).
This paper was supported by the Shanghai Natural
References
[1] Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong. and Xipeng Qiu. L.-vaal: Instituting standardized evaluation for long context language models. In Lei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14388-14411, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[2] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers). pages 1137-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[3] Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench v2: Towards deeper under- standing and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412. 15204, 2024.
[4] Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong. Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, 2025.
[5] Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, and Chao Huang. Seplim: Accelerate large language models by compressing one segment into one separator. arXiv preprint arXiv:2412. /2094, 2024.
[6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[7] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 16359, 2022.
[8] Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of iitormatioonseeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599-4610, 2021.
[9] Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective 1_2 norm-based strategy for KV cache compression. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18476-18499, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[10] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[11] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[12] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances neural information processing systems, 2, 1989.

---

[13] Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15339-15353, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[14] Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, lon Stoica, Xuezhe Ma. and Hao Zhang. How long can context length of open-source Ilms truly promise? In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
[15] Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. A survey large language model acceleration based on kv cache management. arXiv preprint arXiv:2412.1 19442, 2024.
[16] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye. Tianle Cai, Patrick Lewis, and Deming Chen. Snapky: Lim knows what you are looking for before generation. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 22947-22970. Curran Associates, Inc., 2024.
[17] Yuhong Li, Yinghing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2025. Curran Associates Inc
[18] Xiaoran Liu, Qipeng Guo, Yuerong Song, Zhigeng Liu, Kai Lv, Hang Yan, Linlin Li, Qun Liu, and Xipeng Qiu. Farewell to length extrapolation, training- infinite context with finite attention scope. arXiv preprint 24Xiv:2407. 15176, 2024.
[19] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of impor- tance hypothesis for IIm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024.
[20] Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi-state RNNs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18724-18741, Miami, Florida, USA, November 2024. Association for Computational Linguisties.
[21] Zhuoshi Pan. Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Ruhle, Yuqing Yang. Chin- Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024, pages 963-981. Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics.
[22] Siyu Ren and Kenny Q Zhu. On the efficacy of eviction policy for key-value constrained generative language model inference. arXiv preprint 222.:22200.02222.. 2024.
[23] Kimi Team. Kimi k1.5: Scaling reinforcement learning with 1lms. 2025.
[24] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini urXiv:2307.09288, 2023.
[25] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554 2022.
[26] Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, Longyue Wang, and Mi Zhang. D2o: Dynamic discriminative operations for efficient long-context inference of large language models. In The Thirteenth International Conference Learning Representations, 2025.
12

---

[27] Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for Ilms on long-context tasks, 2024.
[28] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis, Efficient streaming language models with attention sinks. ICLR, 2024.
[29] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024.
[30] Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369 2380, 2018.
[31] Yao Yao, Zuchao Li, and Hai Zhao. SirLLM: Streaming infinite retentive LLM. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2611-2624, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[32] Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. Cam: Cache merging for memory-efficiee Ilms inference. In Forty-first International Conference on Machine Learning, 2024.
[33] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing 2023.
Systems,
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state:
Discovery of KV cache asymmetry (key homogeneity value heterogeneity).
Proposal of AsymKV, a training-free compression framework combining key merging and lossless value representation.
Experimental validation showing SOTA performance (e.g.. 43,95 vs. 38.89 for H2O on LongBench).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see $ We discussed the potential engineering adaption problem.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
14

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
The paper includes detailed mathematical derivations for its key compression (Eq. 5) and lossless value compression (Eq. 8) methods. Assumptions, such as the use of approximation for the Hessian, are explicitly stated. Complete proofs are provided in the main text and appendices, supported by references to established techniques like Taylor expansion.
Justification:
the Fisher information matrix
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The experimental setup is thoroughly described, including the models, datasets, and hyperparameters. As AsymKV is training-fres free approach, the compression strategy is clearly outlined, providing sufficient detail for others to replicate the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Will be added in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
Sublic/guidese httai//guuiiee//ouuuuiiissstt
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
Since AsymKV is training-free no training details are required. Meanwhile, the paper provides comprehensive inference settings, including compression rates, chunk sizes, and model configurations. Key hyperparameters, such as max_ max_length and chunk size, are explicitly listed, ensuring clarity.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
16

---

Justification: The paper does not include traditional error bars due to the computational expense of long-context experiments, which allowed for only one run per configuration. However, the statistical significance of results is established through comprehensive eval- uations across multiple models (I.ama-3.11-3 Mistral-7B, Qwen2-7B, Llamma2-7B) and various settings. This extensive cross-model validation demonstrates the consistent perfor- mance advantages of AsymKV across different settings, providing strong evidence for the robustness of the reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative еттог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies that experiments were run on NVIDIA A A100 80GB GPUs and includes details on memory usage in
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httph: https://neurips: wwtprrips.ccppuuueee wwtp://pppppp.cccccooooooc htt.bliiie/hhcccccdellleeesss
Answer: [Yes]
Justification: The research focuses on improving LLM efficiency and does not involve human subjects, sensitive data, or applications with potential harm. fully aligns with the ethical standards outlined in the NeurlPS Code of Ethics.
Guidelines:
17

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper is foundational research and does not directly point to any potential negative social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer:[NA)
Justification:
No new high-risk assets released.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
18

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: the paper credits the original sources of datasets and models, mentioning applicable licenses and adhering to their terms of
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi paperswithcode thcode com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the implementation code of our proposed method to support reproducibility.
Asset Type: Source code (implementation of the proposed method AsymKV).
Intended Use: Research and reproducibility. Enables replication and extension of the reported results.
License: Apache License 2.0
Repository: https: www.c/oeeteeceeeeeeeeeeeeeeee AsynKV
Documentation: Usage instructions, environment setup, and experimental commands are provided in the repository README. md.
Ethical Considerations: No human personal data involved. All external datasets and models comply with their original licenses.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
19

---

Justification:
No human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects are involved in this research, rendering IRB approvals unnecessary.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
The paper clearly states that LLMS serve as the base models for evaluating AsymKV. Their usage is standard and well-documented, requiring no further declaration beyond what provided.
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Analysis: Key-Value Asymmetry in Attentions
lama-2-7b-chat (Average Spearman corr: 0.766)
La 2aa-2-7b-chht
Vistral-7B-Insstru
Mistral- 18-Instruct
(Auerage Spearman corn: 0.396)
(Average Spearann corr: 0.729)
(Avorage Spearman corr: 0.419)
075
0.75
0.5
4.5
05
0.5
0.5
24
0.2
Hebd
(a) Key similarity correla- (b) Value similarity correla- (c) Key similarity correla- (d) Value similarity corre- tion heatmap QASPER tion heatmap on QASPER tion heatmap
lation heatmap
Figure 6: Contrasting distributions of local key distributions versus local value distributions across different datasets and model architectures. Heatmaps show Spearman correlation coefficients between adjacent tokens across layers (y-axis) and attention heads (x-axis). The consistent strong positive correlations for local keys (a.c) and weak/negative correlations for local values (b,d) suggesting these are universal properties of LLM KV caches.
Initial experiments with Llama2-7b-char on ShareGPT 6 data (Fig. D revealed striking asymmetry between local key and value distributions. To obtain a more direct observation, we analyze the Spearman correlation coefficient of sim(key,. key,) and sim (val,, val;) for different j. To validate the universality of these patterns, we conducted comprehensive analysis across diverse settings: (1) different data distributions, including academic papers (QASPER []) and multi-domain questions (2) and (2) various model architectures, specifically Mistral-7B-Instrucee are shown in with additional results for MultiFiclQAA and Qwen2-7B-Inseee presented Appendix B
(MultiFieldQA(en)
and -eeen---B-Insruuee (29]). The results for QASPER and
Fig. 6
Key Homogeneity: Adjacent keys exhibit consistently strong positive correlations across all layers and attention heads (average correlation coefficient 0.7), indicating robust encoding of local semantic relationships in key representations.
Value Heterogeneity: In stark contrast, adjacent values show significantly lower (average correlation coefficient < 0.4) or even negative correlations, suggesting that value vectors encode distinct and complementary aspects of token information. This heterogeneity appears essential for maintaining the model's representational capacity.
B Distribution of Local Keys and Values in More Models and Datasets
To validate that our observations about the asymmetric properties of keys and values are general across different models and datasets, conduct additional experiments the Multifieldga(en) dataset and the Qwen2-7B-Instin
model. The results are shown in Figure 7
consistently exhibit strong diagonal block patterns, indicating high local homogeneity. In contrast, the value similarity heatmaps (Figure 7b 7d show heterogeneous distributions. These results confirm that the asymmetric properties we observed are inherent characteristics transformer attention mechanisms rather than artifacts of specific models datasets.
As shown in Figure 7 the key similarity heatmaps (Figure 7a 7c
C
Solving for the Optimal Key Vector
Continue from
Gradient Term Expansion:
km
Vlkkm, km+1) T
g (k km) + -gT+ (k km+1)
(12)
k km+1
21

---

-lama-2-7bchatt
Liama 2:7b cb-chat (Average Spearman 0.41)
Qwen2: 78 78-Instruct
Qwen2: 18-instruct
(Average Spearman corr: 0.777)
(Average Speerman corr: 0.759)
(Ayerage Speerman corr: 0.413)
1.0
1.0
0.5
0.5
4.
16
14
5.55
0.25
025
0.25
21
.02
-0.2
-0.2
1,
24
15
24
(a) Key correlation on Mul- (b) Value correlation on (c) Key correlation on (d) Value correlation on tiFieldqa(en)
MultiFicllqa(en)
Qwen2-7B
Qwen2-7B
Figure 7: Similarity heatmaps of local keys and values across different models and datasets (LLaMA2- 7B-chat MultiFieldga(en) and Qwen2-7B BBIIntruct ShareGPT). The diagonal blocks in key heatmaps (a, c) indicate strong local homogeneity, while the more scattered patterns in value heatmaps (b, d) demonstrate heterogeneity.
Quadratic Term Expansion:
k km
H
Km km+1
= (k km) TH2 (k km) + (k km)
H2 (k km+1)
2 k km+1
k
(k km+1)
22² km+1)
(13)
Constructing the Total Objective Function
Adding the above terms, the objective function with respect to k is:
1
L(k) Ro Likm, km+1) + Bm (k- km)) + 1m++ (k km+t) + 2 (k km) T (k - km)
(14)
+ (k-km) H22 km+1) + 5(kk - km+1) H (k km+1)
Taking the Derivative of C(k) with Respect and Setting to Zero
Taking the derivative:
OC
Øk Em + gm+1 + H km)
( km+1) + H 12T
12T (k km) + 22 km+1)
(15)
Since the Hessian matrix is symmetric, i.e., 2 21 we have:
OC
Øk gm + gm+1 + (H" + 22 12 2 ) (H km (km + km+1) H2 222kkman km+1)
(16)
Setting the derivative to zero yields the optimal condition:
(H 22
2 k² -H k,. (k,,, km+1) 2 km+ (gm + gm+1)
(17)
Solving for the Optimal Key Vector k*
The optimal key vector k* is obtained as: HH + 2 12 H 1 km (km + km+1) + 2 km+1 (gm + gm++))
(18)
D Comparison of AsymKV with Other New Baselines
We also conducted additional comparisons with SnapKV [17], PyramidKV 44, TOVA 20] D20 26
and L 2-Norm These experiments were performed on the LongBench dataset based on the Llama3-8b- with compression context lex_hengg.
As show in Table 5 AsymKV still demonstrates performance improvements compared to these baselines.
22

---

Table 5: Comparison of AsymKV with other new baselines LongBench.
Single-Doc
Multi-Doc
Sum
Few-shot
Synthetie
Code
Avg.
Llama3- -ee-BBB-Insse Instruct
Full Context
32.19
34.59
24.96
68.48
36.96
54,41
41.46
StreamingI.IMM
27.90
25.92
24.49
65.09
13.87
55.02
35.50
LongCache
28.26
25.64
24.69
65.75
15.50
54.65
35.83
H2O
30.65
32.77
24.61
61.83
37.08
54.87
39.59
LLMIingua-
26.50
30.80
24.10
39.30
22.50
32.20
29.47
CaM
30.49
31.48
24.85
63.83
37.02
55.46
39.81
TOVA
31.82
27.94
24.57
64.34
19.29
54.06
37.04
L2
30.18
27.41
24.70
63.29
37.34
51.78
38.43
D20
30.81
32.87
24.64
67.42
36.67
56,49
40.85
SnapKV
32.17
34.20
25.28
68.57
37.21
53,30
41.36
PyramidKV
31.79
34.02
25.44
68.57
37.24
54,97
41.49
AsymKV
34.45
33.64
26.17
67.94
38.66
56.61
42.32
E Licenses for Existing Assets
We list the assets used in this paper and their licenses below:
24
Ilama2
IIlama
|
AApache 2.0 License
29
,Apache 2.0 License
2
JMM License
14
AApache 2.0 License
30J,CC BY-SA 4.0
MMIT License
GNU General Public License v3.0
6 Illam22
28 MIT License
23


Reviews:
[{"review_id": "pqVc6gddQU", "reviewer": "Reviewer_8hqr", "summary": "This paper introduces AsymKV, a novel training-free KV cache compression method tailored for long-context LLM inference. The key insight lies in the empirical discovery of a local asymmetry in attention KV caches: adjacent keys tend to be similar (homogeneous), while adjacent values are highly diverse (heterogeneous). Based on this observation, the authors propose a compression technique that merges adjacent homogeneous keys while preserving the values using cardinality-aware normalization. AsymKV demonstrates consistent improvements over prior methods such as H2O and StreamingLLM across a range of tasks and base models, especially on long-context understanding and early information retrieval.", "questions": "1. The paper claims that the method is efficient due to selective gradient computation. However, it would be helpful to include a detailed overhead analysis\n2. Can the method be extended to the decoding stage? Given that the outputs of current reasoning models are becoming increasingly long, applying the method only during the prefill stage may be insufficient. While I understand this is beyond the current scope, it would be valuable to hear the authors’ perspective or insights on potential extensions in this direction.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1) The discovery of the homogeneous key vs. heterogeneous value asymmetry in KV caches is both original and significant. It challenges prior uniform treatment of keys and values and lays a foundation for more nuanced cache compression strategies in LLMs. It might leads to more exploration of KV Cache compression.\n2) The adjacent key merging combined with value addition is a simple yet elegant solution that leverages attention linearity. \n3) The authors conduct extensive experiments across multiple base models (LLaMA3.1, Mistral, Qwen2, LLaMA2) and tasks, demonstrating consistent improvements\n\nWeakness:\n1) The comparison lacks several state-of-the-art methods such as SnapKV, PyramidKV, and Ada-KV, which have shown strong performance in recent studies. Moreover, in the cache merging category, only CaM is used as a baseline, while other recent methods (e.g., D2O, KVMerger) are omitted, weakening the empirical claims.\n2) Some key evaluations, notably compression rate analysis (§4.4) and merge strategy ablations (§4.6), are conducted only on a single dataset or task, which may not be fully representative. This reduces the generalizability of the ablation findings.\n\n[1] https://arxiv.org/abs/2404.14469\n[2] https://arxiv.org/abs/2406.02069 \n[3] https://arxiv.org/abs/2407.11550", "was_revised": true, "final_justification": "The authors added the results of more baseline methods, as well as several ablation experiments, making the experiments more complementary. Thus I increase my score.", "rebuttal": "**W1**: *The comparison lacks several state-of-the-art methods such as SnapKV, PyramidKV, Ada-KV, D2O, and KVMerger.*  \n\nWe have added the comparison with **SnapKV**[1], **PyramidKV**[2] , TOVA [3], **D2O** [4] and L2-Norm [5]. These experiments are based on the Llama3-8b-Instruct with compression context max_length=2048. AsymKV still demonstrates performance improvements compared to these baselines.\n| LLaMA3-8B-8K       | Single-DocQA | Multi-DocQA | Sum   | Few-shot  Learning | Synthetic | Code   | average |\n|------------|--------------|-------------|-------|-------------------|-----------|--------|---------|\n| TOVA       | 31.82        | 27.94       | 24.57 | 64.34             | 19.29     | 54.06  | 37.04   |\n| L2-Norm         | 30.18        | 27.41       | 24.70 | 63.29             | 37.34     | 51.78  | 38.43   |\n| D2O        | 30.81        | 32.87       | 24.64 | 67.42             | 36.67     | 56.49  | 40.85   |\n| SnapKV \\*     | 32.17        | **34.20**       | 25.28 | 68.57             | 37.21     | 53.30  | 41.36   |\n| PyramidKV \\*  | 31.79        | 34.02       | 25.44 | **68.57**             | 37.24     | 54.97  | 41.49   |\n| **AsymKV  (ours)**   | **34.45**        | 33.64       | **26.17** | 67.94             | **38.66**     | **56.61**  | **42.32**   |\n\n\\* Results are obtained from [2]\n\n[1] SnapKV: LLM Knows What You are Looking for Before Generation\n\n[2] PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling\n\n[3] Transformers are Multi-State RNNs\n\n[4] D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models\n\n[5] A Simple and Effective L_2 Norm-Based Strategy for KV Cache Compression\n\n\n\n**W2**: *Some key evaluations (§4.4 and §4.6) are conducted only on a single dataset or task, which may not be fully representative.*  \n\nWe added the experiment on the summarization task QMSum with the Llama3.1-8B-Instruct. The results are as follows, which are consistent with the conclusions in §4.4 and §4.6.  \n\n**§4.4. Compression Rate Analysis.** AsymKV demonstrates superior compression capabilities across all rates.  \n\n| Method       | 2%     | 10%    | 20%    | 30%    | 40%    | Full Context |\n|--------------|-------|-------|-------|-------|-------|-------|\n| LongCache    | 18.49 | 19.98 | 20.35 | 20.91 | 21.46 | 23.97 |\n| StreamingLLM | 18.32 | 19.3  | 20.31 | 20.52 | 21.38 | 23.97 |\n| LLMLingua-2   | 17.27  | 19.43  | 20.24  | 21.37  | 20.86  | 23.97  |\n| H2O           | 20.19  | 21.78  | 22.58  | 22.35  | 22.57  | 23.97  |\n| CaM           | 20.12  | 21.69  | 22.39  | 22.42  | 22.62  | 23.97  |  \n| AsymKV     | **20.6**  | **22.65** | **23.60** | **24.25** | **24.41** | 23.97 |\n\n**§4.6. Merge Strategy Ablations.** The theoretically-derived optimal key merge strategy consistently  outperforms other approaches. Using distinct strategies for keys and values  is beneficial.  \n\n| Method                  | Same as Key | Asymmetric |\n|-------------------------|-------------|------------|\n| Mean Merge              | 19.42       | 21.76      |\n| Weighted by Cardinality | 20.45       | 23.03      |\n| Weighted by Attention   | 18.30       | 22.82      |\n| Optimal k* (Eq. (5))    | **21.04**       | **23.46**      |  \n\n**Q1**: *Can detailed overhead analysis of selective gradient computation be added?*  \n\nWe have discussed the computational overhead in Section 3.3.2, where we explain the selective nature of our gradient updates, with empirical validation in Table 4.\n\nTo elaborate further with a more quantitative perspective, our approach computes gradients **only for the keys in the current chunk**, yielding a gradient tensor of size approximately $c \\times d$, where $c$ denotes the chunk size and $d$ is the dimension per token. While backpropagation inherently flows from the output through the model parameters and intermediate states, we mitigate peak memory usage by restricting the `requires_grad` attribute solely to the targeted chunk keys. This ensures that gradients are computed and stored exclusively for those specific tensors, without propagating or materializing gradients for the full model parameters.\n\nBy comparison, a standard forward pass stores these parameters plus KV cache states (size $2 \\times l \\times d$, where $l$ is the maximum sequence length). Thus, our additional overhead is far smaller than the baseline's $O(2 l d + p)$ when $c \\ll l$ (typical in chunking), minimizing unnecessary computations across the sequence.\n\n**Q2**: *Can the method be extended to the decoding stage?*  \n\nWe would like to clarify that our method already applies to the decoding stage. See the **Chunk‑wise Parallel Compression** scheme described in Section 3.3.1. After each new chunk of tokens is generated, we trigger the lightweight, parallel compression on the accumulated KV cache—exactly as during prefill—without interrupting token streaming.  \nFigure 3 already verified its effect—AsymKV generate long sequences in linear time cost."}, {"review_id": "sW75bEgKVI", "reviewer": "Reviewer_B4ak", "summary": "This paper proposes AsymKV, a training-free KV cache compression method that uses some qualitative insights from the authors on the distribution of keys and values in attention mechanisms. The authors observe that the keys of nearby tokens exhibit local homogeneity (i.e., similar attention patterns), while values of nearby tokens can have very different attention distributions. Based on this, they propose two separate KV cache compression strategies: a homogeneity-based merging approach for keys, and a cardinality-aware normalisation for values that preserves attention distributions. AsymKV shows significant improvements compared to baselines like H2O on long-context benchmarks like LongBench.", "questions": "How/when do the key-value asymmetries emerge during inference?\nHow big is the overhead introduced by the backward passes for gradient computations, in terms of time and memory?\nA big issue with many KV cache compression methods is the integration with FlashAttention, since it makes materialising attention scores very computationally expensive. Is this a limitation for AsymKV as well?", "limitations": "Please see my previous comments", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The authors did a great job at identifying and characterising the key-value asymmetry in attention distributions; they show on several datasets that attention distributions of the keys of nearby tokens correlate strongly, while this is often not true for values. The framework for deriving the key merging strategy is sound and well-motivated, and LMA also makes sense. The evaluation on several model families (LLaMA, Mistral, Qwen) and benchmarks (LongBenchV2) is comprehensive (although it would be nice to include some straightforward KV cache compression approach like L2 from https://arxiv.org/abs/2406.11430 as a baseline).\n\nI have some concerns about the use of backward passes for gradient computation -- how significant is the overhead it introduces? Does it limit the applicability of the method in memory-constrained settings (which is where KV cache compression makes the most sense)? Can you please expand on this?\n\nSince the whole paper is based on the observation from the authors of key-value asymmetries, it would be nice to have a bit of analysis on those (when/how do they emerge? How do they look like?) -- at the moment this is largely lacking.", "was_revised": true, "final_justification": "Thank you for your in-depth answers -- you addressed my main concerns (lack of extremely simple but effective baselines and compatibility with Flash Attention). I'm updating my rating accordingly.", "rebuttal": "**W1**: *It would be nice to include some straightforward KV cache compression approach like L2.*\n\nWe have added the comparison with SnapKV[1], PyramidKV[2] , TOVA [3], D2O[4] and **L2-Norm** [5]. These experiments are based on the Llama3-8b-Instruct with compression context max_length=2048. AsymKV still demonstrates performance improvements compared to these baselines.\n| LLaMA3-8B-8K       | Single-DocQA | Multi-DocQA | Sum   | Few-shot  Learning | Synthetic | Code   | average |\n|------------|--------------|-------------|-------|-------------------|-----------|--------|---------|\n| TOVA       | 31.82        | 27.94       | 24.57 | 64.34             | 19.29     | 54.06  | 37.04   |\n| L2-Norm         | 30.18        | 27.41       | 24.70 | 63.29             | 37.34     | 51.78  | 38.43   |\n| D2O        | 30.81        | 32.87       | 24.64 | 67.42             | 36.67     | 56.49  | 40.85   |\n| SnapKV \\*     | 32.17        | **34.20**       | 25.28 | 68.57             | 37.21     | 53.30  | 41.36   |\n| PyramidKV \\*  | 31.79        | 34.02       | 25.44 | **68.57**             | 37.24     | 54.97  | 41.49   |\n| **AsymKV  (ours)**   | **34.45**        | 33.64       | **26.17** | 67.94             | **38.66**     | **56.61**  | **42.32**   |\n\n\\* Results are obtained from [2]\n\n[1] SnapKV: LLM Knows What You are Looking for Before Generation\n\n[2] PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling\n\n[3] Transformers are Multi-State RNNs\n\n[4] D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models\n\n[5] A Simple and Effective L_2 Norm-Based Strategy for KV Cache Compression\n\n**W2&Q2**: *Concerns about the use of backward passes for gradient  computation -- how significant is the overhead it introduces?*\n\nWe have discussed the computational overhead in Section 3.3.2, where we explain the selective nature of our gradient updates, with empirical validation in Table 4.\n\nTo elaborate further with a more quantitative perspective, our approach computes gradients **only for the keys in the current chunk**, yielding a gradient tensor of size approximately $c \\times d$, where $c$ denotes the chunk size and $d$ is the dimension per token. While backpropagation inherently flows from the output through the model parameters and intermediate states, we mitigate peak memory usage by restricting the `requires_grad` attribute solely to the targeted chunk keys. This ensures that gradients are computed and stored exclusively for those specific tensors, without propagating or materializing gradients for the full model parameters.\n\nBy comparison, a standard forward pass stores these parameters plus KV cache states (size $2 \\times l \\times d$, where $l$ is the maximum sequence length). Thus, our additional overhead is far smaller than the baseline's $O(2 l d + p)$ when $c \\ll l$ (typical in chunking), minimizing unnecessary computations across the sequence.\n\n**W3&Q1**: *it would be nice to have a bit of analysis on the observation of key-value asymmetries (when/how do they emerge? How do they look like?)*\n\nWhile the main paper leverages this observation for compression, we address its characteristics and origins below, with pointers to our existing analyses.\n\n**How Do They Look Like?**\n\nWe have provided comprehensive visualizations and quantifications in Appendices A and B, including Spearman correlation heatmaps across multiple layers, heads, models and datasets, revealing strong local key homogeneity (correlations >0.7) versus weak value heterogeneity (correlations <0.4). Notably, in relatively lower layers (1-5), adjacent values tend to exhibit negative correlations.\n\n**How/When Does the Asymmetry Emerge?**\n\nTo our knowledge, no prior work explicitly identifies this local KV asymmetry, but its emergence is a natural consequence during transformer pre-training as models balance local attention biases with diverse representations. Key homogeneity arises from strong local dependencies, as shown in Longformer (Beltagy et al., 2020) [1] and Big Bird (Zaheer et al., 2020)[2] , where adjacent keys share high similarity due to syntactic and semantic coherence. Value heterogeneity, in contrast, stems from the need for distinct token information in residual streams, as explored in Synthesizer (Tay et al., 2021) [3] for high-rank attention signals and the Anthropic framework (Elhage et al., 2021) [4] for circuit-level orthogonality. This asymmetry likely solidifies mid-training, optimizing efficiency in attention while preserving expressivity.\n\n[1] Longformer: The Long-Document Transformer, by AI2\n\n[2] Big Bird: Transformers for Longer Sequences, by Google Research\n\n[3] Synthesizer: Rethinking Self-Attention for Transformer Models, by Google Research\n\n[4] A Mathematical Framework for Transformer Circuits, by Anthropic\n\n**Q3**: *Does integration with FlashAttention (expensive for attention scores) limit AsymKV?*\n\nAsymKV is compatible with the core principle of FlashAttention—leveraging the GPU memory hierarchy (HBM and SRAM), and through tiling, split Q, K, and V into small blocks that are loaded into high-speed SRAM for computation. To adapt the hierarchy to AsymKV, we only need to additionally pass the tiled cardinality vector into SRAM (Eq. (9))."}, {"review_id": "cBz19qqUIH", "reviewer": "Reviewer_Tmu7", "summary": "This paper addresses the phenomenon of asymmetry in KV caches –– adjacent values have distinct heterogeneous distributions, even though adjacent keys are similar in attention weights. Existing methods overlook this asymmetry by merging keys and values identically, while AsymKV instead merges only homogeneous keys using a mathematically derived method that minimizes information loss, and combines the corresponding values via a lossless cardinality-aware normalization to preserve attention outputs exactly. Experiments on LongBench and LongBenchV2 across multiple models (e.g., LLaMA3.1-8B, Mistral-7B) demonstrate that AsymKV outperforms prior state-of-the-art methods like H2O and CaM in both performance and compression efficiency, achieving robust gains even under aggressive compression without retraining.", "questions": "1.\tIn Figure 3, why the result of StreamingLLM is not showing?\n\n2.\tAccording to table 2 and figure 4, why after compression, the results may be better?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1.\tThe discovery of the heterogeneous distributions of key values is innovative. \n\n2.\tThe derivation of the best key merging strategy is mathematically rigorous, and ablation study shows its effectiveness.\n\n3.\tAsymKV does not require model retraining or fine-tuning, making it practical and easy to adopt across existing LLMs.\n\n4.\tAsymKV excels in tasks requiring retention of early context (e.g., Topic Retrieval), where many other methods fail, highlighting its advantage in maintaining semantic continuity.\n\n## Weaknesses\n\n1.\tThe experiments shows marginal superiority to other compression methods in some tasks, for example “Sum” and “Few-shot”, in table 1. And in some tasks this method is beaten by other baselines.", "was_revised": true, "final_justification": "The authors have addressed my question about practical on-chip acceleration ratio. I agree with the authors that with a slight change of kernel design, this method could be seamlessly integrated with FlashAttention. I kept my positive score.", "rebuttal": "**W2**: *On-chip acceleration may be limited compared to the one with Flash-Attention.*\n\nAsymKV is compatible with the core principle of FlashAttention—leveraging the GPU memory hierarchy (HBM and SRAM), and through tiling, split Q, K, and V into small blocks that are loaded into high-speed SRAM for computation. To adapt the hierarchy to AsymKV, we only need to additionally pass the tiled cardinality vector into SRAM (Eq. (9)).\n\n**Q1**: *In Figure 3, why the result of StreamingLLM is not showing?*\n\nWe have shown the results of StreamingLLM in Figure 3. Its line is covered by CAM's line because it is very close to CAM's results.\n\n**Q2**: *According to table 2 and figure 4, why after compression, the results may be better?*\n\nAsymKV acts as a form of regularization: in long contexts, full KV caches often include redundant low-attention tokens that dilute query focus (§3.1). By selectively merging these (based on attention scores), AsymKV reduces noise and quadratic overhead. This phenomenon is also observed in other KV optimization studies; e.g., Zhang et al. (2024) [1] in H2O report slight performance gains from evicting low-attention tokens, attributing it to the regularization effect.  \n\n[1] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models"}, {"review_id": "HwHIZ1DxF6", "reviewer": "Reviewer_AZPM", "summary": "This paper proposes AsymKV, a training-free KV cache compression method for large language models (LLMs) that enables efficient long-context processing by leveraging a previously overlooked asymmetry in attention: keys are locally homogeneous (similar across adjacent positions), while values are heterogeneous. Unlike prior methods that treat keys and values uniformly, AsymKV merges adjacent keys using a second-order optimization technique and preserves values through a mathematically lossless, cardinality-normalized representation. This design minimizes information loss and maintains attention output equivalence. Experiments across various models (e.g., LLaMA3, Mistral) and benchmarks demonstrate that AsymKV provides better accuracy than existing approaches.", "questions": "1. How is the proposed method compared with some other KV compression techniques such as TOVA, DuoAttention, and FastGen?\n2. What is the compression ratio in Figure 3?\n3. Why is the x-axis in Figure 4 denoted as compression ratio? What does it mean when the compression ratio is 100%? Is this a typo?\n4. For Table 2, could you explain why almost all KV compression methods seem to have a very good accuracy, compared to the full context baseline, especially on long sequences?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[Strengths]\n1. The paper empirically validates a key-value asymmetry in attention — keys are homogeneous, values are heterogeneous, and correspondingly proposes a merging method to compress the KV cache.\n2. The accuracy of the proposed method is validated across different models and benchmarks, surpassing the existing KV cache compression techniques such as H2O and CaM.\n\n\n[Weaknesses]\n1. Citations and comparisons to some important related works are missing, such as TOVA[1], DuoAttention[2], FastGen[3], and etc.\n2. The proposed method, although preserves accuracy better than existing methods such as CaM, does not provide better inference speed than existing solutions.\n\n[1] Matanel Oren, et al. Transformers are multi-state rnns, 2024.\n\n[2] Guangxuan Xiao, et al. Duoattention: Efficient long-context llm inference with retrieval and streaming heads, 2024.\n\n[3] Suyu Ge, et al. Model tells you what to discard: Adaptive KV cache compression for LLMs, 2024.", "was_revised": true, "final_justification": "I have read the authors’ rebuttal and increased my score. I will not fight against this paper but the authors should also improve the clarity of some tech details in the final edition. (E.g., clarify the questions raised by reviewers.)", "rebuttal": "**W1&Q1**: *Comparisons to some important related works are missing, such as TOVA, DuoAttention, FastGen.*  \n\nWe have added the comparison with SnapKV[1], PyramidKV[2] , **TOVA** [3], D2O[4] and L2-Norm [5]. These experiments are based on the Llama3-8b-Instruct with compression context max_length=2048. AsymKV still demonstrates performance improvements compared to these baselines.\n| LLaMA3-8B-8K       | Single-DocQA | Multi-DocQA | Sum   | Few-shot  Learning | Synthetic | Code   | average |\n|------------|--------------|-------------|-------|-------------------|-----------|--------|---------|\n| TOVA       | 31.82        | 27.94       | 24.57 | 64.34             | 19.29     | 54.06  | 37.04   |\n| L2-Norm         | 30.18        | 27.41       | 24.70 | 63.29             | 37.34     | 51.78  | 38.43   |\n| D2O        | 30.81        | 32.87       | 24.64 | 67.42             | 36.67     | 56.49  | 40.85   |\n| SnapKV \\*     | 32.17        | **34.20**       | 25.28 | 68.57             | 37.21     | 53.30  | 41.36   |\n| PyramidKV \\*  | 31.79        | 34.02       | 25.44 | **68.57**             | 37.24     | 54.97  | 41.49   |\n| **AsymKV  (ours)**   | **34.45**        | 33.64       | **26.17** | 67.94             | **38.66**     | **56.61**  | **42.32**   |\n\n\\* Results are obtained from [2]\n\n[1] SnapKV: LLM Knows What You are Looking for Before Generation\n\n[2] PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling\n\n[3] Transformers are Multi-State RNNs\n\n[4] D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models\n\n[5] A Simple and Effective L_2 Norm-Based Strategy for KV Cache Compression\n\n**W2**: *Though preserving accuracy better than methods like CaM, does it fail to outperform existing solutions in inference speed?*  \n\nAccording to Figure 4, AsymKV achieves a comparable accuracy with much fewer tokens than CaM. For example, AsymKV (10% tokens) achieves the effect of CaM (30% tokens). Therefore, if the goal is to achieve higher inference efficiency and lower memory usage while not reducing accuracy, a simple solution is to reduce the token budget in AsymKV.\n\n**Q2**: *What is the compression ratio in Figure 3?*  \n\nWe would like to clarify that Figure 3 does not involve compression ratio. All methods are uniformly set with context max_length of 2048 tokens. This is explained in lines 206-207.\n\n**Q3**: *Why is the x-axis in Figure 4 denoted as compression ratio? What does it mean when the compression ratio is 100%?*  \n\nIn line 248, we have already defined the compression ratio as the ratio between the compressed and original token count. Therefore, 100% compression ratio means no compression is applied, and the full context is used.\n\n**Q4**: *For Table 2, could you explain why almost all KV compression methods seem to have a very good accuracy, compared to the full context baseline, especially on long sequences?*  \n\nThis is a misunderstanding. LongBenchV2 consists of 4-choice questions. Most baselines do not show a significant improvement  over random guesses (25%). In particular, for the Long category (with  lengths ranging from 128k to 2M), all methods except AsymKV have almost degraded to random guesses."}]

Author rebuttal:

Meta-review: This paper introduces AsymKV, a training-free KV-cache compression method that exploits KV-cache asymmetry for long-context LLM inference. The method is readily integratable with existing solutions such as FlashAttention, offering practical speed-up benefits. The authors have addressed most concerns raised during the rebuttal, and all reviewers are open to accepting the submission. Given the many new results provided in the rebuttal, these should be incorporated into the camera-ready version.
Decision: Accept (poster)