Title: ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

KV cache

Compression

Long-context
Abstract: Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70\% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded performance. We introduce \method{}, which fundamentally reimagines KV cache compression by treating semantic chunks - rather than isolated tokens - as basic compression units. This approach preserves complete linguistic structures and contextual integrity, ensuring that essential meaning is retained even under aggressive compression. Our innovation includes a novel layer-wise index reuse technique that exploits the higher cross-layer similarity of preserved indices in \method{}, reducing computational overhead and improving throughput by 26.5\%. Comprehensive evaluations on challenging benchmarks: LongBench, Needle-In-A-HayStack, GSM8K, and JailbreakV demonstrate that \method{} outperforms state-of-the-art methods by up to 8.7\% in precision while maintaining the same compression ratio. These results confirm that semantic-aware compression significantly enhances both efficiency and performance for long-context LLM inference, providing a simple yet effective solution to the memory bottleneck problem. \emph{The code is available at \href{https://github.com/NVIDIA/kvpress}{link}.}

ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference

Abstract

Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded performance. We introduce ChunkKV, which fundamentally reimagines KV cache compression by treating semantic chunks - rather than isolated tokens - as basic compression units. This approach preserves complete linguistic structures and contextual integrity, ensuring that essential meaning is retained even under aggressive compression. Our innovation includes a novel layer-wise index reuse technique that exploits the higher cross-layer similarity of preserved indices in ChunkKV, reducing computational overhead and improving throughput by 26.5%. Comprehensive evaluations on challenging benchmarks: LongBench, Needle-InA-HayStack, GSM8K, and JailbreakV demonstrate that ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision while maintaining the same compression ratio. These results confirm that semantic-aware compression significantly enhances both efficiency and performance for long-context LLM inference, providing a simple yet effective solution to the memory bottleneck problem. The code is available at link.

1 Introduction

Large Language Models (LLMs) have become essential for addressing various downstream tasks of natural language processing (NLP), including summarization and question answering, which require the interpretation of a long context from sources such as books, reports, and documents, often encompassing tens of thousands of tokens [1-5]. Recent advances in long-context technology within the field of machine learning (ML) systems [6-8] have significantly improved computational throughputs and reduced latency of LLMs to process increasingly large input context lengths [9, 10] with historical KV cache (key value attentions). However, the memory requirement of the KV cache in serving super-long contexts becomes a new bottleneck [11-14]. For instance, the KV cache for a

single token in a 7B-parameter model requires approximately 0.5 MB of GPU memory, resulting in a 10,000-token prompt consuming around 5 GB of GPU memory.

Question: purple-crested turaco eats what food?

ChunkKV:

𝑆

!

=	∑

𝑓 𝑡

"

#

"$%

,

where	𝑐	 =	{𝑡

%

, … , 𝑡

#

}

Discrete KV methods:

𝑆

"

= 𝑓(𝑡

"

)

ChunkKV with a low sparsity

…… …… …… …… …… …… …… …… …… …… …… …… …… …… …… …… …… ……

…… …… …… …… …… ……

After fruit consumption, they

…… …… …… …… ……

porphyreolophus primarily consumes fruits

…… …… …… …… ……, ……

similar

turacos, the purple-crested turaco have faster minimum transit times when

consuming smaller seed diets than larger seed diets,

…… …… …… ……

…… …… …… …… …… …… …… …… …… …… …… …… …… …… …… …… …… ……

…… …… …… …… …… …… ……

Discrete KV methods with a high sparsity

…… ……

purple-crested turaco

…… …… …… …… …… …… …… …… …… ……

…… …… …… …… ……

Purple-crested turacos

…… …… …… …… …… ……

eat

…… …… ……

turacos,

…… …… …… …… …… …… …… …… …… ……

Turaco

…… …… …… …… …… …… …… …… …… …… …… …… …… …… ……

…… …… …… …… …… …… …… …… …… …… …… …… ……

coexist with various

other animals, including those that might enjoy

strawberries

…… …… …… …… …… …… …… …… …… …… …… …….

ChunkKV with a high sparsity

…… …… …… …… …… …… …… …… …… …… …… …… …… …… …… …… …… ……

…… …… …… …… …… ……

After fruit consumption, they

…… …… …… …… ……

…… …… …… …… …… …… …… …… …… …… …… …… …… …… …… …… …… ……

porphyreolophus primarily consumes fruits

…… …… …… …… ……,

…… …… …… …… …… …… …… …… …… …… …… …… …… …… …… …… …… ……

…… …… …… …… …… …… …… ………… …… …… ……

Discrete KV methods with a low sparsity

…… ……

purple-crested turaco

…… …… …… …… …… …… …… …… …… ……

…… …… …… …… ……

Purple-crested turacos

…… …… …… ……

Bird

…… ……

eat

…… …… ……

turacos,

…… …… …… ……

pulp

…… …… …… …… …… ……

Turaco

…… …… …… …… …… …… …… …… …… …… …… …… …… …… ……

…… …… …… …… ……

water

…… …… …… …… …… …… ……

leaves

…… …… …… …… …… …… …… ……

nuts

…… …… …… …… ……

coexist

with various other animals, including those that might enjoy

strawberries

…… …… …… …… …… …… …… …… …… …… …… …….

Figure 1: Illustration of the impact of the token discrete method and the chunk method on semantic preservation. The discrete method preserves words related to the question but often omits the subject. In contrast, the chunk method retains the subject of the words, maintaining more accurate semantic information. For the equation: S is the score function, and c is a chunk of tokens.

To address the substantial GPU memory consumption caused by KV caching, recent studies consider compressing the KV cache by pruning non-important discrete parts from the prompt tokens [11, 12, 1521]. H2O [11] and SnapKV [15] have shown that retaining less than 50% of the discrete KV cache can significantly reduce GPU memory usage with minimal impact on performance. However, we identify that the previous KV cache compression methods [11, 17] measure token importance isolatedly, neglecting the dependency between different tokens on the characteristics of the real-world language. For example, as shown in Figure 1, focusing on the importance of the token level can excessively focus on words about subjects 'turaco' in the question while omitting crucial information about objects (foods) in the documents, resulting in the loss of essential semantic information. This motivates us to rethink the following question:

How to avoid isolated token importance measurement and preserve the semantic information in KV cache?

Method

KV Cache

Compression

Dynamic

Policy

Layer-Wise

Policy

Semantic

Information

Efficient

Index Reuse

StreamingLLM [8]

✓

H2O [11]

✓

✓

SnapKV [15]

✓

✓

PyramidInfer [19]

✓

✓

✓

PyramidKV [17]

✓

✓

✓

ChunkKV(Ours)

✓

✓

✓

✓

✓

In light of this, we observe that complete semantic information usually appears in a continuous sequence [4, 22-24]. Thus, we introduce a straightforward yet effective ChunkKV, grouping the tokens in a chunk as a basic compressing unit, which should be preserved or discarded as a whole. Thus, it retains the most informative semantic chunks from the original KV cache. As shown in Figure 1, preserving a chunk helps catch the subject, predicate, and object. In Table 13, we quantify the minimal loss and higher recovery of the attention score achieved by the chunk-based approach at the inference stage. Furthermore, we investigate that the preserved KV cache indices by ChunkKV exhibit a higher similarity compared to previous methods. Consequently, we develop a technique called layer-wise index reuse, which reduces the additional computational time introduced by the KV cache compression method. As outlined in Table 1, recent highly relevant KV cache compression methods lack the ability to retain semantic information and efficiently reuse indices .

To evaluate ChunkKV's performance, we conduct comprehensive experiments across multiple cutting-edge long-context benchmarks: long-context tasks including LongBench [25] and Needle-

In-A-HayStack (NIAH) [26], in-context learning tasks such as GSM8K [27] and JailbreakV [28]. And also different models including DeepSeek-R1-Distill-Llama-8B [29],LLaMA-3-8B-Instruct [30], Mistral-7B-Instruct [31], and Qwen2-7B-Instruct [32]. Our experimental results demonstrate that ChunkKV surpasses existing KV cache compression methods in both efficiency and accuracy, primarily due to its ability to preserve essential information through selective chunk retention. These findings establish ChunkKV as a simple yet effective approach to KV cache compression.

We summarize our key contributions as follows:

We identify the phenomenon in which discrete KV cache compression methods inadvertently prune the necessary semantic information.

We propose ChunkKV, a simple KV cache compression method that uses the fragmentation method that retains semantic information, and design the layer-wise index reuse technique to reduce the additional computational time.

We evaluate ChunkKV on cutting-edge long-context benchmarks including LongBench and NeedleIn-A-HayStack, as well as the GSM8K, many-shot GSM8K and JailbreakV in-context learning benchmark, and multi-step reasoning (O1 and R1) LLMs, achieving state-of-the-art performance.

2 Related Work

KV Cache Compression. KV cache compression technology has developed rapidly in the era of LLM, with methods mainly focused on evicting unimportant tokens. The compression process occurs before the attention blocks, optimizing both the prefilling time and GPU memory. Xiao et al. [8] and Han et al. [33] propose that initial and recent tokens consistently have high attention scores between different layers and attention heads. As a result, retaining these tokens in the KV cache is more likely to preserve important information. Furthermore, FastGen [16] evicts tokens based on observed patterns. H2O [11] and SnapKV [15] employ dynamic KV cache compression methods, evaluating the importance of tokens based on attention scores and then evicting the less important ones. As inference scenarios become increasingly complex, dynamic KV cache compression methods demonstrate powerful performance. Recently, Yang et al. [19] and Cai et al. [17] have closely examined the distributions of attention scores during the pre-filling stage of the Retrieval-Augmented Generation (RAG) task, discovering a pyramidal KV cache compression pattern in different transformer layers. In contrast, FlowKV [34] employs a novel multi-turn isolation mechanism that preserves the accumulated compressed KV cache from past turns and only compresses the KV pairs from the most recent turn, effectively mitigating the re-compression of older context and the problem of catastrophic forgetting.

Although these KV cache compression methods have explored efficient GPU memory management while maintaining original performance, our study focuses more on the semantic information of the prompt. We find that chunks of the original KV cache are more important than discrete tokens.

3 ChunkKV

3.1 Preliminary Study of KV Cache Compression

With the increasing long-context capabilities of LLMs, the KV cache has become crucial for improving the inference efficiency. However, it can consume significant GPU memory when handling long input contexts. The GPU memory cost of the KV cache for the decoding stage can be calculated as follows:

where B is the batch size, S is the sequence length of prompt and decoded length, L is the number of layers, N is the number of attention heads, D is the dimension of each attention head, and the first 2 accounts for the KV matrices, while the last 2 accounts for the precision when using float16. Table E shows the configuration parameters for LLaMA-3-8B-Instruct [30]. With a batch size B = 1 and a sequence length of prompt S = 2048 , the GPU memory cost of the KV cache is nearly 1 GB. If the batch size exceeds 24, the GPU memory cost of the KV cache will exceed the capacity of an RTX 4090 GPU. To address this issue, KV cache compression methods have been proposed, with the aim of retaining only a minimal amount of KV cache while preserving as much information as possible. For more details on the LLM configuration parameters, refer to Appendix E.

3.2 Chunk Based KV Compression

To address the limitations of existing KV cache compression methods, we propose ChunkKV, a novel KV cache compression method that retains the most informative semantic chunks. The key idea behind ChunkKV is to group tokens in the KV cache into chunks that preserve more semantic information, such as a chunk containing a subject, verb, and object. As illustrated in Figure 1, ChunkKV preserves the chunks of the KV cache that contain more semantic information. First, we define a chunk as a group of tokens that contain related semantic information. By retaining the most informative chunks from the original KV cache, ChunkKV can effectively reduce the memory usage of the KV cache while preserving essential information. For more information on ChunkKV, please refer to Section A.

Algorithm 1 ChunkKV

Input: Q ∈ R T q × d , K ∈ R T k × d , V ∈ R T v × d , observe window size w , chunk size c , compressed KV cache max length L max

Output:

Compressed KV cache K ′ , V ′

Observe Window Calculation:

A

T

w

:

T

-

K

C

T

q

k

Q

⌈

⌉

T

c Chunk Attention Score Calculation :

for

i

= 1

do

A

ic

C

to

j

=(

i

1)

c

+1

-

i

A

∑

end for

Top-K Chunk Selection :

k

⌊

⌋

L

c Top_K_Indices ← indices of Topk chunks based on A i Compression :

K ′ , V ′ ← index_select ( K , V , Top_K_Indices )

Concatenation

K

concat

(

′

V

concat

(

′

K ′ , V ′

′

0:

:

K

V

′

0:

L

L

w

-

-

w

The algorithm 1 shows the pseudocode for ChunkKV. First, following the approach of H2O [11] and SnapKV [15], we set the observe window by computing the attention scores A ← Q T q -w : T q K T , where Q T q -w : T q is the observe window, K is the Key matrix and the window size w is usually set to { 4 , 8 , 16 , 32 } . Next, the number of chunks C is calculated as C = ⌈ T k c ⌉ , where T k is the length of the Key matrix and c is the chunk size. The attention scores for each chunk are then computed as A i = ∑ ic j =( i -1) c +1 A : ,j for i = 1 , 2 , . . . , C . We use the topk algorithm as the sampling policy for ChunkKV. The topk chunks are selected based on their attention scores, where k = ⌊ L max c ⌋ , and L max is the maximum length of the compressed KV cache. The size of the last chunk is set to min ( c, L max -( k -1) × c ) . The indices of the topk chunks preserve the original sequence order. In the compression step, only the key and value matrices corresponding to the selected indices are retained, resulting in the compressed KV cache. Finally, the observe window of the original KV cache will be concatenated to the compressed KV cache by replacing the last w tokens to keep important information. The compressed KV cache is then used for subsequent attention computations. For implementation, we add vectorized operations, memory optimizations, etc., to optimize the code. Refer to Appendix A.5 for more details.

3.3 Layer-Wise Index Reuse

Furthermore, we investigated the KV cache indices preserved by ChunkKV and found that they exhibit higher similarity compared to previous methods. Figure 2 shows the layer-wise similarity heatmaps of SnapKV and ChunkKV. Each cell represents the similarity between the

:

,j

q

,

,

K

T

V

k

T

-

v

-

w

:

w

:

k

T

T

v

)

)

▷

Attention scores for the observe window

▷

Calculate the number of chunks

▷ Sum of attention scores for each chunk

0

4

8

12

16

20

24

28

Layer

0

4

8

12

16

20

24

28

Layer

SnapKV

0

4

8

12

16

20

24

28

Layer

0

4

8

12

16

20

24

28

ChunkKV

0.2

0.4

0.6

0.8

1.0

preserved KV cache indices of two layers, with deeper colors indicating higher similarity. The results demonstrate that the KV cache indices preserved by ChunkKV are more similar to those in neighboring layers.

As shown in Table 2, ChunkKV consistently achieves a higher average Jaccard similarity between adjacent layers compared to SnapKV in different model architectures, indicating that the retained token index in ChunkKV is more similar to each other. For a more detailed visualization, please refer to Appendix B.1.3.

Based on the above findings of the KV cache index, we propose a training-free layer-wise index reuse method to further reduce the additional cost of KV cache compression, which reuses compressed token indices across multiple layers. We evaluate the efficiency and effectiveness of the layer-wise index reuse method in Section 4.3. Reuse of the layer-

Algorithm 2 Layer-wise Index Reuse for ChunkKV

Input: Number of layers in LLMs N layers, number of reuse layers N reuse

Initialize:

Dictionary to store indices

I

reuse

=

{}

for

l

= 0

to (

N

layers

-

1

)

do

if

l

N

reuse

== 0

then

K

′

l

,

V

′

l

,

I

l

ChunkKV

(

K

l

,

V

l

)

I

reuse

[

l

]

←I

l

else

I

l

←I

reuse

[

⌊

l

N

reuse

⌋

×

N

reuse

]

end if

K

′

l

index_select

(

K

l

,

I

l

)

V

′

l

index_select

(

V

l

,

I

l

)

end for

wise index reduces the KV cache compression time by 20% compared to the FullKV baseline, with a performance drop of only 0.5%.

This layer-wise index reuse method is formally described in Algorithm 2. The ChunkKV compression process returns the compressed KV cache and their respective token indices, denoted as I l . For layer-wise index reuse, we define a grouping of layers such that all N reuse layers share the same token indices for ChunkKV. Specifically, for a group of layers { l, l +1 , . . . , l + N reuse -1 } , we perform ChunkKV on the first layer l to obtain the token indices I l and reuse I l for the subsequent layers l +1 , l +2 , . . . , l + N reuse -1 . The notation K l [ I l ] and V l [ I l ] indicates the selection of key and value caches based on the indices in I l . The efficiency analysis for layer-wise index reuse is provided in Appendix B.1.1.

Theoretical Understanding. We provide a theoretical understanding from the in-context

learning (ICL) [24] to interpret why maintaining the KV cache according to a continuous sequence in ChunkKV is better than according to sparse tokens. Informally speaking, the continuously chunklevel KV cache preserves the whole examples (semantic information) in ICL, thus reducing the requirement on distinguishability, i.e., lower bound of KL divergence between the example and the question (Equation 4 in condition 2). The complete analysis is provided in Appendix C.

4 Experiment Results

In this section, we conduct experiments to evaluate the effectiveness of ChunkKV on KV cache compression in two benchmark fields, with a chunk size set to 10 even for various model architectures. The first is the In-Context Learning benchmark, for which we select GSM8K [27] and Jailbreakv [28, 35, 36] to evaluate the performance of ChunkKV, furthermore, we also include multistep reasoning LLM DeepSeek-R1-Distill-Llama-8B [29] to evaluate the performance of ChunkKV. The In-Context Learning scenario is a crucial capability for LLMs and has been adapted in many powerful technologies such as Chain-of-Thought [37-40]. The second is the Long-Context benchmark, which includes LongBench [25] and Needle-In-A-HayStack (NIAH) [26], both widely used for assessing KV cache compression methods. All experiments were carried out three times, using the mean score to ensure robustness.

4.1 In-Context Learning

The In-Context Learning (ICL) ability significantly enhances the impact of prompts on LLMs. For example, the Chain-of-Thought approach [37] increases the accuracy of the GSM8K of the PaLM model [41] from 18% to 57% without additional training. In this section, we evaluate the performance of ChunkKV on the GSM8K, Many-Shot GSM8K [42], and JailbreakV [28] benchmarks.

GSM8K. In the in-context learning scenario, we evaluated multiple KV cache compression methods for GSM8K [27], which contains more than 1,000 arithmetic questions on LLaMA-3-8B-Instruct, LLaMA-3.1-8BInstruct [30], Qwen2-7B-Instruct [32] and DeepSeek-R1-Distill-Llama-8B [29]. Following Agarwal et al. [42], we consider manyshot GSM8K as a long-context reasoning scenario, which is a more challenging task than long-context retrieval benchmark Long-

Bench [25]. The CoT prompt settings for this experiment are the same as those used by Wei et al. [37], for many-shot GSM8K we set the number of shots to 50, where the prompt length is more than 4k tokens. For more details on the prompt settings, please refer to the APPENDIX G.

Table 3 presents the performance comparison. The results show that ChunkKV outperforms other KV cache compression methods on different models and compression ratios. Table 4 presents the performance comparison of many-shot GSM8K, and also ChunkKV outperforms other KV cache compression methods. The consistent superior performance of ChunkKV in both models underscores its effectiveness in maintaining crucial contextual information for complex arithmetic reasoning tasks by chunk level KV cache rather than the discrete token level.

Jailbreak. In this section, we evaluate the performance of ChunkKV on the JailbreakV benchmark [28], which is a safety Jailbreak benchmark for language models. The prompt settings are the same as those used by Luo et al. [28].

Table 5 presents the performance comparison. The results demonstrate that ChunkKV out-

performs other KV cache compression methods on different compression ratios. This shows that for safety benchmark, the chunk level KV cache is more effective than other discrete token level compression methods.

(a) ChunkKV, accuracy 73.8%

(c) SnapKV, accuracy 58.9%

(d) StreamingLLM, accuracy 23.7%

4.2 Long-Context Benchmark

LongBench and NIAH are two widely used benchmarks for KV cache compression methods. Both benchmarks have a context length that exceeds 10 K . NIAH requires retrieval capability, while LongBench is a meticulously designed benchmark suite that tests the capabilities of language models in handling extended documents and complex information sequences. For more details on LongBench, please refer to the APPENDIX F.

LongBench. We use LongBench [25] to assess the performance of ChunkKV on tasks involving longcontext inputs. We evaluated multiple KV cache eviction methods using the LongBench benchmark with LLaMA-3-8B-Instruct [30], Mistral-7B-Instruct-v0.3 [31], and Qwen2-7B-Instruct [32], with a KV cache compression ratio of 10% . LongBench-ZH provides the Chinese subtask, and Qwen2-7BInstruct also supports Chinese, so we tested Qwen2-7B-Instruct with different KV cache compression methods on the Chinese subtasks.

Tables 6 present the performance gap (in percentage) between each method and the FullKV baseline, where negative values indicate performance degradation compared to FullKV. The table is evaluated in both the LongBench English and Chinese subtasks, where ChunkKV outperforms other compression methods overall. This suggests that ChunkKV's approach of retaining semantic chunks is more effective in preserving important information compared to other discrete token-based compression methods. For the 70B model and Chinese subtask results, please refer to Appendices B.2 and B.5.

Needle-In-A-HayStack. We use NIAH [26] to evaluate the long-context retrieval capability of LLMs. NIAH assesses how well LLM

extracts hidden tricked information from extensive documents, and following LLM-as-a-Judge [43] we apply GPT-4o-mini [44] to assess the accuracy of the retrieved information. We evaluated multiple KV cache eviction methods using NIAH with LLaMA-3-8B-Instruct and Mistral-7B-Instruct-v0.2, setting benchmark context lengths to 8k and 32k tokens.

Table 7 provides statistical results for different compression methods. These findings clearly indicate the effectiveness of ChunkKV in managing varying token lengths and depth percentages, making it a robust choice for KV cache management in LLMs. Figure 3 presents the NIAH benchmark results for LLaMA-3-8BInstruct. The vertical axis represents the depth percentage, while the horizontal axis represents the token length, with shorter lengths on

the left and longer lengths on the right. A cell highlighted in green indicates that the method can retrieve the needle at that length and depth percentage. The detailed visualization of the NIAH benchmark can be found in the Appendix B.3. The visualization results demonstrate that ChunkKV outperforms other KV cache compression methods.

4.3 Index Reuse

This section will evaluate the performance of the layer-wise index reuse approach with ChunkKV from the two aspects of efficiency and performance.

Measuring Efficiency. We evaluated the latency and throughput of ChunkKV compared to FullKV using LLaMA3-8B-Instruct on an A40 GPU. All experiments were conducted with reuse layer is 2, batch size set to 1 and inference was performed using Flash Attention 2, each experiment was repeated 10 times and the average latency and throughput were reported.

The results in Table 8 show that the layer-wise index reuse strategy (ChunkKV_reuse) further boosts performance, achieving up to a 20.7% reduction in latency, and throughput improvements are particularly notable for longer input sequences, with ChunkKV_reuse delivering up to a 26.5% improvement over FullKV. For more detailed results, please refer to Appendix B.8.

Measuring Task Performance.

We evaluate the effectiveness of our proposed layer-wise index reuse technique on LongBench [25] and GSM8K [27]

benchmarks. For these experiments, we use the same configuration as our main LongBench experiments in Section 4.2, with index reuse applied to consecutive layers (reuse layers = 2).

As shown in Table 9, layer-wise index reuse maintains the performance of the models while reducing computational requirements. In LongBench, performance degradation is minimal (less than 0.6%) for both models, while GSM8K shows neutral or slightly positive effects. This validates that semantic chunks selected by ChunkKV remain consistently important across adjacent transformer layers, enabling efficient computation without sacrificing accuracy. An additional analysis of different reuse depths and their impact on throughput is provided in Appendix B.1.2.

Overall, these findings on efficiency and performance suggest that layer-wise index reuse can be an effective technique for optimizing the efficiency-performance trade-off in KV cache compression, with the potential for model-specific tuning to maximize benefits.

4.4 Chunk Size

This section aims to investigate the impact of chunk size on the performance of ChunkKV. Different chunk sizes will lead to varying degrees of compression on the semantic information of the data. We used the same experiment setting as in LongBench and NIAH (Section 4.2). The chunk size is set from the range { 3 , 5 , 10 , 20 , 30 } under the compression rate 10% for LongBench and the 128 KV cache size for NIAH. For more experiments with different compression ratios, see Appendix B.4.

Table 9: Reusing Indexing Performance Comparison on LongBench and GSM8K. △ indicates the performance degradation of index reuse compared to the baseline.

As shown in Table 10, the performance remains relatively stable when the chunk size is between 5 and 20, with the best results consistently achieved at chunk size 10. When the chunk size is too small (e.g., 3), the context is fragmented, leading to a slight drop in performance. Conversely, when the chunk size is too large (for example, 30), the semantic granularity becomes too coarse, and important fine-grained information may be lost, also resulting in performance degradation. This trend is consistent across both LongBench and NIAH benchmarks, as well as across different model architectures, indicating that the optimal chunk size is not highly sensitive to the specific task or model. We also provide a line graph in Appendix B.4 to visually illustrate this trend. Based on these findings, we recommend using a chunk size of 10 as a robust default for most applications. For users with specific requirements, chunk size can be further tuned, but our results suggest that moderate values (5-20) generally offer a good trade-off between semantic preservation and compression efficiency.

4.5 Comparing with KV Quantization

For comprehensively evaluate the effectiveness of ChunkKV, we performed experiments comparing ChunkKV with the KIVI quantization methods [45]. Although both approaches aim to optimize LLM inference, they operate on fundamentally different principles: Quantization reduces KV matrix precision, whereas our eviction method reduces KV matrix size. For more detail, see Appendix B.6.

From an implementation perspective, quantization methods require the full KV cache during prefilling to produce quantized representations, which are then used during decoding. In contrast, ChunkKV employs token removal prior to prefilling, enabling operation with a compressed cache throughout the entire inference process. This distinction creates different efficiency profiles, and each method offers unique advantages.

Due to the fact that KIVI requires an old Python version, the ChunkKV results are not aligned with Table 8. The efficiency results in Table 11 reveal ChunkKV's significant advantages in latency-critical metrics. Although both methods substantially reduce cache size (ChunkKV to 10% and KIVI-2bits to 15.63%), ChunkKV delivers superior metrics for Time to First Token (TTFT) and Token Processing Time (TPOT). Most remarkably, ChunkKV achieves a 164.66s total generation time compared to KIVI's 226.52s at 2-bit quantization, representing a 27.3% improvement in overall inference speed.

4.6 Analysis of Hybrid Compression: Chunk-level vs. Token-level at Different Layer Depths

A critical question raised during the review process was whether the benefits of chunk-level compression diminish in deeper Transformer layers, where semantic information becomes more abstract and diffused. To investigate this, we designed a hybrid compression model that applies different strategies at varying network depths.

Experimental Setup We created a hybrid version of the LLaMA-3-8B-Instruct model. For the first 16 layers (bottom half), we applied our chunk-based ChunkKV. For the final 16 layers (top half), we applied SnapKV, a state-of-the-art token-level compression method. We then compared this hybrid model's performance on the diverse LongBench benchmark against pure ChunkKV and pure SnapKV at identical compression ratios.

Analysis and Insights The results in Table 12 provide two key insights. First, the pure ChunkKV model achieves the highest overall average score. This empirically validates our core hypothesis: preserving local semantic integrity via chunking is a robust and effective strategy across all layers of the model, even where information is highly processed.

Second, the hybrid model reveals a fascinating, task-dependent performance trade-off.

For local information retrieval tasks (e.g., Single- and Multi-Document QA), pure ChunkKV is significantly superior. These tasks often require retrieving precise, intact text fragments. ChunkKV's ability to preserve complete linguistic units prevents critical information from being fragmented, which is essential in these scenarios.

For global understanding tasks (e.g., Summarization and Few-shot Learning), the hybrid model performs best. In these tasks, synthesizing information from across the entire context is key. In the deeper layers, where abstract representations are formed, the token-level SnapKV method may be more adept at retaining a broader, more diffuse set of globally important signals.

This analysis does not undermine our approach but rather enriches it, suggesting that the future of KV cache compression may lie in adaptive, task-aware strategies. However, for a general-purpose and robust solution, pure ChunkKV proves to be the most effective.

5 Conclusion

In this paper, we first indicate that current KV cache methods lack the semantic information of the data, and then we proposed a novel KV cache compression method that preserves semantic information by retaining more informative chunks. Through extensive experiments across multiple state-of-the-art LLMs (including DeepSeek-R1, LLaMA-3, Qwen2, and Mistral) and diverse benchmarks (GSM8K, LongBench, NIAH, and JailbreakV), we demonstrate that ChunkKV consistently outperforms existing methods while using only a fraction of the memory. Our comprehensive analysis shows that ChunkKV's chunk-based approach maintains crucial contextual information, leading to superior performance in complex reasoning tasks, long-context understanding, and safety evaluations. The method's effectiveness is particularly evident in challenging scenarios like many-shot GSM8K and multi-document QA tasks, where semantic coherence is crucial. Furthermore, our proposed layer-wise index reuse technique provides significant computational efficiency gains with minimal performance impact, achieving up to 20.7% latency reduction and 26.5% throughput improvement. These findings, supported by detailed quantitative analysis and ablation studies, establish ChunkKV as a significant advancement in KV cache compression technology, offering an effective solution for deploying LLMs in resource-constrained environments while maintaining high-quality outputs.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:18771901, 2020.

Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. ArXiv preprint , abs/2205.05131, 2022. URL https://arxiv.org/abs/2205. 05131 .

Zhenheng Tang, Xiang Liu, Qian Wang, Peijie Dong, Bingsheng He, Xiaowen Chu, and Bo Li. The lottery LLM hypothesis, rethinking what abilities should LLM compression preserve? In The Fourth Blogpost Track at ICLR 2025 , 2025.

Qian Wang, Zhenheng Tang, Zichen Jiang, Nuo Chen, Tianyu Wang, and Bingsheng He. Agenttaxo: Dissecting and benchmarking token distribution of llm multi-agent systems. In ICLR 2025 Workshop on Foundation Models in the Wild , 2025.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint , abs/2307.09288, 2023. URL https://arxiv.org/abs/2307.09288 .

Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR) , 2024.

Sam Ade Jacobs et al. DeepSpeed Ulysses: System optimizations for enabling training of extreme long sequence Transformer models. ArXiv preprint , abs/2309.14509, 2023. URL https://arxiv.org/abs/2309.14509 .

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=NG7sS51zVF .

Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. ArXiv preprint , abs/2402.08268, 2024. URL https: //arxiv.org/abs/2402.08268 .

Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. ArXiv preprint , abs/2403.04652, 2024. URL https://arxiv.org/abs/2403.04652 .

Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems , 36:34661-34710, 2023.

Yuanbing Zhu, Zhenheng Tang, Xiang Liu, Ang Li, Bo Li, Xiaowen Chu, and Bo Han. OracleKV: Oracle guidance for question-independent KV cache compression. In ICML 2025 Workshop on Long-Context Foundation Models , 2025. URL https://openreview.net/ forum?id=KHM2YOGgX9 .

Qian Wang, Tianyu Wang, Zhenheng Tang, Qinbin Li, Nuo Chen, Jingsheng Liang, and Bingsheng He. Megaagent: A large-scale autonomous llm-based multi-agent system without predefined sops. In The 63rd Annual Meeting of the Association for Computational Linguistics , 2025.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jeanbaptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv preprint , abs/2403.05530, 2024. URL https://arxiv.org/abs/2403.05530 .

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. ArXiv preprint , abs/2404.14469, 2024. URL https://arxiv.org/abs/ 2404.14469 .

Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. ArXiv preprint , abs/2310.01801, 2023. URL https://arxiv.org/abs/2310.01801 .

Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069 , 2024.

Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Najibi. LazyLLM: Dynamic token pruning for efficient long context LLM inference. In Workshop on Efficient Systems for Foundation Models II @ ICML2024 , 2024. URL https: //openreview.net/forum?id=gGZD1dsJqZ .

Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. PyramidInfer: Pyramid KV cache compression for high-throughput LLM inference. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024 , pages 3258-3270, Bangkok, Thailand and virtual meeting, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.195. URL https://aclanthology.org/2024.findings-acl.195 .

Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems , 36, 2024.

Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. ArXiv preprint , abs/2406.10774, 2024. URL https://arxiv.org/abs/2406.10774 .

George A Miller. Information and memory. Scientific American , 195(2):42-47, 1956.

Lance A Ramshaw and Mitchell P Marcus. Text chunking using transformation-based learning. In Natural language processing using very large corpora , pages 157-176. Springer, 1999.

Hongchao Fang and Pengtao Xie. An end-to-end contrastive self-supervised learning framework for language understanding. Transactions of the Association for Computational Linguistics , 10:1324-1340, 2022. doi: 10.1162/tacl_a_00521. URL https://aclanthology.org/ 2022.tacl-1.76/ .

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.172. URL https://aclanthology.org/2024.acl-long.172 .

Gregory Kamradt. Needle In A Haystack - pressure testing LLMs. Github , 2023. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main .

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint , abs/2110.14168, 2021. URL https://arxiv. org/abs/2110.14168 .

Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks. In First Conference on Language Modeling , 2024. URL https://openreview. net/forum?id=GC4mXVfquq .

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025.

Meta. Introducing meta llama 3: The most capable openly available llm to date. https: //ai.meta.com/blog/meta-llama-3/ , 2024. Accessed: 2024-06-07.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825 .

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. ArXiv preprint , abs/2407.10671, 2024. URL https://arxiv.org/abs/2407.10671 .

Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. LM-infinite: Zero-shot extreme length generalization for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages 3991-4008, Mexico City, Mexico, 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024. naacl-long.222 .

Xiang Liu, Hong Chen, Xuming Hu, and Xiaowen Chu. FlowKV: Enhancing multi-turn conversational coherence in LLMs via isolated key-value cache management. In First Workshop on Multi-Turn Interactions in Large Language Models , 2025. URL https://openreview. net/forum?id=rZumU1owkr .

Zichen Tang, Zhenheng Tang, Gaoning Pan, Buhua Liu, Kunfeng Lai, Xiaowen Chu, and Bo Li. Ghost in the cloud: Your geo-distributed large language models training is easily manipulated. In ICML 2025 Workshop on Data in Generative Models - The Bad, the Ugly, and the Greats , 2025. URL https://openreview.net/forum?id=dpDdqgfcTM .

Fanjunduo Wei, Zhenheng Tang, Rongfei Zeng, Tongliang Liu, Chengqi Zhang, Xiaowen Chu, and Bo Han. JailbreakloRA: Your downloaded loRA from sharing platforms might be unsafe. In ICML 2025 Workshop on Data in Generative Models - The Bad, the Ugly, and the Greats , 2025. URL https://openreview.net/forum?id=RjaeiNswGh .

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022.

Kunfeng Lai, Zhenheng Tang, Xinglin Pan, Peijie Dong, Xiang Liu, Haolan Chen, Li Shen, Bo Li, and Xiaowen Chu. Mediator: Memory-efficient llm merging with less parameter conflicts and uncertainty based routing. arxiv preprint arXiv:2502.04411 , 2025.

Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. Active prompting with chain-of-thought for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1330-1350, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.73. URL https://aclanthology.org/2024.acl-long.73 .

Rui Pan, Shuo Xing, Shizhe Diao, Wenhe Sun, Xiang Liu, KaShun Shum, Jipeng Zhang, Renjie Pi, and Tong Zhang. Plum: Prompt learning using metaheuristics. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024 , pages 2177-2197, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.129. URL https://aclanthology.org/2024.findings-acl.129 .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ArXiv preprint , abs/2204.02311, 2022. URL https://arxiv.org/abs/2204.02311 .

Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018 , 2024.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36:4659546623, 2023.

OpenAI. Gpt-4o-mini: Advancing cost-efficient intelligence, 2023. Accessed: 2023-12-14.

Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750 , 2024.

Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118 , 2024.

Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057 , 2024.

Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales? In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Ingo Steinwart. How to compare different loss functions and their risks. Constructive Approximation , 26:225-287, 2007. URL https://api.semanticscholar.org/CorpusID: 16660598 .

Bernardo Ávila Pires and Csaba Szepesvári. Multiclass classification calibration functions. arXiv preprint arXiv:1609.06385 , 2016.

Kleijn and Van der Vaart. The bernstein-von-mises theorem under misspecification. Electronic Journal of Statistics , 6:354-381, 2012. URL https://api.semanticscholar.org/ CorpusID:85548207 .

Erik F. Tjong Kim Sang and Jorn Veenstra. Representing text chunks. In Henry S. Thompson and Alex Lascarides, editors, Ninth Conference of the European Chapter of the Association for Computational Linguistics , pages 173-179, Bergen, Norway, 1999. Association for Computational Linguistics. URL https://aclanthology.org/E99-1023 .

Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. In The Twelfth International Conference on Learning Representations , 2024.

Weizhi Fei, Xueyan Niu, Pingyi Zhou, Lu Hou, Bo Bai, Lei Deng, and Wei Han. Extending context window of large language models via semantic compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024 , pages 5169-5181, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.306. URL https://aclanthology.org/2024.findings-acl.306 .

Antonio Jimeno Yepes, Yao You, Jan Milczek, Sebastian Laverde, and Renyu Li. Financial report chunking for effective retrieval augmented generation, 2024. URL https://arxiv. org/abs/2402.05131 .

Brandon Smith and Anton Troynikov. Evaluating chunking strategies for retrieval. Technical report, Chroma, 2024. URL https://research.trychroma.com/evaluating-chunking .

Anthropic. Introducing contextual retrieval, 2024. URL https://www.anthropic.com/ news/contextual-retrieval .

Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning. ArXiv preprint , abs/2403.17919, 2024. URL https://arxiv.org/abs/2403.17919 .

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth

International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9 .

Zhenheng Tang, Xueze Kang, Yiming Yin, Xinglin Pan, Yuxin Wang, Xin He, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, Amelie Chi Zhou, Bo Li, Bingsheng He, and Xiaowen Chu. Fusionllm: A decentralized llm training system on geo-distributed gpus with adaptive compression. arxiv preprint arXiv:2410.12707 , 2024.

Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=Syx4wnEtvH .

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. ArXiv preprint , abs/2309.03883, 2023. URL https://arxiv.org/abs/2309.03883 .

Haoyi Wu and Kewei Tu. Layer-condensed kv cache for efficient inference of large language models, 2024. URL https://arxiv.org/abs/2405.10637 .

Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.05254 , 2024.

William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan Kelly. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405.12981 , 2024.

Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models. arXiv preprint arXiv:2405.14366 , 2024.

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. ∞ -bench: Extending long context evaluation beyond 100k tokens. ArXiv preprint , abs/2402.13718, 2024. URL https://arxiv. org/abs/2402.13718 .

Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: A zeroshot benchmark for long text understanding. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 79777989, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. findings-emnlp.536. URL https://aclanthology.org/2023.findings-emnlp.536 .

Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. ArXiv preprint , abs/2307.11088, 2023. URL https://arxiv.org/abs/2307.11088 .

Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. ArXiv preprint , abs/2305.16300, 2023. URL https:// arxiv.org/abs/2305.16300 .

Dacheng Li, Rulin Shao, et al. How long can open-source LLMs truly promise on context length?, 2023. URL https://lmsys.org/blog/2023-06-29-longchat .

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics , 12:157-173, 2024. doi: 10.1162/tacl_a_00638. URL https://aclanthology.org/2024.tacl-1.9 .

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? ArXiv preprint , abs/2404.06654, 2024. URL https://arxiv.org/abs/ 2404.06654 .

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https: //openreview.net/forum?id=qVyeW-grC2k .

Xiang Liu, Zhenheng Tang, Hong Chen, Peijie Dong, Zeyu Li, Xiuze Zhou, Bo Li, Xuming Hu, and Xiaowen Chu. Can llms maintain fundamental abilities under kv cache compression? arXiv preprint arXiv:2502.01941 , 2025.

Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. Longgenbench: Long-context generation benchmark. arXiv preprint arXiv:2410.04199 , 2024.

David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 5621-5634, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-emnlp.412. URL https://aclanthology.org/2022. findings-emnlp.412 .

Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 38293846, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ v1/2023.emnlp-main.232. URL https://aclanthology.org/2023.emnlp-main.232 .

Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. Recurrentgpt: Interactive generation of (arbitrarily) long text, 2023.

Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, and Li Guo. Recursively summarizing enables long-term dialogue memory in large language models. arXiv preprint arXiv:2308.15022 , 2023.

Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Compressing prompts for accelerated inference of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 13358-13376, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.825. URL https:// aclanthology.org/2023.emnlp-main.825 .

Huiqiang Jiang, Qianhui Wu, , Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1658-1677, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.acl-long.91 .

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning , pages 38087-38099. PMLR, 2023.

Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems , 6:196-209, 2024.

Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning , pages 31094-31116. PMLR, 2023.

Zeyu Li, Chuanfu Xiao, Yang Wang, Xiang Liu, Zhenheng Tang, Baotong Lu, Mao Yang, Xinyu Chen, and Xiaowen Chu. Antkv: Anchor token-aware sub-bit vector quantization for kv cache in large language models, 2025. URL https://arxiv.org/abs/2506.19505 .

Tomáš Koˇ ciský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics , 6:317-328, 2018. doi: 10.1162/tacl_a_00023. URL https://aclanthology.org/Q18-1023 .

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings

of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4599-4610, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.365. URL https:// aclanthology.org/2021.naacl-main.365 .

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2369-2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259 .

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics , pages 6609-6625, Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.580. URL https://aclanthology.org/2020.coling-main.580 .

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics , 10:539-554, 2022. doi: 10.1162/tacl_a_00475. URL https: //aclanthology.org/2022.tacl-1.31 .

Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. DuReader: a Chinese machine reading comprehension dataset from real-world applications. In Eunsol Choi, Minjoon Seo, Danqi Chen, Robin Jia, and Jonathan Berant, editors, Proceedings of the Workshop on Machine Reading for Question Answering , pages 37-46, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-2605. URL https://aclanthology.org/W18-2605 .

Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1419-1436, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. naacl-main.112. URL https://aclanthology.org/2021.naacl-main.112 .

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5905-5921, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.472. URL https://aclanthology.org/ 2021.naacl-main.472 .

Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 1074-1084, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1102. URL https: //aclanthology.org/P19-1102 .

Han Wu, Mingjie Zhan, Haochen Tan, Zhaohui Hou, Ding Liang, and Linqi Song. VCSUM: A versatile Chinese meeting summarization dataset. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023 , pages 6065-6079, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.377. URL https://aclanthology.org/2023. findings-acl.377 .

Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics , 2002. URL https://aclanthology.org/ C02-1150 .

Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors, Proceedings of the 2nd Workshop on New Frontiers in Summarization , pages 70-79, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-5409. URL https://aclanthology. org/D19-5409 .

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and MinYen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1601-1611, Vancouver, Canada, 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https://aclanthology. org/P17-1147 .

Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian J. McAuley. Longcoder: A longrange pre-trained language model for code completion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 12098-12107. PMLR, 2023. URL https://proceedings.mlr.press/v202/guo23j.html .

Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repositorylevel code auto-completion systems. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=pPjZIOuQuF .

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS Paper Checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately describe ChunkKV, its advantages in semantic preservation, and the experimental results supporting these claims. See Abstract and Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Section I.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provides a theoretical understanding of ChunkKV with assumptions and a proof sketch. See Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper details experimental setups, datasets, models, and evaluation metrics. See Section 4 and Appendix F, E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper states that code will be made available to ensure reproducibility. See Impact Statement. Datasets used are publicly available. See Appendix J.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental settings, model configurations, and dataset details are provided. See Section 4 and Appendix F, E, G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper is focused on the LLM inference, and with same model, prompt, and temperature, the results are consistent.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper mentions GPU types (e.g., A40) and inference setup for efficiency experiments. See Section 4.2 and Appendix B.8.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research uses publicly available datasets and models, and aims to improve LLM efficiency, aligning with ethical research practices. See Impact Statement.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper includes an Impact Statement at Section H.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper focuses on a compression technique rather than releasing new models or datasets with high misuse risk. The models used are standard, publicly available models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Datasets are cited and their licenses are mentioned in Appendix J. Models used are standard and widely known, with their original papers cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper introduces a new method (ChunkKV) but does not release new datasets or standalone models as new assets. The code for the method is planned for release.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The research does not involve crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The research does not involve human subjects, so IRB approval is not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLM only used for writing, editing, and formatting.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

A In-depth Analysis of ChunkKV vs. Discrete Token Methods

A.1 Quantitative Analysis

To rigorously evaluate the effectiveness of ChunkKV compared to discrete token-based methods, we conducted systematic experiments using a LLaMA-3-8B-Instruct model. We randomly selected 100 sequences from the each sub-category of LongBench dataset and analyzed two key metrics across different model layers: KV cache L1 loss and attention cosine similarity. For each sequence, we: 1. Computed the full KV cache and attention patterns without compression as ground truth. 2. Applied ChunkKV, SnapKV, and H2O compression methods with a fixed 10% compression ratio, and the parameters of the three methods are set the same as in Table 17. 3. Measured the differences between compressed and uncompressed versions.

0

5

10

15

20

25

30

Layer

0.6

0.7

0.8

0.9

1.0

L1 Loss

Layer-wise KV cache L1 Loss

0

5

10

15

20

25

30

Layer

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Cosine Similarity

Layer-wise Attention Cosine Similarity

ChunkKV

SnapKV

H2O

Results Analysis As shown in Figure 4, ChunkKV demonstrates superior performance across both metrics:

KV Cache L1 Loss: ChunkKV achieves consistently lower L1 loss compared to SnapKV and H2O, particularly in the early and middle layers (layers 5-25). This indicates better preservation of the original KV cache information through the semantic chunk-based approach.

Attention Cosine Similarity: ChunkKV exhibits higher similarity scores across most layers, with notably strong performance in layers 0-5 and 20-30. This suggests better preservation of attention relationships between tokens, which is crucial for maintaining semantic understanding.

To quantify these improvements, we calculated average metrics across all layers, as shown in Table 13. ChunkKV achieves both the lowest L1 loss and highest attention cosine similarity, outperforming both baseline methods.

Significance of Results While the improvements may appear modest in absolute terms (approximately 2% in L1 loss and 1.5% in cosine similarity), their practical significance is substantial. These metrics reflect the model's ability to maintain crucial semantic relationships and attention patterns, which are essential for complex reasoning tasks. The consistent improvements across different sequences demonstrate that preserving semantic chunks leads to better information retention than selecting individual tokens.

The enhanced performance is particularly evident in the middle layers of the model, which are typically responsible for higher-level semantic processing. This provides concrete evidence for why ChunkKV achieves superior performance on downstream tasks compared to discrete token-based methods.

A.2 Hypothetical Scenario

To provide a deeper understanding of ChunkKV's effectiveness compared to discrete token-based methods, we present a detailed analysis using a hypothetical scenario. This analysis aims to illustrate the fundamental differences between these approaches and explain why ChunkKV is more effective at preserving semantic information in long contexts.

Consider a comprehensive document that contains detailed information on various animals, including their habitats, diets, and behaviors. A user asks the question "What do pandas eat in the wild?"

Both ChunkKV and discrete token-based methods would use this question to calculate attention scores for the document. However, their approaches to selecting and retaining information differ significantly.

A.2.1 Discrete Token-based Method

A discrete token-based method might identify and retain individual tokens with high relevance scores, such as:

'pandas",'eat", 'bamboo", 'wild", 'diet", 'food"

Although these tokens are relevant, they lack context and coherence. The method might discard other essential tokens that provide crucial context or complete the information.

A.2.2 ChunkKV Method

In contrast, ChunkKV would identify and retain semantically meaningful chunks, such as:

'In the wild, pandas primarily eat bamboo shoots and leaves"

'Their diet consists of 99% bamboo, but they occasionally consume other vegetation"

'Wild pandas may also eat small rodents or birds when available"

By preserving these chunks, ChunkKV maintains not only the relevant keywords but also their contextual relationships and additional pertinent information.

A.3 Comparative Analysis

The advantages of ChunkKV become evident when we consider how these retained pieces of information would be used in subsequent processing:

Contextual Understanding : Discrete tokens require the model to reconstruct meaning from isolated words, which could lead to ambiguity. ChunkKV provides complete phrases or sentences, allowing for immediate and accurate comprehension.

Semantic Coherence : ChunkKV preserves the semantic relationships within a chunk, crucial to understanding nuances such as the difference between primary and occasional food sources for pandas.

Information Density : A single chunk can contain multiple relevant tokens in their proper context, potentially retaining more useful information within the same compressed cache size compared to discrete methods.

Reduced Ambiguity : Discrete methods might retain the token 'eat" from various sentences about different animals. ChunkKV ensures that 'eat" is preserved specifically in the context of pandas in the wild.

Temporal and Logical Flow : ChunkKV can maintain the sequence of ideas present in the original text, preserving any temporal or logical progression that may be crucial for understanding.

A.4 Implications for Model Performance

This analysis suggests several key implications for model performance:

Improved Accuracy : By retaining contextually rich information, ChunkKV enables more accurate responses to queries, especially those requiring nuanced understanding.

Enhanced Long-context Processing : Preservation of semantic chunks allows for better handling of long-range dependencies and complex reasoning tasks.

Reduced Computational Overhead : Although both methods compress the KV cache, ChunkKV's approach may reduce the need for extensive context reconstruction, potentially improving inference efficiency.

Versatility : The chunk-based approach is likely to be more effective across a wide range of tasks and domains as it preserves the natural structure of language.

This in-depth analysis demonstrates why ChunkKV is more effective in preserving semantic information in long contexts. By retaining coherent chunks of text, it provides language models with more contextually rich and semantically complete information, leading to improved performance in tasks that require deep understanding and accurate information retrieval from extensive documents.

A.5 Implementation Details

In our implementation of ChunkKV, we focus on maximizing computational efficiency to ensure practical deployment in real-world applications. In algorithm 3, we adopt several key optimization strategies to reduce both computational overhead and memory footprint during inference.

The optimized implementation features several key improvements:

Vectorized Operations. We leverage batch matrix operations for attention score calculation instead of explicit loops, significantly reducing computational overhead. By computing chunk scores through optimized tensor operations (lines 7-11), we achieve substantial acceleration compared to token-bytoken processing.

Memory Optimization. Our implementation employs a binary mask approach (lines 15-20) that avoids redundant memory allocations and reduces fragmentation. This single-pass token selection strategy consolidates both chunk preservation and recent window retention operations, leading to more efficient memory usage during inference.

Boundary Handling. We incorporate robust boundary checks throughout the algorithm (lines 10, 12, 18) to handle edge cases such as incomplete chunks or limited compressed length. This ensures stable performance across variable input lengths without requiring special case handling.

Single-pass Compression. Rather than performing multiple concatenation operations, our implementation builds a comprehensive selection mask that includes both semantically important chunks and recent context tokens. This approach (lines 21-25) minimizes memory copying operations and reduces computational overhead.

Our PyTorch implementation further extends these optimizations by leveraging GPU acceleration for all vector and matrix operations. The integration of CUDA kernels for critical operations like

Algorithm 3 ChunkKV Implementation

attention score calculation and top-k selection enables real-time performance even with long context windows.

B Additional Experiments

B.1 Layer-Wise Index Reuse

B.1.1 Efficiency Analysis

The layer-wise index reuse method significantly reduces the computational complexity of ChunkKV. Without index reuse, ChunkKV would be applied to all N layers layers, resulting in a total compression time of N layers · T compress, where T compress is the time taken to compress one layer. With index reuse, ChunkKV is only applied to N layers N reuse layers, reducing the total time to N layers N reuse · T compress + ( N layers -N layers N reuse ) · T select, where T select is the time taken to select indices, which is typically much smaller than T compress. This results in a theoretical speedup factor of:

Assuming T select is negligible compared to T compress, this simplifies to approximately N reuse . In practice, the actual speedup may vary depending on the specific implementation and hardware, but it can still lead to substantial time savings, especially for models with a large number of layers.

B.1.2 Index Reuse Performance

Figure 5 and Table 14 illustrates the performance of ChunkKV with varying index reuse layers on the GSM8K benchmark. Figure 6 and Table 15 illustrates the performance of ChunkKV with varying index reuse layers on the LongBench benchmark. The experiment reveals that math problems are more sensitive to index reuse layers compared to LongBench. Both LLaMA3-8B-Instruct and Qwen27B-Instruct exhibit significant performance degradation, with LLaMA3-8B-Instruct experiencing a steeper decline after two layers of index reuse than Qwen2-7B-Instruct. This suggests that the Qwen2-7B-Instruct model may be more robust to index reuse.

0

1

2

3

5

10

20

28

30

32

0

10

20

30

40

50

60

70

80

GSM8K Score

Index Reuse Performance on GSM8K

LLaMA-3-8B-Inst

Qwen2-7B-Inst

B.1.3 Layer-Wise Index Similarity

This section details the experiment of layer-wise index reuse similarity described in Section 3.3. The inference prompt is randomly selected from the LongBench benchmark, and the preserved indices for H2O, SnapKV, and ChunkKV are saved in the log file. For multi-head attention, only the indices of the first head are saved. PyramidKV, which has varying preserved index sizes across different layers, is not applicable for this experiment. Then we calculate the Jaccard similarity of the preserved indices of adjacent layers for different models. Table 16 shows the Jaccard similarity of the preserved indices of adjacent layers for different models.

Figures 7-9 (LLaMA-3-8B-Instruct), 10-12 (Mistral-7B-Instruct), and 13-15 (Qwen2-7B-Instruct) display the heatmaps of layer-wise indices similarity of the preserved KV cache indices by H2O, SnapKV and ChunkKV on different models. The pattern of the layer-wise indices similarity heatmap is consistent across different models, aligning with our findings in Section 3.3.

0 1 2 3

5

10

20

28

30

32

Number of Index Reuse Layers

36

38

40

42

44

46

48

LongBench Score

Index Reuse Performance on LongBench

LLaMA-3-8B-Inst

Qwen2-7B-Inst

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

1.00

0.16

0.38

0.40

0.44

0.36

0.26

0.20

0.23

0.17

0.13

0.21

0.23

0.21

0.24

0.41

0.07

0.09

0.17

0.08

0.15

0.21

0.08

0.21

0.19

0.18

0.13

0.21

0.13

0.17

0.24

0.25

0.16

1.00

0.16

0.10

0.13

0.18

0.20

0.21

0.14

0.10

0.14

0.11

0.19

0.09

0.09

0.18

0.08

0.17

0.12

0.13

0.19

0.16

0.10

0.15

0.21

0.16

0.07

0.08

0.10

0.10

0.12

0.13

0.38

0.16

1.00

0.42

0.40

0.37

0.28

0.22

0.23

0.17

0.13

0.19

0.22

0.18

0.19

0.37

0.08

0.09

0.15

0.12

0.16

0.23

0.08

0.25

0.21

0.22

0.14

0.20

0.11

0.17

0.24

0.28

0.40

0.10

0.42

1.00

0.52

0.39

0.27

0.19

0.24

0.19

0.15

0.22

0.20

0.24

0.25

0.35

0.09

0.07

0.18

0.11

0.15

0.22

0.08

0.22

0.15

0.20

0.19

0.25

0.14

0.19

0.27

0.24

0.44

0.13

0.40

0.52

1.00

0.37

0.31

0.21

0.25

0.18

0.16

0.22

0.21

0.21

0.22

0.40

0.07

0.07

0.16

0.10

0.15

0.21

0.08

0.23

0.18

0.21

0.17

0.23

0.14

0.19

0.27

0.26

0.36

0.18

0.37

0.39

0.37

1.00

0.30

0.24

0.23

0.18

0.15

0.22

0.24

0.19

0.22

0.35

0.10

0.13

0.20

0.11

0.18

0.22

0.09

0.24

0.23

0.21

0.14

0.18

0.14

0.15

0.23

0.23

0.26

0.20

0.28

0.27

0.31

0.30

1.00

0.25

0.27

0.17

0.17

0.22

0.26

0.16

0.17

0.29

0.09

0.16

0.17

0.18

0.20

0.21

0.10

0.26

0.27

0.27

0.14

0.18

0.14

0.12

0.22

0.19

0.20

0.21

0.22

0.19

0.21

0.24

0.25

1.00

0.24

0.16

0.22

0.16

0.24

0.15

0.14

0.24

0.10

0.18

0.15

0.15

0.21

0.23

0.12

0.20

0.19

0.21

0.10

0.13

0.12

0.14

0.18

0.16

0.23

0.14

0.23

0.24

0.25

0.23

0.27

0.24

1.00

0.37

0.28

0.40

0.36

0.35

0.33

0.29

0.14

0.19

0.23

0.20

0.22

0.27

0.08

0.28

0.33

0.30

0.19

0.23

0.20

0.15

0.29

0.21

0.17

0.10

0.17

0.19

0.18

0.18

0.17

0.16

0.37

1.00

0.23

0.49

0.33

0.40

0.37

0.20

0.12

0.18

0.25

0.17

0.19

0.22

0.07

0.24

0.31

0.24

0.19

0.24

0.22

0.15

0.25

0.17

0.13

0.14

0.13

0.15

0.16

0.15

0.17

0.22

0.28

0.23

1.00

0.23

0.27

0.26

0.23

0.16

0.16

0.18

0.16

0.18

0.20

0.20

0.11

0.17

0.21

0.18

0.17

0.17

0.16

0.13

0.18

0.13

0.21

0.11

0.19

0.22

0.22

0.22

0.22

0.16

0.40

0.49

0.23

1.00

0.35

0.41

0.37

0.22

0.10

0.18

0.21

0.15

0.18

0.20

0.06

0.30

0.31

0.28

0.17

0.24

0.18

0.16

0.27

0.18

0.23

0.19

0.22

0.20

0.21

0.24

0.26

0.24

0.36

0.33

0.27

0.35

1.00

0.29

0.29

0.27

0.12

0.26

0.22

0.19

0.23

0.26

0.09

0.28

0.36

0.30

0.14

0.21

0.19

0.16

0.24

0.22

0.21

0.09

0.18

0.24

0.21

0.19

0.16

0.15

0.35

0.40

0.26

0.41

0.29

1.00

0.49

0.22

0.14

0.17

0.27

0.16

0.20

0.26

0.08

0.22

0.26

0.21

0.21

0.24

0.20

0.16

0.26

0.17

0.24

0.09

0.19

0.25

0.22

0.22

0.17

0.14

0.33

0.37

0.23

0.37

0.29

0.49

1.00

0.26

0.13

0.19

0.36

0.16

0.21

0.29

0.10

0.20

0.26

0.21

0.23

0.25

0.27

0.15

0.28

0.19

0.41

0.18

0.37

0.35

0.40

0.35

0.29

0.24

0.29

0.20

0.16

0.22

0.27

0.22

0.26

1.00

0.09

0.11

0.20

0.14

0.21

0.31

0.10

0.25

0.27

0.25

0.16

0.22

0.18

0.17

0.32

0.28

0.07

0.08

0.08

0.09

0.07

0.10

0.09

0.10

0.14

0.12

0.16

0.10

0.12

0.14

0.13

0.09

1.00

0.17

0.15

0.25

0.24

0.15

0.14

0.09

0.14

0.13

0.16

0.10

0.16

0.10

0.11

0.07

0.09

0.17

0.09

0.07

0.07

0.13

0.16

0.18

0.19

0.18

0.18

0.18

0.26

0.17

0.19

0.11

0.17

1.00

0.24

0.19

0.26

0.18

0.10

0.17

0.25

0.16

0.10

0.11

0.17

0.10

0.13

0.10

0.17

0.12

0.15

0.18

0.16

0.20

0.17

0.15

0.23

0.25

0.16

0.21

0.22

0.27

0.36

0.20

0.15

0.24

1.00

0.18

0.24

0.31

0.12

0.18

0.25

0.17

0.19

0.19

0.27

0.14

0.22

0.16

0.08

0.13

0.12

0.11

0.10

0.11

0.18

0.15

0.20

0.17

0.18

0.15

0.19

0.16

0.16

0.14

0.25

0.19

0.18

1.00

0.27

0.23

0.14

0.16

0.26

0.22

0.18

0.14

0.22

0.12

0.19

0.11

0.15

0.19

0.16

0.15

0.15

0.18

0.20

0.21

0.22

0.19

0.20

0.18

0.23

0.20

0.21

0.21

0.24

0.26

0.24

0.27

1.00

0.28

0.17

0.20

0.29

0.22

0.19

0.18

0.21

0.14

0.22

0.16

0.21

0.16

0.23

0.22

0.21

0.22

0.21

0.23

0.27

0.22

0.20

0.20

0.26

0.26

0.29

0.31

0.15

0.18

0.31

0.23

0.28

1.00

0.12

0.20

0.30

0.27

0.21

0.22

0.25

0.16

0.28

0.21

0.08

0.10

0.08

0.08

0.08

0.09

0.10

0.12

0.08

0.07

0.11

0.06

0.09

0.08

0.10

0.10

0.14

0.10

0.12

0.14

0.17

0.12

1.00

0.08

0.09

0.08

0.08

0.05

0.10

0.08

0.09

0.06

0.21

0.15

0.25

0.22

0.23

0.24

0.26

0.20

0.28

0.24

0.17

0.30

0.28

0.22

0.20

0.25

0.09

0.17

0.18

0.16

0.20

0.20

0.08

1.00

0.30

0.37

0.13

0.20

0.15

0.15

0.25

0.22

0.19

0.21

0.21

0.15

0.18

0.23

0.27

0.19

0.33

0.31

0.21

0.31

0.36

0.26

0.26

0.27

0.14

0.25

0.25

0.26

0.29

0.30

0.09

0.30

1.00

0.34

0.15

0.26

0.24

0.20

0.27

0.21

0.18

0.16

0.22

0.20

0.21

0.21

0.27

0.21

0.30

0.24

0.18

0.28

0.30

0.21

0.21

0.25

0.13

0.16

0.17

0.22

0.22

0.27

0.08

0.37

0.34

1.00

0.16

0.22

0.20

0.16

0.27

0.23

0.13

0.07

0.14

0.19

0.17

0.14

0.14

0.10

0.19

0.19

0.17

0.17

0.14

0.21

0.23

0.16

0.16

0.10

0.19

0.18

0.19

0.21

0.08

0.13

0.15

0.16

1.00

0.20

0.22

0.13

0.27

0.14

0.21

0.08

0.20

0.25

0.23

0.18

0.18

0.13

0.23

0.24

0.17

0.24

0.21

0.24

0.25

0.22

0.10

0.11

0.19

0.14

0.18

0.22

0.05

0.20

0.26

0.22

0.20

1.00

0.21

0.22

0.27

0.21

0.13

0.10

0.11

0.14

0.14

0.14

0.14

0.12

0.20

0.22

0.16

0.18

0.19

0.20

0.27

0.18

0.16

0.17

0.27

0.22

0.21

0.25

0.10

0.15

0.24

0.20

0.22

0.21

1.00

0.15

0.22

0.19

0.17

0.10

0.17

0.19

0.19

0.15

0.12

0.14

0.15

0.15

0.13

0.16

0.16

0.16

0.15

0.17

0.10

0.10

0.14

0.12

0.14

0.16

0.08

0.15

0.20

0.16

0.13

0.22

0.15

1.00

0.18

0.19

0.24

0.12

0.24

0.27

0.27

0.23

0.22

0.18

0.29

0.25

0.18

0.27

0.24

0.26

0.28

0.32

0.11

0.13

0.22

0.19

0.22

0.28

0.09

0.25

0.27

0.27

0.27

0.27

0.22

0.18

1.00

0.23

0.25

0.13

0.28

0.24

0.26

0.23

0.19

0.16

0.21

0.17

0.13

0.18

0.22

0.17

0.19

0.28

0.07

0.10

0.16

0.11

0.16

0.21

0.06

0.22

0.21

0.23

0.14

0.21

0.19

0.19

0.23

1.00

LLaMA-3-8B-Instruct with H2o

0.2

0.4

0.6

0.8

1.0

B.2 LongBench

The Table 17 shows the average performance of KV cache compression methods in the LongBench English subtask categories. The ChunkKV achieves the best performance on the overall average, and the Multi-Document QA category, which supports that chunk method is more effective for semantic preservation.

Table 18 shows additional experiments with LLaMA-3-70B-Instruct, comparing our ChunkKV method with baselines with KV cache size = 256 on the LongBench dataset:

B.3 Needle-In-A-Haystack

Figure 16 and 17 visualizes the performance of ChunkKV on the NIAH benchmark for LLaMA-38B-Instruct and Mistral-7B-Instruct with a KV cache size of 128 under 8k and 32k context length. The performance of ChunkKV is consistently better as the context length increases.

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

1.00

0.14

0.41

0.50

0.51

0.48

0.25

0.21

0.26

0.23

0.17

0.24

0.28

0.25

0.29

0.47

0.07

0.11

0.16

0.09

0.13

0.25

0.07

0.20

0.19

0.15

0.19

0.18

0.12

0.13

0.28

0.23

0.14

1.00

0.12

0.07

0.10

0.16

0.18

0.24

0.11

0.10

0.14

0.11

0.15

0.10

0.08

0.17

0.08

0.20

0.11

0.16

0.18

0.14

0.09

0.15

0.20

0.17

0.09

0.09

0.09

0.08

0.13

0.08

0.41

0.12

1.00

0.47

0.42

0.42

0.30

0.20

0.23

0.18

0.16

0.21

0.26

0.19

0.21

0.45

0.09

0.11

0.15

0.11

0.17

0.26

0.06

0.21

0.19

0.18

0.19

0.22

0.12

0.15

0.26

0.25

0.50

0.07

0.47

1.00

0.58

0.42

0.26

0.14

0.27

0.20

0.17

0.25

0.24

0.25

0.27

0.41

0.09

0.07

0.17

0.10

0.12

0.24

0.05

0.18

0.14

0.14

0.26

0.25

0.14

0.16

0.31

0.25

0.51

0.10

0.42

0.58

1.00

0.44

0.29

0.17

0.29

0.25

0.19

0.28

0.27

0.27

0.32

0.47

0.09

0.10

0.20

0.13

0.15

0.27

0.08

0.20

0.20

0.17

0.25

0.24

0.14

0.16

0.35

0.23

0.48

0.16

0.42

0.42

0.44

1.00

0.28

0.21

0.25

0.22

0.18

0.22

0.27

0.23

0.26

0.46

0.09

0.13

0.19

0.14

0.17

0.23

0.11

0.21

0.20

0.18

0.19

0.21

0.16

0.12

0.29

0.21

0.25

0.18

0.30

0.26

0.29

0.28

1.00

0.27

0.25

0.21

0.19

0.23

0.30

0.16

0.20

0.34

0.11

0.22

0.22

0.23

0.23

0.29

0.09

0.27

0.27

0.29

0.21

0.24

0.15

0.13

0.28

0.20

0.21

0.24

0.20

0.14

0.17

0.21

0.27

1.00

0.20

0.15

0.23

0.17

0.26

0.15

0.14

0.23

0.10

0.23

0.17

0.19

0.22

0.23

0.10

0.20

0.21

0.24

0.12

0.17

0.10

0.13

0.20

0.14

0.26

0.11

0.23

0.27

0.29

0.25

0.25

0.20

1.00

0.49

0.36

0.46

0.45

0.39

0.40

0.29

0.17

0.24

0.27

0.24

0.22

0.29

0.09

0.36

0.44

0.39

0.25

0.32

0.23

0.17

0.38

0.29

0.23

0.10

0.18

0.20

0.25

0.22

0.21

0.15

0.49

1.00

0.36

0.55

0.39

0.46

0.43

0.23

0.16

0.22

0.25

0.20

0.19

0.25

0.06

0.35

0.39

0.35

0.23

0.31

0.22

0.22

0.32

0.24

0.17

0.14

0.16

0.17

0.19

0.18

0.19

0.23

0.36

0.36

1.00

0.30

0.32

0.30

0.29

0.20

0.19

0.19

0.19

0.27

0.22

0.21

0.10

0.23

0.29

0.26

0.21

0.30

0.16

0.21

0.24

0.20

0.24

0.11

0.21

0.25

0.28

0.22

0.23

0.17

0.46

0.55

0.30

1.00

0.41

0.41

0.38

0.25

0.15

0.19

0.23

0.20

0.17

0.25

0.06

0.36

0.39

0.35

0.24

0.31

0.19

0.18

0.31

0.29

0.28

0.15

0.26

0.24

0.27

0.27

0.30

0.26

0.45

0.39

0.32

0.41

1.00

0.31

0.33

0.32

0.16

0.26

0.26

0.29

0.25

0.31

0.09

0.35

0.45

0.39

0.23

0.33

0.21

0.20

0.34

0.29

0.25

0.10

0.19

0.25

0.27

0.23

0.16

0.15

0.39

0.46

0.30

0.41

0.31

1.00

0.50

0.26

0.15

0.16

0.25

0.15

0.19

0.26

0.07

0.27

0.29

0.21

0.24

0.27

0.21

0.17

0.32

0.22

0.29

0.08

0.21

0.27

0.32

0.26

0.20

0.14

0.40

0.43

0.29

0.38

0.33

0.50

1.00

0.30

0.12

0.14

0.28

0.14

0.18

0.29

0.08

0.25

0.29

0.19

0.29

0.29

0.24

0.16

0.33

0.23

0.47

0.17

0.45

0.41

0.47

0.46

0.34

0.23

0.29

0.23

0.20

0.25

0.32

0.26

0.30

1.00

0.09

0.14

0.23

0.15

0.23

0.32

0.10

0.27

0.27

0.21

0.21

0.23

0.17

0.15

0.34

0.24

0.07

0.08

0.09

0.09

0.09

0.09

0.11

0.10

0.17

0.16

0.19

0.15

0.16

0.15

0.12

0.09

1.00

0.16

0.17

0.24

0.21

0.16

0.16

0.15

0.20

0.19

0.16

0.17

0.17

0.17

0.15

0.12

0.11

0.20

0.11

0.07

0.10

0.13

0.22

0.23

0.24

0.22

0.19

0.19

0.26

0.16

0.14

0.14

0.16

1.00

0.22

0.24

0.24

0.20

0.11

0.25

0.28

0.31

0.13

0.14

0.17

0.11

0.18

0.14

0.16

0.11

0.15

0.17

0.20

0.19

0.22

0.17

0.27

0.25

0.19

0.23

0.26

0.25

0.28

0.23

0.17

0.22

1.00

0.22

0.21

0.39

0.13

0.19

0.24

0.22

0.23

0.22

0.27

0.17

0.27

0.23

0.09

0.16

0.11

0.10

0.13

0.14

0.23

0.19

0.24

0.20

0.27

0.20

0.29

0.15

0.14

0.15

0.24

0.24

0.22

1.00

0.28

0.24

0.11

0.22

0.36

0.32

0.22

0.25

0.22

0.19

0.23

0.16

0.13

0.18

0.17

0.12

0.15

0.17

0.23

0.22

0.22

0.19

0.22

0.17

0.25

0.19

0.18

0.23

0.21

0.24

0.21

0.28

1.00

0.26

0.13

0.21

0.27

0.25

0.18

0.18

0.21

0.14

0.24

0.15

0.25

0.14

0.26

0.24

0.27

0.23

0.29

0.23

0.29

0.25

0.21

0.25

0.31

0.26

0.29

0.32

0.16

0.20

0.39

0.24

0.26

1.00

0.13

0.21

0.30

0.25

0.28

0.25

0.25

0.17

0.33

0.25

0.07

0.09

0.06

0.05

0.08

0.11

0.09

0.10

0.09

0.06

0.10

0.06

0.09

0.07

0.08

0.10

0.16

0.11

0.13

0.11

0.13

0.13

1.00

0.06

0.09

0.08

0.08

0.07

0.08

0.05

0.09

0.05

0.20

0.15

0.21

0.18

0.20

0.21

0.27

0.20

0.36

0.35

0.23

0.36

0.35

0.27

0.25

0.27

0.15

0.25

0.19

0.22

0.21

0.21

0.06

1.00

0.39

0.44

0.18

0.26

0.16

0.18

0.27

0.26

0.19

0.20

0.19

0.14

0.20

0.20

0.27

0.21

0.44

0.39

0.29

0.39

0.45

0.29

0.29

0.27

0.20

0.28

0.24

0.36

0.27

0.30

0.09

0.39

1.00

0.46

0.20

0.35

0.25

0.23

0.34

0.28

0.15

0.17

0.18

0.14

0.17

0.18

0.29

0.24

0.39

0.35

0.26

0.35

0.39

0.21

0.19

0.21

0.19

0.31

0.22

0.32

0.25

0.25

0.08

0.44

0.46

1.00

0.22

0.33

0.19

0.23

0.31

0.23

0.19

0.09

0.19

0.26

0.25

0.19

0.21

0.12

0.25

0.23

0.21

0.24

0.23

0.24

0.29

0.21

0.16

0.13

0.23

0.22

0.18

0.28

0.08

0.18

0.20

0.22

1.00

0.24

0.29

0.17

0.32

0.23

0.18

0.09

0.22

0.25

0.24

0.21

0.24

0.17

0.32

0.31

0.30

0.31

0.33

0.27

0.29

0.23

0.17

0.14

0.22

0.25

0.18

0.25

0.07

0.26

0.35

0.33

0.24

1.00

0.21

0.29

0.29

0.26

0.12

0.09

0.12

0.14

0.14

0.16

0.15

0.10

0.23

0.22

0.16

0.19

0.21

0.21

0.24

0.17

0.17

0.17

0.27

0.22

0.21

0.25

0.08

0.16

0.25

0.19

0.29

0.21

1.00

0.14

0.25

0.24

0.13

0.08

0.15

0.16

0.16

0.12

0.13

0.13

0.17

0.22

0.21

0.18

0.20

0.17

0.16

0.15

0.17

0.11

0.17

0.19

0.14

0.17

0.05

0.18

0.23

0.23

0.17

0.29

0.14

1.00

0.17

0.22

0.28

0.13

0.26

0.31

0.35

0.29

0.28

0.20

0.38

0.32

0.24

0.31

0.34

0.32

0.33

0.34

0.15

0.18

0.27

0.23

0.24

0.33

0.09

0.27

0.34

0.31

0.32

0.29

0.25

0.17

1.00

0.26

0.23

0.08

0.25

0.25

0.23

0.21

0.20

0.14

0.29

0.24

0.20

0.29

0.29

0.22

0.23

0.24

0.12

0.14

0.23

0.16

0.15

0.25

0.05

0.26

0.28

0.23

0.23

0.26

0.24

0.22

0.26

1.00

LLaMA-3-8B-Instruct with SnapKV

0.2

0.4

0.6

0.8

1.0

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

1.00

0.53

0.53

0.57

0.51

0.36

0.45

0.51

0.47

0.31

0.34

0.43

0.48

0.38

0.35

0.33

0.33

0.20

0.23

0.22

0.19

0.26

0.16

0.13

0.13

0.13

0.15

0.14

0.22

0.17

0.20

0.38

0.53

1.00

0.67

0.73

0.66

0.44

0.57

0.59

0.47

0.41

0.43

0.51

0.53

0.39

0.36

0.40

0.39

0.28

0.32

0.32

0.28

0.35

0.23

0.25

0.21

0.21

0.20

0.22

0.33

0.26

0.28

0.54

0.53

0.67

1.00

0.71

0.75

0.57

0.64

0.69

0.60

0.45

0.53

0.56

0.53

0.45

0.41

0.45

0.44

0.28

0.32

0.36

0.30

0.38

0.23

0.23

0.21

0.21

0.23

0.23

0.36

0.28

0.32

0.56

0.57

0.73

0.71

1.00

0.79

0.50

0.64

0.66

0.51

0.41

0.44

0.53

0.57

0.39

0.38

0.40

0.39

0.28

0.32

0.34

0.30

0.38

0.23

0.22

0.22

0.20

0.21

0.22

0.35

0.26

0.30

0.56

0.51

0.66

0.75

0.79

1.00

0.59

0.75

0.71

0.53

0.44

0.50

0.56

0.56

0.39

0.36

0.41

0.39

0.30

0.34

0.34

0.31

0.38

0.24

0.22

0.23

0.20

0.22

0.22

0.35

0.28

0.30

0.57

0.36

0.44

0.57

0.50

0.59

1.00

0.57

0.57

0.44

0.35

0.41

0.45

0.47

0.31

0.27

0.31

0.31

0.23

0.27

0.31

0.24

0.28

0.20

0.20

0.18

0.18

0.19

0.20

0.28

0.24

0.24

0.44

0.45

0.57

0.64

0.64

0.75

0.57

1.00

0.69

0.50

0.40

0.44

0.53

0.54

0.35

0.33

0.38

0.34

0.27

0.31

0.34

0.27

0.33

0.23

0.20

0.21

0.20

0.21

0.20

0.30

0.26

0.25

0.51

0.51

0.59

0.69

0.66

0.71

0.57

0.69

1.00

0.60

0.48

0.54

0.67

0.67

0.48

0.44

0.43

0.43

0.27

0.32

0.32

0.27

0.34

0.22

0.19

0.20

0.20

0.22

0.21

0.33

0.26

0.30

0.51

0.47

0.47

0.60

0.51

0.53

0.44

0.50

0.60

1.00

0.54

0.57

0.62

0.53

0.51

0.45

0.41

0.44

0.24

0.28

0.30

0.24

0.32

0.21

0.18

0.19

0.19

0.23

0.21

0.31

0.25

0.30

0.44

0.31

0.41

0.45

0.41

0.44

0.35

0.40

0.48

0.54

1.00

0.59

0.62

0.48

0.54

0.50

0.51

0.54

0.33

0.36

0.38

0.33

0.38

0.28

0.25

0.24

0.24

0.28

0.27

0.35

0.32

0.36

0.43

0.34

0.43

0.53

0.44

0.50

0.41

0.44

0.54

0.57

0.59

1.00

0.71

0.51

0.53

0.50

0.47

0.44

0.28

0.32

0.35

0.31

0.33

0.26

0.21

0.23

0.21

0.28

0.26

0.34

0.30

0.33

0.44

0.43

0.51

0.56

0.53

0.56

0.45

0.53

0.67

0.62

0.62

0.71

1.00

0.67

0.60

0.54

0.51

0.50

0.31

0.36

0.36

0.31

0.36

0.27

0.23

0.24

0.23

0.31

0.27

0.38

0.31

0.35

0.51

0.48

0.53

0.53

0.57

0.56

0.47

0.54

0.67

0.53

0.48

0.51

0.67

1.00

0.47

0.44

0.41

0.40

0.26

0.31

0.30

0.24

0.28

0.21

0.18

0.20

0.19

0.20

0.20

0.28

0.25

0.26

0.47

0.38

0.39

0.45

0.39

0.39

0.31

0.35

0.48

0.51

0.54

0.53

0.60

0.47

1.00

0.71

0.59

0.56

0.35

0.39

0.34

0.32

0.41

0.30

0.25

0.24

0.23

0.31

0.26

0.34

0.30

0.36

0.43

0.35

0.36

0.41

0.38

0.36

0.27

0.33

0.44

0.45

0.50

0.50

0.54

0.44

0.71

1.00

0.64

0.60

0.40

0.43

0.36

0.39

0.41

0.34

0.25

0.28

0.27

0.38

0.33

0.38

0.33

0.40

0.41

0.33

0.40

0.45

0.40

0.41

0.31

0.38

0.43

0.41

0.51

0.47

0.51

0.41

0.59

0.64

1.00

0.69

0.45

0.50

0.48

0.48

0.50

0.45

0.36

0.36

0.33

0.43

0.38

0.47

0.38

0.48

0.48

0.33

0.39

0.44

0.39

0.39

0.31

0.34

0.43

0.44

0.54

0.44

0.50

0.40

0.56

0.60

0.69

1.00

0.56

0.50

0.54

0.48

0.50

0.44

0.41

0.41

0.39

0.48

0.45

0.47

0.41

0.54

0.44

0.20

0.28

0.28

0.28

0.30

0.23

0.27

0.27

0.24

0.33

0.28

0.31

0.26

0.35

0.40

0.45

0.56

1.00

0.53

0.50

0.59

0.50

0.51

0.45

0.47

0.51

0.47

0.47

0.41

0.41

0.48

0.34

0.23

0.32

0.32

0.32

0.34

0.27

0.31

0.32

0.28

0.36

0.32

0.36

0.31

0.39

0.43

0.50

0.50

0.53

1.00

0.47

0.50

0.48

0.45

0.39

0.36

0.34

0.44

0.38

0.44

0.35

0.45

0.40

0.22

0.32

0.36

0.34

0.34

0.31

0.34

0.32

0.30

0.38

0.35

0.36

0.30

0.34

0.36

0.48

0.54

0.50

0.47

1.00

0.54

0.47

0.44

0.43

0.39

0.43

0.44

0.43

0.48

0.45

0.53

0.40

0.19

0.28

0.30

0.30

0.31

0.24

0.27

0.27

0.24

0.33

0.31

0.31

0.24

0.32

0.39

0.48

0.48

0.59

0.50

0.54

1.00

0.60

0.66

0.51

0.57

0.48

0.51

0.53

0.50

0.39

0.56

0.35

0.26

0.35

0.38

0.38

0.38

0.28

0.33

0.34

0.32

0.38

0.33

0.36

0.28

0.41

0.41

0.50

0.50

0.50

0.48

0.47

0.60

1.00

0.50

0.45

0.44

0.35

0.45

0.40

0.48

0.38

0.50

0.39

0.16

0.23

0.23

0.23

0.24

0.20

0.23

0.22

0.21

0.28

0.26

0.27

0.21

0.30

0.34

0.45

0.44

0.51

0.45

0.44

0.66

0.50

1.00

0.53

0.56

0.51

0.53

0.53

0.44

0.41

0.56

0.33

0.13

0.25

0.23

0.22

0.22

0.20

0.20

0.19

0.18

0.25

0.21

0.23

0.18

0.25

0.25

0.36

0.41

0.45

0.39

0.43

0.51

0.45

0.53

1.00

0.54

0.50

0.47

0.53

0.36

0.33

0.50

0.28

0.13

0.21

0.21

0.22

0.23

0.18

0.21

0.20

0.19

0.24

0.23

0.24

0.20

0.24

0.28

0.36

0.41

0.47

0.36

0.39

0.57

0.44

0.56

0.54

1.00

0.50

0.53

0.57

0.40

0.38

0.50

0.30

0.13

0.21

0.21

0.20

0.20

0.18

0.20

0.20

0.19

0.24

0.21

0.23

0.19

0.23

0.27

0.33

0.39

0.51

0.34

0.43

0.48

0.35

0.51

0.50

0.50

1.00

0.47

0.53

0.35

0.36

0.44

0.26

0.15

0.20

0.23

0.21

0.22

0.19

0.21

0.22

0.23

0.28

0.28

0.31

0.20

0.31

0.38

0.43

0.48

0.47

0.44

0.44

0.51

0.45

0.53

0.47

0.53

0.47

1.00

0.62

0.45

0.44

0.60

0.32

0.14

0.22

0.23

0.22

0.22

0.20

0.20

0.21

0.21

0.27

0.26

0.27

0.20

0.26

0.33

0.38

0.45

0.47

0.38

0.43

0.53

0.40

0.53

0.53

0.57

0.53

0.62

1.00

0.41

0.45

0.60

0.33

0.22

0.33

0.36

0.35

0.35

0.28

0.30

0.33

0.31

0.35

0.34

0.38

0.28

0.34

0.38

0.47

0.47

0.41

0.44

0.48

0.50

0.48

0.44

0.36

0.40

0.35

0.45

0.41

1.00

0.51

0.57

0.44

0.17

0.26

0.28

0.26

0.28

0.24

0.26

0.26

0.25

0.32

0.30

0.31

0.25

0.30

0.33

0.38

0.41

0.41

0.35

0.45

0.39

0.38

0.41

0.33

0.38

0.36

0.44

0.45

0.51

1.00

0.54

0.38

0.20

0.28

0.32

0.30

0.30

0.24

0.25

0.30

0.30

0.36

0.33

0.35

0.26

0.36

0.40

0.48

0.54

0.48

0.45

0.53

0.56

0.50

0.56

0.50

0.50

0.44

0.60

0.60

0.57

0.54

1.00

0.40

0.38

0.54

0.56

0.56

0.57

0.44

0.51

0.51

0.44

0.43

0.44

0.51

0.47

0.43

0.41

0.48

0.44

0.34

0.40

0.40

0.35

0.39

0.33

0.28

0.30

0.26

0.32

0.33

0.44

0.38

0.40

1.00

LLaMA-3-8B-Instruct with ChunkKV

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

1.00

0.12

0.16

0.09

0.13

0.13

0.07

0.13

0.16

0.10

0.14

0.06

0.14

0.09

0.11

0.07

0.08

0.08

0.03

0.11

0.07

0.05

0.05

0.07

0.08

0.08

0.09

0.10

0.05

0.06

0.17

0.08

0.12

1.00

0.17

0.16

0.19

0.14

0.05

0.10

0.13

0.12

0.15

0.12

0.17

0.12

0.17

0.17

0.10

0.22

0.12

0.12

0.12

0.15

0.14

0.07

0.15

0.10

0.14

0.12

0.13

0.17

0.15

0.12

0.16

0.17

1.00

0.14

0.18

0.16

0.07

0.15

0.21

0.12

0.20

0.10

0.17

0.11

0.15

0.08

0.11

0.11

0.06

0.15

0.08

0.08

0.08

0.11

0.10

0.13

0.13

0.14

0.10

0.05

0.13

0.10

0.09

0.16

0.14

1.00

0.19

0.19

0.07

0.11

0.11

0.12

0.09

0.14

0.14

0.10

0.12

0.15

0.08

0.15

0.07

0.08

0.08

0.16

0.16

0.07

0.09

0.09

0.12

0.08

0.08

0.12

0.13

0.08

0.13

0.19

0.18

0.19

1.00

0.18

0.16

0.16

0.19

0.20

0.16

0.13

0.17

0.14

0.19

0.16

0.13

0.17

0.10

0.10

0.12

0.11

0.12

0.09

0.13

0.11

0.13

0.14

0.10

0.15

0.15

0.08

0.13

0.14

0.16

0.19

0.18

1.00

0.08

0.14

0.14

0.16

0.15

0.07

0.24

0.17

0.17

0.17

0.14

0.17

0.07

0.18

0.10

0.14

0.14

0.12

0.12

0.15

0.10

0.09

0.08

0.11

0.16

0.07

0.07

0.05

0.07

0.07

0.16

0.08

1.00

0.10

0.12

0.17

0.09

0.10

0.07

0.11

0.15

0.14

0.10

0.11

0.07

0.06

0.08

0.07

0.06

0.09

0.06

0.09

0.07

0.12

0.06

0.06

0.09

0.07

0.13

0.10

0.15

0.11

0.16

0.14

0.10

1.00

0.21

0.17

0.14

0.06

0.15

0.15

0.14

0.09

0.11

0.10

0.03

0.12

0.07

0.08

0.08

0.13

0.12

0.14

0.13

0.18

0.06

0.04

0.11

0.07

0.16

0.13

0.21

0.11

0.19

0.14

0.12

0.21

1.00

0.18

0.20

0.09

0.18

0.15

0.21

0.10

0.15

0.13

0.04

0.18

0.11

0.08

0.07

0.09

0.13

0.15

0.15

0.18

0.10

0.05

0.13

0.11

0.10

0.12

0.12

0.12

0.20

0.16

0.17

0.17

0.18

1.00

0.14

0.12

0.16

0.16

0.18

0.16

0.17

0.15

0.06

0.13

0.09

0.10

0.10

0.11

0.11

0.13

0.10

0.13

0.09

0.09

0.12

0.08

0.14

0.15

0.20

0.09

0.16

0.15

0.09

0.14

0.20

0.14

1.00

0.10

0.20

0.15

0.21

0.12

0.16

0.14

0.06

0.14

0.14

0.10

0.09

0.13

0.17

0.17

0.15

0.16

0.12

0.09

0.18

0.14

0.06

0.12

0.10

0.14

0.13

0.07

0.10

0.06

0.09

0.12

0.10

1.00

0.08

0.09

0.12

0.11

0.08

0.10

0.08

0.06

0.07

0.08

0.08

0.07

0.08

0.11

0.18

0.15

0.08

0.11

0.12

0.19

0.14

0.17

0.17

0.14

0.17

0.24

0.07

0.15

0.18

0.16

0.20

0.08

1.00

0.13

0.17

0.15

0.10

0.17

0.07

0.14

0.11

0.12

0.12

0.11

0.13

0.14

0.12

0.13

0.10

0.13

0.16

0.08

0.09

0.12

0.11

0.10

0.14

0.17

0.11

0.15

0.15

0.16

0.15

0.09

0.13

1.00

0.17

0.15

0.17

0.15

0.05

0.22

0.15

0.10

0.11

0.11

0.13

0.17

0.16

0.13

0.10

0.09

0.13

0.09

0.11

0.17

0.15

0.12

0.19

0.17

0.15

0.14

0.21

0.18

0.21

0.12

0.17

0.17

1.00

0.33

0.19

0.35

0.11

0.16

0.21

0.18

0.16

0.09

0.19

0.14

0.15

0.17

0.18

0.13

0.17

0.12

0.07

0.17

0.08

0.15

0.16

0.17

0.14

0.09

0.10

0.16

0.12

0.11

0.15

0.15

0.33

1.00

0.16

0.38

0.16

0.09

0.19

0.24

0.24

0.07

0.16

0.10

0.10

0.10

0.16

0.19

0.15

0.07

0.08

0.10

0.11

0.08

0.13

0.14

0.10

0.11

0.15

0.17

0.16

0.08

0.10

0.17

0.19

0.16

1.00

0.16

0.07

0.14

0.18

0.11

0.11

0.08

0.20

0.17

0.13

0.11

0.14

0.12

0.14

0.10

0.08

0.22

0.11

0.15

0.17

0.17

0.11

0.10

0.13

0.15

0.14

0.10

0.17

0.15

0.35

0.38

0.16

1.00

0.16

0.12

0.24

0.25

0.22

0.07

0.17

0.10

0.11

0.11

0.18

0.21

0.16

0.08

0.03

0.12

0.06

0.07

0.10

0.07

0.07

0.03

0.04

0.06

0.06

0.08

0.07

0.05

0.11

0.16

0.07

0.16

1.00

0.03

0.13

0.18

0.16

0.04

0.09

0.04

0.04

0.06

0.13

0.18

0.08

0.04

0.11

0.12

0.15

0.08

0.10

0.18

0.06

0.12

0.18

0.13

0.14

0.06

0.14

0.22

0.16

0.09

0.14

0.12

0.03

1.00

0.13

0.09

0.09

0.11

0.13

0.17

0.12

0.12

0.11

0.04

0.12

0.10

0.07

0.12

0.08

0.08

0.12

0.10

0.08

0.07

0.11

0.09

0.14

0.07

0.11

0.15

0.21

0.19

0.18

0.24

0.13

0.13

1.00

0.17

0.17

0.09

0.21

0.10

0.13

0.11

0.18

0.13

0.13

0.09

0.05

0.15

0.08

0.16

0.11

0.14

0.07

0.08

0.08

0.10

0.10

0.08

0.12

0.10

0.18

0.24

0.11

0.25

0.18

0.09

0.17

1.00

0.30

0.08

0.14

0.09

0.09

0.08

0.19

0.17

0.15

0.07

0.05

0.14

0.08

0.16

0.12

0.14

0.06

0.08

0.07

0.10

0.09

0.08

0.12

0.11

0.16

0.24

0.11

0.22

0.16

0.09

0.17

0.30

1.00

0.07

0.16

0.09

0.09

0.08

0.19

0.18

0.12

0.06

0.07

0.07

0.11

0.07

0.09

0.12

0.09

0.13

0.09

0.11

0.13

0.07

0.11

0.11

0.09

0.07

0.08

0.07

0.04

0.11

0.09

0.08

0.07

1.00

0.09

0.17

0.13

0.11

0.07

0.05

0.10

0.10

0.08

0.15

0.10

0.09

0.13

0.12

0.06

0.12

0.13

0.11

0.17

0.08

0.13

0.13

0.19

0.16

0.20

0.17

0.09

0.13

0.21

0.14

0.16

0.09

1.00

0.16

0.15

0.15

0.14

0.12

0.15

0.12

0.08

0.10

0.13

0.09

0.11

0.15

0.09

0.14

0.15

0.13

0.17

0.11

0.14

0.17

0.14

0.10

0.17

0.10

0.04

0.17

0.10

0.09

0.09

0.17

0.16

1.00

0.25

0.18

0.11

0.08

0.14

0.17

0.09

0.14

0.13

0.12

0.13

0.10

0.07

0.13

0.15

0.10

0.15

0.18

0.12

0.16

0.15

0.10

0.13

0.11

0.04

0.12

0.13

0.09

0.09

0.13

0.15

0.25

1.00

0.21

0.12

0.11

0.13

0.24

0.10

0.12

0.14

0.08

0.14

0.09

0.12

0.18

0.18

0.13

0.16

0.15

0.13

0.13

0.17

0.10

0.11

0.11

0.06

0.12

0.11

0.08

0.08

0.11

0.15

0.18

0.21

1.00

0.11

0.08

0.13

0.19

0.05

0.13

0.10

0.08

0.10

0.08

0.06

0.06

0.10

0.09

0.12

0.08

0.10

0.10

0.18

0.16

0.14

0.18

0.13

0.11

0.18

0.19

0.19

0.07

0.14

0.11

0.12

0.11

1.00

0.13

0.14

0.12

0.06

0.17

0.05

0.12

0.15

0.11

0.06

0.04

0.05

0.09

0.09

0.11

0.13

0.09

0.13

0.19

0.12

0.21

0.18

0.04

0.13

0.17

0.18

0.05

0.12

0.08

0.11

0.08

0.13

1.00

0.12

0.07

0.17

0.15

0.13

0.13

0.15

0.16

0.09

0.11

0.13

0.12

0.18

0.12

0.16

0.13

0.17

0.15

0.14

0.16

0.08

0.12

0.13

0.15

0.12

0.10

0.15

0.14

0.13

0.13

0.14

0.12

1.00

0.12

0.08

0.12

0.10

0.08

0.08

0.07

0.07

0.07

0.11

0.08

0.14

0.19

0.08

0.09

0.12

0.07

0.10

0.08

0.04

0.10

0.09

0.07

0.06

0.10

0.12

0.17

0.24

0.19

0.12

0.07

0.12

1.00

Mistral-7B-Instruct with H2o

0.2

0.4

0.6

0.8

1.0

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

1.00

0.10

0.10

0.13

0.10

0.12

0.03

0.13

0.10

0.09

0.11

0.06

0.12

0.09

0.08

0.08

0.08

0.09

0.03

0.07

0.06

0.08

0.08

0.06

0.08

0.08

0.09

0.05

0.05

0.07

0.13

0.07

0.10

1.00

0.17

0.26

0.22

0.15

0.07

0.11

0.14

0.16

0.17

0.18

0.18

0.13

0.17

0.20

0.11

0.26

0.14

0.11

0.13

0.10

0.08

0.06

0.14

0.10

0.14

0.10

0.11

0.19

0.21

0.11

0.10

0.17

1.00

0.18

0.24

0.17

0.13

0.16

0.25

0.16

0.25

0.11

0.17

0.13

0.18

0.09

0.14

0.15

0.04

0.15

0.12

0.07

0.08

0.10

0.13

0.15

0.14

0.15

0.10

0.07

0.16

0.12

0.13

0.26

0.18

1.00

0.19

0.24

0.06

0.13

0.15

0.16

0.19

0.12

0.19

0.14

0.16

0.17

0.12

0.22

0.10

0.11

0.10

0.11

0.10

0.06

0.13

0.11

0.14

0.10

0.10

0.15

0.21

0.09

0.10

0.22

0.24

0.19

1.00

0.17

0.18

0.16

0.24

0.18

0.22

0.14

0.18

0.15

0.21

0.14

0.17

0.18

0.11

0.13

0.15

0.09

0.08

0.10

0.17

0.14

0.16

0.17

0.10

0.17

0.18

0.12

0.12

0.15

0.17

0.24

0.17

1.00

0.06

0.14

0.15

0.16

0.18

0.09

0.25

0.16

0.16

0.17

0.12

0.18

0.08

0.17

0.11

0.10

0.10

0.12

0.14

0.14

0.13

0.09

0.09

0.12

0.19

0.10

0.03

0.07

0.13

0.06

0.18

0.06

1.00

0.09

0.18

0.14

0.12

0.14

0.07

0.13

0.19

0.12

0.15

0.11

0.10

0.09

0.14

0.07

0.06

0.13

0.13

0.17

0.15

0.20

0.09

0.11

0.12

0.14

0.13

0.11

0.16

0.13

0.16

0.14

0.09

1.00

0.22

0.16

0.19

0.08

0.14

0.15

0.15

0.08

0.13

0.11

0.03

0.12

0.09

0.06

0.06

0.13

0.15

0.15

0.15

0.16

0.06

0.06

0.13

0.10

0.10

0.14

0.25

0.15

0.24

0.15

0.18

0.22

1.00

0.19

0.26

0.11

0.19

0.17

0.26

0.09

0.23

0.15

0.04

0.23

0.16

0.07

0.07

0.13

0.21

0.22

0.21

0.23

0.10

0.09

0.16

0.16

0.09

0.16

0.16

0.16

0.18

0.16

0.14

0.16

0.19

1.00

0.19

0.14

0.18

0.16

0.18

0.13

0.16

0.14

0.06

0.13

0.11

0.07

0.08

0.12

0.15

0.16

0.14

0.15

0.12

0.11

0.15

0.15

0.11

0.17

0.25

0.19

0.22

0.18

0.12

0.19

0.26

0.19

1.00

0.12

0.22

0.18

0.28

0.15

0.22

0.19

0.05

0.19

0.18

0.08

0.08

0.13

0.20

0.19

0.19

0.17

0.12

0.11

0.23

0.16

0.06

0.18

0.11

0.12

0.14

0.09

0.14

0.08

0.11

0.14

0.12

1.00

0.09

0.12

0.16

0.17

0.12

0.17

0.12

0.08

0.12

0.08

0.07

0.11

0.12

0.14

0.18

0.15

0.11

0.16

0.15

0.18

0.12

0.18

0.17

0.19

0.18

0.25

0.07

0.14

0.19

0.18

0.22

0.09

1.00

0.13

0.17

0.14

0.12

0.19

0.07

0.14

0.13

0.08

0.09

0.12

0.14

0.13

0.14

0.13

0.09

0.16

0.17

0.09

0.09

0.13

0.13

0.14

0.15

0.16

0.13

0.15

0.17

0.16

0.18

0.12

0.13

1.00

0.21

0.13

0.22

0.15

0.06

0.28

0.17

0.06

0.08

0.16

0.18

0.24

0.22

0.17

0.09

0.15

0.18

0.17

0.08

0.17

0.18

0.16

0.21

0.16

0.19

0.15

0.26

0.18

0.28

0.16

0.17

0.21

1.00

0.28

0.28

0.34

0.13

0.26

0.28

0.11

0.09

0.16

0.29

0.25

0.24

0.25

0.18

0.18

0.23

0.19

0.08

0.20

0.09

0.17

0.14

0.17

0.12

0.08

0.09

0.13

0.15

0.17

0.14

0.13

0.28

1.00

0.14

0.37

0.23

0.09

0.17

0.14

0.14

0.09

0.15

0.11

0.13

0.11

0.16

0.23

0.24

0.10

0.08

0.11

0.14

0.12

0.17

0.12

0.15

0.13

0.23

0.16

0.22

0.12

0.12

0.22

0.28

0.14

1.00

0.17

0.08

0.23

0.23

0.08

0.08

0.13

0.28

0.24

0.22

0.19

0.12

0.16

0.19

0.17

0.09

0.26

0.15

0.22

0.18

0.18

0.11

0.11

0.15

0.14

0.19

0.17

0.19

0.15

0.34

0.37

0.17

1.00

0.20

0.13

0.22

0.14

0.13

0.09

0.17

0.13

0.16

0.13

0.17

0.24

0.26

0.10

0.03

0.14

0.04

0.10

0.11

0.08

0.10

0.03

0.04

0.06

0.05

0.12

0.07

0.06

0.13

0.23

0.08

0.20

1.00

0.04

0.13

0.13

0.10

0.04

0.11

0.06

0.07

0.09

0.13

0.22

0.13

0.04

0.07

0.11

0.15

0.11

0.13

0.17

0.09

0.12

0.23

0.13

0.19

0.08

0.14

0.28

0.26

0.09

0.23

0.13

0.04

1.00

0.20

0.06

0.07

0.15

0.21

0.23

0.18

0.18

0.12

0.09

0.14

0.14

0.06

0.13

0.12

0.10

0.15

0.11

0.14

0.09

0.16

0.11

0.18

0.12

0.13

0.17

0.28

0.17

0.23

0.22

0.13

0.20

1.00

0.09

0.10

0.11

0.24

0.16

0.17

0.17

0.14

0.15

0.18

0.12

0.08

0.10

0.07

0.11

0.09

0.10

0.07

0.06

0.07

0.07

0.08

0.08

0.08

0.06

0.11

0.14

0.08

0.14

0.13

0.06

0.09

1.00

0.13

0.05

0.10

0.06

0.07

0.07

0.13

0.11

0.12

0.06

0.08

0.08

0.08

0.10

0.08

0.10

0.06

0.06

0.07

0.08

0.08

0.07

0.09

0.08

0.09

0.14

0.08

0.13

0.10

0.07

0.10

0.13

1.00

0.05

0.09

0.06

0.07

0.06

0.11

0.11

0.11

0.07

0.06

0.06

0.10

0.06

0.10

0.12

0.13

0.13

0.13

0.12

0.13

0.11

0.12

0.16

0.16

0.09

0.13

0.09

0.04

0.15

0.11

0.05

0.05

1.00

0.15

0.21

0.18

0.15

0.07

0.10

0.11

0.17

0.08

0.14

0.13

0.13

0.17

0.14

0.13

0.15

0.21

0.15

0.20

0.12

0.14

0.18

0.29

0.15

0.28

0.17

0.11

0.21

0.24

0.10

0.09

0.15

1.00

0.23

0.22

0.23

0.12

0.18

0.19

0.16

0.08

0.10

0.15

0.11

0.14

0.14

0.17

0.15

0.22

0.16

0.19

0.14

0.13

0.24

0.25

0.11

0.24

0.13

0.06

0.23

0.16

0.06

0.06

0.21

0.23

1.00

0.30

0.26

0.10

0.14

0.16

0.21

0.09

0.14

0.14

0.14

0.16

0.13

0.15

0.15

0.21

0.14

0.19

0.18

0.14

0.22

0.24

0.13

0.22

0.16

0.07

0.18

0.17

0.07

0.07

0.18

0.22

0.30

1.00

0.24

0.11

0.17

0.18

0.24

0.05

0.10

0.15

0.10

0.17

0.09

0.20

0.16

0.23

0.15

0.17

0.15

0.13

0.17

0.25

0.11

0.19

0.13

0.09

0.18

0.17

0.07

0.06

0.15

0.23

0.26

0.24

1.00

0.11

0.15

0.15

0.18

0.05

0.11

0.10

0.10

0.10

0.09

0.09

0.06

0.10

0.12

0.12

0.11

0.09

0.09

0.18

0.16

0.12

0.17

0.13

0.12

0.14

0.13

0.11

0.07

0.12

0.10

0.11

0.11

1.00

0.12

0.15

0.10

0.07

0.19

0.07

0.15

0.17

0.12

0.11

0.06

0.09

0.11

0.11

0.16

0.16

0.15

0.18

0.23

0.16

0.24

0.22

0.09

0.15

0.11

0.11

0.10

0.18

0.14

0.17

0.15

0.12

1.00

0.20

0.11

0.13

0.21

0.16

0.21

0.18

0.19

0.12

0.13

0.16

0.15

0.23

0.15

0.17

0.18

0.23

0.24

0.19

0.26

0.13

0.14

0.18

0.12

0.11

0.11

0.19

0.16

0.18

0.15

0.15

0.20

1.00

0.14

0.07

0.11

0.12

0.09

0.12

0.10

0.14

0.10

0.16

0.15

0.16

0.18

0.09

0.17

0.19

0.10

0.17

0.10

0.04

0.14

0.12

0.06

0.07

0.17

0.16

0.21

0.24

0.18

0.10

0.11

0.14

1.00

Mistral-7B-Instruct with SnapKV

0.2

0.4

0.6

0.8

1.0

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

Layer

1.00

0.26

0.27

0.27

0.27

0.24

0.25

0.24

0.24

0.24

0.27

0.21

0.16

0.17

0.16

0.09

0.12

0.14

0.15

0.15

0.13

0.15

0.13

0.15

0.13

0.12

0.13

0.14

0.14

0.14

0.12

0.13

0.26

1.00

0.46

0.47

0.41

0.38

0.37

0.30

0.35

0.31

0.33

0.26

0.21

0.24

0.22

0.14

0.19

0.24

0.19

0.20

0.20

0.20

0.18

0.20

0.21

0.17

0.18

0.23

0.20

0.22

0.21

0.22

0.27

0.46

1.00

0.55

0.52

0.48

0.47

0.42

0.43

0.39

0.41

0.34

0.28

0.29

0.25

0.16

0.20

0.24

0.20

0.21

0.20

0.20

0.18

0.23

0.22

0.20

0.21

0.23

0.21

0.23

0.22

0.23

0.27

0.47

0.55

1.00

0.56

0.51

0.45

0.38

0.43

0.37

0.37

0.31

0.26

0.28

0.27

0.18

0.21

0.27

0.23

0.23

0.23

0.26

0.22

0.24

0.25

0.23

0.24

0.27

0.24

0.26

0.26

0.26

0.27

0.41

0.52

0.56

1.00

0.60

0.49

0.46

0.42

0.41

0.38

0.36

0.30

0.29

0.29

0.22

0.25

0.29

0.24

0.25

0.27

0.26

0.23

0.25

0.29

0.26

0.25

0.29

0.26

0.29

0.29

0.30

0.24

0.38

0.48

0.51

0.60

1.00

0.52

0.51

0.48

0.45

0.43

0.39

0.31

0.32

0.30

0.18

0.21

0.28

0.23

0.24

0.22

0.24

0.23

0.24

0.25

0.23

0.25

0.25

0.25

0.25

0.26

0.28

0.25

0.37

0.47

0.45

0.49

0.52

1.00

0.54

0.57

0.56

0.53

0.46

0.36

0.36

0.35

0.22

0.25

0.34

0.29

0.28

0.28

0.31

0.28

0.28

0.29

0.28

0.29

0.32

0.30

0.30

0.29

0.30

0.24

0.30

0.42

0.38

0.46

0.51

0.54

1.00

0.55

0.55

0.49

0.44

0.36

0.39

0.31

0.19

0.22

0.28

0.25

0.27

0.22

0.27

0.24

0.26

0.25

0.24

0.26

0.27

0.27

0.27

0.25

0.25

0.24

0.35

0.43

0.43

0.42

0.48

0.57

0.55

1.00

0.57

0.52

0.51

0.37

0.44

0.34

0.20

0.24

0.34

0.26

0.29

0.25

0.27

0.25

0.28

0.25

0.27

0.28

0.32

0.29

0.28

0.28

0.27

0.24

0.31

0.39

0.37

0.41

0.45

0.56

0.55

0.57

1.00

0.58

0.62

0.45

0.49

0.42

0.24

0.27

0.36

0.30

0.32

0.29

0.31

0.28

0.30

0.30

0.29

0.30

0.31

0.32

0.32

0.31

0.29

0.27

0.33

0.41

0.37

0.38

0.43

0.53

0.49

0.52

0.58

1.00

0.54

0.40

0.46

0.34

0.17

0.21

0.29

0.24

0.26

0.21

0.25

0.21

0.27

0.23

0.23

0.24

0.26

0.26

0.25

0.24

0.23

0.21

0.26

0.34

0.31

0.36

0.39

0.46

0.44

0.51

0.62

0.54

1.00

0.59

0.56

0.50

0.30

0.33

0.41

0.36

0.39

0.33

0.36

0.32

0.34

0.34

0.34

0.33

0.35

0.36

0.35

0.34

0.35

0.16

0.21

0.28

0.26

0.30

0.31

0.36

0.36

0.37

0.45

0.40

0.59

1.00

0.55

0.57

0.40

0.44

0.46

0.43

0.45

0.42

0.39

0.35

0.34

0.39

0.38

0.36

0.35

0.39

0.40

0.39

0.40

0.17

0.24

0.29

0.28

0.29

0.32

0.36

0.39

0.44

0.49

0.46

0.56

0.55

1.00

0.48

0.28

0.34

0.43

0.33

0.38

0.32

0.34

0.30

0.33

0.32

0.32

0.33

0.36

0.36

0.36

0.34

0.34

0.16

0.22

0.25

0.27

0.29

0.30

0.35

0.31

0.34

0.42

0.34

0.50

0.57

0.48

1.00

0.48

0.48

0.54

0.52

0.50

0.48

0.45

0.40

0.35

0.43

0.42

0.39

0.40

0.40

0.43

0.42

0.43

0.09

0.14

0.16

0.18

0.22

0.18

0.22

0.19

0.20

0.24

0.17

0.30

0.40

0.28

0.48

1.00

0.63

0.49

0.48

0.49

0.53

0.43

0.40

0.30

0.45

0.42

0.36

0.34

0.38

0.44

0.41

0.41

0.12

0.19

0.20

0.21

0.25

0.21

0.25

0.22

0.24

0.27

0.21

0.33

0.44

0.34

0.48

0.63

1.00

0.53

0.49

0.56

0.57

0.43

0.41

0.30

0.44

0.46

0.39

0.39

0.42

0.49

0.44

0.47

0.14

0.24

0.24

0.27

0.29

0.28

0.34

0.28

0.34

0.36

0.29

0.41

0.46

0.43

0.54

0.49

0.53

1.00

0.52

0.59

0.51

0.50

0.47

0.37

0.47

0.46

0.45

0.48

0.47

0.48

0.46

0.49

0.15

0.19

0.20

0.23

0.24

0.23

0.29

0.25

0.26

0.30

0.24

0.36

0.43

0.33

0.52

0.48

0.49

0.52

1.00

0.56

0.50

0.50

0.45

0.36

0.48

0.44

0.40

0.39

0.40

0.43

0.44

0.43

0.15

0.20

0.21

0.23

0.25

0.24

0.28

0.27

0.29

0.32

0.26

0.39

0.45

0.38

0.50

0.49

0.56

0.59

0.56

1.00

0.55

0.53

0.47

0.38

0.50

0.47

0.42

0.42

0.47

0.48

0.48

0.46

0.13

0.20

0.20

0.23

0.27

0.22

0.28

0.22

0.25

0.29

0.21

0.33

0.42

0.32

0.48

0.53

0.57

0.51

0.50

0.55

1.00

0.58

0.50

0.36

0.61

0.54

0.44

0.42

0.46

0.56

0.55

0.53

0.15

0.20

0.20

0.26

0.26

0.24

0.31

0.27

0.27

0.31

0.25

0.36

0.39

0.34

0.45

0.43

0.43

0.50

0.50

0.53

0.58

1.00

0.54

0.43

0.55

0.53

0.48

0.46

0.51

0.51

0.52

0.49

0.13

0.18

0.18

0.22

0.23

0.23

0.28

0.24

0.25

0.28

0.21

0.32

0.35

0.30

0.40

0.40

0.41

0.47

0.45

0.47

0.50

0.54

1.00

0.41

0.52

0.50

0.41

0.44

0.45

0.45

0.46

0.46

0.15

0.20

0.23

0.24

0.25

0.24

0.28

0.26

0.28

0.30

0.27

0.34

0.34

0.33

0.35

0.30

0.30

0.37

0.36

0.38

0.36

0.43

0.41

1.00

0.43

0.40

0.37

0.36

0.38

0.40

0.37

0.38

0.13

0.21

0.22

0.25

0.29

0.25

0.29

0.25

0.25

0.30

0.23

0.34

0.39

0.32

0.43

0.45

0.44

0.47

0.48

0.50

0.61

0.55

0.52

0.43

1.00

0.53

0.45

0.42

0.46

0.54

0.52

0.51

0.12

0.17

0.20

0.23

0.26

0.23

0.28

0.24

0.27

0.29

0.23

0.34

0.38

0.32

0.42

0.42

0.46

0.46

0.44

0.47

0.54

0.53

0.50

0.40

0.53

1.00

0.50

0.48

0.49

0.56

0.51

0.53

0.13

0.18

0.21

0.24

0.25

0.25

0.29

0.26

0.28

0.30

0.24

0.33

0.36

0.33

0.39

0.36

0.39

0.45

0.40

0.42

0.44

0.48

0.41

0.37

0.45

0.50

1.00

0.48

0.46

0.48

0.50

0.48

0.14

0.23

0.23

0.27

0.29

0.25

0.32

0.27

0.32

0.31

0.26

0.35

0.35

0.36

0.40

0.34

0.39

0.48

0.39

0.42

0.42

0.46

0.44

0.36

0.42

0.48

0.48

1.00

0.47

0.50

0.44

0.49

0.14

0.20

0.21

0.24

0.26

0.25

0.30

0.27

0.29

0.32

0.26

0.36

0.39

0.36

0.40

0.38

0.42

0.47

0.40

0.47

0.46

0.51

0.45

0.38

0.46

0.49

0.46

0.47

1.00

0.48

0.48

0.50

0.14

0.22

0.23

0.26

0.29

0.25

0.30

0.27

0.28

0.32

0.25

0.35

0.40

0.36

0.43

0.44

0.49

0.48

0.43

0.48

0.56

0.51

0.45

0.40

0.54

0.56

0.48

0.50

0.48

1.00

0.53

0.53

0.12

0.21

0.22

0.26

0.29

0.26

0.29

0.25

0.28

0.31

0.24

0.34

0.39

0.34

0.42

0.41

0.44

0.46

0.44

0.48

0.55

0.52

0.46

0.37

0.52

0.51

0.50

0.44

0.48

0.53

1.00

0.60

0.13

0.22

0.23

0.26

0.30

0.28

0.30

0.25

0.27

0.29

0.23

0.35

0.40

0.34

0.43

0.41

0.47

0.49

0.43

0.46

0.53

0.49

0.46

0.38

0.51

0.53

0.48

0.49

0.50

0.53

0.60

1.00

Mistral-7B-Instruct with ChunkKV

0.2

0.4

0.6

0.8

1.0

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

Layer

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

Layer

1.00

0.10

0.40

0.16

0.06

0.18

0.16

0.09

0.17

0.09

0.20

0.14

0.10

0.13

0.04

0.11

0.07

0.09

0.12

0.00

0.11

0.04

0.08

0.03

0.14

0.08

0.04

0.11

0.10

1.00

0.09

0.13

0.13

0.13

0.13

0.12

0.11

0.14

0.12

0.13

0.10

0.13

0.07

0.10

0.11

0.11

0.10

0.16

0.11

0.11

0.18

0.09

0.08

0.13

0.11

0.15

0.40

0.09

1.00

0.17

0.07

0.20

0.14

0.11

0.21

0.10

0.22

0.14

0.10

0.12

0.05

0.11

0.08

0.11

0.13

0.00

0.12

0.05

0.08

0.04

0.14

0.09

0.06

0.12

0.16

0.13

0.17

1.00

0.15

0.26

0.20

0.15

0.18

0.12

0.20

0.26

0.17

0.14

0.06

0.17

0.11

0.17

0.20

0.01

0.21

0.08

0.19

0.09

0.17

0.18

0.12

0.19

0.06

0.13

0.07

0.15

1.00

0.16

0.13

0.11

0.16

0.16

0.15

0.14

0.12

0.14

0.09

0.13

0.12

0.16

0.14

0.10

0.13

0.15

0.18

0.16

0.11

0.12

0.15

0.17

0.18

0.13

0.20

0.26

0.16

1.00

0.22

0.16

0.22

0.14

0.20

0.31

0.12

0.15

0.05

0.20

0.13

0.19

0.18

0.00

0.19

0.09

0.18

0.09

0.17

0.14

0.10

0.15

0.16

0.13

0.14

0.20

0.13

0.22

1.00

0.15

0.15

0.14

0.16

0.20

0.11

0.15

0.05

0.15

0.09

0.13

0.18

0.00

0.16

0.08

0.15

0.10

0.21

0.13

0.09

0.13

0.09

0.12

0.11

0.15

0.11

0.16

0.15

1.00

0.15

0.14

0.10

0.16

0.10

0.15

0.07

0.16

0.10

0.13

0.15

0.00

0.17

0.09

0.13

0.09

0.10

0.13

0.09

0.10

0.17

0.11

0.21

0.18

0.16

0.22

0.15

0.15

1.00

0.15

0.29

0.24

0.16

0.21

0.07

0.19

0.14

0.21

0.21

0.01

0.20

0.11

0.14

0.10

0.11

0.15

0.11

0.15

0.09

0.14

0.10

0.12

0.16

0.14

0.14

0.14

0.15

1.00

0.14

0.14

0.14

0.26

0.10

0.12

0.14

0.14

0.10

0.09

0.15

0.16

0.17

0.11

0.10

0.10

0.12

0.14

0.20

0.12

0.22

0.20

0.15

0.20

0.16

0.10

0.29

0.14

1.00

0.27

0.20

0.20

0.10

0.18

0.17

0.22

0.27

0.07

0.19

0.13

0.17

0.08

0.14

0.18

0.11

0.21

0.14

0.13

0.14

0.26

0.14

0.31

0.20

0.16

0.24

0.14

0.27

1.00

0.17

0.19

0.07

0.24

0.16

0.24

0.23

0.00

0.20

0.11

0.21

0.10

0.19

0.19

0.13

0.20

0.10

0.10

0.10

0.17

0.12

0.12

0.11

0.10

0.16

0.14

0.20

0.17

1.00

0.21

0.09

0.19

0.14

0.21

0.17

0.05

0.23

0.13

0.14

0.13

0.12

0.19

0.13

0.15

0.13

0.13

0.12

0.14

0.14

0.15

0.15

0.15

0.21

0.26

0.20

0.19

0.21

1.00

0.09

0.24

0.17

0.20

0.16

0.04

0.20

0.14

0.16

0.13

0.14

0.14

0.11

0.15

0.04

0.07

0.05

0.06

0.09

0.05

0.05

0.07

0.07

0.10

0.10

0.07

0.09

0.09

1.00

0.07

0.16

0.11

0.09

0.10

0.08

0.15

0.14

0.11

0.06

0.08

0.14

0.12

0.11

0.10

0.11

0.17

0.13

0.20

0.15

0.16

0.19

0.12

0.18

0.24

0.19

0.24

0.07

1.00

0.13

0.21

0.17

0.01

0.25

0.11

0.14

0.12

0.15

0.17

0.11

0.15

0.07

0.11

0.08

0.11

0.12

0.13

0.09

0.10

0.14

0.14

0.17

0.16

0.14

0.17

0.16

0.13

1.00

0.23

0.15

0.09

0.13

0.22

0.18

0.15

0.09

0.12

0.17

0.16

0.09

0.11

0.11

0.17

0.16

0.19

0.13

0.13

0.21

0.14

0.22

0.24

0.21

0.20

0.11

0.21

0.23

1.00

0.22

0.04

0.23

0.20

0.18

0.17

0.14

0.16

0.17

0.19

0.12

0.10

0.13

0.20

0.14

0.18

0.18

0.15

0.21

0.10

0.27

0.23

0.17

0.16

0.09

0.17

0.15

0.22

1.00

0.03

0.18

0.11

0.14

0.12

0.18

0.18

0.10

0.18

0.00

0.16

0.00

0.01

0.10

0.00

0.00

0.00

0.01

0.09

0.07

0.00

0.05

0.04

0.10

0.01

0.09

0.04

0.03

1.00

0.01

0.14

0.12

0.10

0.00

0.03

0.16

0.10

0.11

0.11

0.12

0.21

0.13

0.19

0.16

0.17

0.20

0.15

0.19

0.20

0.23

0.20

0.08

0.25

0.13

0.23

0.18

0.01

1.00

0.13

0.17

0.14

0.14

0.16

0.13

0.16

0.04

0.11

0.05

0.08

0.15

0.09

0.08

0.09

0.11

0.16

0.13

0.11

0.13

0.14

0.15

0.11

0.22

0.20

0.11

0.14

0.13

1.00

0.21

0.24

0.08

0.10

0.23

0.17

0.08

0.18

0.08

0.19

0.18

0.18

0.15

0.13

0.14

0.17

0.17

0.21

0.14

0.16

0.14

0.14

0.18

0.18

0.14

0.12

0.17

0.21

1.00

0.17

0.14

0.15

0.25

0.25

0.03

0.09

0.04

0.09

0.16

0.09

0.10

0.09

0.10

0.11

0.08

0.10

0.13

0.13

0.11

0.12

0.15

0.17

0.12

0.10

0.14

0.24

0.17

1.00

0.11

0.10

0.26

0.10

0.14

0.08

0.14

0.17

0.11

0.17

0.21

0.10

0.11

0.10

0.14

0.19

0.12

0.14

0.06

0.15

0.09

0.14

0.18

0.00

0.14

0.08

0.14

0.11

1.00

0.14

0.10

0.13

0.08

0.13

0.09

0.18

0.12

0.14

0.13

0.13

0.15

0.10

0.18

0.19

0.19

0.14

0.08

0.17

0.12

0.16

0.18

0.03

0.16

0.10

0.15

0.10

0.14

1.00

0.11

0.19

0.04

0.11

0.06

0.12

0.15

0.10

0.09

0.09

0.11

0.12

0.11

0.13

0.13

0.11

0.14

0.11

0.17

0.17

0.10

0.16

0.13

0.23

0.25

0.26

0.10

0.11

1.00

0.23

0.11

0.15

0.12

0.19

0.17

0.15

0.13

0.10

0.15

0.14

0.21

0.20

0.15

0.15

0.12

0.15

0.16

0.19

0.18

0.10

0.16

0.17

0.25

0.10

0.13

0.19

0.23

1.00

Qwen2-7B-Instruct with H2o

0.2

0.4

0.6

0.8

1.0

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

Layer

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

Layer

1.00

0.13

0.24

0.22

0.13

0.23

0.20

0.11

0.15

0.16

0.17

0.20

0.12

0.18

0.06

0.17

0.11

0.16

0.12

0.00

0.14

0.10

0.14

0.10

0.18

0.17

0.10

0.15

0.13

1.00

0.12

0.13

0.14

0.13

0.13

0.12

0.10

0.17

0.12

0.14

0.10

0.13

0.07

0.11

0.12

0.11

0.10

0.16

0.10

0.11

0.18

0.10

0.09

0.13

0.10

0.16

0.24

0.12

1.00

0.27

0.16

0.27

0.22

0.12

0.22

0.16

0.21

0.22

0.13

0.15

0.07

0.16

0.12

0.18

0.14

0.00

0.15

0.10

0.15

0.11

0.24

0.15

0.09

0.17

0.22

0.13

0.27

1.00

0.20

0.30

0.25

0.17

0.20

0.20

0.23

0.27

0.16

0.17

0.07

0.20

0.12

0.19

0.16

0.01

0.22

0.12

0.21

0.11

0.17

0.18

0.11

0.21

0.13

0.14

0.16

0.20

1.00

0.17

0.14

0.10

0.15

0.18

0.19

0.16

0.14

0.15

0.09

0.14

0.15

0.17

0.14

0.11

0.13

0.16

0.19

0.16

0.11

0.14

0.16

0.22

0.23

0.13

0.27

0.30

0.17

1.00

0.21

0.16

0.24

0.15

0.23

0.33

0.12

0.15

0.07

0.21

0.12

0.20

0.15

0.00

0.18

0.11

0.19

0.09

0.18

0.16

0.11

0.18

0.20

0.13

0.22

0.25

0.14

0.21

1.00

0.15

0.15

0.15

0.16

0.21

0.11

0.15

0.05

0.15

0.09

0.14

0.15

0.00

0.15

0.08

0.16

0.08

0.23

0.15

0.09

0.14

0.11

0.12

0.12

0.17

0.10

0.16

0.15

1.00

0.15

0.12

0.11

0.15

0.09

0.15

0.06

0.16

0.10

0.13

0.15

0.00

0.17

0.08

0.13

0.09

0.09

0.14

0.09

0.11

0.15

0.10

0.22

0.20

0.15

0.24

0.15

0.15

1.00

0.14

0.28

0.23

0.18

0.20

0.07

0.20

0.12

0.21

0.21

0.01

0.20

0.12

0.17

0.09

0.14

0.19

0.11

0.17

0.16

0.17

0.16

0.20

0.18

0.15

0.15

0.12

0.14

1.00

0.19

0.16

0.16

0.24

0.11

0.13

0.15

0.16

0.11

0.09

0.19

0.17

0.21

0.12

0.13

0.15

0.14

0.22

0.17

0.12

0.21

0.23

0.19

0.23

0.16

0.11

0.28

0.19

1.00

0.29

0.30

0.22

0.13

0.21

0.19

0.30

0.33

0.09

0.22

0.20

0.22

0.12

0.16

0.22

0.16

0.30

0.20

0.14

0.22

0.27

0.16

0.33

0.21

0.15

0.23

0.16

0.29

1.00

0.20

0.21

0.08

0.25

0.15

0.27

0.23

0.00

0.21

0.12

0.20

0.11

0.19

0.20

0.12

0.21

0.12

0.10

0.13

0.16

0.14

0.12

0.11

0.09

0.18

0.16

0.30

0.20

1.00

0.24

0.10

0.24

0.15

0.27

0.26

0.06

0.28

0.15

0.14

0.10

0.12

0.19

0.14

0.18

0.18

0.13

0.15

0.17

0.15

0.15

0.15

0.15

0.20

0.24

0.22

0.21

0.24

1.00

0.10

0.26

0.17

0.23

0.20

0.05

0.23

0.16

0.17

0.12

0.17

0.17

0.13

0.17

0.06

0.07

0.07

0.07

0.09

0.07

0.05

0.06

0.07

0.11

0.13

0.08

0.10

0.10

1.00

0.09

0.17

0.14

0.10

0.11

0.08

0.15

0.16

0.11

0.06

0.07

0.16

0.14

0.17

0.11

0.16

0.20

0.14

0.21

0.15

0.16

0.20

0.13

0.21

0.25

0.24

0.26

0.09

1.00

0.14

0.23

0.19

0.01

0.25

0.12

0.14

0.11

0.14

0.21

0.11

0.17

0.11

0.12

0.12

0.12

0.15

0.12

0.09

0.10

0.12

0.15

0.19

0.15

0.15

0.17

0.17

0.14

1.00

0.24

0.16

0.10

0.13

0.24

0.18

0.18

0.10

0.12

0.19

0.20

0.16

0.11

0.18

0.19

0.17

0.20

0.14

0.13

0.21

0.16

0.30

0.27

0.27

0.23

0.14

0.23

0.24

1.00

0.31

0.05

0.25

0.20

0.19

0.14

0.15

0.19

0.16

0.22

0.12

0.10

0.14

0.16

0.14

0.15

0.15

0.15

0.21

0.11

0.33

0.23

0.26

0.20

0.10

0.19

0.16

0.31

1.00

0.06

0.20

0.15

0.15

0.11

0.14

0.18

0.11

0.21

0.00

0.16

0.00

0.01

0.11

0.00

0.00

0.00

0.01

0.09

0.09

0.00

0.06

0.05

0.11

0.01

0.10

0.05

0.06

1.00

0.02

0.15

0.14

0.15

0.00

0.02

0.15

0.12

0.14

0.10

0.15

0.22

0.13

0.18

0.15

0.17

0.20

0.19

0.22

0.21

0.28

0.23

0.08

0.25

0.13

0.25

0.20

0.02

1.00

0.12

0.16

0.10

0.13

0.17

0.11

0.18

0.10

0.11

0.10

0.12

0.16

0.11

0.08

0.08

0.12

0.17

0.20

0.12

0.15

0.16

0.15

0.12

0.24

0.20

0.15

0.15

0.12

1.00

0.23

0.20

0.10

0.12

0.22

0.24

0.14

0.18

0.15

0.21

0.19

0.19

0.16

0.13

0.17

0.21

0.22

0.20

0.14

0.17

0.16

0.14

0.18

0.19

0.15

0.14

0.16

0.23

1.00

0.16

0.14

0.15

0.23

0.28

0.10

0.10

0.11

0.11

0.16

0.09

0.08

0.09

0.09

0.12

0.12

0.11

0.10

0.12

0.11

0.11

0.18

0.14

0.11

0.15

0.10

0.20

0.16

1.00

0.10

0.09

0.21

0.14

0.18

0.09

0.24

0.17

0.11

0.18

0.23

0.09

0.14

0.13

0.16

0.19

0.12

0.17

0.06

0.14

0.10

0.15

0.14

0.00

0.13

0.10

0.14

0.10

1.00

0.12

0.09

0.13

0.17

0.13

0.15

0.18

0.14

0.16

0.15

0.14

0.19

0.15

0.22

0.20

0.19

0.17

0.07

0.21

0.12

0.19

0.18

0.02

0.17

0.12

0.15

0.09

0.12

1.00

0.10

0.19

0.10

0.10

0.09

0.11

0.16

0.11

0.09

0.09

0.11

0.14

0.16

0.12

0.14

0.13

0.16

0.11

0.19

0.16

0.11

0.15

0.11

0.22

0.23

0.21

0.09

0.10

1.00

0.23

0.15

0.16

0.17

0.21

0.22

0.18

0.14

0.11

0.17

0.22

0.30

0.21

0.18

0.17

0.14

0.17

0.20

0.22

0.21

0.12

0.18

0.24

0.28

0.14

0.13

0.19

0.23

1.00

Qwen2-7B-Instruct with SnapKV

0.2

0.4

0.6

0.8

1.0

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

Layer

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

Layer

1.00

0.49

0.40

0.41

0.28

0.28

0.32

0.26

0.25

0.26

0.31

0.30

0.29

0.29

0.22

0.30

0.19

0.18

0.19

0.17

0.21

0.15

0.14

0.15

0.21

0.16

0.15

0.00

0.49

1.00

0.60

0.45

0.32

0.35

0.40

0.31

0.28

0.27

0.39

0.34

0.31

0.33

0.23

0.33

0.19

0.20

0.20

0.17

0.22

0.16

0.15

0.16

0.20

0.18

0.15

0.02

0.40

0.60

1.00

0.47

0.35

0.39

0.44

0.36

0.30

0.33

0.39

0.38

0.35

0.37

0.26

0.33

0.21

0.23

0.21

0.19

0.23

0.16

0.16

0.17

0.21

0.18

0.16

0.01

0.41

0.45

0.47

1.00

0.36

0.35

0.40

0.32

0.29

0.29

0.35

0.31

0.32

0.32

0.23

0.31

0.19

0.21

0.20

0.18

0.24

0.17

0.16

0.18

0.23

0.18

0.15

0.01

0.28

0.32

0.35

0.36

1.00

0.48

0.35

0.36

0.41

0.33

0.36

0.37

0.34

0.32

0.33

0.35

0.26

0.27

0.25

0.26

0.29

0.25

0.26

0.28

0.27

0.28

0.28

0.06

0.28

0.35

0.39

0.35

0.48

1.00

0.39

0.39

0.42

0.36

0.39

0.45

0.40

0.36

0.37

0.40

0.31

0.33

0.27

0.31

0.30

0.28

0.30

0.28

0.30

0.28

0.30

0.08

0.32

0.40

0.44

0.40

0.35

0.39

1.00

0.38

0.33

0.35

0.39

0.37

0.36

0.35

0.25

0.30

0.23

0.25

0.25

0.23

0.26

0.22

0.22

0.22

0.26

0.23

0.22

0.06

0.26

0.31

0.36

0.32

0.36

0.39

0.38

1.00

0.36

0.39

0.39

0.35

0.34

0.31

0.27

0.30

0.24

0.30

0.24

0.25

0.29

0.23

0.24

0.23

0.27

0.23

0.22

0.05

0.25

0.28

0.30

0.29

0.41

0.42

0.33

0.36

1.00

0.37

0.43

0.44

0.42

0.31

0.40

0.48

0.30

0.31

0.27

0.30

0.30

0.24

0.28

0.26

0.26

0.24

0.27

0.04

0.26

0.27

0.33

0.29

0.33

0.36

0.35

0.39

0.37

1.00

0.39

0.41

0.38

0.35

0.31

0.35

0.29

0.34

0.27

0.28

0.32

0.27

0.26

0.26

0.29

0.27

0.26

0.06

0.31

0.39

0.39

0.35

0.36

0.39

0.39

0.39

0.43

0.39

1.00

0.50

0.48

0.43

0.36

0.44

0.33

0.34

0.30

0.31

0.33

0.24

0.25

0.23

0.29

0.23

0.23

0.03

0.30

0.34

0.38

0.31

0.37

0.45

0.37

0.35

0.44

0.41

0.50

1.00

0.53

0.42

0.45

0.46

0.35

0.36

0.32

0.34

0.37

0.28

0.28

0.28

0.29

0.25

0.27

0.05

0.29

0.31

0.35

0.32

0.34

0.40

0.36

0.34

0.42

0.38

0.48

0.53

1.00

0.52

0.46

0.53

0.38

0.40

0.37

0.36

0.38

0.28

0.27

0.27

0.29

0.25

0.25

0.03

0.29

0.33

0.37

0.32

0.32

0.36

0.35

0.31

0.31

0.35

0.43

0.42

0.52

1.00

0.39

0.47

0.31

0.37

0.35

0.30

0.36

0.25

0.23

0.24

0.28

0.23

0.22

0.02

0.22

0.23

0.26

0.23

0.33

0.37

0.25

0.27

0.40

0.31

0.36

0.45

0.46

0.39

1.00

0.49

0.51

0.43

0.39

0.46

0.39

0.34

0.36

0.34

0.35

0.29

0.34

0.05

0.30

0.33

0.33

0.31

0.35

0.40

0.30

0.30

0.48

0.35

0.44

0.46

0.53

0.47

0.49

1.00

0.40

0.43

0.36

0.36

0.38

0.29

0.30

0.28

0.32

0.25

0.28

0.02

0.19

0.19

0.21

0.19

0.26

0.31

0.23

0.24

0.30

0.29

0.33

0.35

0.38

0.31

0.51

0.40

1.00

0.47

0.41

0.55

0.40

0.42

0.43

0.40

0.38

0.33

0.38

0.09

0.18

0.20

0.23

0.21

0.27

0.33

0.25

0.30

0.31

0.34

0.34

0.36

0.40

0.37

0.43

0.43

0.47

1.00

0.40

0.46

0.42

0.35

0.41

0.34

0.35

0.31

0.34

0.07

0.19

0.20

0.21

0.20

0.25

0.27

0.25

0.24

0.27

0.27

0.30

0.32

0.37

0.35

0.39

0.36

0.41

0.40

1.00

0.47

0.41

0.38

0.35

0.35

0.34

0.31

0.30

0.09

0.17

0.17

0.19

0.18

0.26

0.31

0.23

0.25

0.30

0.28

0.31

0.34

0.36

0.30

0.46

0.36

0.55

0.46

0.47

1.00

0.45

0.51

0.48

0.42

0.42

0.35

0.41

0.10

0.21

0.22

0.23

0.24

0.29

0.30

0.26

0.29

0.30

0.32

0.33

0.37

0.38

0.36

0.39

0.38

0.40

0.42

0.41

0.45

1.00

0.36

0.35

0.32

0.38

0.28

0.29

0.05

0.15

0.16

0.16

0.17

0.25

0.28

0.22

0.23

0.24

0.27

0.24

0.28

0.28

0.25

0.34

0.29

0.42

0.35

0.38

0.51

0.36

1.00

0.52

0.49

0.47

0.44

0.47

0.11

0.14

0.15

0.16

0.16

0.26

0.30

0.22

0.24

0.28

0.26

0.25

0.28

0.27

0.23

0.36

0.30

0.43

0.41

0.35

0.48

0.35

0.52

1.00

0.56

0.48

0.44

0.55

0.11

0.15

0.16

0.17

0.18

0.28

0.28

0.22

0.23

0.26

0.26

0.23

0.28

0.27

0.24

0.34

0.28

0.40

0.34

0.35

0.42

0.32

0.49

0.56

1.00

0.47

0.48

0.55

0.14

0.21

0.20

0.21

0.23

0.27

0.30

0.26

0.27

0.26

0.29

0.29

0.29

0.29

0.28

0.35

0.32

0.38

0.35

0.34

0.42

0.38

0.47

0.48

0.47

1.00

0.43

0.45

0.10

0.16

0.18

0.18

0.18

0.28

0.28

0.23

0.23

0.24

0.27

0.23

0.25

0.25

0.23

0.29

0.25

0.33

0.31

0.31

0.35

0.28

0.44

0.44

0.48

0.43

1.00

0.53

0.15

0.15

0.15

0.16

0.15

0.28

0.30

0.22

0.22

0.27

0.26

0.23

0.27

0.25

0.22

0.34

0.28

0.38

0.34

0.30

0.41

0.29

0.47

0.55

0.55

0.45

0.53

1.00

0.17

0.00

0.02

0.01

0.01

0.06

0.08

0.06

0.05

0.04

0.06

0.03

0.05

0.03

0.02

0.05

0.02

0.09

0.07

0.09

0.10

0.05

0.11

0.11

0.14

0.10

0.15

0.17

1.00

Qwen2-7B-Instruct with ChunkKV

0.2

0.4

0.6

0.8

1.0

Method

Single-Document QA

Multi-Document QA

Summarization

Few-shot Learning

Synthetic

Code

↑

NrtvQA

Qasper

MF-en

HotpotQA

2WikiMQA

Musique

GovReport

QMSum

MultiNews

TREC

TriviaQA

SAMSum

PCount

PRe

Lcc

RB-P

Avg len

18,409

3,619

4,559

9,151

4,887

11,214

8,734

10,614

2,113

5,177

8,209

6,258

11,141 9,289

1,235

4,206

LlaMa-3-8B-Instruct, KV Size = Full

FullKV

25.70

29.75

41.12

45.55

35.87

22.35

25.63

23.03

26.21

73.00

90.56

41.88

4.67

69.25

58.05

50.77

41.46

LlaMa-3-8B-Instruct, KV Size Compression Ratio =

10%

StreamingLLM

20.62

13.09

22.10

36.31

28.01

15.61

21.47

21.05

19.39

62.00

84.18

40.27

4.62

69.10

58.84

55.26

35.74

H2O

24.80

17.32

31.80

40.84

33.28

18.90

22.29

22.29

21.82

40.00

90.51

40.55

5.79

69.50

58.04

55.26

37.06

SnapKV

25.08

22.02

37.95

43.36

35.08

20.29

22.94

22.64

21.37

71.00

90.47

40.15

5.66

69.25

58.69

56.50

40.15

PyramidKV

25.58

20.77

35.85

43.80

33.03

21.45

23.68

22.26

21.85

71.50

90.47

41.66

5.84

69.25

58.52

55.91

40.08

ChunkKV

24.89

22.96

37.64

43.27

36.45

20.65

22.80

22.97

20.82

71.50

90.52

40.83

5.93

69.00

60.49

57.48

40.51

LlaMa-3-8B-Instruct, KV Size Compression Ratio =

20%

StreamingLLM

23.35

18.97

32.94

42.39

29.37

18.76

25.78

21.92

25.16

71.00

88.85

40.82

5.04

69.00

56.46

51.12

38.80

H2O

25.60

21.88

35.36

42.06

32.68

19.72

23.54

22.77

22.72

45.50

90.57

41.67

5.51

69.25

54.97

50.95

37.79

SnapKV

25.50

25.95

38.43

44.12

35.38

20.49

24.85

23.36

23.51

72.50

90.52

40.91

5.23

69.25

56.74

51.75

40.53

PyramidKV

25.36

26.88

37.99

44.21

35.65

21.43

25.52

23.43

23.47

72.00

90.56

41.45

5.26

69.50

56.55

50.93

40.63

ChunkKV

26.13

28.43

38.59

44.46

34.13

21.06

24.72

23.11

22.91

71.50

90.56

41.51

5.09

69.00

58.17

52.51

40.74

LlaMa-3-8B-Instruct, KV Size Compression Ratio =

30%

StreamingLLM

24.49

22.53

35.30

44.33

32.81

19.00

27.12

22.19

25.93

72.50

89.84

41.75

5.41

69.00

60.40

55.13

40.48

H2O

25.87

23.03

37.06

43.71

33.68

20.93

24.56

23.14

23.58

50.50

90.77

41.96

4.91

69.25

59.38

55.39

39.23

SnapKV

25.15

28.75

39.28

43.57

36.16

21.58

25.56

23.19

24.30

73.00

90.52

41.70

4.96

69.25

60.27

55.74

41.43

PyramidKV

25.42

27.91

38.81

44.15

36.28

21.72

26.50

23.10

24.28

72.00

90.56

41.87

4.67

69.50

60.09

55.19

41.37

ChunkKV

25.88

29.58

38.99

43.94

34.16

21.70

26.50

23.15

23.95

72.00

90.56

42.47

5.34

69.25

61.68

56.35

41.59

FullKV

29.07

41.58

52.88

49.37

39.01

28.58

34.93

25.68

27.74

76.00

88.59

47.59

6.00

98.50

61.41

62.39

48.08

10%

StreamingLLM

25.15

25.47

30.08

44.39

32.49

19.40

24.11

20.85

19.55

65.00

88.21

44.83

4.50

79.50

59.48

58.82

40.11

H2O

29.35

33.39

50.39

49.58

36.76

27.42

25.16

24.75

22.12

42.00

89.00

47.04

5.50

98.50

57.58

59.24

43.61

SnapKV

28.54

36.88

53.42

50.15

38.17

27.99

26.67

25.21

22.33

72.00

89.36

45.44

5.50

99.00

59.79

61.63

46.38

PyramidKV

29.40

35.39

52.96

49.93

38.67

28.63

27.59

24.99

22.77

74.00

90.02

46.07

4.00

98.50

58.54

60.88

46.39

ChunkKV

29.75

36.82

53.99

50.33

38.72

29.01

27.03

24.76

21.42

76.00

88.73

46.49

5.00

98.00

59.98

61.47

46.71

Qwen2-7B-Instruct, KV Size = Full

FullKV

25.11

42.64

44.29

14.25

13.22

9.08

36.38

23.43

26.53

77.00

89.99

44.88

6.75

75.92

60.17

61.84

40.71

Qwen2-7B-Instruct, KV Size Compression Ratio =

10%

StreamingLLM

25.15

45.42

41.46

13.66

11.95

8.72

32.79

21.49

26.24

77.50

89.15

44.54

7.50

50.50

60.03

60.91

38.56

H2O

26.17

44.33

42.54

12.81

12.46

9.15

33.24

22.69

25.94

76.50

89.44

44.32

8.00

76.00

61.28

62.39

40.45

SnapKV

26.84

45.96

45.79

14.27

13.35

9.91

32.62

22.70

25.83

77.00

89.19

44.71

7.50

71.50

60.35

61.37

40.55

PyramidKV

27.51

44.45

43.59

13.35

13.13

9.12

32.28

22.60

25.45

77.00

89.44

44.53

7.00

73.50

60.91

61.24

40.31

ChunkKV

26.48

44.19

45.04

15.94

12.60

10.52

32.38

22.87

25.91

77.50

89.22

44.78

8.50

76.50

60.64

61.32

40.88

Method

Single-Document QA

Multi-Document QA

Summarization

Few-shot Learning

Synthetic

Code

↑

NrtvQA

Qasper

MF-en

HotpotQA

2WikiMQA

Musique

GovReport

QMSum

MultiNews

TREC

TriviaQA

SAMSum

PCount

PRe

Lcc

RB-P

Avg len

18,409

3,619

4,559

9,151

4,887

11,214

8,734

10,614

2,113

5,177

8,209

6,258

11,141 9,289

1,235

4,206

LlaMa-3-70B-Instruct, KV Size = Full

FullKV

27.56

46.00

49.75

50.86

56.16

29.77

31.66

21.80

27.42

75.50

92.58

46.08

12.00

73.50

45.02

68.74

47.15

LlaMa-3-70B-Instruct, KV Cache Size = 256

H2O

26.42

37.52

36.23

47.82

49.34

24.30

26.96

20.67

26.50

72.00

92.16

43.56

12.00

73.00

47.21

63.13

43.67

SnapKV

26.79

42.23

48.78

50.89

55.46

28.14

23.85

22.20

24.62

73.00

93.05

44.53

12.00

72.50

54.24

65.60

46.11

ChunkKV

27.37

45.95

49.23

50.51

55.49

30.28

30.47

22.63

26.30

75.50

92.58

46.51

12.00

73.50

49.51

68.05

47.24

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

2000

2100

2200

2300

2400

2500

2600

2700

2800

2900

3000

3100

3200

3300

3400

3500

3600

3700

3800

3900

4000

4100

4200

4300

4400

4500

4600

4700

4800

4900

5000

5100

5200

5300

5400

5500

5600

5700

5800

5900

6000

6100

6200

6300

6400

6500

6600

6700

6800

6900

7000

7100

7200

7300

7400

7500

7600

7700

7800

7900

8000

Token Limit

0.0

11.0

22.0

33.0

44.0

56.0

67.0

78.0

89.0

100.0

0.0

0.2

0.4

0.6

0.8

1.0

Score

(a) ChunkKV, accuracy 73.8%

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

2000

2100

2200

2300

2400

2500

2600

2700

2800

2900

3000

3100

3200

3300

3400

3500

3600

3700

3800

3900

4000

4100

4200

4300

4400

4500

4600

4700

4800

4900

5000

5100

5200

5300

5400

5500

5600

5700

5800

5900

6000

6100

6200

6300

6400

6500

6600

6700

6800

6900

7000

7100

7200

7300

7400

7500

7600

7700

7800

7900

8000

Token Limit

0.0

11.0

22.0

33.0

44.0

56.0

67.0

78.0

89.0

100.0

0.0

0.2

0.4

0.6

0.8

1.0

Score

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

2000

2100

2200

2300

2400

2500

2600

2700

2800

2900

3000

3100

3200

3300

3400

3500

3600

3700

3800

3900

4000

4100

4200

4300

4400

4500

4600

4700

4800

4900

5000

5100

5200

5300

5400

5500

5600

5700

5800

5900

6000

6100

6200

6300

6400

6500

6600

6700

6800

6900

7000

7100

7200

7300

7400

7500

7600

7700

7800

7900

8000

Token Limit

0.0

11.0

22.0

33.0

44.0

56.0

67.0

78.0

89.0

100.0

0.0

0.2

0.4

0.6

0.8

1.0

Score

(c) SnapKV, accuracy 58.9%

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

2000

2100

2200

2300

2400

2500

2600

2700

2800

2900

3000

3100

3200

3300

3400

3500

3600

3700

3800

3900

4000

4100

4200

4300

4400

4500

4600

4700

4800

4900

5000

5100

5200

5300

5400

5500

5600

5700

5800

5900

6000

6100

6200

6300

6400

6500

6600

6700

6800

6900

7000

7100

7200

7300

7400

7500

7600

7700

7800

7900

8000

Token Limit

0.0

11.0

22.0

33.0

44.0

56.0

67.0

78.0

89.0

100.0

0.0

0.2

0.4

0.6

0.8

1.0

Score

(d) H2O, accuracy 47.9%

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

2000

2100

2200

2300

2400

2500

2600

2700

2800

2900

3000

3100

3200

3300

3400

3500

3600

3700

3800

3900

4000

4100

4200

4300

4400

4500

4600

4700

4800

4900

5000

5100

5200

5300

5400

5500

5600

5700

5800

5900

6000

6100

6200

6300

6400

6500

6600

6700

6800

6900

7000

7100

7200

7300

7400

7500

7600

7700

7800

7900

8000

Token Limit

0.0

11.0

22.0

33.0

44.0

56.0

67.0

78.0

89.0

100.0

0.0

0.2

0.4

0.6

0.8

1.0

Score

(e) StreamingLLM, accuracy 23.7%

400

800

1200

1600

2000

2400

2800

3200

3600

4000

4400

4800

5200

5600

6000

6400

6800

7200

7600

8000

8400

8800

9200

9600

10000

10400

10800

11200

11600

12000

12400

12800

13200

13600

14000

14400

14800

15200

15600

16000

16400

16800

17200

17600

18000

18400

18800

19200

19600

20000

20400

20800

21200

21600

22000

22400

22800

23200

23600

24000

24400

24800

25200

25600

26000

26400

26800

27200

27600

28000

28400

28800

29200

29600

30000

30400

30800

31200

31600

32000

Token Limit

0.0

11.0

22.0

33.0

44.0

56.0

67.0

78.0

89.0

100.0

0.0

0.2

0.4

0.6

0.8

1.0

Score

(a) ChunkKV, accuracy 99.8%

400

800

1200

1600

2000

2400

2800

3200

3600

4000

4400

4800

5200

5600

6000

6400

6800

7200

7600

8000

8400

8800

9200

9600

10000

10400

10800

11200

11600

12000

12400

12800

13200

13600

14000

14400

14800

15200

15600

16000

16400

16800

17200

17600

18000

18400

18800

19200

19600

20000

20400

20800

21200

21600

22000

22400

22800

23200

23600

24000

24400

24800

25200

25600

26000

26400

26800

27200

27600

28000

28400

28800

29200

29600

30000

30400

30800

31200

31600

32000

Token Limit

0.0

11.0

22.0

33.0

44.0

56.0

67.0

78.0

89.0

100.0

0.0

0.2

0.4

0.6

0.8

1.0

Score

400

800

1200

1600

2000

2400

2800

3200

3600

4000

4400

4800

5200

5600

6000

6400

6800

7200

7600

8000

8400

8800

9200

9600

10000

10400

10800

11200

11600

12000

12400

12800

13200

13600

14000

14400

14800

15200

15600

16000

16400

16800

17200

17600

18000

18400

18800

19200

19600

20000

20400

20800

21200

21600

22000

22400

22800

23200

23600

24000

24400

24800

25200

25600

26000

26400

26800

27200

27600

28000

28400

28800

29200

29600

30000

30400

30800

31200

31600

32000

Token Limit

0.0

11.0

22.0

33.0

44.0

56.0

67.0

78.0

89.0

100.0

0.0

0.2

0.4

0.6

0.8

1.0

Score

(c) SnapKV, accuracy 91.6%

400

800

1200

1600

2000

2400

2800

3200

3600

4000

4400

4800

5200

5600

6000

6400

6800

7200

7600

8000

8400

8800

9200

9600

10000

10400

10800

11200

11600

12000

12400

12800

13200

13600

14000

14400

14800

15200

15600

16000

16400

16800

17200

17600

18000

18400

18800

19200

19600

20000

20400

20800

21200

21600

22000

22400

22800

23200

23600

24000

24400

24800

25200

25600

26000

26400

26800

27200

27600

28000

28400

28800

29200

29600

30000

30400

30800

31200

31600

32000

Token Limit

0.0

11.0

22.0

33.0

44.0

56.0

67.0

78.0

89.0

100.0

0.0

0.2

0.4

0.6

0.8

1.0

Score

(d) H2O, accuracy 88.2%

400

800

1200

1600

2000

2400

2800

3200

3600

4000

4400

4800

5200

5600

6000

6400

6800

7200

7600

8000

8400

8800

9200

9600

10000

10400

10800

11200

11600

12000

12400

12800

13200

13600

14000

14400

14800

15200

15600

16000

16400

16800

17200

17600

18000

18400

18800

19200

19600

20000

20400

20800

21200

21600

22000

22400

22800

23200

23600

24000

24400

24800

25200

25600

26000

26400

26800

27200

27600

28000

28400

28800

29200

29600

30000

30400

30800

31200

31600

32000

Token Limit

0.0

11.0

22.0

33.0

44.0

56.0

67.0

78.0

89.0

100.0

0.0

0.2

0.4

0.6

0.8

1.0

Score

(e) StreamingLLM, accuracy 44.3%

B.4 Chunk Size

Table 19 and 20 show the performance of ChunkKV with different comperession ratios and different chunk sizes on the LongBench and NIAH. We conducted extensive experiments across different compression ratios and KV cache sizes to shows the effectiveness of ChunkKV and the chunk size is robust.

LongBench Performance vs Chunk Size

1

3

5

10

20

30

36

38

40

42

44

46

48

LongBench Score

LLaMA-3-8B

Mistral-7B

Qwen2-7B

LLaMA-3-8B Full KV

Mistral-7B Full KV

Qwen2-7B Full KV

Figure 18 shows the performance of the ChunkKV with different chunk size on the LongBench and NIAH benchmarks. The three colorful curves represent three LLMs with different chunk sizes, and the colorful dashed line is the corresponding FullKV performance. From Figure 18, we can observe that the LongBench performance of ChunkKV is not significantly affected by the chunk size, with

performance variations less than 1%. The three curves are closely aligned, indicating that chunk sizes in the range of { 10 , 20 } exhibit better performance.

Table 21 and Figure 19 shows that the ChunkKV with different chunk sizes on GSM8K displays the same curve pattern as LongBench. The CoT prompt length for GSM8K is only 1K tokens, so the optimal chunk size range is smaller.

3

5

10

20

60.0

62.5

65.0

67.5

70.0

72.5

75.0

77.5

80.0

GSM8K Score

LLaMA-3-8B-Instruct 80.0 Qwen2-7B-Instruct GSM8K Performance vs Chunk Size

Full KV

3

5

10

20

60.0

62.5

65.0

67.5

70.0

72.5

75.0

77.5

GSM8K Score

Full KV

B.5 Multi-Lingual

Table 22 is the Chinese support model Qwen2-7B-Instruct evaluated on the LongBench Chinese subtask, where ChunkKV achieves better performance than other compression methods and the full KV cache performance. Both the English and Chinese results indicate that ChunkKV is a promising approach for maintaining crucial information in the KV cache.

B.6 KV Cache Quantization

For comprehensively evaluate the effectiveness of ChunkKV, we conducted experiments comparing ChunkKV with quantization methods KIVI [45]. While both approaches aim to optimize LLM

inference, they operate on fundamentally different principles: quantization reduces KV matrix precision, whereas our eviction method reduces KV matrix size.

From an implementation perspective, quantization methods require the full KV cache during prefilling to produce quantized representations, which are then used during decoding. In contrast, ChunkKV employs token removal prior to prefilling, enabling operation with a compressed cache throughout the entire inference process. This distinction creates different efficiency profiles, and each method offers unique advantages.

The following tables present our detailed analysis comparing ChunkKV with KIVI, across both performance and efficiency metrics on the LLaMA-3-8B-Instruct model. It should be noted that due to KIVI's dependency on older Python versions, there may be some discrepancies between the efficiency results reported here and those in Table 8 of our paper.

Table 23 demonstrates that ChunkKV with 30% compression ratio achieves comparable performance (41.59) to KIVI with 8-bit quantization (41.43) across LongBench subtasks. Notably, ChunkKV shows particularly strong results in code-related tasks, while KIVI maintains better performance in single-document QA scenarios.

These results highlight the complementary strengths of eviction and quantization approaches. ChunkKV's semantic chunk-based method delivers particularly strong efficiency benefits while maintaining competitive performance with state-of-the-art quantization techniques.

B.7 Comparison with Orthogonal and Training-Based Methods

To better position ChunkKV in the broader landscape of inference optimization, we conducted additional experiments comparing our training-free eviction method against Palu [46], a recent training-based compression method. The experiments were conducted on the LongBench benchmark with the LLaMA-3-8B-Instruct model.

The results in Table 24 highlight key differences. Palu, as a training-based method, experiences significant performance degradation on the diverse LongBench benchmark, which represents an Out-Of-Distribution (OOD) challenge relative to its training data. In contrast, our training-free method, ChunkKV, demonstrates robust performance.

B.8 Efficiency results

Table 25 shows the efficiency results for different KV cache strategies with varying output lengths and the metrics are Time to First Token (TTFT) and Token Processing Time (TPOT).

C Theoretical Understanding

In this section, we provide the theoretical interpretation from the perspective from the In-context learning (ICL) to further understand how ChunkKV outperforms token-level KV cache compression.

Pretraining Data Distribution. Given a set of concepts Θ and a concept θ ∈ Θ , we define the pretraining data is sampled from p ( o 1 , . . . , o T ) = ∫ θ ∈ Θ p ( o 1 , . . . , o T | θ ) p ( θ ) dθ [24]. Each token o is sampled from a vocabulary O . For simplicity, we write o 1: t = o 1 . . . o t .

Language Modeling. Current LLMs [1, 5, 24] usually utilize the next word prediction as the language modelling, which predicts the next token o t given the previous tokens o 1 . . . o t -1 for all t = 1 , . . . , T . Formally, a language modelling can be writen as the distribution f ( o t | o 1: t -1 ) . And it is pretrained on

a huge corpus sampled from the pretraining distribution p ( o 1 , . . . , o t +1 ) [24]. Considering the large scale of the model size and dataset size, it can be assumed that the f ( o 1 . . . o t +1 ) has been aligned with the p ( o 1 . . . o t +1 ) [24].

Prompt Distribution. Following [24], a prompt is composed of an input token sequence x followed by an output token y . Then, the i -th training example † that can appear in any place in the whole prompt o 1: T is defined as O i consisting of an input x i = O i [1 : k -1] (the first k -1 tokens) followed by the output y i = O i [ k ] at the end, where the length k is fixed for simplicity.

The i -th training example is independently generated as follows: 1) Generate a start hidden state h start i from a prompt start distribution p prompt; 2) Given h start i , generate the example sequence O i = [ x i , y i ] from p ( O i | h start i , θ ⋆ ) . The test input x test = x n +1 is sampled similarly. Then, the prompt consists of a sequence of training examples ( S n ) followed by the example x test :

In-context learning setups and Assumptions. We follow other settings and assumptions in [24]. With the greedy decoding [47], sampling the next token from the language modeling f ( o t | o 1: t -1 ) becomes the predictor as y = arg max o t f ( o t | o 1: t -1 ) .

̸

Thus, for [ S n , x test ] , the in-context learning predictor can be written as f n ( x test ) := arg max y p ( y | S n , x test ) , which outputs the most likely prediction over the pretraining distribution conditioned on the prompt distribution . Its expected 0-1 error with n examples is L 0-1 ( f n ) = E x test ,y test ∼ p prompt [ 1 [ f n ( x test ) = y test ]] .

We define p i θ ( o ) := p ( O [ i ] = o | O [1 : i -1] , θ ) of the i -th token with previous tokens and the analogous distribution p i prompt := p prompt ( O [ i ] = o | O [1 : i -1]) under the prompt distribution. Following [24], there is a distinguishability condition formalizes when in-context learning occurs giving the concept θ .

The distinguishability condition is dependent on a KL divergence between the previous two distributions and the error terms ϵ θ resulting from the distribution mismatch between the prompt and the pertaining distributions for each example. Letting p i θ ( o ) and p i prompt correspond to the concept θ and θ ⋆ .

̸

Condition 1 (distinguishability [24]) . The θ ⋆ is distinguishable if for all θ ∈ Ω , θ = θ ⋆ ,

where the KL i ( θ ⋆ || θ ) := E O [1: i -1] ∼ p prompt [ KL ( p i prompt || p i θ )] .

Noises from KV Cache Compression. Naturally, because of the sparsified KV cache, some history tokens in o 1: t -1 at different layers lost its attention score calculation with respect to the next word prediction o t . We can regard this as the noise added onto the o 1: t -1 . Thus, distincting θ ⋆ from θ requires larger KL divergence. Following [48], we provide the following second condition about the distinguishability with the KV cache sparsity.

̸

Condition 2 (distinguishability under sparsified KV cache) . With the noise introduced by the sparsified KV cache of the sparse ratio r , the distribution mismatch between the prompt and the pretraining distribution that is approximated by LLM is enlarged, resulting in a varied requirement with error term ξ θ ( r ) for θ ∗ being distinguishable if for all θ ∈ Θ , θ = θ ∗ ,

Lemma 1 (noisy-relaxed bound in [24, 48]) . let B denotes the set of θ which does not satisfy Condition 1. We assume that KL ( p prompt ( y test | x test )) || p ( y test | x test , θ ) is bounded for all θ and that θ ⋆ minimizes the multi-class logistic risk as,

If then

where g ( ν ) = 1 2 ( (1 -ν ) log(1 -ν ) + (1 + ν ) log(1 + ν ) ) is the calibration function [49, 50] for the multiclass logistic loss for ν ∈ [0 , 1] .

Following [24, 51], KL divergence is assumed to haver the 2nd-order Taylor expansion with the concept θ . Then, we have the following theorem and proof.

Theorem 1. [24, 48] Let the set of θ which does not satisfy Equation 3 in Condition 1 to be B . Assume that KL divergences have a 2nd-order Taylor expansion around θ ⋆ :

where I j,θ ∗ is the Fisher information matrix of the j -th token distribution with respect to θ ⋆ . Let γ θ ∗ = max j λ max ( I j,θ ∗ ) min jλ min ( I j,θ ∗ ) where λ max , λ min return the largest and smallest eigenvalues. Then for k ≥ 2 and as n →∞ , the 0-1 risk of the in-context learning predictor f n is bounded as

Proof. [24] By the Taylor expansion on θ , we have for any θ in B that

We can bound the last KL term ( k -th token) with the above term:

Rearranging above equation, and with KL k ( θ ⋆ || θ ) = E x test ∼ p prompt [ KL ( p prompt ( y test | x test ) ∥ p ( y test | x test , θ ))] , there is

Combining Equation 16 with Equation 6 into Lemma 1 completes the proof.

KVCache Sparsification. Revisiting the Equation 4 in Condition 2, the ξ θ ( r ) is enlarged with the sparsity ratio r . The higher compression ratio r (means that more KV cache are discarded), the more noise ξ θ ( r ) . Then it leads to the higher bound of the lim n →∞ L 0 -1 ( f n ) in Equation 5 in Lemma 1. Next, we discuss how KV cache compression influences the Equation 4.

Token-level Importance Measure. The token-level KV cache methods usually calculate the importance of different tokens. Then, the KV cache with indexes that have higher importance will

be preserved. Such indexes are normaly choosed as the attention score. Considering that the case in Figure 1, where each token in the i -th training † example sequence ( O i = [ x i , y i ] ) might be compressed, and tokens are sparsified concretely without out dependency to other tokens. However, in the generation process of the i -th training example, O i = [ x i , y i ] is sampled from p ( O i | h start i , θ ⋆ ) and p j θ ( o ) := p ( O [ j ] = o | O [1 : j -1] , θ ) of the j -th token with previous tokens and the analogous distribution p j prompt := p prompt ( O [ j ] = o | O [1 : j -1]) . And the KL divergence is defined as KL j ( θ ⋆ || θ ) := E O [1: j -1] ∼ p prompt [ KL ( p j prompt || p j θ )] , which means that in a training example O i = [ x i , y i ] = O i [1 : k ] , each token O i [ j ] has strong dependency with O i [1 : j -1] , noises on previous any j -th token will influence the distinguishability on the following tokens, i.e. requiring larger { KL u ( θ ⋆ || θ ) } u>j .

Onthe other hand, the token-level sparsity enlarges the requirement on the distinguishability uniformly for each example O i (the case in Figure 1), which uniformly loses the bound of L 0-1 ( f n ) as in Equation 9.

Chunk-level Importance Measure. Different from token-level importance measure, ChunkKV regards tokens in a continuous window as a basic unit that should be left or discarded as a whole. The preserved window can be regarded as saving the complete O i = [ x i , y i ] without noise. Thus, ChunkKV reduces the noise ξ θ ( r ) for the preserved O i , which lowers the bound of L 0-1 ( f n ) .

More intuitively, ChunkKV focus on reducing the noise on some complete training examples, but some other examples overall with low importance will be discarded. Then, the model identifies the x test from those clean and more related training examples O i and neglect those O i with less importance.

Note that here, we do not provide the rigorous proof on how KV cache sparsity enhances the requirement of the distinguishability and how different KL j ( θ ⋆ || θ ) on O i = [ x i , y i ] influences the bound L 0-1 ( f n ) . We left this as the future work to analyze how KV cache sparsity influences the in-context learning.

D Additional Related Work

Chunking Method. The chunking methodology is widely used in the field of NLP due to its simplicity and effectiveness [52]. In the era of LLMs, chunking is primarily applied in data preprocessing. For example, Shi et al. [53] suggest grouping related training data into chunks to achieve better convergence curves to pre-train LLMs. Fei et al. [54] apply a topic-based chunking method to improve the semantic compression of prompts. Furthermore, chunking plays an important role in the Retrieval-Augmented Generation (RAG) field [55-57]. It serves to divide documents into units of information with semantic content suitable for embedding-based retrieval and processing by LLMs.

Layer-Wise Technique. The layer-wise technique is widely used in the training and inference of large language models (LLMs). LISA [58] is a layer-wise sampling method based on observations of the training dynamics of Low-Rank Adaptation (LoRA)[35, 36, 38, 59, 60] across layers. LAMB[61] is a layer-wise adaptive learning rate method that speeds up LLM training by stabilizing training convergence with large batch sizes. DoLa [62] employs layer-wise contrasting to reduce hallucinations during LLM inference.

KV Cache Sharing. Recent work has explored various strategies for sharing KV caches across transformer layers. Layer-Condensed KV Cache (LCKV) [63] computes KVs only for the top layer and pairs them with queries from all layers, while optionally retaining standard attention for a few top and bottom layers to mitigate performance degradation. Similarly, You Only Cache Once (YOCO) [64] computes KVs exclusively for the top layer but pairs them with queries from only the top half of layers, employing efficient attention in the bottom layers to maintain a constant cache size. In contrast, Cross-Layer Attention (CLA) [65] divides layers into groups, pairing queries from all layers in each group with KVs from that group's bottom layer. MiniCache [66] introduces a novel method that merges layer-wise KV caches while enabling recovery during compute-in-place operations, optimizing KV cache size. These methods illustrate various trade-offs between computation, memory usage, and model performance when sharing KV caches across transformer layers.

Long-Context Benchmarks. The landscape of long-context model benchmarks has evolved to encompass a wide range of tasks, with particular emphasis on retrieval and comprehension capabilities. Benchmarks for understanding have made significant strides, with ∞ -Bench [67] pushing the boundaries by presenting challenges that involve more than 100,000 tokens. LongBench [25] has introduced bilingual evaluations, addressing tasks such as long-document question answering, summarization, and code completion. Complementing these efforts, ZeroSCROLLS [68] and L-Eval [69] have broadened the scope to include a diverse array of practical natural language tasks, including query-driven summarization.

In parallel, retrieval benchmarks have largely relied on synthetic datasets, offering researchers precise control over variables such as the length of input tokens. This approach minimizes the impact of disparate parametric knowledge resulting from varied training methodologies. A significant body of recent work has concentrated on the development of synthetic tasks specifically for retrieval evaluation [26, 70-73]. In addition, researchers have explored the potential of extended contexts in facilitating various forms of reasoning [74]. Liu et al. [75] explores the influence of the KV cache method beyond long-context benchmarks. LongGenBench [76] advances beyond traditional benchmarks by redesigning the format of questions and necessitating that LLMs respond with a single, cohesive long-context answer.

This dual focus on synthetic retrieval tasks and comprehensive understanding benchmarks reflects the field's commitment to rigorously assessing the capabilities of long-context models across diverse linguistic challenges.

Prompting Compression. In the field of prompt compression, various designs effectively combine semantic information to compress natural language. [77] utilize soft prompts to encode more information with fewer tokens. [78] present AutoCompressor, which uses soft prompts to compress the input sequence and extend the original length of the base model. Both [79] and [80] recurrently apply LLMs to summarize input texts, maintaining long short-term memory for specific purposes such as story writing and dialogue generation. The LLMLingua series [54, 81, 82] explores the potential of compressing LLM prompts in long-context, reasoning, and RAG scenarios. [54] use pre-trained language models to chunk the long context and summarize semantic information, compressing the original context.

KV Cache Quantization. Among quantization approaches for KV cache optimization, SmoothQuant [83] stands out as a notable post-training technique. By implementing balanced transformations between activation and weight quantization difficulties, this method enables 8-bit KV cache compression with minimal performance degradation. However, research by [84] has demonstrated that SmoothQuant encounters substantial accuracy deterioration when compressed beyond 8 bits. In parallel developments, the FlexGen system [85] implements group-wise quantization at 4 bits for both key and value caches, while KIVI [45] introduces an innovative approach with asymmetric 2-bit per-channel quantization that requires no additional tuning phase. Additionally, AnTKV [86] introduces a two-stage token-aware vector quantization framework, leveraging gradient-weighted centroid learning and anchor-token selection to achieve ultra-low-bit KV cache quantization with minimal accuracy loss.

E Statistics of Models

F Statistics of Datasets

LongBench is a meticulously designed benchmark suite that evaluates the capabilities of language models in handling extended documents and complex information sequences. This benchmark was created for multi-task evaluation of long-context inputs and includes 17 datasets covering tasks such as single-document QA [87, 88], multi-document QA [89-92], summarization [93-96], fewshot learning [97-99], synthetic tasks and code generation [100, 101]. The datasets feature an average input length ranging from 1 K to 18 K tokens, requiring substantial memory for KV cache management.

Table 28 shows the statistics of the datasets that we used in our experiments.

Table 28: Dataset Statistics. # TRAIN and # TEST represent the number of training and test samples, respectively. *: The size of the NIAH test set varies based on the context length and step size, typically around 800 samples per evaluation.

G Prompt

Table 29 shows the prompt for the Figure 1.

The prompt for demonstration

. . . . . .

. . . . . .

The purple-crested turaco (Gallirex porphyreolophus) or, in South Africa, the purple-crested loerie, (Khurukhuru in the Luvend . a (Vend . a) language) is a species of bird in the clade Turaco with an unresolved phylogenetic placement. Initial analyses placed the purple-crested turaco in the family Musophagidae, but studies have indicated that these birds do not belong to this family and have been placed in the clade of Turacos with an unresolved phylogeny. It is the National Bird of the Kingdom of Eswatini, and the crimson flight feathers of this and related turaco species are important in the ceremonial regalia of the Swazi royal family. This bird has a purple-coloured crest above a green head, a red ring around their eyes, and a black bill. The neck and chest are green and brown. The rest of the body is purple, with red flight feathers. Purple-crested turacos are often seen near water sources, where they can be observed drinking and bathing, which helps them maintain their vibrant plumage. Purple-crested turacos are considered to be large frugivores that are known to carry cycad seeds from various plant species long distances from feeding to nesting sites. After fruit consumption, they regurgitate the seeds intact where they can germinate nearby. G. porphyreolophus primarily consumes fruits whole like many other large frugivores which are suggested to be necessary for effective ecosystem functioning. Among similar turacos, the purple-crested turaco have faster minimum transit times when consuming smaller seed diets than larger seed diets, and G. porphyreolophus has been shown to have significantly faster pulp (seedless fruit masses) transit time than another closely related Turaco when fed only the pulp of larger-seeding fruits than smaller-seeding fruits. In addition to their frugivorous diet, these birds are occasionally seen foraging for other food items such as nuts and leaves, which provide essential nutrients. They are also known to coexist with various other animals, including those that might enjoy strawberries and other similar fruits. The purple-crested turaco's role in seed dispersal is crucial, and their interaction with different elements of their habitat, including water and diverse plant materials, highlights their importance in maintaining ecological balance.

. . . . . .

. . . . . .

Here we provide the CoT prompt exemplars for GSM8K which is used in section 4.1.

GSM8K experiemnt CoT Prompt Exemplars

Question: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?

Olivia had 23 dollars. 5 bagels for 3 dollars each will be 5 x 3 = 15 dollars. So she has 23 - 15 dollars left. 23 - 15 is 8. The answer is 8.

H Impact Statement

Our study does not involve human subjects, data collection from individuals, or experiments on protected groups. The models and datasets used in this work are publicly available and widely used in the research community. We have made efforts to ensure our experimental design and reporting of results are fair, unbiased, and do not misrepresent the capabilities or limitations of the methods presented.

In our work on KV cache compression for large language models, we acknowledge the potential broader impacts of improving efficiency in AI systems. While our method aims to reduce computational resources and potentially increase accessibility of these models, we recognize that more efficient language models could also lead to increased deployment and usage, which may have both positive and negative societal implications. We encourage further research and discussion on the responsible development and application of such technologies.

We declare no conflicts of interest that could inappropriately influence our work. All experiments were conducted using publicly available resources, and our code will be made available to ensure reproducibility. We have made every effort to cite relevant prior work appropriately and to accurately represent our contributions in the context of existing research.

I Limitations

While ChunkKV demonstrates strong performance and efficiency, we acknowledge several limitations that provide avenues for future research.

First, the core design of ChunkKV prioritizes the preservation of semantic coherence by treating chunks as indivisible units. This approach may be less suitable for tasks requiring absolute semantic fidelity, where every single token can be critical. For example, in domains such as legal or biomedical document analysis, discarding any part of the text, even if it appears less relevant based on attention scores, could lead to the loss of crucial information. Our method is therefore optimized for tasks where capturing the core gist is more important than verbatim retention of every detail.

Second, our experiments with a hybrid compression strategy (using ChunkKV in early layers and a token-level method in deeper layers) revealed a nuanced, task-dependent performance trade-off. While pure ChunkKV excelled in local information retrieval tasks (e.g., Single- and Multi-Document QA), the hybrid model showed surprising strength in tasks requiring global understanding (e.g., Summarization). This suggests that a one-size-fits-all compression strategy may not be optimal for all scenarios. The ideal approach might involve dynamically adapting the compression granularity based on the perceived task requirements.

Finally, our current implementation relies on fixed-size chunks for computational efficiency. While our ablations show this is a robust strategy, adaptively determining chunk boundaries based on linguistic cues (e.g., sentence endings) could potentially improve semantic integrity further, although this would introduce additional inference latency that must be carefully managed.

J Licenses

For the evaluation dataset, all the datasets, including, GSM8K [27], LongBench [25] are released under MIT license. NIAH [26] is released under GPL-3.0 license.

Peer reviews:
comments: SUMMARY:
The paper introduces ChunkKV, a new KV cache compression method for efficient LLM inference. Unlike existing token-level compression techniques, ChunkKV groups tokens into semantic chunks, preserving linguistic structures and contextual integrity. The method includes a layer-wise index reuse technique to reduce computational overhead. Evaluations across benchmarks (LongBench, NIAH, GSM8K, JailbreakV) and models (LLaMA-3, Mistral, Qwen2) demonstrate ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision while maintaining compression ratios, with additional throughput improvements of 26.5%.

STRENGTHS AND WEAKNESSES:
## Strengths

1. The idea is easy to understand. Compared to token-level KV cache compression, chunk-level compression could avoid pruning certain essential tokens.

2. The study is very comprehensive. Authors have evaluated their method across different benchmarks, as well as providing many visualizations and analysis.

3. This paper is easy to follow. The overall writing is satisfying.

## Weakness

1. The use of fixed-size chunks may limit adaptability. Dynamic chunking could further improve performance but would increase complexity.

2. According to Figure 19, it is quite challenging to choose the optimal chunk size for different LLMs, as they behave quite differently  when using different chunk sizes. For example, Qwen2-7B is robust but LLaMA-3-8B performs badly at larger chunk size. Authors are encouraged to provide a discussion here.

QUESTIONS:
Please refer to the Weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ChunkKV, a novel approach for compressing the Key-Value (KV) cache in Large Language Models (LLMs) to enable efficient inference with long-context inputs. Unlike prior work that evaluate individual token importance, ChunkKV treats contiguous chunks of tokens as the basic unit for compression. The method utilises a layer-wise index reuse technique to leverage cross-layer similarities, reducing computational overhead. Authors evaluate their methods on various benchmarks and that it surpasses other compression methods performance-wise.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper addresses an important challenge in LLM inference: the high memory demands of the Transformer's KV Cache.\n* The layer-wise index reuse technique is a valuable contribution, effectively leveraging cross-layer similarities in KV Compression to enhance efficiency. It is particularly interesting to observe similarities between layers in the sets of retained chunks.\n* The authors provide a thorough evaluation across various benchmarks and model types. Their method demonstrates consistent performance improvements compared to existing baselines.\n\nWeaknesses:\n* The primary motivation behind the ChunkKV method is that chunks of tokens, rather than individual tokens, are semantically connected and form coherent units of information. However, the authors' justification is not fully convincing. For instance, they state, \"As shown in Figure 1, preserving a chunk helps catch the subject, predicate, and object.\" This argument may not fully apply to Transformer models using self-attention (which the authors evaluate), since self-attention distributes information across tokens. Therefore, after several Transformer layers, it is not clear that specific token information remains localized within the original tokens; it could be processed and stored in entirely different tokens appearing later in the sequence. Furthermore, the method groups tokens into fixed-size chunks, which imposes a clear limitation because semantic units naturally vary in length. Consequently, fixed-size chunks are prone to misalignment, causing fragmentation where meaningful units of information (like sentences) either span multiple chunks or combine unrelated units into a single chunk.\n* The authors use SnapKV as a baseline, and their approach closely resembles SnapKV. Specifically, SnapKV assigns token importance scores and smooths these scores using an averaging or max kernel with a fixed window size, which appears conceptually similar to ChunkKV’s method of aggregating contiguous tokens into chunks and then summarizing importance scores at the chunk level. However, the authors do not provide details about hyperparameter settings (such as kernel size) for SnapKV and PyramidKV in their experiments. A clear comparison or ablation distinguishing ChunkKV's chunking from SnapKV's smoothing approach would significantly strengthen the paper. Without this, the proposed method seems overly similar to SnapKV, diminishing the novelty.\n* The paper lacks a statistical analysis of results, making it unclear whether the performance improvements reported for ChunkKV are statistically significant. The authors mention, \"All experiments were carried out three times, using the mean score to ensure robustness,\" but do not specify details about how these repetitions differed. Moreover, the authors state in the checklist, \"The paper is focused on LLM inference, and with the same model, prompt, and temperature, the results are consistent.\" This explanation is insufficient because the paper introduces a new KV cache compression method and compares its performance against existing methods on downstream benchmarks. In such a scenario, providing measures of uncertainty (e.g., standard deviation or confidence intervals) is crucial for determining whether the observed improvements truly reflect the effectiveness of the proposed method or merely result from variability.\n\nQUESTIONS:\n- Could you discuss the difference between smoothing done by SnapKV and your chunking approach?\n- Could you report the standard deviation of your evaluations to understand if the performance improvements are statistically significant?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ChunkKV, a novel method for KV cache compression by preserving semantically meaningful chunks of tokens rather than isolated tokens. Traditional compression methods often prune tokens based on isolated importance scores, which can disrupt semantic coherence and degrade performance. ChunkKV addresses this limitation by grouping tokens into coherent chunks and selecting top-k chunks based on attention scores to retain critical contextual information. Additionally, the paper proposes a layer-wise index reuse technique that leverages the similarity of retained chunks across layers to reduce redundant computation and improve throughput. Extensive experiments across a variety of LLMs (LLaMA-3, DeepSeek, Mistral, Qwen2) and benchmarks (GSM8K, LongBench, JailbreakV, and Needle-In-A-Haystack) demonstrate that ChunkKV consistently outperforms prior token-level KV compression approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The core idea of performing KV cache compression at the semantic unit level rather than at the individual token level is novel and well-motivated, effectively addressing the issue of fragmented context in long sequences.\n- The paper presents comprehensive experimental evaluations across multiple benchmarks (e.g., GSM8K, LongBench, NIAH, JailbreakV) and base models, demonstrating that ChunkKV consistently outperforms prior methods, particularly under aggressive compression settings.\n- An additional contribution is the layer-wise index reuse strategy, supported by empirical analysis of inter-layer similarity. This approach effectively reduces computational overhead and improves throughput without sacrificing performance.\n\n**Weaknesses**\n1. While ChunkKV performs well in scenarios where unimportant segments can be discarded, its effectiveness may be limited in tasks where every sentence is critical, such as legal or biomedical document processing, where full semantic fidelity is required.\n2. Although the paper successfully targets semantic-level redundancy, it does not leverage token-level redundancy, which can still exist and offer potential for further optimization. ChunkKV, by design, may miss opportunities in cases where fine-grained token pruning is more efficient.\n\nQUESTIONS:\n1. What are the efficiency results (e.g., latency and throughput) for ChunkKV under 20% and 30% compression ratios? Tables 3-7 suggest that ChunkKV achieves performance comparable to or even exceeding the FullKV baseline at these compression levels. Including the corresponding efficiency metrics would provide valuable insight into the trade-offs between compression and runtime performance.\n2. Can ChunkKV be integrated with token-level compression methods? If so, could such a hybrid approach lead to additional performance gains by leveraging both semantic and fine-grained redundancy?\n\nLIMITATIONS:\nPlease see weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn summary, the authors propose a KV cache compression method called ChunkKV, where the KV matrices are compressed in the unit of continuous chunks instead of individual tokens to keep the semantic information. After the KV matrices are divided into chunks, the proposed method follows a similar way to H2O and SnapKV to select the important chunks. Furthermore, the authors propose layer-wise index reuse to further reduce the inference cost. The proposed method is compared to H2O, SnapKV, and PKV on multiple long-context tasks such as Longbench, NIAH, GSM8K and shows better performance given the same KV compression ratio.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. Although the proposed ChunkKV is simple, the idea of keeping semantic information in the KV cache makes sense to me and I think it is beneficial for future research in the community\n\n2. The authors conducted extensive evaluations on long-context tasks in the paper. The proposed method shows better long-context task performance than baselines such as H2O and SnapKV given the same KV cache compression rate\n\nWeakness:\n1. The chunk selection algorithm lacks flexibility to some extent. For example, the chunks in the KV cache are defined in a fixed and uniform way. Thus, it is not guaranteed that each chunk contains enough semantic information. A chunk might cover the end and beginning of two sentences. \n\n2. There appears to be no systematic method for selecting the optimal chunk size for a model and a task. Although most experiments indicate that 10 is the best chunk size, there do exist exceptions. For example, chunk size 3 works the best for GSM8k according to Table 20. \n\n3. The evaluation looks incomplete to me. For example, KIVI is missing for the performance evaluation in Tables 4, 5, 6, and 7. Similarly, H2O and SnapKV are missing for the latency/throughput evaluation in Table 11. Also, some more recent KV cache compression methods such as Palu[1] is not compared.\n\n4. The ablation study of $N_{reuse}$ for the layer-wise index reuse is missing. For most of the paper, $N_{reuse}$ is set to 2 but it would be good to know what the effect is when making it larger.\n\n5. The input sequence length is only scaled up to 8k in Tables 8 and 11. What would be the performance if it is scaled up to 16k or 32k?\n\n[1] Chang, Chi-Chih, et al. \"Palu: Compressing kv-cache with low-rank projection.\" arXiv preprint arXiv:2407.21118 (2024).\n\nQUESTIONS:\nPlease refer to the weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}