ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
Authors: Xiang Liu
Zhenheng Tang
Peijie Dong
Zeyu Li
Liuyue
Bo Li
Xuming Hu
Xiaowen Chu
Venue: NeurIPS 2025 poster ()
Keywords: LLM
KV cache
Compression
Long-context
Primary area: general_machine_learning

Abstract: Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70\% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded performance. We introduce \method{}, which fundamentally reimagines KV cache compression by treating semantic chunks - rather than isolated tokens - as basic compression units. This approach preserves complete linguistic structures and contextual integrity, ensuring that essential meaning is retained even under aggressive compression. Our innovation includes a novel layer-wise index reuse technique that exploits the higher cross-layer similarity of preserved indices in \method{}, reducing computational overhead and improving throughput by 26.5\%. Comprehensive evaluations on challenging benchmarks: LongBench, Needle-In-A-HayStack, GSM8K, and JailbreakV demonstrate that \method{} outperforms state-of-the-art methods by up to 8.7\% in precision while maintaining the same compression ratio. These results confirm that semantic-aware compression significantly enhances both efficiency and performance for long-context LLM inference, providing a simple yet effective solution to the memory bottleneck problem. \emph{The code is available at \href{https://github.com/NVIDIA/kvpress}{link}.}

ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
Xiang LI
Zhenheng TANG* * Bo LI Xuming HU
Peijic DONG
Zeyu LI
Yue LIU
Xiaowen CHU
The Hong Kong University of Science and Technology (Guangzhou)
CSE, The Hong Kong University of Science and Technology
Guangzhou HKUST Fok Ying Tung Research Institute
O Terminus Technologies
(x1iu886, [xliu886.pdomg212, pdw.gddnn 22112222,,,115588cconeeee wwwpmneet..uuutteeee www.hhuuu.ttc.c.c edu.
{zhtang.ml ml, bli]ecse. ust. {xuminghu, Brchus"Ckkust-oo edu, cn
Abstract
Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70% of total memory during inference. Although existing compression methods reduce memory by evaluating the importance of individual tokens, they overlook critical semantic relationships between tokens, resulting in fragmented context and degraded per- formance. We introduce ChunkKVV which fundamentally reimagines KV cache compression by treating semantic chunks rather than isolated tokens as basic compression units. This approach preserves complete linguistic structures and contextual integrity, ensuring that essential meaning is retained even under ag- gressive compression. Our innovation includes novel layer-wise index reuse technique that exploits the higher cross-layer similarity of preserved indices in ChunkKV, reducing computational overhead and improving throughput by 26.5%. Comprehensive evaluations on challenging benchmarks: LongBench, Needle-In- A-HayStack, GSMSK, and Jailbreak V demonstrate that ChunkKV outperforms state-of- of-the--rtttttt methods by up to 8.7% in precision while maintaining the same compression ratio. These results confirm that semantic-aware compression signifi- cantly enhances both efficiency and performance for long-context L.M inference, providing a simple yet effective solution the memory bottleneck problem. The code is available at link
Introduction
Large Language Models (LLMs) have become essential for addressing various downstream tasks of natural language processing (NLP), including summarization and question answering, which require the interpretation of a long context from sources such as books, reports, and documents, often encompassing tens of thousands of tokens [155] Recent advances in long-context technology within the field of machine learning (ML) systems 6 have significantly improved computational throughputs and reduced latency of LLMs process increasingly large input context lengths 10] with historical KV cache (key value attentions). However, the memory requirement of the KV cache in serving super-long contexts becomes new bottleneck [1.44. For instance. the KV cache for
"Equal Contribution. Corresponding Author,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

single token in 7B parameeter model requires approximately 0.5 MB of GPU memory, resulting in 10,000-token prompt consuming around GB of GPU memory.
Question: purple-crested turaco eats what food?
Discrete KV methods 5 (00)
Chunkky: & El.00) where Par
Discrote KV mechods with low sparsty
Chunkiy with a lew sparthy
Purple crenco Euroee
Afler frii oomsumption they
sim
tursoos
minimum transii
Waller
consuming smallrr seed diets than larger seed dicts
thil
with verious other enimels including those that might enjo Mlowworrs
KV mechods With high spansiy
Chur WSV with a nigh spa sity
ero-e-cesse Gressed 2220009
Afhar that consumption they
cORIEE with varloux
other onimals uding chose that might
Figure I: Illustaaioon of the impact of the token discrete method and the chunk method on semantic preservation. The discrete method preserves words related to the question but often omits the subject. In contrast, the chunk method retains the subject of the words, maintaining more accurate semantic information. For the equation: S is the score function, and is chunk of tokens.
To address the substantial GPU memory consumption caused by KV caching, recent studies consider compressing the KV cache by pruning non-important discrete parts from the prompt tokens [1] [2] 15
20. H2O [1T] and SnapKV [T5] have shown that retaining less than 50% of the discrete KV cache can significantly reduce GPU memory usage with minimal impact on performance. However, we identify that the previous KV cache compression methods [11 17 measure token importance isolatedly, neglecting the dependency between different tokens on the characteristics of the real-world language. For example, as shown in F focusing on the importance of the token level can excessively focus on words about subjects "turaco" in the question while omitting crucial information about objects (foods) in the documents, resulting in the loss of essential semantic information. This motivates us to rethink the following question:
How avoid isolated token importance measurement and preserve the semantic information in KV cache?
Table I: Comparison of Methods on KV Cache Compression.
KV Cache
Dynamic Policy
Layer-Wise
Semantic Information
Efficient
Method
Compression
Policy
Index Reuse
Streamingl.L.M
8
H20 (T
SnapKV
15
PyramidInfer
19
PyramidKV V
17
/
ChunkK V(Ours)
In light of this, we observe that complete semantic information usually appears in continuous sequence [4] 22,244. Thus, we introduce straightforward yet effective ChunkKV, grouping the tokens in a chunk as basic compressing unit, which should be preserved or discarded as whole. Thus, it retains the most informative semantic chunks from the original KV cache. As shown in Figure [ preserving a chunk helps catch the subject, predicate, and object. In Table 13 we quantify the minimal loss and higher recovery of the attention score achieved by the chunk-based approach at the inference stage. Furthermore, we investigate that the preserved KV cache indices by ChunkKVV exhibit a higher similarity compared to previous methods. Consequently, we develop a technique called layer-wis, index reuse, which reduces the additional computational time introduced by the KV cache compression method. As outlined in Table recent highly relevant KV cache compression methods lack the ability relain semantic information and efficiently reuse indices.
To evaluate ChunkKV's performance, we conduct comprehensive experiments across multiple cutting-edee long-contex, benchmarks: long-context tasks including LongBench (25) and Needle-

---

In-A Dn-A-HayStacck (NIAH) (26]. in-context learning tasks such as GSM8K 27 and Jailbreaky V -i-becc--R--isill-------
[28]
And also different models including
29 LLLMA-3--88 3---M-------nsstruc 30
Settral-B-Instturee [3I, and Ayee2-7BBInnstuuee 32 Our experimental results demonstrate that
ChunkKV surpasses existing KV cache compression methods in both efficiency and accuracy, primarily due to its ability to preserve essential information through selective chunk retention. These
findings establish ChunkKV as a simple yet effective approach to KV cache compression.
We summarize our key contributions as follows:
We identify the phenomenon in which discrete KV cache compression methods inadvertently prune the necessary semantic information.
We propose ChunkKV, a simple KV cache compression method that uses the fragmentation method that retains semantic information, and design the layer-wise index reuse technique to reduce the additional computational time.
We evaluate ChunkKV on cutting-edge long-context benchmarks including LongBench and Needle- In-A-Han A-HayStack, as well as the GSM8K, many-shot GSM8K and Jailbreak in-context learning benchmark, and multi-step reasoning (Ol and RI) LLMs, achieving state-of-the-an performance.
2 Related Work
KV Cache Compression. KV cache compression technology has developed rapidly in the era of LLM, with methods mainly focused on evieting unimportant tokens. The compression process occurs before the attention blocks, optimizing both the prefilling time and GPU memory. Xiao et al. [8] and Han et al. 33 propose that initial and recent tokens consistently have high attention scores between different layers and attention heads. As a result, retaining these tokens in the KV cache is more likely to preserve important information. Furthermore, FastGen [16] evicts tokens based on observed patterns. H2O [11] and SnapKV V [15 employ dynamic KV cache compression methods, evaluating the importance of tokens based on attention scores and then evicting the less important ones. As inference scenarios become increasingly complex, dynamic KV cache compression methods demonstrate [17] have closely examined the
powerful performance. Recently, Yang et al. [9 and Cai et al. distributions of attention scores during the pre-filling stage of the detrieval-Augmentedd Generation (RAG) task, discovering pyramidal KV cache compression pattern in different transformer layers. In contrast, FlowKV [B4] employs novel multi-ture turtii-tuuumu isolation mechanism that preserves the accumulated compressed KV cache from past turns and only compresses the KV pairs from the most recent turn, effectively mitigating the re-compression of older context and the problem of catastrophic forgetting.
Although these KV cache compression methods have explored efficient GPU memory management while maintaining original performance, our study focuses more on the semantic information of the prompt. We find that chunks of the original KV cache are more important than discrete tokens.
3 ChunkKV
3.1
Preliminary Study of KV Cache Compression
With the increasing long-contex capabilities of LLMs, the KV cache has become crucial for improv- ing the inference efficiency. However, it can consume significant GPU memory when handling long input contexts. The GPU memory cost of the KV cache for the decoding stage can calculated as follows:
MRV = 2 x xx>xxxxxxxXxxxX
(I)
where B is the batch size, S is the sequence length of prompt and decoded length, L is the number of layers, N is the number of attention heads, 1 the dimension of each attention head, and the first
accounts for the KV matrices, while the last 2 accounts for the precision when using 16. Table E shows the configuration parameters for 1--MMA------Istruu- [30]. With a batch size sequence length of prompt S = 2048, the GPU memory cost of the KV cache is nearly 1 GB. If the batch size exceeds 24, the GPU memory cost of the KV cache will exceed the capacity of an RTX 4090 GPU. To address this issue, KV cache compression methods have been proposed, with the aim of retaining only minimal amount KV cache while preserving as much information as possible. For more details on the LLM configuration parameters, refer to Appendix B
1 and
3

---

3.2 Chunk Based KV Compression
To address the limitations of existing KV cache compression methods, we propose ChunkKV, novel KV cache compression method that retains the most informative semantic chunks. The key idea behind ChunkKV to group tokens in the KV cache into chunks that preserve more semantic information, such chunk containing subject, verb, and object. As illustrated in Figure D ChunkKV preserves the chunks of the KV cache that contain more semantic information. First, we define a chunk as a group of tokens that contain related semantic information. By retaining the most informative chunks from the original KV cache, C can effectively reduce the memory usage of the KV cache while preserving essential information. For more information ChunkKV, please refer to Section A
Algorithm ChunkKV
Input: Q E TRxx K E RTxxd V E RRxxdd observe window size chunk size compressed KV cache max length Timux
Output: Compressed KV cache K', V
Observe Window Calculation:
A f- QT. K
Attention scores for the observe window Calculate the number of chunks
Ce L&
Chunk Attention Score Calculation: for i to C do
A; f Σ) 1)=(-1))++1 A.j
Sum of attention scores for each chunk
end for
Top-K Chunk Selection:
Lmu
Top K Indices t- indices of Top-k chunks based on A;
Compression:
K', V + index_selee feeee__sllettK___ K, V. ToTp__ <_ndices)
Concatenation:
K' f- concat K 0: 0:Temu wi KT₃ -w:T)
V +- concat V V0:Lam wi V VT, ww:T.
K' V
The algorithm I shows the pseudocode for ChunkKV. First, following the approach of H2O and
SnapKV [15,, we set the observe window by computing the attention scores A f Qr, w: K" where Q - is the observe window, K is the Key matrix and the window size is usually set to {4, 8, 323. Next, the number of chunks
calculated as C = [ where Ta is the length of the Key matrix and e: is the chunk size. The attention scores for each chunk are then computed A; Eie Dj=(f-1)e+1 A., for 1, C. We use the top-k algorithm the sampling policy for ChunkKV. The top-k chunks are selected based on their attention scores, where k = سسح and Lmmax is the maximum length of the compressed KV cache. The size of the last chunk set to min(c, I_max (k- 1) x c). The indices of the top-k chunks preserve the original sequence order. In the compression step, only the key and value matrices corresponding to the selected indices are retained, resulting in the compressed KV cache. Finally, the observe window of the original KV cache will be concatenated to the compressed KV cache by replacing the last w tokens to keep important information. The compressed KV cache is then used for subsequent attention computations. For implementation, we add vectorized operations, memory optimizations, etc., to optimize the code. Refer to Appendix A.5 for more details.
3.3 Layer- Wise Index Reuse
Table Retained KV Cache Indices Similarity of Adjacent Layers for Different Models.
Furthermore, we investigated the KV cache in- dices preserved by ChunkKV and found that they exhibit higher similarity compared to pre- vious methods. Figure 2 shows the layer-wise similarity heatmaps of SnapKV and ChunkKV. Each cell represents the similarity between the
Method
H20
SnapKV
ChunkKV
L.aMA-3-8R
25.31%
27.95%
57.74%
Qwen2-7B
14.91%
16.50%
44.26 %
Mistral-7B
15.15%
15.78%
52.16%

---

SnapkV
ChunkkV
12
Layer
18
20
74
28
12
16 Layer
20
24
20
4
12
16 Layer
20
24
20
Figure Layer-wise similarity heatmaps the preserved KV cache indices by SnapKV (left) and ChunkKV (right) LLaMA-3-8B-Inse visualization can be found in Appendix B.1.3
Deep colors indicate higher similarity. More
preserved KV cache indices of two layers, with deeper colors indicating higher similarity. The results demonstrate that the KV cache indices preserved by ChunkKV are more similar to those in neighboring layers.
As shown in Table 2 ChunkKV consistently achieves higher average Jaccard similarity between adjacent layers compared to SnapKV in different model architectures, indicating that the retained token index in ChunkKV more similar to each other. For more de- tailed visualization, please refer to Appendix B.I.3
Algorithm Layer-wise Index Reuse for ChunkKV
Input: Number of layers LLMs Nayess number of reuse layers Nreuse
Initialize: Dictionary to store indices Treuse = 0 for = to (Mayyrs
do if mod N == 0 then
K/. V
ChunkKV (KK, Vi)
Treuso [1] t Ii
Based on the above findings of the KV cache index, we propose a training-free free layer-wise index reuse method further reduce the addi- tional cost of KV cache compression, which reuses compressed token indices across mul- tiple layers. We evaluate the efficiency and effectiveness of the layer-wise index reuse method in Section 4.3 Reuse of the layer- wise index reduces the KV cache compression time by 20% compared the FullKV baseline, with
else
I₁ +- Ireuee|
Nreuse]
Nmac
end if
K < index_ dedee__selett__ee Ki
V + index_select (i,
end for
performance drop of only 0.5%.
This layer-wise index reuse method is formally described in Algorithm 2 The ChunkKV compression process returns the compressed KV cache and their respective token indices, denoted as Z, For layer-wist index reuse, we define a grouping of layers such that all Nreuse layers share the same token indices for ChunkKV. Specifically, for group of lay- ers {1, +1.++1,..... 1,..., Naeuse - 1}, we perform ChunkKV the first layer to obtain the to- ken indices Z, and reuse I₁ for the subsequent layers /+1,/+2, .... (+ Nresse 1. The nota- tion K and Vi[Ti] indicates the selection of key and value caches based on the indices in I, The efficiency analysis for laser-wise index reuse is provided in Appendix B.I.I
Table 3: GSM8K Performance Comparison.
# SL.M=StreamingLLMM SKV=SnapKV,
PKV=PyramidKV
ChunkKV (Ours)
Ratio
SLM
H2O
SKV
PKV
DeepSeek RI Distill Llama-8B FullKV: 69.4% ?
10%
51.6%
55.6%
57.6%
62.6%
65.7% %
LlaMa-3. 1-88----Btnteee Full 79.5% f
30%
70.5%
72.2%
76.1%
77.1%
77.3%
20%
63.8%
64.0%
68.88 68.8%
71.4%
77.6%
10%
47.8%
45.0%
50.3%
48.2%
65.7%
1laMa-3-8B-Instrenn FullKV: 76.8% f
30%
70.6%
73.6%
70.2%
68.2%
74.6%
Qwen2-77 BB-Isrrutttt FullK
71.1%
30%
70.8%
61.2%
70.8%
64.7%
73.5%
Theoretical Understanding. We provide a theoretical understanding from the in-context
5

---

learning (ICL) 24 to inteppret why maintaining the KV cache according to continuous sequence in ChunkKV is better than according sparse tokens. Informally speaking, the continuously chunk- level KV cache preserves the whole examples (semantic information) in ICI., thus reducing the requirement on distinguishabiltty i.e., lower bound of KI. divergence between the example and the question (Equation 4 in condition 2) The complete analysis is provided in Appendix C
4 Experiment Results
In this section, we conduct experiments evaluate the effectiveness of ChunkKV on KV cache compression in two benchmark fields, with a chunk size set to 10 even for various model archi- tectures. The first is the In-Context Learning benchmark, for which we select GSM8K 27 and
Jailbreakv (28 35 36 to evaluate the performance of ChunkKV, furthermore, we also include multi- step reasoning L.M DeepSeek- -iieeek-RR-Dian ---isttlllLLlama---- 8B [29] to evaluate the performance of ChunkKV. The In-Context Learning scenario is a crucial capability for LLMs and has been adapted in many powerful technologies such as Chain-of-Thoughh (37-40), The second is the Long-Context bench- mark, which includes LongBench [25 and Needle-In-A-e An-A-HayStack (NIAH) (66]. both widely used for assessing KV cache compression methods. All experiments were carried out three times, using the mean score to ensure robustness.
4.1 In-Context Learning
The In-Context Learning (ICL) ability significantly enhances the impact of prompts on LLMs. For example, the Chain-of-Thought approach (37] increases the accuracy of the GSM8K of the PaLM model 441 from 18% to 57% without additional training. this section, we evaluate the performance of ChunkKV on the GSM8K, Many-Shou GSM8K [4), and Jailbreak V (28] benchmarks.
GSM8K. In the in-context learning scenario, evaluated multiple KV cache compres- sion methods for GSM8K (27), which con- tains more than 1,000 arithmetic questions on LLaMA-3-8B-Instruu Instruct 130. Qwen2-7B-Instnun
Table 4: Many-Shot (50-shot) GSM8K Performance Comparison.
ChunkKV (Ours)
Ratio
SLM
H2O
SKV
PKV
1.L.MA-31-88 1-8B-
32 and
DeepSeck-R1 1
FullKV:
71.2% 1
DeepSeek-R1 R1
[29] Follow-
10%
63.2%
54.2%
54.1%
59.2%
68.2%
ing Agarwal al. [42). we consider many- shot GSM8K as a long-context reasoning
L. 3.1BBB I-Sttrut FullKV: 82.4% f
scenario, which is a more challenging task than long-context retrieval benchmark Long- Bench (25) The CoT prompt settings for this experiment are the same as those used by Wei et al. [37] for many-shot GSM8K we set the number of shots to 50, where the prompt length is more than 4k tokens. For more details on the prompt settings, please refer to the APPENDIX G
10%
74.3%
51.2%
68.2%
70.3%
79.3% %
Table presents the performance comparison. The results show that ChunkKY V outperforms other KV cache compression methods different models and compression ratios. Table 4 presents the performance comparison of many-shot GSM8K, and also ChunkKy V outperforms other KV cache compression methods. The consistent superior performance of ChunkKV in both models underscores its effectiveness in maintaining crucial contextual information for complex arithmetic reasoning tasks by chunk level KV cache rather than the discrete token level.
Jailbreak. In this section, we evaluate the performance of ChunkKV on the JailbreakV benchmark (28). which is safety Jailbreak benchmark for language models. The prompt settings are the same as those used by Luo et al. 228..
Table 5: Jailbreak Performance Comparison.
ChunkKy (Ours)
Ratio
SLM
H2O
SKV
PKV
LlaMa-3. 1-88-Inntrud Full 88.9% f
20% 10%
65.0%
71.7%
88.0%
87.5%
89.0% 87.9 %
Table 5 presents the performance comparison. The results demonstrate that ChunkK out- performs other KV cache compression methods on different compression ratios. This shows that for safety benchmark, the chunk level KV cache more effective than other discrete token level compression methods.
53.1 1%
65.4%
84.3%
85.5%

---

(a) ChunkKV, accuracy 73.8%
(h) PyramidKV, accuracy 65.1%
(c) SnapKV, accuracy 58.9%
(d) StreamingI.LMM, accuracy 23.7%
Figure 3: NIAH benchmark for LLaMA3-8B Instructt with KV cache size=128 under 8k context length.
4.2 Long-Context Benchmark
LongBench and NIAH are two widely used benchmarks for KV cache compression methods. Both benchmarks have context length that exceeds 10K. NIAH requires retrieval capability, while LongBench is meticulously designed benchmark suite that tests the capabilities of language models in handling extended documents and complex information sequences. For more details on LongBench, please refer to the APPENDIX F
LongBench. We use LongBench [25] to assess the performance of ChunkKV on tasks involving long- context inputs. We evaluated multiple KV cache eviction methods using the LongBench benchmark with CAA-MAA---B--nsteee
[32] with KV cache compression ratio of 10%. LongBench-Z4 provides the Chinese subtask, and Qwen2-7B-
[30].
[31. and Owen2-7B-Instreee
Instruct also supports Chinese, so we tested Qwen2-7BB IIInrruuct with different KV cache compression methods on the Chinese subtasks.
Tables 0 present the performance gap (in percentage) between each method and the FullKV baseline. where negative values in- dicate performance degradation compared to FullKV. The table is evaluated in both the LongBench English and Chinese subtasks, where ChunkKV outperforms other compres- sion methods overall. This suggests that ChunkKV's approach of retaining semantic chunks is more effective in preserving im- portant information compared to other dis- crete token-basee based compression methods. For the 70B model and Chinese subtask results, please refer to Appendices B.2 and B.5
Table 6: KV cache compression methods the Long- Bench benchmark. Results show performance gap compared to FullKV baseline (negative values indi- cate worse performance).
ChunkKV (Ours)
Ratio
SLM
H2O
SKV
PKV
LlaMa-3
FullKV: 41.46
10%
-13.80%
-10.61%
-3.16%
-3.33%
-2.29%
20%
-6.42%
-8.85%
-2.24%
-2.00%
-1.74%
30%
-2.36%
-5.8%%
-0.07%
+0.31
Mistral- Instruct--O.... FullK
10%
-16.58%
-9.30%
-3.54%
-3.52%
-2.85%
Bwen2-7B-Instruce
40.71 1
10%
-5.28%
-0.4%%
-0.3%%
-0.98%
+0.42%
Needle-In-A -Hayy- We use NIAH [26 evaluate the long-context retrieval capabil-
Owen2-7B-Instreeen on LongBencch FullKV:
10%
-15.95%
-5.31%
+0.18%
-5.31%
+2.20%
ity of LLMs. NIAH assesses how well LLM
extracts hidden tricked information from extensive documents, and following LLM-as-a-Judee we apply GPT-4o-mini [44) to assess the accuracy of the retrieved information. We evaluated multiple KV cache eviction methods using NIAH with ----M-----B-Instreeee and
45
setting benchmark context lengths to 8k and 32k tokens.
Table 7 provides statistical results for different compression methods. These findings clearly indicate the effectiveness of ChunkKV in man- aging varying token lengths and depth percent- ages, making it robust choice for KV cache management in LLMs. Figure presents the
Table 7: NIAH Performance Comparison.
KV cache Size
ChunkKY (Ours)
SLM
H2O
SKV
PKV
LlaMa-3. 188--Issstttt FullKV: 74.6%
512
32.0%
68.6%
71.2 %
72.6%
74.5%
256
28.0%
61.7%
68.8%
69.5%
74.1% %
NIAH benchmark results for LLaMA-3-8B- Instruct. The vertical axis represents the depth percentage, while the horizontal axis repre- sents the token length, with shorter lengths on
128
23.7%
47.9%
58.9%
65.1%
73.8%
96
21.5%
41.0%
56.2%
63.2%
70.3%
Mistrul-7B-Inssnn FullKVV 99.8%
128
44.3%
88.2%
91.6%
99.3%
99.8%

---

the left and longer lengths on the right. A cell highlighted in green indicates that the method can retrieve the needle at that length and depth percentage. The detailed visualization of the NIAH benchmark can be found in the Appendix B.3 The visualization results demonstrate that ChunkKY V outperforms other KV cache compression methods.
4.3 Index Reuse
This section will evaluate the performance of the layer-wise index reuse approach with ChunkK from the two aspects of efficiency and performance.
Measuring Efficiency. We evaluated the latency and throughput of ChunkKV compared to FullKV using LL--M3--B--nnan
on an A40 GPU. All experiments were conducted with reuse layer is 2, batch size set to I and inference was performed using Flash Attention 2. each experiment was repeated 10 times and the average latency and throughput were reported.
The results in Thow that the layer-wiss index reuse strat- egy (ChunkKV (ChuukKV_reuse) further boosts performance, achieving up to a 20.7% reduction la- tency, and throughput improve- ments are particularly notable for longer input sequences, with ChunkKV_reau reuse delivering up to 26.5% improvement over Ful- IKV. For more detailed results, please refer Appendix
Table & Lateney and throughput comparison between C and FullK V under different input-output configurations. Percent- ages in parentheses indicate improvements over FullK baseline.
Sequence Length
Performance Metrics
Method
Input
Output
Lateney(s) |
Throughput(T/S)
FullKV
4096
1024
43.60
105.92
ChunkKV
4096
1024
37.52 (13.9%)
118.85 (12.2%)
ChunkKV __reuse
4096
1024
37.35 (14.3%)
124.09 (17.2%)
FullKV
4096
4096
175.50
37.73
ChunkKV
4096
4096
164.55 (6.2%%
40.58 (7.6%)
ChunkKV rreuse
4096
4096
162.85 (7.2%
41.12 (9.0%)
[B.8 B.8
FullKY
8192
1024
46.48
184.08
Measuring Task Performance.
ChunkKV
8192
1024
37.83 (18.6%))
228.96 (24.4%)
Chunk BannKKVrreuseee
8192
1024
36.85 (20.7%)
232.99 (26.5%)
We evaluate the effectiveness of our proposed layer-wise in- dex reuse technique on Long- Bench [25] and GSM8K (27) benchmarks. For these experiments, we use the same configuration
FullKV
8192
4096
183.42
55.93
ChunkKV
8192
4096
164.78 (10.2%)
65. 14 (16.5%
ChunkKV_reee rrruse
8192
4096
162.15 (11.6%)
66.05 (18.1%)
main LongBench experi- 4.2 with index reuse applied to consecutive layers (reuse layers = 2).
ments in Section
As shown in Table layer-wise index reuse main- tains the performance of the models while reducing computational requirements. In LongBench, perfor- mance degradation minimal (less than 0.6%) for both models, while GSM8K shows neutral or slightly positive effects. This validates that semantic chunks selected by ChunkKV remain consistently important across adjacent transformer layers, enabling efficient computation without sacrificing accuracy. An ad- ditional analysis of different reuse depths and their impact throughput is provided in Appendix [B.122
Table 9: Reusing Indexing Performance Com-
parison on LongBench and GSM8K. A indi- cates the performance degradation of index reuse compared to the bascline.
ChunkKV
Model
Baseline
Index Reuse
LongBench
LLaMA-3-8B-A
40.51
40.27_p.50%
40.27
-0.0%%
MBstral-7B-Inst
46.71
46.43 0.50%
Qwen2-7B-Inst
40.88
0.20%
Overall, these findings on efficiency and performance suggest that layer-wise index reuse can be an effective technique for optimizing the trade-off in KV cache compression, with the potential for model-specific tuning to maximize benefits.
GSM8K
I.LMM-3-8B--nstt
74.5
40.6.
Qwen2-7B-Inst
71.2
71.2.p.00%
4.4 Chunk Size
This section aims to investigate the impact of chunk size on the performance of ChunkKV. Different chunk sizes will lead to varying degrees compression on the semantic information the data. We used the same experiment setting as in LongBench and NIAH (Section 4.2). The chunk size is set from the range (3, 5, 10, 20, 30) under the compression rate 10% for Lenghench and the 128 KV cache size for NIAH. For more experiments with different compression ratios, see Appendix B.4
8

---

Table 10: LongBench and NIAH Results with Chunk Size Ablation
Chunk KV
Model
Full KV
SnapkV
H20
size=3
size=5
size=10
size=20
size=30
LongBench 1
1.aMA-3-8B-Instuue
41.46 48.08
40.49
40.47
40.51
40.05
39.57
40.15
37.06
Mistral-7B-Instruus
46.45
46.51
46.71
46.42
45.98
46.38
43.61
NIAH
JLaMA-3-8B-Inttrue Mistral-7B BBIIntructt
74.6
65.6
69.1
73.8
72.0
71.2
58.9
47.9
99.8
98.1
99.2
99.8
99.8
99.1
91.6
88.2
As shown Table 10 the performance remains relatively stable when chunk size between 5 and 20, with the best results consistently achieved at chunk size 10. When the chunk size is too small (e.g.. 3), the context is fragmented, leading to a slight drop performance. Conversely, when the chunk size is too large (for example. 30), the semantic granularity becomes too coarse, and important fine-grained information may be lost, also resulting in performance degradation. This trend is consistent across both LongBench and NIAH benchmarks, as well as across different model architectures, indicating that the optimal chunk size not highly sensitive to the specific task or model. We also provide line graph in Appendix B. visually illustrate this trend. Based on these findings, we recommend using a chunk size of 10 as a robust default for most applications. For users with specific requirements, chunk size be further tuned, but our results suggest that moderate values (5-20) generally offer a good trade-off between semantic preservation and compression efficiency.
4.5 Comparing with KV Quantization
For comprehensively evaluate the effectiveness of ChunkKV, we performed experiments comparing ChunkKV with the KIVI quantization methods [5). Although both approaches aim to optimize LLM inference, they operate on fundamentally different principles: Quantization reduces KV matrix precision, whereas our eviction method reduces KV matrix size. For more detail, see Appendix B.6
From an implementation perspective, quantization methods require the full KV cache during prefilling to produce quantized representations, which are then used during decoding. In contrast, ChunkKV employs token removal prior to prefilling, enabling operation with a compressed cache throughout the entire inference process. This distinction creates different efficiency profiles, and each method offers unique advantages.
Due to the fact that KIVI requires an old Python version, the ChunkKV results are not aligned with Table The efficiency results in Table Ia reveall ChunkKV's significant advantages latency-critical metrics. Although both methods substantially reduce cache size (ChunkKV V to 10% and KIVI-2bits to 15.63%), ChunkKV delivers superior metrics for Time First Token (TTFT) and Token Processing Time (TPOT), Most remarkably, ChunkKV achieves a 164.66s total generation time compared to KIVI's 226.52s bib quantization, representing a 27.3% improvement in overall inference speed.
Table 11: Efficiency Results for ChunkKV and KIVI on LlaMa-3-8B-Insiiie
Configuration
Prompt Length
Output Length
Compression Ratio nbits
Prefilling Time
Cache Size(GB) |
TTFT
TPOT
Total Gen.
(8)
(ms)
Time(s) |
FullKV
8192
4096
1.5621
1,0000
1.6013
45.9421
184.2934
KIVI
8192
4096
2bits
1.4024
0.1563
1.4325
54,9561
226.5234
KIVI
8192
4096
4bits
1.3916
0,2813
1.4146
52.0510
214.5634
ChunkKV
8192
4096
10%
1.3653
0.1000
1.3914
39.8702
164.6600
4.6 Analysis of Hybrid Compression: Chunk-level vs. Token-level at Different Layer Depths
critical question raised during the review process was whether the benefits chunk-level compres- sion diminish in deeper Transformer layers, where semantic information becomes more abstract and diffused. To investigate this, we designed a hybrid compression model that applies different strategies at varying network depths.

---

Experimental Setup We created a hybrid version of the LLaMA-3-88P -aMMA-3-8B-Instructt Instrucct model. For the first 16 layers (bottom half), we applied our chunk- ChunkKV. For the final 16 layers (top half), we applied SnapK V, a state-of-the-arr token-level compression method. We then compared this hybrid model's performance on the diverse LongBench benchmark against pure ChunkKV and pure SnapKV at identical compression ratios.
Table 12:
Performance of the hybrid compression model on LongBench. While the hybrid model shows strengths in global understanding tasks, pure ChunkKV achieves the best overall performance, validating the robustness of preserving semantic chunks even in deep layers.
Method
Single-Doc QA
Multi-Doc QA Summarization
Few-shot
Synthetic
Code
Avg. Score
FullKV
32.19
34.59
24.96
68.48
36,96
54.41
41.46
SnapKV (10% Ratio)
28.11
32.55
24.12
67.81
36.01
55.67
40.15
Hybrid Model (10% Ratio)
28.38
24.54
67.87
36.37
55.32
ChunkKV (10% Ratio)
28.50
33.46
22.20
67.62
37.47
58.98
40.51
Analysis and Insights The results in Table 12 provide two key insights. First, the pure ChunkKV model achieves the highest overall average score. This empirically validates our core hypothesis: preserving local semantic integrity via chunking is robust and effective strategy across all layers of the model, even where information is highly processed.
Second, the hybrid model reveals fascinating, task-dependent performance trade-off.
For local information retrieval tasks (e.g.. Single- and Multi-Document QA), pure ChunkKV is significantly superior. These tasks often require retrieving precise, intact text fragments. ChunkK ability to preserve complete linguistic units prevents critical information from being fragmented, which is essential in these scenarios.
For global understanding tasks (e.g., Summarization and Few-shot Learning). the hybrid model performs best. In these tasks, synthesizing information from across the entire context is key. In the deeper layers, where abstract representations are formed, the token-level SnapKV method may be more adept at retaining a broader, more diffuse set of globally important signals.
This analysis does not undermine our approach but rather enriches it, suggesting that the future of KV cache compression may lie in adaptive, task-aware strategies. However, for a general-purpon and robust solution, pure ChunkK V proves to be the most effective.
5
Conclusion
In this paper, we first indicate that current KV cache methods lack the semantic information of the data, and then we proposed novel KV cache compression method that preserves semantic information by retaining more informative chunks. Through extensive experiments across multiple ofute-of-thh-arr LLMs (including DeepSeek-Ro LLaMA-3, Qwen2, and Mistral) and diverse bench- marks (GSM8K, LongBench, NIAH, and Jailbreak V), we demonstrate that ChunkKV consistently outperforms existing methods while using only a fraction of the memory. Our comprehensive analysis shows that ChunkK chunk-based approach maintains crucial contextual information, leading to superior performance complex reasoning tasks, long-context understanding, and safety evaluations. The method's s effectiveness is particularly evident in challenging scenarios like many-shot GSM8K and multi-document QA tasks, where semantic coherence is crucial. Furthermore, our proposed layer-wise index reuse technique provides significant computational efficiency gains with minimal performance impact, achieving up to 20.7% latency reduction and 26.5% throughput improvement. These findings, supported by detailed quantitative analysis and ablation studies, establish ChunkKV as a significant advancement in KV cache compression technology, offering an effective solution for deploying LLMs in resource-comstrained environments while maintaining high-quality outputs.
Acknowledge
This work was supported by the NSFC grant 62432008; RGC RIF grant R6021-20; an RGC TRS grant T43-513/23N-2. RGC CRF grants C7004-22G, C1029-22G and C6015-23G; NSFC/RGC grant CRS HKUST601/24 and RGC GRF grants 16207922, 16207423 and 16203824; National
10

---

Natural Science Foundation of China (Grant No.62506318); Guangdong Provincial Department of Education Project (Grant No.2024KONCX029) Scientific Research Projects for the Higher- educational Institutions (Grant No.20243 1122332000 1096), Education Bureau of Guangzhou Municipality; Brangzhou-HKUSTICE Joint Funding Program (Grant No. n0.2025AA0333557, Education Bureau of Guangzhou Municipality:the Guangzhou Municipal Joint Funding Project with Universities and Enterprises under Grant No. 2024A03J0616 and Guangzhou Municipality Big Data Intelligence Key
Lab (202300300112).
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-
1901. 2020.
[2] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. ArXiv preprint, abs/2205.05131, 2022. URI.httss:/ htt.ttttss//ari......t org/ 2205 05131
[3] Zhenheng Tang, Xiang Liu, Qian Wang, Peijie Dong, Bingsheng He, Xiaowen Chu, and Bo Li. The lottery LLM hypothesis, rethinking what abilities should LLM compression preserve? In The Fourth Blogpost Track at ICLR 2025, 2025.
[4] Qian Wang, Zhenheng Tang, Zichen Jiang, Nuo Chen, Tianyu Wang, and Bingsheng He. Agenttaxo: Dissecting and benchmarking token distribution of Ilm multi-agent systems. In ICLR 2025 Workshop on Foundation Models in the Wild, 2025.
[5] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.09228, 2023. URL
https://axxiv.ct org www.////////eeeeeeee..eeeeeeeeee /2307 09288
[6] Tri Dao. FlashAttention- Faster attention with better parallelism and work partitioning, In International Conference on Learning Representations (ICLR), 2024.
[7] Sam Ade Jacobs et al. DeepSpeed Ulysses: System optimizations for enabling training of extreme long sequence Transformer models. ArXiv preprint, abs/2309. 14509, 2023. URL
https://arxiv. org/ ab//////////....///// 2309 14509
[8] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, The Twelfth International Conference on Learning Representations, 2024. URL
[9] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. ArXiv preprint, abs/2402.08266, 2024. URL https
// //rxiiiv.a org abs wwtp///...... 240..0826888888
[10] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang. Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01, ai. ArXiv preprint, abs/2403.0465 2024. utt.tttss:///axxt org abs/2403. 2etps//220000522222.
[11] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710, 2023.
[12] Yuanbing Zhu, Zhenheng Tang, Xiang Liu, Ang Li, Bo Li, Xiaowen Chu, and Bo Han. OracleKV: Oracle guidance for inntrendeend KV cache compression. In ICML 2025 Workshop on Long- Context Foundation Models, 2025. URL https://oponnrvvier net/ Korum?id-KKK2Y0Ge..
[13] Qian Wang, Tianyu Wang, Zhenheng Tang, Qinbin Li, Nuo Chen, Jingsheng Liang, and Bingsheng He. Megaagent: A large-scale autonomous lmm-based multi-agent system without predefined sops. In The 63rd Annual Meeting of the Association for Computational Linguistics, 2025.

---

[14] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean- baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieser. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv preprint, abs/2403. 0055330 2024. URI.https://arxiv org/ abs 2203.4433..0 05530
[15] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. ArXiv preprint, abs/2404. 14469, 2024.
//apxivv///xxv org/ abs/
2404. 14469)
[16] Suyu Ge. Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. ArXiv preprint, abs/23 10.01300,01801,
2023. URL. htt.ttttts//xiiiii http://////xxiooo//aaaaab/a/a 2310. 01801
[17] Zefan Cai, Yichi Zhang. Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on 2024.
pyramidal information fümneling. arXiv preprint
[18] Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Najibi. LazyLLM: Dynamic token pruning for efficient long context LLM inference. In Workshop on Efficient Systems for Foundation Models II @ ICML2024, 2024. URL https
/ //openreview htt.//oorrmiiddg2221--gag
[19] Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. Pyramid- Infer: Pyramid KV cache compression for high-throughput LLM inference. In Lui-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computa- tional Linguistics ACL 2024, pages 3258-3270, Bangkok. Thailand and virtual meeting, 2024. Association for Computational Linguistics. doi: https://aclanthologg org/ 2024. findings- 195.
1/2024. wwt.02...iniins-aaaa findings-acl.199 URL
[20] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anas- tasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for 11m kv cache compression test time. Advances in Neural Informa- tion Processing Systems, 36, 2024.
[21] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context 11m inference. ArXiv preprint, 2tts/2406.00744 2024. URL htttp:://axx:>>> org/ars /25/
[22] George A Miller. Information and memory. Scientific American, 195(2):42 47, 1956.
[23]
Lance A Ramshaw and Mitchell P Marcus. Text chunking using transformation- based learning. In Natural language processing using very large corpora, pages 157-176. Springer, 1999.
[24] Hongchao Fang and Pengtao Xie. An end-to-end contrastive self-supervisee learning frame- work for language understanding. Transactions of the Association for Computational Linguis- tics, 10: 10:1324-3440, 2022. t.a.ta.c11..1
doi: 10.1162/tacl a_ a_00221. URL
orgg/
[25] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang. Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 111931137 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 12.28453/w/2222..ccl 177. org/ wwwpg/////0000.. ac1- 172
URL U.RLpptts:/accaaathollgg
[26] Gregory Kamradt. Needle In A Haystack pressure testing LLMs. Github, 2023. URL htt.aa/eLLLLeeeeeeeeeeeeeeaaaaae
https://githur. com/
[27] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/21 14166, 2021. URL https://arxxi
14168
[28] Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks. In First Conference on Language Modeling, 2024. URL https::tps: //pperrviiee
12

---

[29] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025.
[30]
Meta. Introducing meta Ilama 3: The most capable openly available Ilm to date. https
//ai //aimmtta. http///oommmmeeeeeee 11ama-333 3// 2024. Accessed: 2024-06-07.
[31] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.on org/ wwtgg/abs/3110.. 205.0/6250
[32] An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. ArXiv preprint, abs/2407.10671, 2024.
/tattps///ariv....t abw///// 12t.//22077.061.1..
[33] Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. LM-infinite: Zero-shot extreme length generalization for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 3991-4008, Mexico City, Mexico, 2024. Association for Computational Linguistics. URL httphtttt//aclattbhtott wwwpoooooooooog////10 org/2024 naacl-long 222
[34] Xiang Liu, Hong Chen, Xuming Hu, and Xiaowen Chu. FlowKV: Enhancing multi-turn conversational coherence in LLMs via isolated key-value cache management. In First Workshop on Multi-Turn Inteructions in Large Language Models, 2025. httttttt////:rrrreeeet werffoommmmmdrr-ui
[35] Zichen Tang, Zhenheng Tang, Gaoning Pan, Buhua Liu, Kunfeng Lai, Xiaowen Chu, and Bo Li. Ghost in the cloud: Your geo-distributed large language models training is easily manipulated. In ICML 2025 Workshop on Data in Generative Models The Bad, the Ugly, /ttptttps::/ppppe/eeee
and the Greats, 2025.
www./oruuiiideeeeeeoooo nett/forum?id-qpdaagt..o
[36] Fanjunduo Wei, Zhenheng Tang. Rongfei Zeng. Tongliang Liu, Chengqi Zhang, Xiaowen Chu, and Bo Han. JailbreakloRA: Your downloaded loRA from sharing platforms might be unsafe. In ICML 2025 Workshop on Data in Generative Models The Bad, the Ugly, and the Greats, 2025. URL.
NswGh
[37] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thougle prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[38] Kunfeng Lai, Zhenheng Tang, Xinglin Pan, Peijie Dong, Xiang Liu, Haolan Chen, Li Shen, Bo Li, and Xiaowen Chu. Mediator: Memory-efficient 11m merging with less parameter confficts and uncertainty based routing. arxiv preprint 2rXiv:2502.04411 2025.
[39] Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. Active prompting with chain-of-thought for large language models. In Lun-Wa Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1330-1350, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/1/2200 1/2.6553/1/222aaa---mooo URL /estps:/aaclanttinn wot.aaannnnoooooooor/22244.4.. wtwp////../..ceoo ac1-
[40] Rui Pan, Shuo Xing, Shizhe Diao, Wenhe Sun, Xiang Liu, KaShun Shum, Jipeng Zhang, Renjie Pi, and Tong Zhang. Plum: Prompt learning using metaheuristics. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024, pages 2177-2197, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 10. htt.://.....nnnng.......... URL htttt::s//acaaathllloo org/2024. wwwniiinnnnncl.... 129
[41] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ArXiv preprint, Abs/2204.00311,, 2022. URL
https: /ttp:///xxvvvvvvv www./ogg///////oo..e/t.../...eet abs./204..... 22tp//2240002111111
13

---

[42] Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, et al. Many-shot in-context learning. arXiv preprint arXiv:2404. 1018, 2024.
[43] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023.
[44] OpenAT. Gpt-4o-mini: Advancing cost-efficient intelligence, 2023. Accessed: 2023-12-14.
[45] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint wwxiv:2402.0275555 2024.
[46] Chi-Chih Chang. Wei-Cheng Lin, Chien- Yu Lin, Chong- Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118. 2024.
[47] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. arXiv preprint urXiv:2402.02057 2024.
[48] Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. Can language models perform robust reasoning in chain-of- tf-hhught prompting with noisy rationales? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[49] Ingo Steinwart. How compare different loss functions and their risks. Constructive Ap- proximation, 26:225-287, 2007. URL WRL.https://apt semanticscholar .rtpoooooaooo//ooruSDD..... 16660598
[50] Bernardo Ávila Pires and Csaba Szepesvári. Multiclass classification calibration functions. arXiv preprint
2016.
[51] Kleijn and Van der Vaart. The bernstein voneeii---oo---ee von- von-mises theorem under misspecificaation. Elec- tronic Journal of Statistics, 6:354 2012. URL
org/
CorpusID: 85548207
[52] Erik Tjong Kim Sang and Jorn Veenstra. Representing text chunks. Henry Thompson and Alex Lascarides, editors, Ninth Conference of the European Chapter of the Associa- fir Computational Linguistics, pages 173-179, Bergen, Norway, 1999. Association for Computational Linguistics. /LL/Ittps://accaatthoooo.
org 2ttg//99-10223
[53] Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. In The Twelfth International Conference on Learning Representationss 2024.
[54] Weizhi Fei, Xueyan NIu, Pingyi Zhou, Lu Hou, Bo Bai, Lei Deng, and Wei Han. Extending context window of large language models via semantic compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Lin- guisties ACL 2024, pages 5 1189-118, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 10. 18653/v 1/2024. findings-acll.3000 URL https://acclattollea
org/ 2024..024..2 findings-acl... .36
[55] Antonio Jimeno Yepes. Yao You, Jan Milezek, Sebastian Laverde, and Renyu Li. Financial report chunking for effective retrieval augmented generation, 2024. URL https://arxiv
org/ hrggabss24422 05131
[56] Brandon Smith and Anton Troynikov, Evaluating chunking strategies for retrieval. Technical re- port, Chroma, 2024. URL https///reeee oty.ahara....cco///aaaaaaaaaanga chunking
cow
[57] Anthropic Introducing contextual retrieval, 2024. URL http::::/www..nthpp.....o
Bewea//cattextuul-retiieeee
[58] Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning. ArXiv preprint, abs/2403. 2024. URL holLihttss///raae
org/ abs/2403. //1.///43...999999..1
[59] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth
14

---

International Conference on Learning Representationns ICLR 2022, Virtual Event, April 25-29. 2022. OpenReview.net, 2022.
net/foormmid--eee VeeeFFF
[60] Zhenheng Tang. Xueze Kang. Yiming Yin, Xinglin Pan, Yuxin Wang, Xin He, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, Amelie Chi Zhou, Bo Li, Bingsheng He, and Xiaowen Chu. Fusionllm: A decentralized I1m training system on geo-distributed gpus with adaptive compression. arxiv preprint arXiv:2410. 12707, 2024.
[61] Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL.Ltttps://openrevvee wttpieeww.eeeeeeeeeeeeeeeeeeeeee Seaet/fooumiiddSS4xtaa
[62] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. ArXiv preprint, abs/2309 03883, 2023. URL htttt://tta::t:
org/ abs/2309. 038833
[63] Haoyi Wu and Kewei Tu. Layer-condensed kv cache for efficient inference of large language models, 2024. URL. httpps///aaat www.://gg///aata.eo.ooooooo /2055. 10637
[64] Yutao Sun, Li Dong, Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. arXiv preprint wrXiv:2405.05254 2024.
[65] William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ra- gan Kelly. Reducing transformer key-value cache size with cross-layer attention. arXiv preprint arXiv:2405. 12981, 2024.
[66] Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Mini- cache: Kv cache compression in depth dimension for large language models. arXiv preprint arXiv:2405. 011124455.1466.. 14366, 2024.
[67] Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. oo-bench: Extending long context evaluation beyond 100k tokens. ArXiv preprint, abs/2402. 2024. URL httpp://///xiii org/ 2b25/222/2 13718
[68] Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS; A zero- shot benchmark for long text understanding. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977- 7989, Singapore. 2023. Association for Computational Linguistics. doi: 10.18653/w1/2023 findings-emnlp.536
wwwphholooooooooooooooooooo 2023. findaffinniinnnmmmii
536
[69] Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. ArXiv preprint, abs/2307. 1088, 2023. |RL/ttts::/// /tttts:/tt/riiii// /wwpiv.xivvv. wwwps/.orrgbbbb//bbbbbee .10088
[70] Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random- -access infinite context length for transformers. ArXiv preprint, abs/2305. 2023. URL /ttpps://
arxiv. abs/ 2205. 16300
[71] Dacheng Li, Rulin Shao, et al. How long can open-source LLMs truly promise on context length?, 2023. http::tttt::mmmme ottps/.rggbboooo/ 2023- 06- 29- 2911ohgh
[72] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. doi: 10.1162/tacl a 00638.. .og. 2024/2024.22 t.c1-1.9
[73] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? ArXiv preprint, abs/2404.06654, 2024. BtLLsttps::/ 2404.06655
org/ /
[74] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena benchmark for efficient transformers. In 9th International Conference on Learning Representations, ICI.R 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httpss //openreview.c www.:ewww..eeeeeeeeeeeeeeeeeeeee net./forumiiddggyeeeee.... grC2k
15

---

[75] Xiang Liu, Zhenheng Tang, Hong Chen, Peijie Dong, Zeyu Li, Xiuze Zhou, Bo Li, Xuming Hu, and Xiaowen Chu. Can llms maintain fundamental abilities under kv cache compression? arXiv preprint arXiv:2502.01999 2025.
[76] Xiang Liu, Peijie Dong. Xuming Hu, and Xiaowen Chu. Longgenbench: Long-context generation benchmark. arXiv preprint arXiv:2410.04
2024.
[77] David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and con- trastive conditioning for controllability and toxicity reduction in language models. Find- ings of the Association for Computational Linguistics: EMNIP 2022, pages 5621-5634, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 1/t.665331/222...nnnnnn... Sadings-emmlp.... 412
URL htttps::/aclanthhlll0g org org/2022
[78] Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Dangi Chen. Adapting language mod- els to compress contexts, In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3829- 3846, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ v1/2023. 112022.emnll-male 23.. httthttps::///
orgg20233333 emnlp-man 232
[79] Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yi- fan Hou, Ryan Cotterell, and Mrinmaya Sachan. Recurrentgpt: Interactive generation of (arbitrarily) long text, 2023.
[80] Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, and Li Guo. Recursively summarizing enables long-term dialogue memory in large language models. arXiv preprini arXiv:2308. 15022, 2023.
[81] Huiqiang Jiang, Qianhui Wu, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Compressing prompts for accelerated inference of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pages 1 Singapore, December 2023. Association for Computational Linguistics. doi: 10. http:3//11222.mmlmmmmmooo m2in.825. URL https:// wrg//023..././ 2023. emnlp-main 825
aclanthology.c org
[82] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin- Yew Lin, Yuqing Yang, and Accelerating and enhancing LLMs in long context scenarios via
Lili Qiu. LongLLMLingun prompt compression. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1658-1677 Bangkok, Thailand, August 2024. Association for Computational org/2024. 2024. acl-long.99
Linguistics. URL btattps:/aclanttol..g
[83] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099. PMLR, 2023.
[84] Yilong Zhao, Chien- Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng. Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate I1m serving. Proceedings of Machine Learning and Systems, 1:19920 2024.
[85] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with single gpu. In International Conference on Machine Learning, pages 31094-311 11443-11166 16. PMLR, 2023.
[86] Zeyu Li, Chuanfu Xiao, Yang Wang. Xiang Liu, Zhenheng Tang, Baotong Lu, Mao Yang. Xinyu Chen, and Xiaowen Chu. Antkv: Anchor token-aware sub-bit vector quantization for kv cache in large language models, 2025. URL tRtpsstss://axiii org/ abs/ 19505
[87] Tomáš Kočiský. Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gá- bor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018. doi: URI.
10. 1622/tcc_a__00000
[88] Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz. Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings
16

---

[94] Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A new benchmark for query-basee based multi-domaie meeting summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilck Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5905-5921, Online, 2021. Association for Computational org/ of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599-461V Online, 2021. Association for Computational Linguistics. doi: 10. (0.18653/v1// 105533//122111aallmmmo
URL https://
aclanthology org.oog///22/ www.nnallllaa manecl-main.... 365
[89] Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 20069-38000. Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. (0.18653/v1// 1/D55//11D188 1- 1259. URL http:: http///aaaaathoooogyy.o / org/D18 1259
[90] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprchensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference Computational Linguistics, pages 6609-6625, Barcelona, Spain (Online), 2020. International Committee Computational Linguistics. doi: 10. 18653vv 1/20///////22oollongg main.580. URL
https://aclantholloe wwwpoologg.orgg/2/00 http://20...ooonn-.-.oo mong-main..888 580
[91] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022. doi: 10.1162/tacl_a_000 Lattaaall_0000000 00475. URL https
//aclanthology. orgooo////0 202gg/222.c20 tacl-1.c..
[92] Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang. Hua Wu, Qiaoqiao She. Xuan Liu, Tian Wu, and Haifeng Wang. DuReader: Chinese machine reading comprehension dataset from real-world applications. In Eunsol Choi, Minjoon Seo, Danqi Chen, Robin Jia, and Jonathan Berant, editors, Proceedings of the Workshop on Machine Reading for Question Answering, pages 37-46, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.188/3/vl 1B///////1-/2///
URL URRLBpctp::/accaantollgg hwwpchhhologyooogogoeooooo org Brg/WW8-220 2605
[93] Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419-1436, Online, 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2211 naacl-main. 1 12. URL /tttss://aclanteet
org/20211 www.aaallaaaa main 112
Linguistics. doi: 10.18603/11 1/2021 waa.cm-mmii447777 URL 2021 www.naaaalaa.aa m
[95] Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Anna Korhonen, David Traum, and Lluís Märquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10. 11111886611119111111 1102. URL https www.aaathoooooooooo111---1 //aclanthology www.oorggg/.... 1102
[96] Han Wu, Mingjie Zhan, Haochen Tan, Zhaohui Hou, Ding Liang, and Linqi Song. VCSUM:
A versatile Chinese meeting summarization dataset. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguisties: ACL 2023, pages 6065-6079, Toronto, Canada, 2023. Association for Computational Linguis- ties. doi: 10. 1/.188/33112223, 2tt.:3//////2....nnnnnggoool.l. findings-acl.377. URL. findings-an acl. .77
17

---

[97] Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th Interna- tional Conference on Computational Linguistics, 2002. UtL//ttps///accaaaooollg org/
C02-1150
[98]
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors. Proceedings of the 2nd Workshop on New Frontiers Summarization, pages 70-79, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v I/D19-5499 URL https: ltttp:://allanthologg
5409
[99] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1601-1611, Vancouver, Canada, 2017. Association for Computational Linguistics. doi: 10.18653/11P11-11144 URL https: aatp:///lllaathoooroooo
org/P17- 1147
[100]
Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. Longcoder: A long- range pre-trained language model for code completion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardtt Sivan Sabato, and Jonathan Scarlett, editors, Interna- tional Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 12098-12107. PMIR, 2023. URL.httss::/proceddiii
mlr www.ppressssse v202/ guo23j .html
[101] Tianyang Liu. Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository- level code complcomnn systems. The Twelfth International Conference on Learning Representations, 2024. htthttps: wtwp:rrvviiieeeeeefrreeeeee .et.://ooruuriiipppp)/0ooooo
18

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately describe ChunkKV, its advantages in semantic preservation, and the experimental results supporting these claims. See Abstract and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification:
Please refer to Section !
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sellispecifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides a theoretical understanding of ChunkKV with assumptions and a proof sketch. See Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper details experimental setups, datasets, models, and evaluation metrics.
See Section and Appendix E
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The paper states that code will be made available to ensure reproducibility.
See Impact Statement. Datasets used are publicly available. See Appendix J
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nis.cc/public//uust http://iccggiieeeeeeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
21

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings, model configurations, and dataset details are provided. See Section and Appendix G
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: The paper is focused on the LLM inference, and with same model, prompt, and temperature, the results are consistent.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper mentions GPU types (e.g., A40) and inference setup for efficiency experiments. See Section 4.2 and Appendix B.8
22

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compule required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research uses publicly available datasets and models, and aims to improve LLM efficiency. aligning with ethical research practices. See Impact Statement.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper includes an Impact Statement at Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
23

---

Answer: [NA]
The paper focuses on a compression technique rather than releasing new models datasets with high misuse risk. The models used standard, publicly available models.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Datasets are cited and their licenses are mentioned Appendix ! Models used are standard and widely known, with their original papers cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: The paper introduces a new method (ChunkKV) but does not release new datasets standalone models as assets. The code for the method is planned for release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the detaset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The research does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The research does not involve human subjects, so IRB approval is not applica- ble.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM only used for writing, editing, and formatting.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standard components.
Please refer to our LLM policy (https: hctps////cooaeeeeee/ee222222 for what should or should not be described.
25

---

Appendix
In-depth Analysis of ChunkKV vs. Discrete Token Methods
27
A.1
Quantitative Analysis
27
A.2
Hypothetical Scenario
28
A.3
Comparative Analysis
28
Implications for Model
Performance
29
A.5
Implementation Details
29
Additional Experiments
30
B.1
Layer-Wise Index Reuse
30
B.2
LongBench
33
................................
B.3
Needle-In-A Hodse-ln-AHayystckk
................................
33
Chunk Size
B.5
Multi- Mulgi-Lingual
42
B.6
KV Cache Quantization
42
B.7
Comparison with Orthogonal Training- Based Methods
B.8
Efficiency results
44
Theoretical Understanding
44
Additional Related Work
47
Statistics of Models
48
Statistics of Datasets
49
Prompt
49
H Impact Statement
50
Limitations
50
Licenses
51
26

---

A In-depth Analysis of ChunkKV vs. Discrete Token Methods
A.1 Quantitative Analysis
To rigorously evaluate effectiveness of ChunkKV compared to discrete token-based methods, we conducted systematic experiments using LLaMA-3-8B-Inttnur model. We randomly selected 100 sequences from the each sub-category of LongBench dataset and analyzed two key metrics across different model layers: KV cache L1 loss and attention cosine similarity. For each sequence, we: 1. Computed the full KV cache and attention patterns without compression as ground truth. 2. Applied ChunkKV, SnapKV, and H2O compression methods with fixed 10% compression ratio, and the parameters the three methods are set the same as in Table 17 3. Measured the differences between compressed and uncompressed versions.
Layer-wise cache
Layer-wise Attention Cosine Similarity
1.0
ChunkkV
SnapkV
H20
0.5
Cosine Similarity
Loxs
0.4
0.8
L1
0.3
0.7
0.2
0.0
35
Layer
Layer
Figure 4: Laser-wise comparison of L1 loss and attention cosine similarity between ChunkKV and discrete methods in Document QA sub-category of LongBench.
Results Analysis As shown in Figure ChunkKV demonstrates superior performance across both metrics:
KV Cache L1 Loss: ChunkKV achieves consistently lower L1 loss compared SnapK and H2O, particularly in the early and middle layers (layers 5-25). This indicates bet- ter preservation of the original KV cache information through the semantic chunk-based approach.
Attention Cosine Similarity: ChunkKV exhibits higher similarity scores across most layers, with notably strong performance in layers 0-5 and 20-30. This suggests better preservation of attention relationships between tokens, which is crucial for maintaining semantic understanding.
To quantify these improvements, calculated average metrics across all layers, shown in Table ChunkKV achieves both the lowest L1 loss and highest attention cosine similarity, outperforming
13
both baseline methods.
Significance of Results While the improvements may appear modest absolute terms (approximately 2% in LI loss and 1.5% in cosine similarity), their practical significance is substantial. These metrics reflect the model's ability to maintain crucial semantic relationships and attention patterns, which essential for complex reasoning tasks. The consistent improvements across different sequences demonstrate that preserving semantic chunks leads to better information retention than selecting individual tokens.
The enhanced performance is particularly evident in the middle layers the model, which are typically responsible for higher-level semantic processing. This provides concrete evidence for why ChunkKV achieves superior performance on downstream tasks compared discrete token-based methods.
27

---

Table 13: Detailed comparison of KV cache metries across different task categories in LongBench.
Single-Documar QA
Multi-Doeument
Few-shot Learning
Synthetic Code
Method
Summarization
QA
KV Cache LI Loss +
ChunkKV
0,8741
0.8748
0,8770
0.8861
0.8726
SnapKV
0.8921
0.8933
0.8930
0.8917
0.8938
H2O
0.8905
0.8917
0.8913
0.8906
0.8915
Attention Score Cosine Similarity f
ChunkKV SnapKV H2O
0.3567
0.3651
0.3841
0.4330
0.3805
0.3513
0.3594
0.3771
0.4305
0.3759
0,3491
0,3572
0.3750
0.4284
0.3740
A.2
Hypothetical Scenario
To provide a deeper understanding of ChunkKV's effectiveness compared discrete boken-based methods, we present detailed analysis using hypothetical scenario. This analysis aims to illustrate the fundamental differences between these approaches and explain why ChunkKV V is more effective at preserving semantic information in long contexts.
Consider comprehensive document that contains detailed information on various animals, including their habitats, diets, and behaviors. A user asks the question "What do pandas eat in the wild?"
Both ChunkKV and discrete boken-based methods would use this question to calculate attention scores for the document. However, their approaches to selecting and retaining information differ significantly.
A.2.1 Discrete Token-based Method
discrete token-based method might identify and retain individual tokens with high relevance scores, such as:
"pandas", "eat", "bamboo", "wild", "diet", "food"
Although these tokens are relevant, they lack context and coherence. The method might discard other essential tokens that provide crucial context or complete the information.
A.2.2 ChunkKV Method
In contrast, ChunkKV would identify and retain semantically meaningful chunks, such as:
"In the wild, pandas primarily eat bamboo shoots and leaves" "
"Their diet consists of 99% bamboo, but they occasionally consume other vegetation"
"Wild pandas may also eat small rodents or birds when available"
By preserving these chunks, ChunkKV maintains not only the relevant keywords but also their contextual relationships and additional pertinent information.
A.3 Comparative Analysis
The advantages of ChunkKV become evident when we consider how these retained pieces of information would be used in subsequent processing:
1. Contextual Understanding: Discrete tokens require the model to reconstruct meaning from isolated words, which could lead to ambiguity. ChunkK provides complete phrases or sentences, allowing for immediate and accurate comprehension.
28

---

2. Semantic Coherence:
ChunkKV preserves the semantic relationships within a chunk, crucial to understanding nuances such the difference between primary and occasional food sources for pandas.
A single chunk can contain multiple relevant tokens in their proper context, potentially retaining more useful information within the same compressed cache size compared to discrete methods.
Information Density:
4. Reduced Ambiguity: Discrete methods might retain the token "eat" from various sentences about different animals. ChunkKV ensures that "eat" is preserved specifically in the context of pandas in the wild.
5. Temporal and Logical Flow: ChunkKV can maintain the sequence of ideas present in the original text, preserving any temporal logical progression that may be crucial for understanding.
A.4 Implications for Model Performance
This analysis suggests several key implications for model performance:
Improved Accuracy: By retaining contextually rich information, ChunkKV enables more accurate responses to queries, especially those requiring nuanced understanding.
Enhanced Long-context Processing: Preservation of semantic chunks allows for better handling of long-range dependencies and complex reasoning tasks.
Reduced Computational Overhead: Although both methods compress the KV cache, ChunkKV's approach may reduce the need for extensive context reconstruction, potentially improving inference efficiency.
Versatility: The chunk-based approach is likely to be more effective across wide range of tasks and domains as it preserves the natural structure of language.
This in-depth analysis demonstrates why ChunkKV is more effective in preserving semantic informa- tion long contexts. By retaining coherent chunks of text, it provides language models with more contextually rich and semantically complete information, leading improved performance in tasks that require deep understanding and accurate information retrieval from extensive documents.
A.5 Implementation Details
In our implementation of ChunkKV, we focus on maximizing computational efficiency to ensure practical deployment in real-world applications. In algorithm we adopt several key optimization strategies reduce both computational overhead and memory footprint during inference.
The optimized implementation features several key improvements:
Vectorized Operations. We leverage batch matrix operations for attention score calculation instead of explicit loops, significantly reducing computational overhead. By computing chunk scores through optimized tensor operations (lines 7-11), we achieve substantial acceleration compared to token-by- token processing.
Memory Optimization. Our implementation employs a binary mask approach (lines 15-20) that avoids redundant memory allocations and reduces fragmentation. This single-pass token selection strategy consolidates both chunk preservation and recent window retention operations, leading to more efficient memory usage during inference.
Boundary Handling. We incorporate robust boundary checks throughout the algorithm (lines 10, 12, 18) to handle edge cases such as incomplete chunks or limited compressed length. This ensures stable performance across variable input lengths without requiring special case handling.
Single-pass Compression. Rather than performing multiple concatenation operations, our imple- mentation builds comprehensive selection mask that includes both semantically important chunks and recent context tokens. This approach (lines 21-25) minimizes memory copying operations and reduces computational overhead.
Our PyTorch implementation further extends these optimizations by leveraging GPU acceleration for all vector and matrix operations. The integration of CUDA kernels for critical operations like
29

---

Algorithm 3 ChunkKV Implementation
Input: Q E RT,xd K E R V R..dd observe window size w, chunk size c. compressed
KV cache max length Lmax
2: Output: Compressed KV cache K', V
3: Gobserve f- Qr.-w:t
Extract observe window queries RexTx attention scores Number of chunks
4: A < Jobserve K
5: C L&
6: Allocate chunk soonnksssoessseeeeeee E R to store chunk importance
7: Vectorized Chunk Score Calculation:
8: for 00 do
9: start i.c
10: end +- min((i 11.1).. .,, Ts)
11:
chunk_scores/ +- start start 20 Agi
rnn
Vectorized sum
12: end for
= X-min(] [_ .C)
Limit k to available chunks Indices of top-k chunks
kept_chunks
15: Build selection mask:
mask < -Teros(T&)
Binary mask for token selection
17: for each i in cepp__chunkseee do
18: start 4 i.c
19: end +- min(min + 1) Ts)
20: mask(start: end] +-
> Mark entire chunk as kept
21: end for
22: Ensure recent context is preserved:
23: mask[T; w Ty] <- 1
Always keep most recent tokens
24: Efficient compression:
25: indices nanzero(mask)
Get indices of tokens to keep
26: K' < index_select K, indices)
27: V f- index belex_select(// V, indices)
28: Return K', V
attention score calculation and top-k selection enables real-time performance even with long context windows.
B
Additional Experiments
B.1
Layer-Wise Index Reuse
B.1.1 Efficiency Analysis
The layer- wise index reuse method significantly reduces the computational complexity of ChunkKV. Without index reuse, ChunkKV would be applied to all Mayers layers, resulting in total compression
time of Nuyers T commpress. where T Trompress is the time taken to compress layer. With index reuse,
ChunkKV is only applied to New
Nuwes Niwe layers, reducing the total time to New Teomposss + (Ninyen
Nujes Teetects where Trelect is the time taken to select indices, which typically much smaller than Nasue
compress. This results in theoretical speedup factor of:
N Nayers
T Trompress
Speedup
Naver Nease
T compress
(Nuyers
Nussn Neane
Trclect
Assuming Trelect is negligible compared to Trompress. this simplifies approximately Nreme In practice, the actual speedup may vary depending on the specific implementation and hardware, but can still lead to substantial time savings, especially for models with large number of layers.
30

---

B.1.2 Index Reuse Performance
5 and Table [ iillustrat the performance of ChunkKV with varying index reuse layers on the the performance ChunkKV with varying index reuse layers the LongBench benchmark. The experiment reveals that math problems are more sensitive to index reuse layers compared to LongBench. Both LLaMA3-BB-Instnee and Qwen2- 7B-Instruct exhibit significant performance degradation, with LLLMA3-8B-IInstt experiencing a steeper decline after two layers of index reuse than Qwen2-7B-Insee Qwen2-7B Instrruttt model may be more robust to index reuse.
GSM8K benchmark. Figure
This suggests that the
Index Reuse Performance GSM8K
LLaMA-3-8B-Im
70
Twen2-7B-Inss
GSMBK Score
50
30
10
2
10
20
20
30
32
Index Reuse Depth
Figure 5: GSM8K Performance Comparison with different index reuse layers
Table 14: Reusing Indexing Performance Comparison GSMSK
Number of Index Reuse Layers
Model
2
3
5
8
10
20
28/32
LLaMA-3-SB-Instruet
74.5
74.6
65.9
44.1
15.3
2.20
1.60
1.80
71.2
71.2
73.0
69.4
67.4
71.1
54.0
49.4
Table 15: Reusing Indexing Performance Comparison on LongBench
Number of Index Reuse Layers
Model
10
20
28/32
LLaMM-33-BB-Insrruccc
40.51
40.27
39.45
40.05
39.59
37.42
35.67
Mistral-7BB-Intrrue
46.71
46.43
46.48
45.90
44.36
41.84
39.56
Qwen2-7B-Instruct
40.88
40.76
40.91
41.58
40.19
37.42
36.79
B.1.3 Layer-Wise Index Similarity
This section details the experiment of layer-wise index reuse similarity described in Section 3.3 The inference prompt is randomly selected from the LongBench benchmark, and the preserved indices for H2O, SnapKV, and ChunkKV are saved in the log file. For multi-head attention, only the indices of the first head are saved. PyramidKV, which has varying preserved index sizes across different layers, is not applicable for this experiment. Then we calculate the Jaccard similarity of the preserved indices adjacent layers for different models. Table 16lshows the Jaccard similarity the preserved indices of adjacent layers for different models.
Figures 79
and 13 13 Qwen2-7B-Instrreee display the heatmaps of layer-wise indices similarity of the preserved KV cache indices by H2O, SnapKV and ChunkK on different models. The pattern the layer-wise indices similarity heatmap
TO 12
consistent across different models, aligning with our findings in Section 3.3
31

---

Index Reuse Performance on LongBench
48
LLaMA-3-8B-Mm
46
Mistral-7B-Ine
Qwen2-7B-Inst
LongBench Score
44
42
40
38
36
0
2
10
20
28
30
32
Number of Index Reuse Layers
Figure 6: Comparison with different index reuse layers on LongBench.
Table 16: Retained KV Cache Indices Similarity of Adjacent Layers for Different Models.
Method
H20
ChunkKV SnapKV
LLaMA-3-4 MAMMA---8B-Instruue 25.31%
27.95%
57.74%
1ant2--B-Insttuee
14.91%
16.50%
44.26%
Cestral-B--Instrue
15.15%
15.78%
52.16%
32

---

LLaMA-3-8B-Inse with H2o
6.6
Layer
14
12
13
1
7
Layer
Figure 7: Layer-wise similarity heatmaps of the preserved KV cache indices by H2O on LLaMA-3- 8B-Instruct
B.2 LongBench
The Table 17 shows the average performance of KV cache compression methods in the LongBench English subtask categories. The ChunkKV achieves the best performance on the overall average, and the Multi-Document QA category, which supports that chunk method is more effective for semantic preservation.
Table 18 shows additional experiments with LLaMA-3-70B-Instruu comparing our ChunkKV method with baselines with KV cache size 256 on the LongBench dataset:
B.3 Boolle-In-A-Hassteer
Figure 16]and and 17 visualizes the performance ChunkKV on the NIAH benchmark for LLaMA-3- 8B-Instruct and distral-7B-Inntrrus with KV cache size of 128 under 8k and 32k context length. The performance of ChunkKV is consistently better as the context length increases.
33

---

LLMM--38B-Instruun with SnapKV
Layer
12
Layer
Figure 8: Layer-wise similarity heatmaps of the preserved KV cache indices by SnapKV on LLaMA- 3-8B-Instruct
ROMAA-3-B-IIntrrun with ChunkKV
Layer
23
Layer
Figure 9: Layer-wise similarity heatmaps of the preserved KV cache indices by ChunkKV on 1LLMA-3-8--Instte
34

---

Mistral-7B-Insteee with H2o
cow B.C.
Layer
12
-013
13 1-008
Layer
Figure 10: Layer-wise similarity heatmaps of the preserved KV cache indices by H2O on Mistral-7B- Instruct
Mistral-7B B- B-Instruct with SnapKV
con
8 1.000
Layer
Figure 11: Layer-wise similarity heatmaps of the preserved KV cache indices by SnapKV Mistral- 7B-Instruct
35

---

Mistral-7B-Instaee with ChunkKV
Layer
-014
12
521
Layer
Figure 12: Layer-wise similarity heatmaps of the preserved KV cache indices by ChunkKV on Mistral-7B-IIntruuet
Qwen2-7B BB-nstructt with H2o
3.0
Layer
12
18
13
22
Layer
Figure 13: Layer-wise similarity heatmaps of the preserved KV cache indices by H20 Qwen2-7B. Instruct
36

---

with SnapKV
Layer
73
13
73
7
923
SAR 821
821
Layer
Figure 14: Layer-wise similarity heatmaps of the preserved KV cache indices by SnapKV Qwen2- 7B-Instruct
Qwen2-77 Brven2-7B-Insrrune In-Intut with ChunkKV
14
52
72
13
22
Layer
Figure 15: Layer-wise similarity heatmaps of the preserved KV cache indices by ChunkKV Qwen2-7B-Instne
37

---

Table 17: Comprehensive performance comparison of KV cache compression methods across LongBench English subtasks. Results are shown for various models and tasks, highlighting the effectiveness of different compression techniques.
Single Documen:
QA
Mult Decument Q4
Summnnntation
Few-shot Learning
Synthetic
Cide
Method
Nnv04
Qpaper
Mi
Hespar2A
2WiaMMA
Misique
GevRopont
QMSun
MaldNows
TREE
TrivinJA
SAMSonn
PCamn
RB-P
Asg.
Avg
18,419
3619
4,590
9.511
11,214
10,014
2.13
5,77
8,200
6,258
9,289
4,306
180--2-80--asmaen,, Size Full
FellKV
25,70
20:75
35.87
22.35
25.63
26.11
73.00
69
50.77
41.46
Compression
StreamingL.LM
55.36
24.90
37.06
StapKy
25.08
23.02
20.20
71:00
05
PeramidKV
25.58
20.77
71.50
24.89
22.96
20.82
71.50
57.48
40.51
Instruct,
Compeession
207
LL
23.35
18.97
29:37
21,92
71.00
51.12
H20
43.06
90.57
41.67
50.95
37.79
S
25.50
20.49
72.50
69
51.25
PyramidKy
25.36
26.05
37.99
44.21
35.65
21.43
25.52
23.43
23.47
72.
56
41.45
5.26
69
50.93
40.63
Chunk
26.13
28:43
38.89
44.46
34:13
21:06
24.73
25:11
22.91
71:50
90.56
41:51
3.09
69
58.17
52.51
40.74
1iiMa-3-SB-Innstmuc KV
Size Compression Rario 300%
LM
24.89
22.53
35.30
44.33
32.81
19.00
27.12
22.19
25.93
72.50
41.75
5.41
69.00
60.10
55.13
40.48
1120
25.87
23.03
37.06
43.71
33.68
20.93
24.56
23.14
23.58
50.33
90.77
41.96
4.91
69:25
59.38
55.39
39:23
V
25.15
28.75
39.28
43.57
30.16
21.5%
25.56
23.19
24.30
73.00
90-52
41.70
4.96
69:25
60.27
55.74
41.43
PsramidKV
25.42
27.91
38.81
44.15
36.28
21.72
26.50
23.10
24.28
72.00
90.56
41,87
4.57
69.50
60.09
55.19
41.37
Chonkyy
25.88
29.58
38.99
43.94
34.16
21.70
26.50
23.15
23.95
72.00
90.56
42.47
5.34
69:25
61.68
56.35
41.59
Miscral 7B Instruct v0.3, KV
Size - Full
PalKyV
29.07
41.58
52.88
49.37
39.01
28.58
34.93
25.08
27.74
76.00
88.59
47.59
6,00
98.50
61.41
02.39
48.08
Mistral-78. 7B-i--tttt----000 v0.33 KV Stze Compression Ratio
10%
StrcamingL..M
25.15
25.47
30.08
44.39
32.49
19:40
24.11
20.85
19.55
65.00
88.21
43.83
4.50
79.50
59.48
58.82
40.11
H20
29.35
33.39
50.39
49.58
36.76
27.42
25.16
28.75
22.12
42.00
89.00
47.04
5.50
08. 51
57.58
59.24
43.61
StapKy
28.54
36.88
53.42
$0:15
38.11
22.00
26.67
25.21
22.33
73.00
89.36
45.44
5.50
99.00
59.70
61.63
46. 38
ParamidKV
20.40
35.89
$2.96
40.93
38.67
28.63
27.59
24.99
22.77
24.00
90.02
46.07
4.00
98.
00:50
58.54
60.88
46. 30
ChunkkV
29.75
36.82
$3.99
$0.33
38.72
29.01
27.03
24.76
21.42
76.00
88.73
46.49
5.00
98:00
59.98
61.37
46.71
Owen2
7B Instruct KV Size - Poll
FellKV
25.11
43.64
44.20
14:25
13:22
9.08
36.38
23.43
26.53
77.00
89.99
46.88
6.75
75.92
60.17
61.84
40.71
Qwen2-7B-hhe 70-mn----- KV Sire Compression Ratio
10%%
LMreamingLLM
25.15
45.42
41.46
13.66
11.95
8.72
32.79
21.49
26.24
77.50
89.15
46.54
7.50
50.50
60.03
60.91
38.56
H2U
26.17
44.33
42.54
12.81
12:46
9.15
33,24
22.69
25.94
76.50
89.44
44.32
8,00
76.00
61.28
02.39
40.45
StapKV
26.84
45.96
45.79
14.27
13.35
9.91
32.62
22.70
25.83
77.00
89.19
46.71
7.50
71.30
60.35
61.37
40.55
ParamidKV
27.51
44.45
43.90
13:35
13.13
9.12
32.28
22,60
25.45
77.00
89.84
44.53
7.00
73.50
60.1
61.24
40.31
Chinkyy
26.45
44.19
45.04
15.94
12:00
10.52
32.38
22,87
25.91
77,50)
89.23
44.78
8.50
76.30
60.64
61.32
40.58
Table 18: Comprehensive performance comparison of ChunkKV across LongBench English subtasks with 70B model.
Single-Deeumatt QA
Mull--Deumeer QA
Summarization
Few-shot Learing
Syuthetis
Ciod
Method
N
Quper
Mi--nn
HespaiQA
JWMMMQA
Mosqque
GavRsuort
QMSam
MultiNeess
TREC
TavlagA
SAMSiu
PCemm
PRS
L
RILF
Arg.
Avg. len
18,409
3,619
4,589
9,151
4,887
11,214
8,734
10614
2
5,77
8,209
6,258
11,141
9,289
1,235 4
L4MM----00--nssrree KV Size - Full
FullkV
27.56
46.00
49.75
50.96
56.16
29.37
31.66
21,80
27.42
75.50
92.58
46.08
12,00
73.50
45,02
68.74
47.15
LiaMa-3-7P 3-70B Instruct, KV Cache Size
256
HEO
26.42
37.52
36.23
47.80
49.5
49.34
24 30
26.96
20.67
36.50
72.00
92.16
43.56
12,00
73.00
47.21
63.13
43.67
SnapKV
26.79
42.25
48.75
50.89
25.14
23.85
22
24.62
73.00
44.53
12.00
72.50
54.24
65.60
46.11
ChankKV
27.37
45.95
49.23
22.63
12,00
49.51
47.24
38

---

(a) ChunkKV, accuracy 73.8%
(h) PyramidKV, accuracy 65.15 %
(c) SnapKV, V, accuracy 58.9%
(d) H20, accuracy 47.9%
(e) StreamingLLM, accuracy 23.7%
Figure 16: NIAH benchmark for LLaMA-3 3-B---3---nnntteett with KV cache size=128 under context length
39

---

(a) ChunkKV, accuracy 99.8%
(h) PyramidKY, accuracy 99.3%
(c) SnapK accuracy 91.6%
(d) H20, accuracy 88.2%
(e) StreamingLLM, accuracy 44.3%
Figure 17: NIAH benchmark for Mistral-7B 7B-IIntruct with KV cache size=128 under 32k context length
40

---

B.4 Chunk Size
Table 19 and show the performance of ChunkKY with different comperession ratios and different chunk sizes on the LongBench and NIAH. We conducted extensive experiments across different compression ratios and KV cache sizes to shows the effectiveness of ChunkKV and the chunk size is robust.
Table 19: LongBench Performance with Different Chunk Sizes and Compression Ratios for LLaMA- 3-8B-Instruct
Compression
Chunk Size
Rate
3
5
10
15
20
30
10%
37.32
40.49
40.47
40.51
40.21
40.05
39.57
20%
38.80
40.66
40.57
40.74
40.53
40.46
40.04
30%
39.23
41.02
41.29
41.59
41.38
41.33
41.02
Table 20: NIAH Performance with Different Chunk Sizes and KV Cache Sizes for LLaMA-3-8B.
Instruct
KV Cache
Chunk Size
Size
3
10
15
20
30
96
41.0
63.2
65.2
70.3
67.2
65.3
53.1
128
47.9
65.6
69.1
73.8
72.3
72.0
71.2
256
61.7
70.3
71.2
74.1
73.2
72.3
71.1
512
68.6
72.6
72.5
74.5
74.3
74.0
72.6
LongBench Performance vs Chunk Size
48
46
LongBench Score
LLaMA-3-8B
Mistral-7B
44
Qwen2-78
LLaMA-3-88 Full KV
Mistral-7B Full
42
Qwen2-78 Pull
40
38
36
20
Chunk Size
Figure 18: LongBench Performance Comparison with different chunk size under 10% compression rate.
Figure shows the performance the ChunkKV with different chunk size on the LongBench and NIAH benchmarks. The three colorful curves represent three L.MSs with different chunk sizes, and the colorful dashed line the corresponding FullK performance. From Figure [8] we can observe that the LongBench performance of ChunkKV is not significantly affected by the chunk size, with
41

---

performance variations less than 1%. The three curves are closely aligned, indicating that chunk sizes the range of (10, 20} exhibit better performance.
Table 21 and Figure 19 shows that the ChunkKV with different chunk sizes on GSM8K displays the same curve pattern LongBench. The CoT prompt length for GSM8K is only 1K tokens, the optimal chunk size range is smaller.
GSM8K Performance Chunk Size
HLMM--3-8B-nsttrn
Qwen2-71 ---nstruttt
80.0
80.0
77.5
77.5
75.0
75.0
GSMBK Score
GSV Score
72.5
72.5
70.0
70.0
67.5
67.5
65.0
65.0
62.5
Full KV
62.5
Full KV
60.0
60.0
3
10
20
10
20
Chunk Size
Chunk
Figure 19: GSM8K Performance Comparison with different chunk size
Table 21: GSM8K Performance Comparison with different chunk sizes
Chunk Size
Model
Full KV
3
5
10
20
LLaMA-3-8D-Instructt Swen2-7B-Instruse
74.6
74.5
73.9
63.2
76.8
73.5
71.2
71.8
71.7
71.1
B.5 Multi-Lingual
Table 22 is the Chinese support model Qwen2-7B --nstructt evaluated on the LongBench Chinese subtask, where ChunkKV achieves better performance than other compression methods and the full KV cache performance. Both the English and Chinese results indicate that ChunkKV a promising approach for maintaining erucial information in the KV cache.
Table 22: Performance comparison of Chinese subtask on LongBench for Qwen2- Swen2-7B-Instruct.
Single
Single-Document Document QA
Multi Document QA Summarization
Few shot Learning
Synthetic
Method
MF-2h
DuReader
VCSum
LSHT
PR-2h
Avg.
Avg len
6,701
15,768
15,380
22,337
6,745
Qwen2-70 7B Instruct, KV Size Full
FullKV
39.17
23.63
16.21
43.50
70.50
38.60
Qwen2-7B 1an22-B--Instruee KV Size Compression Ratio 10%
StreamingLLM
38.05
23.24
15.92
40.50
44.50
32.44
H20
37.99
19.58
16.16
41.67
67.35
36.55
SnapKV
44.25
20.27
16.24
44.50
68.10
38.67
PyramidKV
36.57
20.56
16.15
43.50
66.50
36.55
ChunkKV
45.92
20.15
16.37
43.75
71.10
39.45
B.6
KV Cache Quantization
For comprehensively evaluate the effectiveness of ChunkKV, we conducted experiments comparing ChunkKV with quantization methods KIVI [45). While both approaches aim to optimize LLM
42

---

inference, they operate on fundamentally different principles: quantization reduces KV matrix precision, whereas our eviction method reduces KV matrix size.
From an inplementation perspective, quantization methods require the full KV cache during prefilling to produce quantized representations, which then used during decoding. In contrast, ChunkK employs token removal prior to prefilling, enabling operation with a compressed cache throughout the entire inference process. This distinction creates different efficiency profiles, and each method offers unique advantages.
The following tables present detailed analysis comparing ChunkKV with KIVI, across both performance and efficiency metrics on the LLaMA-3-8B- In-tructt model. should be noted that due to KIVI's dependency on older Python versions, there may be some discrepancies between the efficiency results reported here and those in Table g of our paper.
Table 23: Comprehensive performance comparison of ChunkKV and quantization methods across
LongBench English subtasks,
Single Document QA
Muli Document QA
Summarizatio
Few shot Learning
Synthetic
Code
Mathed
NIsQA
Qaeper
Mf--n
HosporDA
ZWAUMOA
Masiqes
GevRepon
@MSom
MultNewa
TREC
TrivisQA
SAMSum
PComt
PRa
KB-P
Arg. f
18,409
3
4,550
9,151
11,114
R,74
10,644
2,119
5,177
8,200
6,258
11,14
9,089
1,235
4,206
MoMa----8--mmmman Size
25.63
23.03
ChaakKV with conipiciion
ChonkKV-10%
20.65
22.00
22:07
5.93
57.48
40.51
ChankKV-20%%
34.72
22.91
ChonkKV:30%
21.70
25/15
quartistioon
22.05
22
4.25
72
46
40.54
KIVI-thäts
72.50
4.35
50.91
41.11
45.50
23.01
Table 23 demonstrates that C with 30% compression ratio achieves comparable performance (41.59) to KIVI with 8-bit quantization (41.43) across LongBench subtasks. Notably, ChunkKV shows particularly strong results in code-related tasks, while KIVI maintains better performance in single-document QA scenarios.
These results highlight the complementary strengths of eviction and quantization approaches. ChunkKV's semantic chunk-based method delivers particularly strong efficiency benefits while maintaining competitive performance with state-of-the-arr quantization techniques.
B.7 Comparison with Orthogonal and Training-Bai Based Methods
To better position ChunkKV in the broader landscape of inference optimization, conducted additional experiments comparing our training-fit free eviction method against Palu [46), recent training-based compression method. The experiments were conducted on the LongBench benchmark with the LLaMA-3-88 --B-------Innnturct model.
Table 24: Performance comparison of ChunkKV with Palu (traiing-baseed) on the LongBench benchmark (LLaMA-3-8B-NN) Inntrrucc..... ChunkKV demonstrates a superior performance-efficiene trade-off compared training- based methods on this diverse benchmark and remains competitive with orthogonal quantization techniques.
Method
Single-Doe QA
Multi-Doc QA
Summarization
Few-shot
Synthetic
Code
Average Score
FullKV (FP16)
32.19
34.59
24.96
68.48
36.96
54.41
41.46
Eviction Methods (Qurx)
ChunkKV (10% Ratio)
28.50
33.46
22.20
67.62
37.47
58.98
40.51
CbunkKV (30% Ratio)
31.48
33.27
24.53
68.34
37.30
59.02
41.59
Training- Methods Palu (50% Ratio)
8.77
7.43
20.71
61.62
8.59
18.43
22.48
Pala (70% Ratio)
10.06
8.31
23.71
68.64
35.00
38.97
28.84
The results in Table 24 highlight key differences. Palu, as a training-based method, experiences significant performance degradation on the diverse LongBench benchmark, which represents an Out-Of--istributiinn (OOD) challenge relative to its training data. contrast, our training-free method, ChunkKV, demonstrates robust performance.
43

---

B.8 Efficiency results
Table 25 shows the efficiency results for different KV cache strategies with varying output lengths and the metrics are Time to First Token (TTFT) and Token Processing Time (TPOT).
Table 25: Efficiency Results for Different KV Cache Strategies with Varying Output Lengths
Method
Input
Output
TTFT(S)
TPOT(ms/token) +
Output Length 1024
FullKV
4096
1024
1.17
42.58
StreamingLLMM
4096
1024
1.02 (12.8%)
35.94 (15.6%)
SnapKV
4096
1024
1.04 (11.1%)
36.24 (14.9%)
ChunkKV
4096
1024
1.03 (12.0%)
36.64 (13.9%)
ChunkKV_ rrrusc
4096
1024
1.00 (14.5%)
36.47 (14.3%)
Output Length = 4096
FullKV
4096
4096
1.17
42.85
StreamingLLM
4096
4096
1.09 (6.8%)
40.01 (6.6%)
SnapKV
4096
4096
1.10 (6.0%)
40.24 (6.1%)
ChunkKV
4096
4096
1.10 (6.0%)
40.17 (6.3%)
ChunkKV _eeuse
4096
4096
1.08 (7.7%)
40.16 (6.3%)
Table 26: Latency and throughput comparison on ILLaMA-3-8B ----------B--sttreet,, including long-conter scenarios up to 16k tokens. Percentages in parentheses indicate improvements over the FullKV baseline. ChunkKV with layer-wise reuse (ChunkKV_reuse consistently delivers the best efficiency gains, especially in long-context settings.
Sequence Length
Performance Metries
Method
Input
Output
Latency(s) L Throughput(T/S) t
4k Context Length
FullKV
4096
1024
43.60
105.92
SnapKV
4096
1024
37.92 (13.0%)
120.42 (13.7%)
ChunkKV
4096
1024
37.52 (13.9%)
118.85 (12.2%)
ChunkKV rruuee
4096
1024
37.35 (14.3%)
124.09 (17.2 %)
16k Context Length
FullKV
16384
1024
49.60
323.24
SnapKV
16384
1024
39.20 (21.0%)
381.11 (17.9%)
ChunkKV
16384
1024
38.82 (21.7%)
381.61 (18.1 %)
ChunkKV reuse
16384
1024
36.96 (25.5%)
389.21 (20.4%)
C
Theoretical Understanding
In this section, we provide the theoretical interpretation from the perspective from the In-context learning (ICI.) to further understand how ChunkK V outperforms token-level KV cache compression.
Pretraining Data Distribution. Given set of concepts ④ and a concept 0 O define the pretraining data is sampled from p(o,,..., OT) Sage p( (0) (())p()))) (0 d0 24 Each token is sampled from a vocabulary O. For simplicity, we write 01., = 01 O,-
Language Modeling. Current L.MMs modelling, which predicts the next token o, given the previous tokens 0 ... 0t-1 for all 1..... T. Formally, a language modelling can be writen the distribution f(or/01:- And it is pretrained on
24 usually utilize the next word prediction the language
44

---

a huge corpus sampled from the pretraining distribution p(o1..... 0++1) 124. Considering the large scale of the model size and dataset size, it can be assumed that the f( ....))1) has been aligned with the p(0 ... 0++1) 24
Prompt Distribution. Following 24 prompt is composed of input token sequence followed by an output token y. Then, the i-th training example that can appear in any place in the whole prompt 0L is defined as O consisting of an input x; Og [1 k 1] (the first k 1 tokens) followed by the output
O; [k] at the end, where the length k is fixed for simplicity.
The i-th training example is independently generated follows: 1) Generate start hidden state h from a prompt start distribution Pprompti 2) Given ht! generate the example sequence O; = [a,, vil from p(O,) 1 (r). The test input Fiest 2n+1 is sampled similarly. Then, the prompt consists of sequence of training examples (S,) followed by the example Tiest:
SSm Tress
(x1, 1/1 T2, U2, ................................ Tin Une, Frest
Prrcmpt-
In-context learning setups and Assumptions. We follow other settings and assumptions in [24].
With the greedy decoding (47), sampling the next token from the language modeling f( (01:t-) becomes the predictor as y = arg max,, f(or(o1.t-))
Thus, for [S,, Fueul the in-context learning predictor be written (Fues)
arg max, p(y|S Sn, Fiss), which outputs the most likely prediction over the pretraining distribu- tion conditioned on the prompt distribution. Its expected 0-1 error with n examples is (on) E taes Prompp [1[f (Fies) * Vesul]
We define ph (0) =- p(O[i] o00 i 1) of the i-th token with previous tokens and the analogous distribution Perompt == Pprompe [O[i) = ol0[1 : ---1 1]) under the prompt distribution. Following (24), there is a distinguishability condition formalizes when in-context learning occurs giving the concept A.
The distinguishability condition is dependent on a KL divergence between the previous two distri- butions and the error terms resulting from the distribution mismatch between the prompt and the pertaining distributions for each example. Letting pi(o) and Pprompt correspond to the concept 0 and 0"
Condition 1 (distinguishability 24
The O is distinguishable for all 0 Q, 0 0",
k
Σ
KL,(0*100) > EB,
(3)
where the KL, := Eojts- Pprromps (P lp
1ph]]
Noises from KV Cache Compression. Naturally, because of the sparsified KV cache, some history tokens in 01:1-1 at different layers lost its attention score calculation with respect the next word prediction Oç- We can regard this as the noise added onto the 01:-11 Thus, distincting 0 from requires larger KL divergence. Following (48), we provide the following second condition about the distinguishability with the KV cache sparsity.
Condition 2 (distinguishability under sparsified KV cache). With the noise introduced by the sparsified KV cache of the sparse ratio r, the distribution mismatch between the prompt and the pretraining distribution that approximated by LLM is enlarged, resulting in a varied requirement with error term Eo(r) for 0 being distinguishable if for all 0 E O 0 & 0",
Σ (L*(10))0)
co Er(r,, where (a(r) x r.
Lemma 1 (noisy-relaxed bound in 24 Condition I
let B denotes the set of O which does not satisfy
We assume that (Proore (Steat (rest) ) (pp ([eesr[eeeleee Tress 0) bounded for all 0 and that 0 minimizes the multi-class logistic risk as,
LCE(()= =
Ersyor Pprompps (pprompe (Vrest (ress) looo P(yees Frest 0)]-
(5)
Here, training example in prompts means happens during the prompt learning.
45

---

If
Extre Pprompy
(prrompp (Sheal (Zeer)) |p (Beer Frest, 0)) S (Ee + Ea(r)),
e
(6)
then
lim Lo-1(fn) S inf Lo-1(f) + g (sup(ce)) n-+00
(7)
BEB
(01- 1) v) + (1 + v) log(1 (1 + v)) is the calibration function the multiclass logistic loss for E [0, 1].
where = =
49 50 for
Following 22 51, KI. divergence is assumed haver the 2nd-order Taylor expansion with the concept Then, we have the following theorem and proof.
Theorem 124 48/ Let the set of 0 which does not satisfy Equation 3 Condition be B.
Assume that Kl. divergences have a 2nd-order Taylor expansion around 0":
<j>1, 1, KL. (10) = 2 (e--0" (8)-8)++)++ 8 (
(8)
where 15.9- is the Fisher information matrix of the j-th loken distribution with respect 0* Let max, ),0 where Amax, Ami return the largest and smallest eigenvalues. Then for ls 2 2
= min fuaal13,5 )-) and + 00, the 0-1 risk of the in-context learning predictor In is bounded as
1/0- (0
suppgn(co + {o(r))
lim Lar(fn) I inf Lo-(f) +
(9)
Proof.
24 By the Taylor expansion on 0, we have for any in B that
KL, (L))0110)
E (0- 0*) 15.0 1 - 0*) + (k- 1)0((10-00))))) 07)3))
(10)
2
2=2
1=2
2
(k: 1)dmin (1 3.00 )110 1112
(11)
2
(co + {o(r))
110 1-1 <
(12)
[k 1)(minn AminnTT (T,,o=)))) ))
We bound the last KL term (k-thh token) with the above term:
1
KI. (0~
2 (0-0r)T TT, (0 - (0) + 0(10 0
(13)
2 (max Amax (1j,0- ))10)))]00000000 0" 0| 0" 11)
1
(14)
(co (r)) Amax
(15)
(k 1) min, Amin 11.0.)
Rearranging above equation, and with KL (10) Exar Es Ppncep: [K (pprompi (Bhess (rea)) Ilp( Muest Prest 0))] there
Bezuer TLL (Pprampp (Jtesl (Ytest (eetess)lze 1lppplppppppp (Mtss Trest 0)) (eo + (max) Amax (1,00) O(1))
(I: - 1) min, Amin (1.0
(16)
Combining Equation 16 with Equation Giint Lemma completes the proof.
KV Cache Sparsification. Revisiting the Equation in Condition
2 the (r) is enlarged with the
sparsity ratio The higher compression ratio r (means that more K cache discarded), the more noise (r). Then it leads to the higher bound of the lim,- ( in Equation in Lemmal Next, discuss how KV cache compression influences the Equation 4
Token-level Importance Measure, The token-level KV cache methods usually calculate the im- portance of different tokens. Then, the KV cache with indexes that have higher importance will
46

---

be preserved. Such indexes are normaly choosed as the attention score. Considering that the case in Figure where each token in the i-th training [ example sequence (O; Vi)) might be compressed, and tokens are sparsified concretely without out dependency to other tokens. However, in the generation process of the i-th training example, O = [at, y/] is sampled from p( 0*) and (o) == p(O[j]] = o00[1 : j - 1]. 0) of the j-th token with previous tokens and the analo- gous distribution Pyrompt Pprompt [OL) = 0|
1]). And the KL divergence is defined as (L (10) = -0] [L (X [r]]], which means that in a training example O = [ y/] = O, : k). each token O,[i] has strong dependency with 0, : j - I], noises on previous any j-th token will influence the distinguishability the following tokens, i.e. requiring larger (KL.(6" (1=1))))
On the other hand, the token-level sparsity enlarges the requirement on the distinguishability uniformly for each example O, (the case in Figure which uniformly loses the bound of (f-) as in Equation
Chunk-level Importance Measure. Different from token-level importance measure, ChunkKV regards tokens in continuous window as basic unit that should be left discarded whole. The preserved window can be regarded as saving the complete O = [ay, ysl without noise. Thus, ChunkKV reduces the noise (r( for the preserved O,, which lowers the bound of (fx)-
More intuitively, ChunkKV focus on reducing the noise on some complete training examples, but some other examples overall with low importance will be discarded. Then, the model identifies the Tist from those clean and more related training examples O and neglect those O with less importance.
Note that here, we do not provide the rigorous proof on how KV cache sparsity enhances the requirement of the distinguishability and how different (L ()) on O; [.,, mil influences the bound La-1 (f)). We left this as the future work analyze how KV cache sparsity influences the in-context learning.
D Additional Related Work
Chunking Method. The chunking methodology is widely used in the field of NLP due to its simplicity and effectiveness [52]. In the era of LLMs, chunking is primarily applied in data pre- processing. For example, Shi al. (53] suggest grouping related training data into chunks to achieve better convergence curves to pre-train LLMs. Fei et al. (54] apply topic-based chunking method to improve the semantic compression of prompts. Furthermore, chunking plays an important role the Generation (RAG) field [55) It serves to divide documents into units of retrieval and processing by LLMs.
information with semantic content suitable for embedding-
Layer-Wiss Technique. The layer-wise technique is widely used in the training and inference of large language models (LLMs). LISA (58) a layer-wise sampling method based on observations of the training dynamics of Low-Rank Adaptation (LoRA) A) [35] 36 38
59 60 across layers. LAMB [6]]
layer-wire wise adaptive learning rate method that speeds up LLM training by stabilizing training
convergence with large batch sizes. DoLa [62] employs layer- wise contrasting to reduce hallucinations during LLM inference.
KV Cache Sharing. Recent work has explored various strategies for sharing KV caches across transformer layers. Layee--Cndensed KV Cache (LCKV) [6] computes KVs only for the top layer and pairs them with queries from all layers, while optionally retaining standard attention for few top and bottom layers to mitigate performance degradation. Similarly, You Only Cache Once (YOCO) [64] computes KVs exclusively for the top layer but pairs them with queries from only the top half of layers, employing efficient attention in the bottom layers maintain constant cache size. In contrast, Cross-Layer Attention (CLA) (65) divides layers into groups, pairing queries from all layers in each group with KVs from that group's bottom layer. MiniCache [66] introduces novel method that merges layer-wise KV caches while enabling recovery during compute-ing -nnpute-ii--lace operations, optimizing KV cache size. These methods illustrate various trade-offs between computation, memory usage, and model performance when sharing KV caches across transformer layers.
I Herr, training means prompt learning 24
47

---

Long-Context Benchmarks. The landscape of long-contes model benchmarks has evolved to encompass a wide range of tasks, with particular emphasis on retrieval and comprehension capabili- ties. Benchmarks for understanding have made significant strides, with co-Bench 67 pushing the boundaries by presenting challenges that involve more than 100,000 tokens. LongBench [25] has introduced bilingual evaluations, addressing tasks such as long-document question answering, sum- marization, and code completion. Complementing these efforts, ZeroSCROLLS [68] and L-Eval [69 have broadened the scope to include diverse array of practical natural language tasks, including query-driven summarization.
In parallel, retrieval benchmarks have largely relied on synthetic datasets, offering researchers precise control over variables such as the length of input tokens. This approach minimizes the impact of disparate parametric knowledge resulting from varied training methodologies. A significant body of recent work has concentrated on the development of synthetic tasks specifically for retrieval evaluation [26] 73. In addition, researchers have explored the potential of extended contexts in facilitating various forms of reasoning 774. Liu al. 775 explores the influence of the KV cache method beyond long-context benchmarks. LongGenBench [76] advances beyond traditional benchmarks by redesigning the format of questions and necessitating that LLMs respond with a single, cohesive long-context answer.
This dual focus on synthetic retrieval tasks and comprehensive understanding benchmarks reflects the field's 's commitment to rigorously assessing the capabilities of long-context models across diverse linguistic challenges.
Prompting Compression. In the field of prompt compression, various designs effectively combine semantic information compress natural language. (77) utilize soft prompts encode more information with fewer tokens. (78) present AutoCompressor, which uses soft prompts to compress the input sequence and extend the original length of the base model. Both [79] and [80] recurrently apply LLMs to summarize input texts, maintaining long short-term memory for specific purposes such
as story writing and dialogue generation. The LLMLingua series [54 8T explores the potential of compressing LLM prompts in long context, reasoning, and RAG scenarios.
[54] use pre- treitra
language models chunk the long context and summarize semantic information, compressing the original context.
KV Cache Quantization. Among quantization approaches for KV cache optimization, {3 stands out as a notable post-training technique. By implementing balanced transformations between activation and weight quantization difficulties, this method enables 8-bit KV cache compression with minimal performance degradation. However, research by [84) has demon- strated that SmoothQuant encounters substantial accuracy deterioration when compressed beyond bits. In parallel developments, the FlexGen system
SmoothQuant
(85] implements group-wise quantization at 4 bits for both key and value caches, while KIVI [45) introduces an innovative approach with asymmetric 2-bit per-channel quantization that requires no additional tuning phase. Additionally, AnTKV [86] introduces a two-stage token-aware vector quantization framework, leveraging waadientwweighte centroid learning and anchor-token selection to achieve ultra-low-bie KV cache quantization with minimal accuracy loss.
E
Statistics of Models
Table 27 provides configuration parameters for LLMS that we evaluated in our experiments.
Model Name
LLLMA-3-8B-IIstrrucc
Mistral-7B--Instruue--e & 0.3
Qwen2-7B-
L (Number of layers)
32
32
28
N (Number of atlention heads)
32
32
28
(Dimension of each head)
128
128
128
Table 27: Models Configuration Parameters
48

---

F Statistics of Datasets
LongBench is meticulously designed benchmark suite that evaluates the capabilittes of language models in handling extended documents and complex information sequences. This benchmark was created for multi-task evaluation of long-context inputs and includes 17 datasets covering tasks such as single-document QA shot learning (97-99], synthetic tasks and code generation [100] [OI]. The datasets feature an average input length ranging from 1K to 18K tokens, requiring substantial memory for KV cache management.
[87 88 multi-document QA [89-92), summarization [23/96], few-
Table 28 shows the statistics of the datasets that we used in our experiments.
DATASET
# TRAIN
# TEST
GSM8K
27
7,473
1,319
LONGBENCH
4,750
NIAH [26
800
JAILBREAKY V
28
28,000
Table 28: Dataset Statistics. # TRAIN and TEST represent the number of training and test samples, respectively. *; The size the NIAH test set varies based on the context length and step size, typically around 800 samples per evaluation.
G
Prompt
Table 29
shows the prompt for the Figure I
The prompt for demonstration
The purple-crested turaco (Gallirex porphyreolohhs) or. South Africa, the purple-crested locrie, (Khurukhuru in the Luvenda (Venda) language) is species of bird the clade Turaco with unresolved phylogenetic placement. Initial analyses placed the purple-crested turaco the family Musophagidae, but studies have indicated that these hirds do belong this family and have been placed in the clade of Turacos with unresolved phylogeny. is the National Bird of the Kingdom of Eswatini, and the crimson flight feathers of this and related turaco species important in the ceremonial regalia of the Swazi royal family. This bird has a purple-colourd crest above green head, red ring around their eyes, and black bill. The neck and chest are green and hrown. The rest of the body is purple, with red flight feathers. Burple-crested turacos are often seen near water sources, where they can be observed drinking and bathing, which helps them maintain their vibrant plumage. Purple-crested turacos are considered to be large frugivores that are known to carry cycad seeds from various plant species long distances from feeding to nesting sites. After fruit consumption, they regurgitate
seeds intact where they germinate nearhy. G. porphyreolohhs primarily consumes fruits whole like many other large frugivores which suggested to be necessary for effective ecosystem functioning. Among similar turacos, the purple-crested turaco have faster minimum transit times when consuming smaller seed diets than larger seed diets, and G. porphyreolpphus has been shown to have significantly faster pulp (seedless fruit masses) transit time than another closely related Turaco when fed only the pulp of larger-seeding fruits than smaller-seed seeding fruits. In addition to their frugivorous diet, these birds are occasionally scen foraging for other food items such as nuts and leaves, which provide essential nutrients. They are also known to coexist with various other animals, including those that might enjoy strawberries and other similar fruits. The purple-crested turaco's role seed dispersal is crucial, and their interaction with different elements of their habitat, including water and diverse plant matcrials, highlights their importance in maintaining ecological balance.
Table 29: The prompt for demonstration
Here provide the CoT prompt exemplars for GSM8K which is used in section 4.1
49

---

GSM8K experiemnt CoT Prompt Exemplars
Question: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?
There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 - 15 =
6. The answer is
Question: If there are cars the parking lot and 2 more cars arrive, how many cars are the parking lot?
There are originally cars. more cars arrive. +-2 5. The answer is S.
Question: Leah had 32 chocolates and her sister had 42. If they ate how many pieces do they have left in total??
Originally, Leah had 32 chocolates. Her sister had 42. So in total they had 32 + 42 74, After eating 35, they had 74-..
35 = 39. The answer is 39. Question: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops did Jason give to Denny? Jason started with 20 lollipops. Then he had 12 after giving some to Denny. So he gave Denny 20 - 12 = 8. The answer is 8.
Question: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does he have now?
Shawn started with toys. he got toys each from his mom and dad, then that more toys. 5 + 4 9. The answer is
There were nine computers in the server room. Five more computers were installed each day, from monday to thursday. How many computers are now in the server room?
Question:
There were originally 9 computers. For each of 4 days, 5 more computers were added. So = 20 computers were added. + 20 is 29. The answer is 29.
Question: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost more. How many golf balls did have al the end of wednesday?
Michael started with 58 golf balls, After losing 23 tuesday, had 58 23 35. After losing more, he had 35 33 golf balls. The answer is 33.
Question: Olivia has $23, She bought five hagels for $3 each. How much money does she have left?
Olivia had 23 dollars. 5 bagels for 3 dollars cach will be x 3 15 dollars. So she has 23 15 dollars left. 23 15 is 8. The answer is 8.
Table 30: GSM8K CoT Prompt Exemplars
H Impact Statement
Our study does not involve human subjects, data collection from individuals. experiments on protected groups. The models and datasets used in this work are publicly available and widely used in the research community. We have made efforts to ensure our experimental design and reporting of results are fair, unbiased, and do not misrepresent the capabilities or limitations of the methods presented.
In our work on KV cache compression for large language models, we acknowledge the potential broader impacts of improving efficiency in Al systems. While our method aims to reduce compu- tational resources and potentially increase accessibility of these models, we recognize that more efficient language models could also lead to increased deployment and usage, which may have both positive and negative socictal implications. We encourage further research and discussion on the responsible development and application of such technologies.
We declare no conflicts of interest that could inappropriately influence our work. All experiments were conducted using publicly available resources, and our code will be made available to ensure reproducibility. We have made every effort to cite relevant prior work appropriately and to accurately represent our contributions in the context of existing research.
I Limitations
While ChunkKV demonstrates strong performance and efficiency, we acknowledge several limitations that provide avenues for future research.
First, the core design of ChunkKV prioritizes the preservation of semantic coherence by treating chunks as indivisible units. This approach may be less suitable for tasks requiring absolute semantic fidelity, where every single token can critical. For example, in domains such as legal or biomedical document analysis, discarding any part of the text, even if it appears less relevant based on attention scores, could lead to the loss of crucial information. Our method is therefore optimized for tasks where capturing the core gist is more important than verbatim retention of every detail.
50

---

Second, our experiments with hybrid compression strategy (using ChunkKV in early layers and token-level method in deeper layers) revealed nuanced, susk-dependent performance trade-off. While pure ChunkKV excelled in local information retrieval tasks (e.g., Single- and Multi-Document QA), the hybrid model showed surprising strength in tasks requiring global understanding (e.g.. Summarization). This suggests that a nee-size-ittsall compression strategy may not be optimal for all scenarios. The ideal approach might involve dynamically adapting the compression granularity based on the perceived task requirements.
Finally, our current implementation relies on iiee-size chunks for computational efficiency. While our ablations show this is a robust strategy, adaptively determining chunk boundaries based on linguistic cues (e.g., sentence endings) could potentially improve semantic integrity further, although this would introduce additional inference latency that must be carefully managed.
J Licenses
For the evaluation dataset, all the datasets, including, GSM8K [27], LongBench [5] are released under MIT license. NIAH (26) is released under GPL-3.0 license.
51


Reviews:
[{"review_id": "tBVlrb8E19", "reviewer": "Reviewer_mXJZ", "summary": "In summary, the authors propose a KV cache compression method called ChunkKV, where the KV matrices are compressed in the unit of continuous chunks instead of individual tokens to keep the semantic information. After the KV matrices are divided into chunks, the proposed method follows a similar way to H2O and SnapKV to select the important chunks. Furthermore, the authors propose layer-wise index reuse to further reduce the inference cost. The proposed method is compared to H2O, SnapKV, and PKV on multiple long-context tasks such as Longbench, NIAH, GSM8K and shows better performance given the same KV compression ratio.", "questions": "Please refer to the weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. Although the proposed ChunkKV is simple, the idea of keeping semantic information in the KV cache makes sense to me and I think it is beneficial for future research in the community\n\n2. The authors conducted extensive evaluations on long-context tasks in the paper. The proposed method shows better long-context task performance than baselines such as H2O and SnapKV given the same KV cache compression rate\n\nWeakness:\n1. The chunk selection algorithm lacks flexibility to some extent. For example, the chunks in the KV cache are defined in a fixed and uniform way. Thus, it is not guaranteed that each chunk contains enough semantic information. A chunk might cover the end and beginning of two sentences. \n\n2. There appears to be no systematic method for selecting the optimal chunk size for a model and a task. Although most experiments indicate that 10 is the best chunk size, there do exist exceptions. For example, chunk size 3 works the best for GSM8k according to Table 20. \n\n3. The evaluation looks incomplete to me. For example, KIVI is missing for the performance evaluation in Tables 4, 5, 6, and 7. Similarly, H2O and SnapKV are missing for the latency/throughput evaluation in Table 11. Also, some more recent KV cache compression methods such as Palu[1] is not compared.\n\n4. The ablation study of $N_{reuse}$ for the layer-wise index reuse is missing. For most of the paper, $N_{reuse}$ is set to 2 but it would be good to know what the effect is when making it larger.\n\n5. The input sequence length is only scaled up to 8k in Tables 8 and 11. What would be the performance if it is scaled up to 16k or 32k?\n\n[1] Chang, Chi-Chih, et al. \"Palu: Compressing kv-cache with low-rank projection.\" arXiv preprint arXiv:2407.21118 (2024).", "was_revised": true, "final_justification": "I appreciate the authors' response and most of my concerns are resolved. I will raise my score.", "rebuttal": "We thank you for your detailed feedback and for recognizing our extensive evaluations and the sensibility of our core idea. We address your weaknesses below.\n\n## **W1: Lack of flexibility in fixed-size chunks.**  \nWe agree this is a crucial design trade-off. Our choice of fixed-size chunks is a deliberate one, prioritizing **computational efficiency and simplicity**, which are paramount for an inference-time technique like KV cache compression. As we state in our limitations (Section I), adaptive chunking methods (e.g., based on sentence boundaries) would introduce significant and variable latency during inference, potentially negating the efficiency gains. Our extensive experiments (Section 4) show that this simple, fixed-size strategy is surprisingly effective and robust across various tasks and models. Below, we conduct new experiments with different chunk size results for LLaMA-3-8B-Instruct on LongBench. The results shows that ChunkKV is effective and robust across different chunk sizes.\n\n| Ratio / Chunk Size | 1 | 3 | 5 | 10 | 15 | 20 | 30 |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **10%** | 37.32 | 40.49 | 40.47 | **40.51** | 40.21 | 40.05 | 39.57 |\n| **20%** | 38.80 | 40.66 | 40.57 | **40.74** | 40.53 | 40.46 | 40.04 |\n| **30%** | 39.23 | 41.02 | 41.29 | **41.59** | 41.38 | 41.33 | 41.02 |\n\n## **W2: No systematic method for selecting chunk size.**  \nThis is a fair point. We have expanded our discussion in Section 4.4 and Appendix B.4 (Figure 19, Table 20). Our results show that while the optimal chunk size can vary slightly (e.g., smaller for GSM8K's shorter contexts), a size of **10 remains a robust default** for most long-context tasks. For GSM8K, we've clarified that its shorter context and dense reasoning steps favor smaller chunks, which aligns with our findings. This approach is consistent with prior work in this area. For instance, the authors of SnapKV[1] also do not provide a systematic method for choosing the pooling size, and instead specify a custom size for each benchmark task.\n  \n## **W3: Incomplete evaluation (missing KIVI, H2O/SnapKV latency, Palu).**  \n**KIVI:** We already compared ChunkKV with KIVI in original paper section B.6. Our goal was to show that our *eviction* method offers competitive performance and superior efficiency. Since quantization and eviction are orthogonal approaches, we believe a full comparison on all benchmarks is not essential, but our head-to-head comparison in the below table already highlights the trade-offs.  \n**Palu [2]:** We thank you for pointing out this very recent work. Palu is a novel KV-Cache compression framework that utilizes low-rank projection to decompose the Key and Value weight matrices and caches their compressed intermediate states. We conducted new experiments to compare the performance between ChunkKV and Palu. The results are shown below.  \n**LLaMA-3-8B-Instruct on LongBench**\n\n| Method | Single-Doc QA | Multi-Doc QA | Summa. | Few-shot Learning | Synthetic | Code | Avg |\n|:---|---:|---:|---:|---:|---:|---:|---:|\n| FullKV | 32.19 | 34.59 | 24.96 | 68.48 | 36.96 | 54.41 | 41.46 |\n| ChunkKV-10% | 28.50 | 33.46 | 22.20 | 67.62 | 37.47 | 58.98 | 40.51 |\n| ChunkKV-20% | 31.05 | 33.22 | 23.58 | 67.86 | 37.05 | 55.34 | 40.74 |\n| **ChunkKV-30%** | **31.48** | **33.27** | **24.53** | **68.34** | **37.30** | **59.02** | **41.59** |\n| Palu-30% | 2.89 | 1.63 | 9.28 | 2.76 | 0.49 | 11.22 | 4.57 |\n| Palu-50% | 8.77 | 7.43 | 20.71 | 61.62 | 8.59 | 18.43 | 22.48 |\n| Palu-70% | 10.06 | 8.31 | 23.71 | 68.64 | 35.00 | 38.97 | 28.84 |\n| KIVI-2bits (Quant) | 31.92 | 34.28 | 24.83 | 66.99 | 34.12 | 53.21 | 40.54 |\n| KIVI-4bits (Quant) | 32.06 | 34.43 | 26.09 | 68.17 | 34.43 | 53.38 | 41.11 |\n| KIVI-8bits (Quant) | 32.18 | 34.56 | 24.94 | 68.45 | 36.92 | 54.38 | 41.43 |  \n\n**H2O/SnapKV Latency:** This is a great suggestion. We have now conduct new  latency/throughput comparisons for these baselines in below table. The results show ChunkKV maintains a competitive edge, especially with our reuse technique.\n\n| Method | Input Length | Output Length | Latency(s) ↓ | Throughput(T/S) ↑ |\n| :--- | :---: | :---: | :---: | :---: |\n| FullKV | 4096 | 1024 | 43.60 | 105.92 |\n| SnapKV | 4096 | 1024 | 37.92(13.0%) | 120.42(13.6%) |\n| ChunkKV | 4096 | 1024 | 37.52(13.9%) | 118.85(12.2%) |\n| ChunkKV_reuse | 4096 | 1024 | **37.35(14.3%)** | **124.09(17.2%)** |\n| FullKV | 4096 | 4096 | 175.50 | 37.73 |\n| SnapKV | 4096 | 4096 | 163.92(6.6%) | 39.98(6.0%) |\n| ChunkKV | 4096 | 4096 | 164.55(6.2%) | 40.58(7.6%) |\n| ChunkKV_reuse | 4096 | 4096 | **162.85(7.2%)** | **41.12(9.0%)** |\n| FullKV | 16384 | 1024 | 49.60 | 323.24 |\n| SnapKV | 16384 | 1024 | 39.20(21.0%) | 381.11(17.9%) |\n| ChunkKV | 16384 | 1024 | 38.82(21.7%) | 381.61(18.0%) |\n| ChunkKV_reuse | 16384 | 1024 | **36.96(25.5%)** | **389.21(20.4%)** |\n\n## **W4: Missing ablation study for layer-wise index reuse.**  \nThank you for this excellent suggestion. We have already compared the `N_reuse` in section B.4. The results explore a wider range of reuse depths and their impact on different models and tasks, providing a much richer analysis.\n\n**LongBench with different `N_reuse` and comparison ratio=10%**\n\n| Model \\ N_reuse | 1 | 2 | 3 | 5 | 10 | 20 | 28/32 |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| LLaMA-3-8B-Instruct | 40.51 | 40.27 | 39.45 | 40.05 | 39.59 | 37.42 | 35.67 |\n| Mistral-7B-Instruct | 46.71 | 46.43 | 46.48 | 45.90 | 44.36 | 41.84 | 39.56 |\n| Qwen2-7B-Instruct | 40.88 | 40.76 | 40.91 | 41.58 | 40.19 | 37.42 | 36.79 |\n\n## **W5: Input sequence length scaling.**  \nWe appreciate the suggestion to test longer sequences. We have run new efficiency experiments on inputs of 16K length, with the results included in our **response of W3**.\n\n\nReference:  \n> [1] Li, Yuhong, et al. \"Snapkv: Llm knows what you are looking for before generation.\" Advances in Neural Information Processing Systems 37 (2024): 22947-22970.  \n> [2] Chang, Chi-Chih, et al. \"Palu: Compressing kv-cache with low-rank projection.\" arXiv preprint arXiv:2407.21118 (2024)."}, {"review_id": "ZooNBYCOmg", "reviewer": "Reviewer_xEys", "summary": "The paper introduces ChunkKV, a novel method for KV cache compression by preserving semantically meaningful chunks of tokens rather than isolated tokens. Traditional compression methods often prune tokens based on isolated importance scores, which can disrupt semantic coherence and degrade performance. ChunkKV addresses this limitation by grouping tokens into coherent chunks and selecting top-k chunks based on attention scores to retain critical contextual information. Additionally, the paper proposes a layer-wise index reuse technique that leverages the similarity of retained chunks across layers to reduce redundant computation and improve throughput. Extensive experiments across a variety of LLMs (LLaMA-3, DeepSeek, Mistral, Qwen2) and benchmarks (GSM8K, LongBench, JailbreakV, and Needle-In-A-Haystack) demonstrate that ChunkKV consistently outperforms prior token-level KV compression approaches.", "questions": "1. What are the efficiency results (e.g., latency and throughput) for ChunkKV under 20% and 30% compression ratios? Tables 3-7 suggest that ChunkKV achieves performance comparable to or even exceeding the FullKV baseline at these compression levels. Including the corresponding efficiency metrics would provide valuable insight into the trade-offs between compression and runtime performance.\n2. Can ChunkKV be integrated with token-level compression methods? If so, could such a hybrid approach lead to additional performance gains by leveraging both semantic and fine-grained redundancy?", "limitations": "Please see weaknesses and questions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The core idea of performing KV cache compression at the semantic unit level rather than at the individual token level is novel and well-motivated, effectively addressing the issue of fragmented context in long sequences.\n- The paper presents comprehensive experimental evaluations across multiple benchmarks (e.g., GSM8K, LongBench, NIAH, JailbreakV) and base models, demonstrating that ChunkKV consistently outperforms prior methods, particularly under aggressive compression settings.\n- An additional contribution is the layer-wise index reuse strategy, supported by empirical analysis of inter-layer similarity. This approach effectively reduces computational overhead and improves throughput without sacrificing performance.\n\n**Weaknesses**\n1. While ChunkKV performs well in scenarios where unimportant segments can be discarded, its effectiveness may be limited in tasks where every sentence is critical, such as legal or biomedical document processing, where full semantic fidelity is required.\n2. Although the paper successfully targets semantic-level redundancy, it does not leverage token-level redundancy, which can still exist and offer potential for further optimization. ChunkKV, by design, may miss opportunities in cases where fine-grained token pruning is more efficient.", "was_revised": true, "final_justification": "I'll maintain my positive rating.", "rebuttal": "We are grateful for your positive assessment and for recognizing the novelty and significance of our work.\n\n##  **Q1: Efficiency results.**  \nThis is an excellent point. Since our method achieves near-FullKV performance at these ratios, showing the efficiency gains is crucial. We have below table to include latency and throughput metrics 10% compression ratios. These results demonstrate that ChunkKV provides significant efficiency improvements.\n\n| Method | Input Length | Output Length | Latency(s) ↓ | Throughput(T/S) ↑ |\n| :--- | :---: | :---: | :---: | :---: |\n| FullKV | 4096 | 1024 | 43.60 | 105.92 |\n| SnapKV | 4096 | 1024 | 37.92(13.0%) | 120.42(13.6%) |\n| ChunkKV | 4096 | 1024 | 37.52(13.9%) | 118.85(12.2%) |\n| ChunkKV_reuse | 4096 | 1024 | **37.35(14.3%)** | **124.09(17.2%)** |\n| FullKV | 4096 | 4096 | 175.50 | 37.73 |\n| SnapKV | 4096 | 4096 | 163.92(6.6%) | 39.98(6.0%) |\n| ChunkKV | 4096 | 4096 | 164.55(6.2%) | 40.58(7.6%) |\n| ChunkKV_reuse | 4096 | 4096 | **162.85(7.2%)** | **41.12(9.0%)** |\n\n##  **Q2: Integration with token-level methods.**  \nThis is a fascinating and insightful question. We believe a hybrid approach is a very promising direction for future research. One could potentially use ChunkKV to preserve core semantic blocks and then apply a token-level method within less important chunks for fine-grained pruning. We have conducted this idea in an experiment. The **first half of the layers are ChunkKV, and the second half of the layers are a token-level method**. The results show that the hybrid approach is effective. The below table shows the results. We can see that this hybrid approach is not better than ChunkKV, but it is still a promising direction.\n\n**LLaMA-3-8B-Instruct on LongBench**\n| Method | Single-Doc QA | Multi-Doc QA | Summa. | Few-shot Learning | Synthetic | Code | Avg.↑ |\n|:---|---:|---:|---:|---:|---:|---:|---:|\n| ChunkKV-10% | 28.50 | 33.46 | 22.20 | 67.62 | 37.47 | 58.98 | 40.51 |\n| ChunkKV-20% | 31.05 | 33.22 | 23.58 | 67.86 | 37.05 | 55.34 | 40.74 |\n| hybrid_kv-10% | 28.38 | 30.37 | 24.54 | 67.87 | 36.37 | 55.32 | 39.80 |\n| hybrid_kv-20% | 29.10 | 29.96 | 25.00 | 68.16 | 36.20 | 55.30 | 39.98 |\n\nThe improved performance of the hybrid method on summarization and few-shot tasks is particularly noteworthy. These tasks, unlike others that can be solved with localized information, demand a holistic understanding of the entire context. This observation leads to a key insight: for tasks where the answer lies within specific text fragments, ChunkKV is more effective in the deeper layers to preserve the cache for those critical segments. In contrast, for tasks requiring a global textual understanding, a token-level compression approach in the deeper layers appears to be more beneficial."}, {"review_id": "22YYrWgs19", "reviewer": "Reviewer_D3CB", "summary": "The paper introduces ChunkKV, a novel approach for compressing the Key-Value (KV) cache in Large Language Models (LLMs) to enable efficient inference with long-context inputs. Unlike prior work that evaluate individual token importance, ChunkKV treats contiguous chunks of tokens as the basic unit for compression. The method utilises a layer-wise index reuse technique to leverage cross-layer similarities, reducing computational overhead. Authors evaluate their methods on various benchmarks and that it surpasses other compression methods performance-wise.", "questions": "- Could you discuss the difference between smoothing done by SnapKV and your chunking approach?\n- Could you report the standard deviation of your evaluations to understand if the performance improvements are statistically significant?", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The paper addresses an important challenge in LLM inference: the high memory demands of the Transformer's KV Cache.\n* The layer-wise index reuse technique is a valuable contribution, effectively leveraging cross-layer similarities in KV Compression to enhance efficiency. It is particularly interesting to observe similarities between layers in the sets of retained chunks.\n* The authors provide a thorough evaluation across various benchmarks and model types. Their method demonstrates consistent performance improvements compared to existing baselines.\n\nWeaknesses:\n* The primary motivation behind the ChunkKV method is that chunks of tokens, rather than individual tokens, are semantically connected and form coherent units of information. However, the authors' justification is not fully convincing. For instance, they state, \"As shown in Figure 1, preserving a chunk helps catch the subject, predicate, and object.\" This argument may not fully apply to Transformer models using self-attention (which the authors evaluate), since self-attention distributes information across tokens. Therefore, after several Transformer layers, it is not clear that specific token information remains localized within the original tokens; it could be processed and stored in entirely different tokens appearing later in the sequence. Furthermore, the method groups tokens into fixed-size chunks, which imposes a clear limitation because semantic units naturally vary in length. Consequently, fixed-size chunks are prone to misalignment, causing fragmentation where meaningful units of information (like sentences) either span multiple chunks or combine unrelated units into a single chunk.\n* The authors use SnapKV as a baseline, and their approach closely resembles SnapKV. Specifically, SnapKV assigns token importance scores and smooths these scores using an averaging or max kernel with a fixed window size, which appears conceptually similar to ChunkKV’s method of aggregating contiguous tokens into chunks and then summarizing importance scores at the chunk level. However, the authors do not provide details about hyperparameter settings (such as kernel size) for SnapKV and PyramidKV in their experiments. A clear comparison or ablation distinguishing ChunkKV's chunking from SnapKV's smoothing approach would significantly strengthen the paper. Without this, the proposed method seems overly similar to SnapKV, diminishing the novelty.\n* The paper lacks a statistical analysis of results, making it unclear whether the performance improvements reported for ChunkKV are statistically significant. The authors mention, \"All experiments were carried out three times, using the mean score to ensure robustness,\" but do not specify details about how these repetitions differed. Moreover, the authors state in the checklist, \"The paper is focused on LLM inference, and with the same model, prompt, and temperature, the results are consistent.\" This explanation is insufficient because the paper introduces a new KV cache compression method and compares its performance against existing methods on downstream benchmarks. In such a scenario, providing measures of uncertainty (e.g., standard deviation or confidence intervals) is crucial for determining whether the observed improvements truly reflect the effectiveness of the proposed method or merely result from variability.", "was_revised": true, "final_justification": "Motivation of the work is nice, but I feel that the contribution is minor. ChunkKV build upon SnapKV, and uses fixed chunks as unit of compression. The improvements coming from chunking isn't robust across tasks and the improvement upon SnapKV isn't significant. I feel that the greater analysis and contribution is needed to clear NeurIPS bar.", "rebuttal": "We thank you for your critical and thought-provoking review. We have carefully considered your concerns and provide clarifications and new evidence below. We hope to convince you of our paper's merits.\n\n## **W1: On the Motivation of ChunkKV and the Limitation of Fixed-Size Chunks**  \n1.  **On Information Diffusion and the Importance of Preserving Local Semantics**: We thank you for this insightful critique regarding information diffusion in Transformers. To directly test your hypothesis—that chunking is beneficial in early layers but token-level pruning might be better in later, more diffused layers—we designed a new hybrid strategy. We split the LLaMA-3-8B model, applying **ChunkKV to the first 16 layers and a token-level method to the final 16 layers**. The results were illuminating:\n\n    **LLaMA-3-8B-Instruct on LongBench**\n    | Method | Single-Doc QA | Multi-Doc QA | Summa. | Few-shot Learning | Synthetic | Code | **Avg.↑** |\n    |:---|---:|---:|---:|---:|---:|---:|---:|\n    | FullKV | 32.19 | 34.59 | 24.96 | 68.48 | 36.96 | 54.41 | 41.46 |\n    | ChunkKV-10% | 28.50 | 33.46 | 22.20 | 67.62 | 37.47 | 58.98 | 40.51 |\n    | ChunkKV-20% | 31.05 | 33.22 | 23.58 | 67.86 | 37.05 | 55.34 | 40.74 |\n    | hybrid_kv-10% | 28.38 | 30.37 | 24.54 | 67.87 | 36.37 | 55.32 | 39.80 |\n    | hybrid_kv-20% | 29.10 | 29.96 | 25.00 | 68.16 | 36.20 | 55.30 | 39.98 |\n\n    As the table shows, while the hybrid approach is effective, **it is consistently outperformed by using the pure ChunkKV method across all layers**. This provides strong empirical evidence that our core assumption holds: preserving local semantic units as indivisible chunks is a more robust strategy for maintaining performance, even in deeper layers where information is more abstract. This result directly supports our central motivation.\n\n2. Regarding the **limitation of fixed-size chunks**, we agree this is a crucial design trade-off. We have clarified in the manuscript that this choice was made deliberately to prioritize **inference efficiency and vectorization**, as adaptive methods would introduce prohibitive latency. Our extensive ablations (Sec 4.4, Appx B.4) demonstrate this simple strategy is remarkably robust.\n\n##  **W2 & Q1：On the Similarity to SnapKV and Novelty**\n\nThank you for pushing us to clarify the distinction from SnapKV. The two methods are fundamentally different in their compression mechanism:  \n* **SnapKV**: Performs **\"score smoothing, then token-level decision.\"** It refines the importance score of each individual token using its neighbors, but the final keep/drop decision is still made on a token-by-token basis.  \n* **ChunkKV**: Performs **\"unit aggregation, then chunk-level decision.\"** It fundamentally changes the **unit of compression**.  It changes the **basic unit of compression** from a token to a chunk. We compute a single score for the entire chunk, and the decision is binary: the entire chunk is either kept or discarded.\n\nThis \"all-or-nothing\" chunk-level policy is the core of our novelty, as it prevents the fragmentation of local linguistic structures. We will add this discussin to Appendix A. We used the default hyperparameters from the official SnapKV repository for all baselines and will state this explicitly in the revision.\n\n##  **W3 & Q2: On the Lack of Statistical Analysis**\nWe agree that reporting mean scores alone is insufficient. The below table is LongBench with standard deviation. We will update rest tables in the manuscript with these statistics.\n\n| Method | Comp. Ratio | Avg. Score (± Std Dev) |\n| :--- | :---: | :---: |\n| FullKV | - | 41.46 ± 0.05 |\n| H2O | 10% | 37.06 ± 0.15 |\n| SnapKV | 10% | 40.15 ± 0.11 |\n| ChunkKV | 10% | **40.51 ± 0.09** |"}, {"review_id": "pfBYCK19d8", "reviewer": "Reviewer_dJtL", "summary": "The paper introduces ChunkKV, a new KV cache compression method for efficient LLM inference. Unlike existing token-level compression techniques, ChunkKV groups tokens into semantic chunks, preserving linguistic structures and contextual integrity. The method includes a layer-wise index reuse technique to reduce computational overhead. Evaluations across benchmarks (LongBench, NIAH, GSM8K, JailbreakV) and models (LLaMA-3, Mistral, Qwen2) demonstrate ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision while maintaining compression ratios, with additional throughput improvements of 26.5%.", "questions": "Please refer to the Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The idea is easy to understand. Compared to token-level KV cache compression, chunk-level compression could avoid pruning certain essential tokens.\n\n2. The study is very comprehensive. Authors have evaluated their method across different benchmarks, as well as providing many visualizations and analysis.\n\n3. This paper is easy to follow. The overall writing is satisfying.\n\n## Weakness\n\n1. The use of fixed-size chunks may limit adaptability. Dynamic chunking could further improve performance but would increase complexity.\n\n2. According to Figure 19, it is quite challenging to choose the optimal chunk size for different LLMs, as they behave quite differently  when using different chunk sizes. For example, Qwen2-7B is robust but LLaMA-3-8B performs badly at larger chunk size. Authors are encouraged to provide a discussion here.", "was_revised": true, "final_justification": "The proposed ChunkKV achieves favourable speed up in relateively short context window. However, as mentioned by other reviewers, this work still has a space for futural improvements, e.g. discussions and analysis. Furthermore, when applying into long context tasks, ChunkKV does not show significant advantage.", "rebuttal": "Thank you for the constructive feedback and your positive evaluation of our paper's clarity and comprehensive study.\n\n##  **W1: Fixed-size chunks limiting adaptability.**  \nWe acknowledge this limitation and discuss it in Section I. As mentioned to mXJZ-W1, this was a conscious design choice to ensure low inference overhead. While dynamic or adaptive chunking is an interesting research direction, its added complexity and latency could be detrimental for this specific application. Our extensive experiments (Section 4) show that this simple, fixed-size strategy is surprisingly effective and robust across various tasks and models.\n\n##  **W2: Difficulty in choosing optimal chunk size.**     \n1.  **Task-Specific Sensitivity:** Our results show that while the optimal chunk size can vary slightly (e.g., smaller for GSM8K's shorter contexts), a size of 10 remains a robust default for most long-context tasks. For GSM8K, we've clarified that its shorter context and dense reasoning steps favor smaller chunks, which aligns with our findings.\n2.  **Model-Specific Robustness:** As you observed, Qwen2-7B is more robust to chunk size variation on GSM8K. We hypothesize this could be due to a combination of architectural differences and tokenizer efficiency. Qwen2's tokenizer is more efficient for mathematical or logical syntax, it might produce more semantically consistent token groupings, making it less sensitive to chunk boundaries[1,2]. This is an interesting avenue for future work. However, for general-purpose use, our broader results on LongBench and NIAH (Table 10) show that a chunk size of 10 is a robust choice across all tested models.\n\nWe believe these clarifications improve the paper and provide better guidance for future users of ChunkKV.\n\n\nReference:  \n> [1] Bai, Jinze, et al. \"Qwen technical report.\" arXiv preprint arXiv:2309.16609 (2023).  \n> [2] Yang, An, et al. \"Qwen3 technical report.\" arXiv preprint arXiv:2505.09388 (2025)."}]

Author rebuttal:

Meta-review: The paper introduces ChunkKV, a novel method for KV cache compression that operates on semantic 'chunks' rather than individual tokens. The core idea is to preserve contextual integrity, which in turn enables a unique and effective layer-wise index reuse technique to significantly improve inference efficiency. The method is evaluated across a comprehensive suite of long-context benchmarks (LongBench, NIAH, GSM8K) and multiple modern LLMs.

Reviewers universally praised the novelty and intuition of the semantic chunking concept, the thoroughness of the experimental evaluation, and the practical value of the layer-wise reuse technique. The primary debate during the review process centered on the magnitude of the contribution, specifically the performance margin over strong baselines like SnapKV. The authors provided an exceptionally thorough rebuttal with substantial new experiments—including 16k context efficiency benchmarks, comparisons against recent (Palu) and orthogonal (KIVI) methods, and statistical analysis—which successfully addressed most initial concerns and led to two reviewers (mXJZ, D3CB) raising their scores. The discussion successfully distilled the paper's core contribution not as a mere accuracy improvement, but as a superior accuracy-efficiency package unlocked by a new compression paradigm.

# Summary Of Reasons To Publish:
1) ChunkKV introduces a novel and intuitive paradigm for KV cache compression. Shifting the basic unit from isolated tokens to semantic chunks is a well-motivated approach to mitigating the context fragmentation problem in existing methods.
2) The proposed layer-wise index reuse technique is a valuable and direct contribution enabled by the structural properties of chunking. It provides a new pathway for optimization that token-level methods cannot easily leverage, resulting in significant and empirically verified throughput gains (up to 26.5%).
3) The paper is supported by a comprehensive and robust empirical evaluation across a wide range of models and challenging benchmarks, consistently demonstrating superior or competitive performance against state-of-the-art baselines. The authors' extensive rebuttal further strengthened these claims.

# Summary Of Suggested Revisions:
To further enhance the paper's impact and clarify the points debated during the review, the following revisions for the final version are recommended:
1) The introduction and conclusion should more explicitly frame the central contribution as the combined "accuracy-efficiency package." The paper should clearly articulate that its value lies not just in the accuracy margin over methods like SnapKV, but critically in the novel optimization pathway (layer-wise reuse) that this new representation unlocks. This will directly address the final lingering concerns from Reviewers D3CB and mXJZ.
2) The presentation of the 16k context efficiency results should be refined to prevent misinterpretation. Clearly highlight the performance of the full ChunkKV_reuse method against all baselines (including FullKV and SnapKV) to unambiguously demonstrate its advantages in latency reduction and throughput increase, addressing the final point of confusion from Reviewer dJtL.
3) The limitations section should be expanded to incorporate the insightful discussion points. This includes acknowledging the trade-off in tasks requiring absolute semantic fidelity (e.g., legal/biomedical documents), as noted by Reviewer xEys, and the nuanced, task-dependent performance of hybrid strategies, as revealed in the discussion with Reviewer D3CB.
4) The future work section should be updated to include the promising directions that emerged from the review process, such as the more sophisticated two-stage hybrid compression strategy (first chunk-level, then token-level) proposed by Reviewer xEys.
Decision: Accept (poster)