SnapKV: LLM Knows What You are Looking for Before Generation
Authors: Yuhong Li
Yingbing Huang
Bowen Yang
Bharat Venkitesh
Acyr Locatelli
Hanchen Ye
Tianle Cai
Patrick Lewis
Deming Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Key-Value Cache Compression
Natural Language Processing
Inference
Machine Learning
Primary area: natural_language_processing

Abstract: Large Language Models (LLMs) have made remarkable progress in processing extensive contexts, with the Key-Value (KV) cache playing a vital role in enhancing their performance. However, the growth of the KV cache in response to increasing input length poses challenges to memory and time efficiency. To address this problem, this paper introduces SnapKV, an innovative and fine-tuning-free approach that efficiently minimizes KV cache size while still delivering comparable performance in real-world applications.

We discover that each attention head in the model consistently focuses on specific prompt attention features during generation. Meanwhile, this robust pattern can be obtained from an `observation' window located at the end of the prompts. Drawing on this insight, SnapKV automatically compresses KV caches by selecting clustered important KV positions for each attention head. Our approach significantly reduces the growing computational overhead and memory footprint when processing long input sequences. Specifically, SnapKV achieves a consistent decoding speed with a 3.6x increase in generation speed and an 8.2x enhancement in memory efficiency compared to baseline when processing inputs of 16K tokens. At the same time, it maintains comparable performance to baseline models across 16 long sequence datasets. Moreover, SnapKV can process up to 380K context tokens on a single A100-80GB GPU using HuggingFace implementation with minor changes, exhibiting only a negligible accuracy drop in the Needle-in-a-Haystack test. Further comprehensive studies suggest SnapKV's potential for practical applications.

SnapKV: LLM Knows What You Are Looking for before Generation
Yuhong Life Yingbing Huangl* Bowen Yang² Bharat Venkitesh² Acyr Locatelli
Hanchen Ye Tianle Caf Patrick Lewis" Deming Chen'
University of Illinois Stbana-Champaigu Cohere Princeton University (leeyh, yh21, hanchen8, dchen)@illinois.c *tanlee. caanne..aa@ppinncee www.paiinnetttneeeeee.e.
[bowen, bharat, acyr, patrick)@cohere... com
Abstract
Large Language Models (LLMs) have made remarkable progress in processing extensive contexts, with the Value (KV) cache playing vital role in enhancing their performance. However, the growth of the KV cache response to increasing input length poses challenges to memory and time efficiency. To address this problem, this paper introduces SnapKV, an innovative and fine-tuning-fic approach that efficiently minimizes KV cache size while still delivering comparable accuracy in real-world applications.
We discover that each attention head in the model consistently focuses on specific prompt attention features during generation. Meanwhile, this robust pattern can be obtained from an 'observation' window located at the end of the prompts. Drawing this insight, SnapKV automatically compresses KV caches by selecting clustered important KV positions for each attention head. Our approach significantly reduces the growing computational overhead and memory footprint when processing long input sequences. Specifically, SnapKV achieves consistent decoding speed with a 3.6x increase in generation speed and an 8.2x enhancement in memory efficiency compared to the baseline when processing inputs of 16K tokens. At the same time, maintains comparable performance the baseline models across 16 long sequence datasets. Moreover, SnapKV can process up 380K context tokens single A100-80GB GPU using HuggingFace implementation with minor changes, exhibiting only negligible accuracy drop in the Needle-in-a-Hayeee comprehensive studies suggest SnapKV's potential for practical applications. Our code is available at https: htt.gaaa..co//aattteeaattttttaaa
test. Further
Introduction
Many leading LLMs have started to handle longer contexts, overcoming the difficulties in context maintenance and attention mechanism scalability, such GPT-4 [1] and Command-R 2 with context length 128K, Claude-3 (3] with 200K, and Gemini-Pro-I. with IM [4). Despite their impressive capabilities, LLMs still face significant challenges when dealing with long context prompts. Specifically, the KV cache in attention calculation becomes less efficient when processing long context. During inference time, as prompt length increases, the decoding latency per step grows linearly due to the attention calculation across past KVs. Moreover, the large KV cache requires significant memory capacity, increasing hardware demands and limiting model scalability.
"qual contribution
38th Conference Neural Information Processing Systems (NeurIPS 2024).

---

Input Sequence KVs
Help me analyze the Q4 report company.... Can you help rephrase my email?
Prefix
Obs. window
The company's expenses for the fourth quarter of 2023 is xoLxx billion. This figure seen the context of...
Voting & Selecting
Attention Weight Calc.
Important Fastures
want buy gift for my mom...
Obs. window
don't understand wh cache LLMB... Can you tell me the details of R&D expense of Q4?
Clustering
Concatenating Fentures
Olos.
"Snap"
Compresaed KVs
The graph shows the simplified workflow of SnapKV, where the orange area represents the cluster of features per head selected by SnapKV. These features are then used to form new Key-Value pairs concatenated with the features in the observation window. Together, the selected prefix and observation windows constitute the new KV cache utilized for the generation.
Figure 1:
There are many approaches to mitigate these problems, such KV cache eviction during generation stage 1 However, most of these methods lack a detailed evaluation in long-context settings. Moreover, they mainly focus on compressing the KV cache appended during decoding steps, while overlooking the realistic problem of compressing KV cache for prompts, which is typically the bottleneck in memory efficiency. practical applications like chatbots and agents, where prompts range from multi-turn conversations to extensive articles or codebases [ prompts are often much larger than generated responses such as summaries and code pieces, thus creating significant inference latency and memory utilization overhead. Additional challenge lies in compressing KV cache for such vast prompts without losing crucial information for accurate generation, especially in scenarios with various noisy contexts.
our paper, we find an vital attention allocation phenomenon: only a subset of prompt tokens convey essential information for response generation, and these tokens remain unchanged during generation. To validate the robustness, design extensive experiments across diverse prompts in terms of length, format, and content. From our observations, derive innovative and intuitive method, SnapKV, which can smartly identify the attention allocation pattern and compress the KV cache for long sequence prompts without compromising the model's accuracy. With its comprehensive design, SnapKV demonstrates its effectiveness on various datasets and can be easily integrated into popular deep-learning frameworks with just a few code adjustments. Our contributions are as follows:
We design experiments to explore the attention allocation pattern during generation, focusing on two key questions:
1. Is there consistent attention allocation pattern for input sequence tokens?
2. Is it feasible identify this pattern prior to the generation stage?
Our finding suggests that for LLMs, the attention allocation of most input sequence tokens stay consistent during generation. Thus, LLMs knows what you are looking for before generation.
Inspired by our observations above, we develop an efficient and fine-tuning-free algorithm, SnapKV, which efficiently identifies critical attention features and compresses KV cache corre- spondingly with minimal model modification (See Fig.
We evaluate SnapKV across diverse LLMs and long- long-sequence datasets. SnapKV shows comparable accuracy with full KV caching method while achieving improved decoding speed and memory efficiency. Meanwhile, we conduct the pressure test with Needle leeee-in-----sstee Hays-aast to further demonstrate its memory efficiency and information retrieval ability.
Related Works
Many previous works compress the KV cache by selectively dropping KVs using different algorithms. In StreamLLM (5), only the most recent tokens and attention sinks (first few tokens) are retained

---

Overiap rates windows prompt
Overiap windows generated
0.9
0.9
Overlap (%6)
Overlop (%6
0.8
window
0.6
0.6
window
Prompt 3263.80
Aug Prompt Len: 3263.80
Avg
Avg Turm:
0.5
Avg Context Len: 955.78
0.5
Avg Context Len: 953.78
0.25
Samples
0.25
Samples
0.00
0.00
10 12
15 Layer
Window *
Figure The overlap rates between attention fea- tures of the input sequence, selected by various windows along the input and during generation, with each line representing model layer.
Figure The layer-wise overlap rates between input sequence attention features selected by the last window of input sequence and those selected by 4 windows along generation.
reduce the KV cache size, making it lose the important information carried by the discarded middle tokens f Heavy-Hitter Oracle (H20) (6] introduces a policy that greedily drops KVs during scoring function derived from cumulative attention. While this approach effectively compresses the KVss appended to the cache during generation, it overlooks compression of prompt KVs, which is crucial for reducing memory and computational overhead. Building on a similar concept, Adaptive KV Compression (FastGen) [8] implements a dual-phase algorithm that encompasses four KV cache compression policies. Initially, identifies optimal policies through profiling results obtained from prompt encoding. Subsequently, dynamically evicts caches during the generation phase based on these policies. Nonetheless, it faces the similar problem with [7] focuses identifying and retaining pivotal tokens that exhibit consistent attention weight pattern with previous token windows during generation steps. However, this method concentrates solely on the window of previous pivotal tokens in generation and neglects the extensive prompt that contains essential information for generating accurate responses. This oversight could lead to an inability to extract detailed information from prompts.
generation based
H2O. ScissorHands
In summary, existing methods have not effectively addressed the challenges encountered in real- world applications, where prompts exceptionally long yet require accurate information retrieval. Although these techniques may reduce the KV cache size during generation, they do not address the primary challenges of understanding complex prompt contexts, leaving critical issues unresolved.
3 Observations
In this section, we present our observations regarding the attention allocation patterns in the Query- Key matrix during token generation. Our analysis utilizes samples from Ultrachat [12], multi-turns, high-quality instruction dataset consisting of 1.4 million dialogues. We further filter the sequences with response length greater than 512 and prompt length greater than 3k. Our findings concluded into two key observations follows:
Pattern can be identified before generation. In this experiment, we split the attention features of input sequence of each layer into multiple windows, each with 128 tokens, and calculate the averaged attention weights of the last 20 windows separately. To understand the attention allocation patterns along input sequences, we calculate the overlap rates between important attention features of input sequence (those with high average attention weights) identified by each window and the actual ones used by generation. The experimental results shown in Fig. 2

---

We observe that the last window of input sequence recognizes highly similar attention allocation pattern with the actual generation.
Pattern is consistent during generation. We study if the positions of features identified as crucial in the last window of input sequence maintain their significance in the subsequent token generation. In the experiment, we split the generated tokens into 4 windows for every layer, each spanning 188 tokens, to compute the averaged overlap rates of these windows versus the last window of input sequence. As shown in Fig- B active attention features of input sequence obtained from the last window exhibit remarkable consistency throughout the generation process, evidenced by high overlap rates.
SnapKV
In the attention mechanism, the growth in prompts will significantly increase time complexity for generation due to the Query-Key matrix multiplication. SnapKV addresses this issue by maintaining a constant amount of prompt KVs during generation, significantly reducing serving times for long- context LLMs. To structure our method coherently, we propose the following terminologies:
Prompt Length (Xpromr):: The total length of the user-provided input.
Observation Window (Lobs): The last segment of the prompt. This window is crucial for analyzing the influence of different contexts on attention allocation patterns.
Prefix Length (= ) The length of the input preceding the observation window. It part of the prompt and does not include the observation window. Overall, we have:
Leramge
Lprefix + Lohs
(1)
The process of calculating attention weights for each query within the observation window across all heads, aggregating these weights highlight the prefix positions that considered most significant. For a single batch of sequence, formally:
Voting:
Like
C
W [x
(2)
1=0
I 1= Topa (C,
(3)
where To, k) selects the indices I of the top k values in tensor C per head. l is defined as L(1 pp) x [.prefi J. where stands for the compression rate. The tensor Wohs E IR L x Lyntti
represents the subset of the prompt softmax cittmax-normaized attention features over N heads.
Hit Rate: We define attention features above predefined threshold 0 during generation as important features. The hit rate, H, is the number of important features successfully selected by the previous voting process over the total number of important features. H quantifies the effectiveness of the voting mechanism and is calculated as follows:
Myotc obs = zeros_ like Asur)
(4)
My obs [f]
(5)
Matneshold = 1(A (A > 0)
(6)
O Mohreshold cur A Myote
(7)
O
H Mitrestold.cco cur
(8)
A IR Nx Fredis represents the attention features between the current generated query and prefix keys. M selects attention features by indices. The threshold operation filters A to retain only features with values over 0, indicating important attention activations. The O measures the overlap between attention features selected by Mihreshold our and My obs- quantifying the alignment of the current attention with previously identified important features. The hit rate H is then computed as the ratio of the sum of overlap to the sum of important features Mytreshold.com providing metric for the efficacy of the attention mechanism in recognizing and emphasizing important attention features within the context. We use H(Mauassold curs My to denote combination of Eq. 7 and Eq.

---

4.1 Observation Window-based Algorithm
The core approach of SnapKV involves identifying and selecting the most crucial attention features per head to create the compressed KV cache. Listing | shows the PyTorch-style pseudo code of SnapKV. Overall, SnapKV operates through two stages as follows:
Vote for important previous features. By the voting process defined above (Eq. we select the important attention features based on the observation window. Sec. highlights the consistency of the attention allocation pattern within observation windows throughout the generation, suggesting that these selected attention features are also vital for subsequent generation. Furthermore, we implement clustering retain the features surrounding the selected attention features (Sec. 4.3
Line 77 shows the pseudo code of the voting process.
Update and store compressed keys and values. We concatenate the selected attention features with all features within the observation window, which encompasses all features containing the necessary prompt information. Line 18 shows the compressing process. The concatenated Vss are stored for later use in generation, thereby saving memory usage.
anap kv _quary__taata atatea, tay_atatas, value, atatea, windov_ sax =ax_ wawpcityypromgee kernel_ _i__)) bez, sun_heade g.les, head_din query_states, chape
Ensure assert ataats..aapp.
prompt phase.
== atary, [-2] q_len nax cax_capacity, ppronpt: return key_stater, value_states
Compute sttention veights of observing vindow's queries and prefix context 's Keye. -vindov_cizei, :], key_states,
atta_Keights conputo_atta ooppute.ttaaagorr.aaateee attention, _aak))
10
Sun the seight along the query dinension.
vote stin_veights l..., -vindov_eize:, :"xindox_sizo (din=-2) # Apply 1D pooling for clustering.
12
pool_vote poolld (vote, tarmel_aiza=kerral # Selact top-x indicas based on the pooled veights to identify isportant pasitiona. indices pote_vote. topk wwwpxxtapccttyp.......
padding=karael, 2t12//// strida=1)
14
window_size din--1), sin=-i).indiees y Expand the indices to natch the head dimensien for gathering.
16
17
indices indices. unsquezze (moguezo(--)oorpaad(-1 (-pand(-1, -1, head_din)
18
# father the compreased past kay and value states based on the selectad indices. :-vindov_size, :l :l.gather indes=indices)
19
c_past__omprese key_statee
c_past_comppess value_etater :-xindov_eize, gather (din=2, indice)) k_obs key_states -vindor_size:,
21
valua_states -xindox_aize:
23
key_states = torch. (lrk
k_obs], din-2)
value_states parch.cat(rrppatt (prch,ccc(s_ppest__coopee v_abs), din-2)
25
return key_states, valte_states
Listing Implementation of SnapKV in pseudo PyTorch style.
4.2 Robustness Analysis of Hit Rate
To understand the robustness of the observation window-based algorithm, we analyze its hit rate multiple long documents QA datasets including QMSum [3]. query-based multi-domal meeting summarization; Openreview [14]. collection of papers from openreview net; SPACE
15). an extractive opinion summarization in quantized transformer spaces. The model we probe is -iattall-7--Instruct-v
Overall, we want to answer the following two questions:
Does the nature of instructions in the prompt affect the hit rate? Does the context and instruction positioning affect the hit rate?
4.2.1 Contextual Dependency of Patterns
We analyze whether instructions will affect the selection of important features even the provided context the same. Our experiment utilizes different instructions on the same document and selects the important features based on the observation window that consists of both the instructions and their corresponding responses. Then we calculate the hit rates between important features selected by different instruction-responss pairs (A, B) within the same document by using HM A M
5

---

Overlap important positions for different the documents
1.0
0.8
QMBum
0.4
Openreviow
SPACE
08/10694 43/18953
0.2
Aug Context Lam: 320.79922.5544427.5
1otal Part Num: 654/63/360
0.0
15
20
Layer
Figure 4: The layer-wise overlap of important positions utilized by different question-answer pairs in the same dataset.
for different datasets
beginning
different datasets with question the and
1.0
Hit rate (%6
rate
GMSum
QNSum
0.4
Open medeww
0.4
Open
SPACE
SPACE
Prompt 9700072222022220000011.
Aug Frompt
0.2
Aug Contest 120. 3.54/227.86
0.2
Aug Contest Lee: 7912 8.44427.86
Total Samples Num: 766/144
Total Samples Num:: 17 7769/444
0.0
0.0
10
15
30
20
Layer
Layer
Figure 5: The layer-wis average hit rate of important positions used by prompts with questions at the beginning and the end.
as previously defined. By varying the instructions, we observe that different instructions prioritize different prefix attention features, indicated by the descending trend in hit rates shown in Fig. Our findings reveal interesting aspect of KV cache in LLMs: the important attention features change with different instructions. This variability challenges the effectiveness of static compression methods that depend on constant weighted importance or fixed policies 7 6 g Thus, the complex relationship between context and related KV cache emphasizes the need for context-aware compres- sion strategies and highlights the capability of SnapKV that recognizes this dynamic. In contrast, context-inde indeeedend compression fail in capturing the dynamic, resulting in misalignment between the attention distribution during profiling and inference, diminishing the generation quality of LLMs.
4.2.2 Invariance to Instruction Positions
Our analysis also extends the significance of instruction positioning on the interpretability L.LMs and their selection of important features. We calculate the average hit rate for the responses using the same observation window size in the previous experiment. Our results shown in Fig. indicaae that across all three datasets, the hit rates are consistently high regardless of whether instructions are positioned before or after extensive supplementary contexts. This consistency suggests that SnapKV is able to identify attention allocation patterns regardless of the question's 's positions.
4.3 Efficient Clustering via Pooling
In LLMs, information retrieval and generation rely on features with high attention weight and are supplemented by copying the rest of features context using induction heads [16]. Hence, naively selecting the top features results in retaining only portions of details and then losing the completeness of the information. For example, such compression might cause the LLMs retrieve only the country code of phone number and hallucinate the rest. Our experiment also revealed that only selecting the features with the highest weights is insufficient (Sec. 5.2). Such sparse selection risks compromising

---

LWM-Text-Chat-IM with SnapKV
1.0
0.0
11.0
0.8
22.0
Depth Percent
33.0
0.6
0.4
67.0
78.0
89.0
100.0
7923
20330
33564
72852
821007
11177
Taken Limit
Figure 6: Needle-in-a-Haae Hays-aak test performance comparison on single A100-80GB GPU, native HuggingFace implementation with only few lines of code changed. The x-axis denotes the length of the document (the "haystack") from 1K to 380K tokens; the y-axis indicates the position that the "needle" (a short sentence) is located within the document. For example, 50% indicates that the needle is placed in the middle of the document. Here LWMChat with SnapKV is able to retrieve the needle correctly before 140k and with only little accuracy drop after. Meanwhile, the original implementation encounters OOM error with 33k input tokens (white dashed line).
the contextual integrity encapsulated in between features, thereby reducing accuracy. Based on the insights, we propose a fine-grained clustering algorithm utilizing a pooling layer shown in Line [3
5 Experiments
In our experimental setup, we explore the performance SnapKV across models that can handle extended prompt sequence contexts. First, we deliver pressure test and benchmark the speed of WWM-Text-Chat- [17], which state-of-the-art regarding its context length. We then conduct an ablation study
understand the influence of pooling the model's information retrieval performance. We assess model performances using the LongBench [88] dataset. Further experiments on compatibility with other acceleration strategies, such as parallel decoding 19), are elaborated in Appendix A To assess the overhead of SnapKV during the prefilling stage, we present time and memory analysis results Appendix B
5.1 Benchmarks on LWM-Text-Chat-IN
LWM-Text-Chat- 17 a instruction- finn----ettttedddd model with up to one million context length.
In this section, we conduct pressure test on this model and examine its algorithmic efficiencies.
5.1.1 Needle-in-a-Haasttack
The Needle-in-a-Haystacl 220 challenges the model to accurately retrieve information from specific sentence ("needle") concealed within extensive document (the "haystack"), with the sentence placed at a random location. Typically, sentences that are inserted in the middle of prompts are harder to retrieve. To rigorously evaluate SnapKV's capabilities, we extended the document length to 380k tokens which is the longest content that can be processed by a single A A100-80GE GPU. We configured the prompt KV cache size to 1024, enabling SnapKV to select the most crucial 1024 attention features from the prompt for answer generation, with maximum pooling kernel size of and observation window size of 16, both of which are hyperparameters that can be customized. The compelling outcomes in Fig. from the Bocell-in-a-Haystee test underscore SnapKV's potential to preciscly manage small details extremely long input contexts with 380x compression ratio.
7

---

120
Decode Latency (meitaken)
Optimized Batch
100
Optimized Batch
Optimized Batch
Optimized Batch 8
Baseline Batch
Baseline Batch
Baseline BaselnneBBtcch 4
(OOM)
ODM
Common SolA models max seq length
4096
8192
16384
32 768
65536
131072
262144
Input Sequence Length
Figure 7: Decoding latency comparison of bascline implementation and SnapKV optimized solutions on various batch sizes. The x-axis denotes the input sequence length; the y-axis indicates decoding latency (ms/token). All experiments are conducted on an A100 80GB GPU. The red dotted line denotes the common context length of state-of-the-ar long sequence models.
5.1.2 Decoding Speed and Memory Bound
We further benchmark the speed of LiM-Text-Chat- under different batch-size settings using SnapKV. We the maximum KV cache size as 2048 for SnapKV, and fix the generation length 512 to ensure fair comparison. There two main takeaways from experiment on decoding speed and prompt sequence length on various batch sizes, as shown in Fig. 7 First, as the input sequence length increases, the decoding latency of the baseline implementation escalates linearly. Conversely, the ShapkV-optimized model maintains a constant decoding speed since the compressed KV cache size prompt stays the same regardless of input sequence length and there is no extra update during the inference. For instance, at a sequence length of 16k and batch size of 2, the decoding time for the baseline model surpasses 100 ms, whereas for SnapKV-optimized model, the decoding time consistently remains below 40 ms, achieving approximately 3.6x speedup. Second, with the same batch size, the model integrated with SnapKV can decode significantly longer sequences. For example, at batch size of 2, the baseline model encounters an OOM error beyond 16k input tokens, whereas the SnapkV-enhanced model extends this limit to 131k input tokens, indicating an approximately 8.2x improvement. This demonstrates SnapKV's effectiveness in minimizing memory consumption.
5.2 Ablation Study of Effectiveness of Pooling
We perform an ablation study on technique, a straightforward but efficient method for consolidating information through cluster- ing. Our evaluation utilizes the modified LongEval-Lines benchmark [2I]. incorporating randomly generated pairs and averaged scores. LongEval-Linge presents a greater challenge compared to Needle-in- -n----------a -nya-aayt because it involves identifying key-value pairs in noisy contexts of the same format, while in Needle-in-at -aast--an the relevant information is more distinctly separated from other contexts. We apply max pooling with a kernel size of and use the observation window with a size 16, which are hyperparameters and could be customized according to different models. As illustrated in our results (Fig. g we find that pooling significantly enhances retrieval accuracy compared methods not utilizing pooling. We hypothesize that this because the initial portions of critical token clusters are weighted higher by attention mechanisms. Typically, large language models tend to copy the tokens surrounding the initial portions to keep the contextual integrity. However, naively compressed KV cache breaks this mechanism and could lead to partially correct
to assess the impact our pooling

---

Mistral-7B-Instruee 78-Instruuct-vO v0.2 without Paoling
Mistral- 78 instruct-vO. with Pooling
1.0
11.0
11.0
0.8
Depth Percent
22.0
33.0
0.6
44.0
56.0
56.0
0.4
67.
78.0
8
0.2
89.0
1
4000
Pope
12000
1800
240000
24000
0.0
9t0p
Woop
11009
16000
30029
22000
24009
27009
0.0
Ikken
Figure 8: Ablation study of pooling on LongEval-Lines. The evaluation includes inputs, each com- prised of lines formatted as "line makeshi wwwissifftpengeee penguui::: REGISTER, CONTENT is <10536>", where the key is an adjective-noun pair and the value is random 5-digit number. The model needs to retrieve the value based on a given key. The x-axis denotes the length of the input; the y-axis indicates the position the groundtruth, from 5K to 30K tokens. With the pooling, the model can retrieve correct values before 16k and performs significantly better than the one without pooling.
Table 1: Performance comparison of SnapKV and H2O across various LLMs Long
Single Document QA
Multi Document QA
Summanizaiin
Few shot Learning
Synthenk
Code
LLMs
NAAQA
Quappt
MF-en
HespuiQN
2WiaMMA
Musiqu
GevRepart
QMSum
MoltiNeus
TREC
TriraAA
5AMSom
PConni
RB-F
All KV
18.15
25.56
40.94
24.57
19.39
10.49
27.97
24.9
24,81
71.0
60.9
39.73
3.17
3.5
44,4
43.82
Stapky 1024
23.73
10:77
34.44
30.64
Stapky: 2048
17.92
25.03
41.38
24.49
19.38
11.34
21.5
24.22
24.36
70.0
61.11
39.91
2.17
44.36
46.9
BtagKV 4096
17.92
25.47
40.76
24.92
19.53
11.27
25.34
25.42
24.58
70.5
61.08
39.62
3.17
4.0
44.44
44.88
H20:
7.2
25.73
35.05
14,66
30,89
22.76
26.61
48,83
$4.89
59,05
Longrbo
Snapky: 1034
26.6
34.93
0.0
StapkV: 2048
19.28
25.81
40.26
35.31
23.75
13:44
26.3
22.29
25.73
66.0
79.93
39.59
0.0
31.0
$6.05
58.61
4196
14.35
26.45
81.25
54.79
11.76
66.0
26.82
33.06
49.28
42.77
27.33
19:27
32.83
24.25
27.06
71.0
36.23
42.95
2.75
36.98
55.51
$2.88
53ama
BtagKV: 1024
25.54
29-51
49.25
40.94
25.7
19.42
25.89
23.82
26.11
69.5
$6.48
42.05
2.95
88.55
55.65
51.87
StapKV
48.6
SnapKy- 4096
26.41
49.81
42.32
27.93
18.76
30.74
24.19
27.08
71.0
36.25
43.01
2.73
52.65
H10:: 6096
22.61
29:06
47.22
36.54
20.6
16.25
30.0
23.8
26.75
70.5
36.16
42.97
3.46
36.38
53.72
51.1
All KV
26.81
37.06
51.55
47.77
32:46
26.99
34.25
36,15
27.91
76.0
911.7
46.95
5.5
180,
69.07
69:65
Snapky: 1024
SnapKV 2048
27.12
51.91
30.19
27.8
76.0
Snagky 4096
26.46
37.03
$2.62
47.71
33.35
26:45
33.64
25.85
27.94
75.5
90.71
47.14
5.5
100.0
68.81
69.56
H20: 6096
32.09
16.5
29.76
23.53
74.5
7.06
63.52
Credit Jin al.
22
for the template used in the table.
results (Fig. 8). Note that throughout our experiments, the choice between max pooling and average pooling did not yield significant differences performance.
5.3 Experiments on LongBench
We evaluate SnapKV on these four models using LongBench [8] multi-task benchmark designed rigorously evaluate long context understanding capabilities across various datasets, spanning single and multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. The average prompt length of LongBench ranges from 5k to 7k, and more details can be found in Ap- pendix D. We choose Mistral-78-IIssrrut--vo Instruct-vo. 2, Mixtral-8x788 CwrrBB-Instuutttte baselines. For each model, we test SnapKV with various settings: compressing KV caches in the prompt to 1024, 2048, and 4096 tokens. We use max pooling with kernel size 7 and observation window size 32. Table illustrates negligible performance drop from models with SnapKV com- pared with original implementations for 16 different datasets, even with prompt-KV with 1024 tokens. Some models outperform the baseline. Our results substantiate that SnapKV can grasp the key information in the long context and give comprehensive summaries with details. Moreover, our
with million context length, LongChat-7b-v1. -0tttt-----v11323.4..- 5-332,
with 32k context length as our

---

Table 2:
The sensitivity analysis was conducted on Mistral-7B- - 2 with a prompt KV cache size set to 1024, evaluating its performance LongBench across different observation window sizes and pooling kernel dimensions. In previous experiments, a configuration with an observation window size w=32 and kernel size k=7 was employed the baseline. w=32 k=1 refers to SnapKV without pooling, where we focus on tasks that do not involve information retrieval since we already demonstrate that in Sec. 5.2
Single-Doc. Doc. QA
MobiiDoo. QA
Summarization
Fow-shor Learning
Synthetie
Code
L.Ms *
Qasper
MF-cn
HupoiQAA
2WikiMQA
GovReport
MultiNows
TREC
TriviaQA
SAMSum
PCount
RB-P
w=32 k=7
29,51
49.25
40.94
25.7
25.89
26.11
69.5
86.48
42.06
2.98
88.56
55.65
51.87
we16 kmm
27.14
48.9
41.02
27.06
28.2
26.13
67.0
86.84
40.9
4.51
91.56
60.55
50.25
Mistral
We64 km
27.28
48.99
40.95
26.95
26.41
26.18
67.0
86.84
40.85
4.44
91.56
60.79
50.25
w=32 k=5
26.79
48.7
40.07
26.74
29.65
24.55
64.29
86.73
40.21
4.74
90.49
$7.06
48.57
W=32 k=9
27.18
49.19
41.39
26.55
26.58
24.61
65.33
86.87
39.74
4.51
91.56
60.56
50.25
33.23
67.33
84.84
50.25
results also indicate the effectiveness of SnapkV in compressing the prompt KV cache. For these models, the average input token length is around 13k. Thus, using 1024, SnapKV achieves an average compression rate of 92%, and using 4096, it reaches 68%, all with negligible drops in accuracy. We compare SnapKV and H2O the LongBench dataset to further demonstrate the performance of SnapKV. To fairly evaluate the accuracy, we set the prompt capacity for H2O to 4096. As Table shows, SnapKV delivers significantly better performance than H2O. Even with 1024 prompt KV caches, SnapKV on Mistral-7B-Ii P7B-Innstrutt-oe .2 achieves better performance than H2O with 4096 caches on out of 16 benchmarks.
5.4 Sensitivity Analysis on Hyperparameters
In SnapKV, we introduce two key hyperparameters: observation window size and pooling ker- nel size. To further assess the robustness of our method, we perform a sensitivity analysis on Mistral-7B- 1estral-77-Inssructt--0 2 with these hyperparameters using the LongBench dataset. As shown in Table different configurations yield the best score across various types of tasks, with no single configuration consistently outperforming others. This indicates that SnapKV demonstrates robustness across a range of configurations.
Additionally, to better understand the effectiveness of the pooling strategy, we conduct an experiment with a kernel size 1, representing a configuration without pooling. This analysis focuses primarily on non-retrieval tasks, complementing the retrieval task results Sec. [5.2] The findings indicate that, in eight out of nine tasks, the model accuracy with pooling exceeds that of configurations without pooling, underscoring the importance of pooling in SnapKV.
6
Discussions
SnapKV an effective yet straightforward solution that compresses the KV cache to mitigate the computational and memory burdens processing extensive prompts. Observing that specific tokens within prompts gain consistent attention from each head during generation, our methodology not only retrieve crucial information but also enhances processing efficiency. Despite its strengths, SnapKV's scope is primarily confined to the generative aspect of models, specifically targeting the KV cache during the generation. This limitation implies that SnapKV cannot extend a model's 's long context capability if the model inherently struggles with long contexts or exhibits poor performance. Additionally, SnapKV's s design does not cover the processing of the prompt inference, which limits its effectiveness in scenarios where the system cannot handle prompts of extensive length. Nonetheless, our contributions offer significant insights and tools for the community, paving the way for more refined approaches managing the challenges of large-scale language modeling. The appendix provides more experiments with parallel decoding and the discussion about generation speedup.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
[2] Cohere. Command r: Retrieval-auggmente generation at production scale, March 2024. URL https://txt. cohere.e......o... wwm/coomande
[3] Anthropic. The claude 3 model family: Opus. sonnet, haiku, March 2024. URL
http://wwww....//...o.......oooa Model_ Card Claude_3, pd
[4] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.055300 2024.
[5] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint 2..i::2309./455.3 2023.
[6] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024.
[7] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of impor- tance hypothesis for I1m kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024.
[8] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023.
[9] Bing Liu and Sahisnu Mazumder. Lifelong and continual learning dialogue systems: learn- ing during conversation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 15058-15063, 2021.
[10] Ramakrishna Bairi, Athary Sonwane, Aditya Kanade, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B Ashok, Shashank Shet, al. Codeplan: Repository-leve coding using llms and planning, arXiv preprint arXiv:2309. 12499, 2023.
[11] Jingru Jia, Zehua Yuan, Junhao Pan, Paul McNamara, and Deming Chen. Decision-making be- havior evaluation framework for llms under uncertain context. arXiv preprint arXiv:2406.05972, 2024.
[12] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint urXiv: /4233, 2023.
[13] Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: new benchmark for query-based multi-domain meeting summarization. arXiv preprint arXiv:2104.05938, 2021.
[14] Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint 2xXiv:2307.11088 2023.

---

[15] Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. Extractive opinion summarization in quantized transformer spaces. Transactions of the Association for Computational Linguistics, 9:277-293, 2021.
[16] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint urXiv:2209. 1895, 2022.
[17] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268, 2024.
[18] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
[19] Tianle Cai, Yuhong Li, Zhengyang Geng. Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple IIm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774 2024.
[20] G Kamradt. Needle in paystack-pressure testing llms, 2023.
[21] Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source Ilms truly promise? In NeurIPS 2023 Workshop Instruction Tuning and Instruction Following, 2023.
[22] Hongye Jin, Xiaotian Han, Jingfeng Yang. Zhimeng Jiang, Zirui Liu, Chia- Chang, Huiyuan Chen, and Xia Hu. Lim maybe longlm: Self-extend Ilm context window without tuning. arXiv preprint arXiv:2401.0132. 2024.
[23] Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018.
[24] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
[25] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving. Jean- Caan-taptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:23002.01318. 2023.
[26] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative IIm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.0978.1 2023.
[27] Aonan Zhang, Chong Wang, Yi Wang, Xuanyu Zhang, and Yunfei Cheng. Recurrent drafter for fast speculative decoding in large language models. arXiv preprint arXiv:2403.099/1 2024.
[28] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
[29] Tri Dao. Flashattention--2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08669, 2023.
[30] Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-decoding for long-context inference, 2023.
[31] Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information questions and answers anchored in research papers. arXiv preprint axXiv:2105.03011, 2021.
12

---

A Case Study: Compatibility with Parallel Decoding
In this section, we provide novel perspective on employing KV cache compression synergistically with parallel decoding (2). Parallel decoding leverages lightweight model or adaptor to draft initial tokens, which are subsequently verified by larger L.LSs. This strategy effectively reduces memory overhead, a critical concern given the autoregressive nature of LLMs that renders them more memory-intensive than computationally demanding. Specifically, in LLMs, each decoding step involves generating single token, with the transfer of weights between High Bandwidth Memory (HBM) and cache contributing to significant overhead
28 29).
cutting-edge parallel decoding frame- work that utilizes multiple classifiers and tree attention mechanisms for drafting tokens, subsequently verified by LLMs. One of the challenges identified is the issue of speculative decoding in processing long sequences since generating multiple tokens per decoding step introduces computational bot- tlenecks during long sequence processing, such as query-key matrix multiplication tiling [30]. By maintaining a constant size for the KV cache associated with prompts during generation, SnapKV enhances generation efficiency.
Our investigation incorporates SnapKV with Medusa 19
Empirical results shown in Figure highhiitt the performance across various prompt lengths, with Misttrl--8-Inntruct--e Instruct-vo..r. undergoing a maximum of 128 generation steps unless preemptively halted. The experiments utilized a subset of the QASPER [11, with fixed prompt instructing the
LLM to summarize the paper. The truncation strategy adopted aligns with LongBench [T8 standards, by removing the context in the middle achieve the desired sequence length for benchmarking.
The findings indicate slowdown in Medusa's performance sequence lengths extend, chal- lenge effectively mitigated by SnapKV's intervention, which achieved 1.3x speedup for sequences with 10k length compared to Medusa and a 2.2x speedup compared to the native decoding. This improvement underscores the potential combining KV cache compression with parallel decoding frameworks enhance LLM efficiency, particularly long-contess scenarios.
Latency Prompt Length
Medusa SnapKV
Medusa
45
Baseline
35
30
25
20
15
Prompt Length (c)
Figure 9: Comparison of generation latency (ms/token). The baseline is the Huggingface implemen- tation of naive decoding.
TGl traind Kedusa heads
13

---

B Overhead Analysis of Prefilling Stage
We evaluate the prefilling time and memory usage Bestral-BBIInstructe .2 with input sequence lengths ranging from 5k to 45k in Fig. [0] The results show no overhead in either aspect.
SnapKV only introduces extra top-k and pooling operations which are trivial regarding computation complexity compared with original prefilling calculations.
Prefil Time
1+10
Prefill Hemory Usage
0.10
SnapKV Prefil Time
SnapkY Prefil Memory Usage
Original Model Prefll lime
Original Model Prefill Memory Usage
Timeet
0.06
0.04
0.02
3:
5000
10600 15ã00
2o0co 25000 s0000 35000 e0000 45000 Sequence Length
5000 10600 15600 200co 25000 33000 35000 eocio 45000
Sequence Lergth
Figure 10: The prefilling time and maximum memory allocated comparison between 2 with and without SnapKV on an H100.
C Discussion of Generation Time Speedup
To better assess SnapKV's effectiveness across different stages, we documented a detailed time breakdown for Mistral- -Baaaall----- -isttral-7--Issruuct--oo 2 during both the prompting and generation stages. We configured the model to consistently generate 512 tokens, facilitating a direct comparison with the prompting stage. As illustrated in Fig. generation time dominates the whole processing time for L.LMs over input sequences, introducing significant overhead. While the generation time for the original model increases with input length, SnapKV maintains consistent decoding speed regardless of input length, significantly reducing generation time. Especially, SnapKV is able to achieve balanced prompting time and generation time with input length smaller than 100k.
Prompting Latency v.s. Generation Time
prompting latency
100000
generation latency SnapKV generation latency without SnapKV
80000
Latengy(ms)
60000
40000
20000
10k
15k
20k
70k
75k
80k
85k
90k
95k
Input Sequence Length
Figure 11: The prompting time and generation time comparison between Mistral model with and without SnapKV.
14

---

D Statistics of LongBench Dataset
Here we provide more statistical information on LongBench Table to provide more insights on our results in 5.3
Table Sequence Length Statistics for LongBench Benchmark
Dataset
Min
Max
Avg
Qasper
1443
14722
4620
HotpotQA
111
12480
6658
2WikimQA
881
23442
7141
GovReport
111
12480
6658
MultiNews
374
27973
6000
MF-en
505
10337
4559
TREC
746
13034
5475
TriviaQA
804
15960
6685
SAMSum
936
12403
6170
PCount
1407
14537
6117
PRe
2358
10607
6115
Lec
386
14106
4283
RB-P
785
18864
6067
15

---

E
Visulization of the Generated Context
Prompt(length 20581) from Qasper
Ground Truth
You scientific article and question. Answer the question as concisely as you can, using a single phrase sentence possibbe. the question cannot answered based on the information in the article, write "unanswerable", the question yes/no question. answer "yes", "no", or "unanswerable" Do not provide any explanation.
Ground truth not established in the paper
SnapKV: 1024
The ground truth fake not claimed the article.
Characterizing Political Fake News Twitter by its Meta-DataJull Juta.llio
LópezAxal Oehmichen Miguel Molina- Solanal a.mmador,
SnapKV: 2048
axelfrancois. oehmichent1. mmolinas@imperial..eo Imperial College London This article presents a preliminary approach
The ground truth fake news is not claimed article as the categorization far from perfect due to the ambiguity fake news and human judgement involved the process.
characterizing political fake news on Twitter through the analysis of their mela-data. particular, focus collected on the day of the election of Donald Trump as president the United the meta-data embedded within those tweets in order differences and tweets not containing them.
than 1.5M
SnapKV: 4096
We
The dataset is not claimed ground truth due to the ambigully fake news and human judgement involved in the categorization process.
containing
KV
Question: How the ground truth for fake news established?
The distaset is not claimed to ground truth due to the ambiguity fake and human judgement involved in the categorization process.
Answer:
Prompt(length 31500) from HotpotQA
Ground Truth
nswer the question based on the given passages. Only the and do not output any other words.
Ellie Kemper
SnapKV: 1024
The following given passages
Passage
List Unbraakable Kimmy Schmidt episodes Unbreakable Kimmy Schmidt is an American sitcom by Tina
Ellie Kemper
Unbreakable Kimmy Schmidt.
Robert Carlock, starring Ellie Kamper the tile role, that has streamed on Netflix since March
13-episode first season on NBC for spring 2015, the show was sold to given
Originally
SnapKV: 2048
fourth season conciuded on January 25. 2019. On May 8, an interactive special, which promiered on May 12, 2020. During the course series, episodes Unbreakablo Kimmy Schmidt were released over four seasons.
Ellie Kemper
Unbreakable Kimmy Schmikt.
SnapKV: 4096
Ellie Kemper
Unbreakable Kimmy Schmidt.
Answer question based on the given passages. Only the answer and do not output other words.
All KV
Who stars in the title role of Netfixx show co co-created by Robert Carlock?
Question:
Kemper stars
Unbreakable Kimmy
Schmidt.
Answer:
Figure 12: Visualization of generation examples from Samsum, Qasper, HotpotQA datasets with 1astral-B-innsruee-ro0 intt-innstuuttoo...
mistral-7B-
Results are compared between ground truth, SnapKV with 1024 prompt tokens, with 2048, with 4096, the baseline model with full KV cache.
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiatiin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation. While "[Yes] is generally preferable to "[No] it is perfectly acceptable to answer "[No] provided provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] " not grounds for rejection. While the questions are phrased binary way, acknowledge that the answer is often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
the contribution and scope can be found in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
17

---

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
we discuss the limitations and future directions in conclusion Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: we not include any theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
18

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: we include needed experiment details in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: we provide psudo code in Listing, to implement SnapKV and include needed experiment details in Sec. 5
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (tttps://nnis..c cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
www.iiisssiioiiiiiioooomm for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
for every experiment, mention all the needed details in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
20

---

Justification:
all experiments detailed results are included in Sec.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
mention computer resources for experiments and provide the memory and latency information, which could all be found in Sec. 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp/pp//aacrrp.ccccppp/ccccca
Question:
Answer: [Yes]
21

---

we keep all information credential and strictly follow the NeurIPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: we discuss the contribution and weakness in discussion Sec, 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: we not release any new models or datasets in the paper.
Guidelines:
22

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we credit all mentioned assets in reference
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, wwwpsswiteeetteeeeeeeeeeeep thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out the asset's 's creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: we provide a new algorithm that can used on many existing models, and do not introduce any more assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
23

---

The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: do include crowdsourcing nor research with human subjects. All based on existing automatic evaluator and models.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: we do not include crowdsourcing nor research with human subjects. All are based on existing automatic evaluator and models.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "Qh2Gjw8hoO", "reviewer": "Reviewer_Zgi7", "summary": "The paper introduces a method for minimizing KV cache size in LLMs. The authors offer the insight that attention heads focus on specific prompt attention features (tokens and their feature representation) during generation. This pattern can be discovered via an observation window at the end of the prompt. The new method, SnapKV, compresses KV caches by selecting clustered important KV positions for each attention head. As a result, the computational overhead and memory footprint are significantly reduced for long inputs. The method is evaluated across various LLMs and datasets showing its efficiency in practical applications.", "questions": "How sensitive is the method to the observation window size, or the pooling kernel size?\nIt would be useful to include some more details about the performance measures used in Table 1.", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses a significant challenge in LLM efficiency for long context processing.\nThe insight about consistent attention patterns is useful and may generate new ideas.\nThe results show some impressive performance improvements and ability to handle very long sequences.\nThe method is fine-tuning free, and can be integrated into existing frameworks.", "weaknesses": "The impact on model accuracy for very long context (more than 100K tokens) is not thoroughly explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1. How sensitive is the method to the observation window size, or the pooling kernel size? It would be useful to include some more details about the performance measures used in Table 1.*\n\n**Answer:** Thanks for the valuable feedback. To address your comment, we experiment with SnapKV on various observation window sizes and pooling kernel sizes. The results can be found in PDF Table 1. Different configuration performs differently in various kinds of datasets as expected.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DcwXDOY3Y7", "reviewer": "Reviewer_wNH7", "summary": "This paper presents a training-free KV cache compression approach. This approach builds off of the observation that the model consistently focuses on particular features during generation, and that these features can be detected when the prompt is passed in. Their approach uses a small “observation window” of queries at the end of the prompt to detect which previous keys must be retained, and then prunes out other key / value pairs. They also incorporate a pooling-based approach in order to maintain contextual information around important tokens. Through these methods, they attain generation speedups and reduced memory consumption for long context length tasks, while maintaining accuracy.", "questions": "- For Table 1, it would be good to also include the average input prompt sizes for the different tasks (in order to get a sense of the relative compression that is attained by going to fixed cache sizes of 1K, 2K, 4K)", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Their paper provides multiple important insights about how models use their input context; namely, the insight that the important tokens during the prompt processing phase stay consistent throughout the generation process, as well as the observation that different instructions prioritize different parts of the input prompt (which therefore necessitates a dynamic approach)\n- Their approach compresses the KV cache for the input prompt, which is crucial for many long context length applications (where the majority of the context length is used up by a long prompt), while maintaining accuracy\n- Their approach is relatively simple and is therefore compatible with existing kernel methods (and can be implemented using only Pytorch-level code changes)\n- Significant accuracy improvements over prior work (H2O) when processing long context length tasks", "weaknesses": "- This paper doesn’t accelerate the prompt processing step (even though it is explicitly targeting long prompt processing), which could present a bottleneck for tasks with long input lengths and short generation lengths\n- The paper lacks benchmarking experiments to justify that their approach doesn’t add any overheads to the prompt processing step (due to having to use the observation window to identify important tokens, which likely needs to be applied separately from FlashAttention)\n- They do not provide sufficient justification for their pooling approach, and their ablations are insufficient to show its effectiveness (they show accuracy benefits on one task, but for other tasks with less sparse attention distributions than retrieval this may actually degrade performance by prioritizing less important tokens that are near important ones). It would be good to ablate the impacts of pooling on non-retrieval tasks as well (eg. few-shot ICL) which may have a flatter attention distribution, and where pooling may lead to retrieving less of the actual important information that is required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1. The paper lacks benchmarking experiments to justify that their approach doesn’t add any overheads to the prompt processing step.*\n\n**Answer:** Thanks for the valuable feedback. To address your comment, we evaluate the prefilling time and memory usage on Mistral-7B with input sequence lengths ranging from 5k to 45k in PDF Figure 1 & 2. The results show no overhead in either aspect. SnapKV only introduces extra topk and pooling operations which are trivial regarding computation complexity compared with original prefilling calculations.\n\n*Q2. They do not provide sufficient justification for their pooling approach, and their ablations are insufficient to show its effectiveness (they show accuracy benefits on one task, but for other tasks with less sparse attention distributions than retrieval, this may degrade performance by prioritizing less important tokens that are near important ones). It would be good to ablate the impacts of pooling on non-retrieval tasks as well (eg. few-shot ICL) which may have a flatter attention distribution, and where pooling may lead to retrieving less of the actual important information that is required.*\n\n**Answer:** We conduct ablation experiments on the effectiveness of pooling on the LongBench dataset and the results are presented in PDF Table 1 as w=32 k=1. In eight out of nine tasks, the model accuracy with pooling is better than those without pooling.\n\n*Q3. For Table 1, it would be good to also include the average input prompt sizes for the different tasks.*\n\n**Answer:** Thanks for the feedback, we provide more information about the LongBench benchmark in PDF Table 3. In general, the sequence lengths approximately range from 100 to 23000 with an average of 5817.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cVJrqgD7lT", "reviewer": "Reviewer_Akz1", "summary": "This paper analyzes the issue of high storage pressure in KV caches within long-context scenarios and proposes a method for KV cache compression. Specifically, it calculates the attention weights between the last 16 Q tokens and K, and utilizes pooling and topK to determine the KV pairs to retain. The paper tests four models—LWM-1M, Mistral-32K, LongChat-32K, and Mixtral-32K—on two tasks, LongBench and Needle In A Haystack. The results show a 2% drop on the GovReport task and a reduction to 140K context on the Needle In A Haystack task, while in other tasks, the results are generally consistent with full attention. The decoding stage latency is reduced by 3.6x.", "questions": "1. Do you have additional baseline results for the Needle In A Haystack test, such as those for H20 or the full-attention model?\n2. Are there any results from more demanding long-context benchmarks, such as RULER[1] or InfiniteBench[2]? I am particularly curious about how SnapKV would perform on tasks akin to KV retrieval[2]. Furthermore, do you have experimental results from other, more capable long-context language models, such as Yi-200K[3] or LLaMA-3-1M[4]?\n3. Have you conducted experiments on contexts longer than those reported, and if so, could you share those results?\n4. Regarding the speedup data in Sec 5.1.2, are these results based on the vLLM freamwork? If not, could you provide results for vLLM, as they would more accurately reflect the degree of speedup improvement in real-world scenarios?\n5. Typo: In Fig. 8, the right image's label \"without\" should be corrected to \"with.\"\n\n- [1] RULER: What’s the Real Context Size of Your Long-Context Language Models?\n- [2] InfiniteBench: Extending Long Context Evaluation Beyond 100K Tokens\n- [3] https://huggingface.co/01-ai/Yi-34B-200K\n- [4] https://huggingface.co/gradientai/Llama-3-8B-Instruct-Gradient-1048k", "limitations": "Please refer to the weaknesses section.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The problem studied in the paper is significant and has practical value.\n- The motivation of the paper is sound, supported by ample experimental evidence.", "weaknesses": "1. The experimental section is limited, with tests conducted only on two benchmarks, making it difficult to demonstrate the method's generalizability and effectiveness. For the Needle In a Haystack task, only the results of LWM-1M with SnapKV are tested, lacking comparison with full attention, other baselines, and results from other models. Additionally, the tested context windows are relatively short, only up to 380K.\n2. The paper uses a pooling method in the approximate stage and claims it is for more efficient pooling. However, Fig. 8 shows significant performance improvements with pooling, leading to doubts about whether the important Keys identified by the observation Query remain unchanged during the decoding stage, especially in retrieval tasks or tasks like Needle in the haystack.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Q1. The paper uses a pooling method in the approximate stage and claims it is for more efficient pooling. However, Fig. 8 shows significant performance improvements with pooling, leading to doubts about whether the important Keys identified by the observation Query remain unchanged during the decoding stage, especially in retrieval tasks or tasks like Needle in the haystack.*\n\n**Answer:** Our observation reflects the consistent pattern during the decoding stage. The reason behind the effectiveness of pooling is that it could keep the integrity of information. For example, there is a number 123456, and SnapKV finds 34 are the important tokens. If we only retrieve 34, it cannot keep the original number, but with pooling, we can select 12 and 45 as well to retain the whole information.\n\n*Q2. Are there any results from more demanding long-context benchmarks, such as RULER[1] or InfiniteBench[2]? I am particularly curious about how SnapKV would perform on tasks akin to KV retrieval[2]. Do you have experimental results from other, more capable long-context language models, such as Yi-200K[3] or LLaMA-3-1M[4]?*\n\n**Answer:** Thanks for the valuable feedback. Because of the time limit, we are unable to evaluate these benchmarks, but we evaluate on Llama-3-1M as you referred. We evaluate one dataset for every category in the LongBench benchmark in PDF Table 2. The results suggest that SnapKV (compressed to 1024 tokens with the observation window equal to 32 and pooling kernel equal to 7) can be adapted to the Llama model and match or even outperform the original model on various tasks.\n\n*Q3. Do you have additional baseline results for the Needle In A Haystack test, such as those for H20 or the full-attention model? Have you conducted experiments on contexts longer than those reported, and if so, could you share those results?*\n\n**Answer:** For the Needle In a Haystack task, the original LWM model will encounter an OOM error after 30k sequence length. Additionally, we test the Needle In a Haystack on H2O.  However, H2O needs to compute the attention weight of every decoding step during the generation to decide which KV cache to drop. This mechanism of H2O makes it incompatible with FlashAttention (aims to not store full attention weight), which results in even faster OOM error (less than 30k) on a single A100. This result also reveals that H2O cannot solve the long context problem in general. In addition, all our tests are based on a single A100 with 80G GPU, which limits the sequence length to 380K.\n\n*Q4. Regarding the speedup data in Sec 5.1.2, are these results based on the vLLM framework? If not, could you provide results for vLLM, as they would more accurately reflect the degree of speedup improvement in real-world scenarios?*\n\n**Answer:** Thanks for the suggestion. Because LLM serving frameworks like vLLM incorporate many optimizations on parallelization, memory management, etc., it is unfair for end-to-end comparison. [Link][1] is an independent study across different LLM serving frameworks including vLLM, TensorRT, etc. It shows the benefit of this framework on input sequence lengths ranging from 20 to 5000. As SnapKV typically compresses KV cache to 1024 to 4096, we expect the LLM serving framework and SnapKV can mutually benefit from each other, and achieve stacked speedup.\n\n[1]: https://www.inferless.com/learn/exploring-llms-speed-benchmarks-independent-analysis", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for all the questions and suggestions. The PDF file contains all the tables and figures mentioned in the rebuttal.

Meta-review: This paper discussed an important problem in improving the efficiency of long context inference. The solution is solid with good novelty, and the experimental section is well designed to verify the design of the proposed solution.
Decision: Accept (poster)