ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
Authors: Yefei He
Luoming Zhang
Weijia Wu
Jing Liu
Hong Zhou
Bohan Zhuang
Venue: NeurIPS 2024 poster (2024)
Keywords: Model Compression
Large Language Models
Model Quantization
Efficient Models
Primary area: generative_models

Abstract: KV cache stores key and value states from previous tokens to avoid re-computation, yet it demands substantial storage space, especially for long sequences. 
  Adaptive KV cache compression seeks to discern the saliency of tokens, preserving vital information while aggressively compressing those of less importance.
  However, previous methods of this approach exhibit significant performance degradation at high compression ratios due to inaccuracies in identifying salient tokens. 
  Additionally, the compression process introduces excessive overhead, substantially increasing memory burdens and the generation latency.
  In this paper, we present ZipCache, an accurate and efficient KV cache quantization method for large language models (LLMs). 
  First, we construct a strong baseline for quantizing KV cache. Through the proposed channel-separable tokenwise quantization scheme, the memory overhead of quantization parameters are substantially reduced compared to fine-grained groupwise quantization.
  To enhance the compression ratio, we propose normalized attention score as an effective metric for identifying salient tokens by considering the lower triangle characteristics of the attention matrix. The quantization bit-width for each token is then adaptively assigned based on their saliency.
  Moreover, we develop an efficient approximation method that decouples the saliency metric from full attention scores, enabling compatibility with fast attention implementations like FlashAttention.
  Extensive experiments demonstrate that ZipCache achieves superior compression ratios, fast generation speed and minimal performance losses compared with previous KV cache compression methods. For instance, when evaluating Mistral-7B model on GSM8k dataset, ZipCache is capable of compressing the KV cache by $4.98\times$, with only a 0.38% drop in accuracy. In terms of efficiency, ZipCache also showcases a 37.3% reduction in prefill-phase latency, a 56.9% reduction in decoding-phase latency, and a 19.8% reduction in GPU memory usage when evaluating LLaMA3-8B model with a input length of 4096. Code is available at https://github.com/ThisisBillhe/ZipCache/.

ZipCacher Accurate and Efficient KV Cache Quantization with Salient Token Identification
Yefei He!
Luoming Zhang!
Weijia Wu
Jing Liu
Hong Zhou
Bohan Zhuang!.
IZhejiang University, China
National University of Singapore, Singapore ZIP Lab, Monash University, Australia
Abstract
KV cache stores key and value states from previous tokens avoid re-computation, yet it demands substantial storage space, especially for long sequences. Adaptive KV cache compression seeks to discern the saliency of tokens, preserving vital information while aggressively compressing those of less importance. However, previous methods of this approach exhibit significant performance degradation at high compression ratios due to inaccuracies in identifying salient tokens. Ad- ditionally, the compression process introduces excessive overhead, substantially increasing memory burdens and the generation latency. In this paper, present ZipCache, an accurate and efficient KV cache quantization method for large lan- guage models (LLMs). First, construct strong baseline for quantizing KV cache. Through the proposed shannel-sepprable tokenwise quantization scheme, the memory overhead of quantization parameters substantially reduced com- pared to fine-grained groupwise quantization. To enhance the compression ratio, we propose normalized attention score as effective metric for identifying salient tokens by considering the lower triangle characteristics of the attention matrix. The quantization bit-width for each token is then adaptively assigned based their saliency. Moreover, we develop an efficient approximation method that decouples the saliency metric from full attention scores, enabling compatibility with fast attention implementations like FlashAttention. Extensive experiments demonstrate that ZipCache achieves superior compression ratios, fast generation speed and min- imal performance losses compared with previous KV cache compression methods. For instance, when evaluating Mistral-7B model on GSM8k dataset, ZipCache is capable of compressing the KV cache by 4.98x, with only a 0.38% drop in accuracy. In terms efficiency. ZipCache also showcases a 37. 37.3% reduction prefill-phase latency, a 56.9% reduction in decoding- latency, and a 19.8% reduction in GPU memory usage when evaluating LLaMA3-SB model with a input length of 4096. Code is available at https://gituu..com/
Introduction
LLMs with the next-token prediction
next-tooen-prrdictton scheme have achieved remarkable advancements in various
text-related tasks, such as language understanding [13, 34, 10], content creation [1, 5, 36], coding [3, 29, 42] and mathematics [33, 23, 35]. In this generation scheme, the forthcoming token interacts with all previous tokens via the attention mechanism [38], where the query, key and value states will be
Corresponning author, Email: zhouhong 2 2ju@zju.edu.co bohan bww.n..huang@@ wwwang@gmail..comm
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

calculated for each token. As the past tokens will not be altered, previously computed key and value states can be stored as KV cache to prevent re-computations, significantly improving the generation speed. However, as the batch size and the input context length grows, the stored KV cache emerges as new memory bottleneck for I.L.s. For example, when serving a parammerre L.LM [1] with a batch size of 64 and a context length of 4096, the KV cache can occupy 1.2TB of memory space, while the model weights only require 350GB. Meanwhile, the size of KV cache will continue to increase as decoding progresses. Therefore, the compression of KV cache is crucial for the efficient deployment of LLMs.
Recent compression methods for KV cache can be broadly categorized into two types. The first type of methods compresses the KV cache uniformly, without considering the significance of individual tokens. To preserve performance. these methods often rely on either high-precision quantization [21] or maintaining recent tokens in full-precision [32], which undoubtedly compromise the compression ratio. Additionally, if salient tokens are not among the most recent ones, such as in information retrieval tasks, it may result in degraded performance. The other type of methods [46, 43, 16] compress KV cache adaptively by identifying salient tokens and compresses them separately. This approach aligns with the observation that a minority of tokens contribute the majority of attention scores [41], potentially achieving higher compression ratios than non-adaptive methods. However, current adaptive KV cache compression methods [46, 43] use accumulated attention scores as a metric of token saliency, which is insufficient in two aspects. First, accumulated attention scores is inaccurate in identifying important tokens. Due to the presence of attention masks, the attention matrix is lower triangular matrix. Earlier tokens tend to have larger softmax attention values and more attention scores to be accumulated, as illustrated in Figure 3. Under this metric, the saliency of the most recent tokens can never surpass that of the first token, thereby introducing bias in determining token saliency. Additionally, to obtain accumulated attention scores, full attention matrices must be explicitly computed and stored, which can be inefficient for serving LLMs. Given an input context length of (, fast attention implementations such as FlashAttention [8, 7] only require O()) memory by computing attention output in blocks without retaining complete attention matrices. By contrast, storing full attention matrices requires ((1²) memory, and the large number of memory accesses significantly slows down the inference speed, as depicted in Figure 4.
To address these challenges, we introduce ZipCache, an efficient KV cache compression method that attains ex- ceptionally high compression ratios by accurate salient token identification. Figure presents overview of
100
y comparisons among ZipCache and diverse KV cache compression methods. We start by designing an efficient quantization baseline for com- pressing the KV cache. To preserve performance, prede- cessor methods [32, 21] employ fine-grained groupwise quantization, which involves independent quantization for a small channel group within each token. However, this method necessitates storing extensive quantization parameters and results in significant memory overhead. By contrast, we introduce channel-separabl quanti- zation scheme that decouples the quantization along channel and token dimensions. This method signifi- cantly reduces the quantization overhead without com- promising performance. To accurately recognize salient tokens, we introduce a new token saliency metric based on normalized attention scores, which alleviates the bias towards earlier tokens that accumulate more val- ues. All tokens, without exception, will be quantized to the target bit-width based their estimated saliency, boosting the overall compression ratio. Moreover, to ease integration with fast attention implementations, we introduce an efficient approximation of the token saliency metric. This approximation only relies on computing and storing attention scores from a few number of tokens, which we refer as probe tokens. An effective probe token selection strategy is then introduced to minimize performance loss. As a result, the majority of tokens can benefit from fast attention implementations, significantly enhancing the generation speed.
Accuracy (%)
ZipCache
MiKV
KIVI
91
GEAR
88
H2O
3
5
Latency (s)
Figure 1: Accuracy and efficiency compar- isons across various KV cache compression methods. Data is collected with LLaMA3- 8B model on Line Retricval dataset. Among these methods, ZipCache achieves the high- est accuracy. generation speed and compres- sion ratio. Details can be found in the sup- plementary material.

---

In summary, our contributions are as follows:
We establish efficient channel-separable quantization scheme for KV cache, which significantly reduces the overhead of quantization parameters without compromising performance compared to fine-gran fine-grained groupwise quantization approach.
We propose an accurate metric for assessing token saliency based on normalized attention scores. All tokens adaptively quantized according to their assessed saliency, thereby improving the overall compression ratio.
We further develop efficient approximation method for the token saliency metric that integrates seamlessly with fast attention implementations, enhancing generation speed.
By integrating these three techniques, we present ZipCache, accurate and efficient framework for KV cache compression. Extensive experiments demonstrate that ZipCache reaches a new state-of-the-ar performance for KV cache compression in terms of compression ratio, accuracy and generation efficiency.
Related Work
2.1 Model Quantization
Quantization is a prevalent technique for compressing deep neural networks by representing model weights and activations with lower numerical bit-widths. This technique can be categorized into two primary approaches based on the necessity of fine-tuning: post-training quantization (PTQ) [26, training (QAT) [28, 31]. For large language models (LLMs), where
17, 14] and quantization-awe fine-tuning can be data- and computation-intensiv PTQ is often the preferred method [40, 45, 27]. In this paper, we also quantize KV cache in post-training manner. For both approaches, quantization can be implemented at various levels of granularity, including channelwise, tokenwise, and groupwise approach. Typically, a finer quantization granularity involves the independent quantization of smaller parameter groups, which often results in improved performance albeit at the cost of more quantization parameters and increased memory overhead. In the context of LLMs, fine- ane-grained quantization is frequently utilized due to the presence of outliers [22, 45]. However, for KV cache compression, this will greatly reduce the overall compression ratio.
Mixed precision quantization [39, 44, 12, 2] allocates varying bit-widths to distinct parts of a model or tensor, enabling more compact compression. This approach originates from the observation that model components exhibit differing sensitivities to quantization. Consequently, components with low sensitivity can utilize reduced bit-widths without impairing performance. For LLMs, previous studies [46, 43, 30, 18) have shown significant disparities in the importance of tokens, indicating that heavy compression of non-critical tokens has minimal impact overall performance. This insight highlights the applicability of mixed precision quantization for compressing the KV cache.
2.2 KV Cache Compression
While KV cache effectively prevents re-computation and significantly enhances generation speed, its memory footprint is notably substantial with long-context input. To alleviate this, many efforts have been made to reduce the KV cache size. Based on the compression method, these methods can be categorized into two groups: token dropping [46, 16, 30] and KV cache quantization [43, 21, 32]. The former identifies and drops unimportant tokens in the KV cache. For example, H2O [46] only maintain 20% heavy-hitted tokens and 20% recent tokens while evicting the rest. However, discarding tokens permanently erases their information, which proves to be suboptimal for tasks such as retrieval [43]. Conversely, the latter category employs quantization on the cached key and value states, and mixed precision quantization can further be applied once token importance is identified [43]. To tackle the outliers present in the KV cache, these methods extract the outlier as full precision [21] or use finer- irei-grained quantization scheme [32], which increases the quantization overhead. In this study, we propose an efficient fraanel-seeprable quantization scheme with reduced quantization overhead and strong performance. Additionally, both categories of methods commonly adopt accumulated attention scores as the metric for token importance [46, 43). However, we observe that this criterion is inaccurate and can result in significant performance deterioration at low bit-widths. In contrast, we achieve superior compression performance by utilizing a more accurate metric for identifying salient tokens.
3

---

3 Preliminary
3.1 Attention Block in LLMs
Given an input prompt, the generation process of LLMs can be broadly categorized into two distinct phases: the prefill phase, which computes and stores the KV cache for input tokens, and the decoding phase, where new tokens are generated through a next-token-predietee X and an attention block with its weight matrices Wq. WK and Wv, the prefill phase can be formulated as:
scheme. Given input data
Q XWQ.
== XWK.
V XWv.
QK Vae
A = Softmax
O AV.
Here, dg is the dimension of the key, and A refers the attention scores. K and V will be stored as KV cache. For clarity, we have omitted the output projection.
For the decoding phase, given x as the embedding vector of the current token, the query q becomes vector and the KV cache matrices will be updated as follow:
q xw Q. K
Concat (K, xWK),
V Concat Concat(V,xXV) (V.xWy).
(3)
The attention output are then computed as follows:
qk Vrl
Softmax
aV.
(4)
To ensure clarity and consistency, we introduce notation to define the hyper-parameters used in the paper. Specifically, we denote the batch size as b, the number of attention heads as h, the sequence length as I, and the head dimension as
3.2 Model Quantization
Uniform quantization is adopted in our study and all experiments. Given floating-pon point vector x, it can be uniformly quantized to b-bit as follows:
Qu k) (clip(( _
1 +,, 1) 2) 88.
Here, 1.1 denotes the round operation, max(x) and -1 1 are quantization 2
parameters. It should be noted that the quantization parameters are stored in full-precision, which can lead to significant overhead if the quantization is fine-graid grained.
Method
4.1 A Strong Baseline for KV Cache Quantization
Tokenwise quantization, as depicted in Figure 2(b) is prevalent in quantizing large language models (LLMs) due to the distinct representations of individual tokens. However, it has been widely observed, as illustrated in Figure 2(a), that outliers emerge within the channel dimensions and value matrices [43, 32), posing challenges for tokenwise quantization. To address this, recent work [32] resorts to groupwise quantization, where outlier channels are processed in distinct groups, illustrated in Figure 2(c). However, this fine-grained quantization approach introduces excessive memory overhead, thereby significantly impacting the compression ratio. For instance, considering X gexhxixd as the data be quantized and group size of n, tokenwise quantization only results in 2bl quantization parameters, while groupwise quantization would yield 2bhld quantization parameters. Since these parameters are usually stored in full precision, this overhead would constitute substantial portion of the storage cost for quantized data.
key
Motivated by depthwise separable convolution [19], we introduce an efficient channel-separable tokenwise quantization scheme, which disentangles the channel and token dimensions. As shown in

---

Key
Value
232
250
Token Index
E w/-hd/n
plond
Rh
Ri
1790
1230
2000
7000
Channel Channel Index
R
0.5 1.0 15 2.0
(a) Visualizaaion of Key and Value States
(b) Tokenwise
(c) Groupwise
(c) Channel-separaale
Quantization
Quantization
Tokenwise Quantization
Figure 2: Visualization and different quantization granularities for key and value states. Here, omit the batch dimension for simplicity. For keys, channel outliers emerge, yet token representations exhibit minimal differences. For values, both channel outliers and distinct token representations exist.
Figure 2(d), our approach initiates by normalizing each channel of data X with scaling factor For the i-th channel in X, the normalization process can be formulated as:
X; C;
X,
where c, //ax X,|)-
After normalization, each channel scaled a closed magnitude, mitigating the influence of outliers during tokenwise quantization. Subsequently, tokenwise quantization can be reliably applied and the scales are multiplied back restore the magnitude of each channel. The process of -hannel-separable tokenwise quantization is summarized in the supplementary material. Within this quantization scheme, the total number of quantization parameters amounts hd + 2bl, representing notable reduction compared to groupwise quantization, while effectively balancing the outlier channels and the representation of each token.
Table 1: Performance comparisons of different quantization granularities for KV cache. The KV cache is quantized to 4-bit and the compression ratio is calculated with 6 8, hd = = 4096 and n 32. Data is collected with LLaMA3-8P model on GSM8k dataset.
Key Cache
Value Cache
Quantization Parameters
Compression Ratio
Acc.(%)
Quantization Granularity
Quantization Granularity
0 4bhld/n
1x
55.88
Groupwise
Groupwise
3.2x
54.51
Tokenwise
Tokenwise
4b
3.99x
49.81
Channelwise
Tokenwise
2hd 25l
4.00x
52.77
Channelwise
-haanel-separable Tokenwise
3hd 26d
4.00x
54.74
As referred to Figure 2(a), since the differences in token representations small in key cache, employ channelwise quantization for the key cache to further reduce overhead and employ channel- separable tokenwise quantization for the value cache. As depicted in Table 1, this configuration yields superior performance with reduced quantization overhead compared with groupwise quantization, thereby establishing robust baseline for KV cache quantization.
4.2 Accurate Salient Token Identification
Adaptive KV cache compression [46, 43, 16] aims to discern the saliency of each token, keeping information of salient tokens while evicting aggressively compressing the rest, to achieve higher compression ratio. These salient tokens, also referred to as "Heavy Hitters" [46], are often identified based on accumulated attention scores. Given attention matrix A g the saliency of token i estimated by:
Pi
A
(7)
km
5

---

Identified by
Identified by
6g5 live are there +p
0.00
There trees in the grove..
Question:
Saliency
0.33
0.67
0.00
0.00
0.00
Let's think step step...
0.5
Question: If there are 3 vars... I et' think step step
0.35
0.30
0.35
0.00
0.00
0.04
0.76
Question: Leah had 32 chocolates... Let's think step by stop...
0.0
200
400
600
800
1.0
0.02
0.02
0.04
0.06
0.36
Question: Olivia has S23... think step by step...
Saliency
So
there
aire
five
eggs
0.5
Pr
1.74
1.05
0.53
0.87
0.86
Question: Janet's docks lay eggs
day...How much in dollars does she
Pt
0.35
0.25
0.18
0.41
0.86
make every day at the farmers' market?
0.0
200
400
600
800
-.0% salient tokens
Token Index (c)
(a)
(b)
Figure 3: (a) A toy example to illustrate accumulated attention scores and normalized attention scores. Initial tokens have larger attention scores and more values to be accumulated. (b) A sample from GSM8k dataset with chain-of-thoughh (CoT) prompting. (c) The probability of each token being selected as salient token, measured by both accumulated and normalized attention scores. Tokens correspond to the final question are identified as low saliency by accumulated attention scores.
Tokens with large saliency values are then considered salient tokens. However, this approach has inherent limitations due the lower triangular nature of the attention score matrix, as illustrated in Figure 3(a). There are two primary issues. Firstly, earlier tokens benefit from having more values accumulated since the elements above the diagonal are all zero. For instance, in sequence of length 1, the initial token accumulates positive values, whereas the final token only accumulates one. Secondly, Softmax function converts real numbers into probabilities, the attention matrix tending to have higher values, as fewer numbers are involved in the Softmax calculation. Consequently, the accumulated attention score of the final token will always be smaller than that of the first, which exceeds 1. To address this, previous works, such as H2O [46], always maintain recent caches in full precision. Nevertheless, this solution suboptimal since recent tokens are not necessarily the most sigmificant ones.
that the earlier rows of
To enhance the evaluation of each token's 's saliency, introduce accurate token saliency metric based on normalized attention scores pr:
ki
p.
(8)
nnZ A.
Here, (..()..) denotes the number of non-zero elements in the i-th column of A. As evidenced in Figure 3(a), normalizing the accumulated attention scores mitigates the influence of excessively large values in the initial rows of the attention score matrix, thereby delivering more precise assessment. To validate the efficacy of our new metric, we input a sample from GSM8k dataset with chain-of- thoughts (Co'T) prompting to the LLLMA3-88 8B model and identify saliency of each token by Eq. 7 and Eq. 8, respectively. As depicted in Figure 3(b) and (c), the salient tokens are at the end of the prompt. which correspond to the question for LLM to answer. However, these tokens identified as low saliency by accumulated attention scores. Under the KV cache compression framework. these tokens would either be discarded or quantized to extremely low bit-width, resulting in significant performance deterioration. In contrast, our method accurately identifies the salient tokens. Additional experimental results regarding the accuracy of our method will be detailed in Section 5.2.
4.3 Efficient Approximation of Saliency Metric
As analyzed in Section 4.2, adaptive KV cache compression requires the explicit computation of full attention scores, as referred to Figure 4(b,, which clashes with fast attention implementations like FtashAttention [8, 7, 9]. As shown in Figure 4(c), FlashAttention computes attention outputs in tiles without storing the intermediate attention scores. To reconcile the efficiency of FlashAttention with the substantial compression offered by adaptive KV caching, we devise an effective approximation for Eq. 8 as measure of token saliency. Specifically, we sample small group of tokens, designated

---

(b) Standard Ulention
Internedaate attention scores
Probe tokens
A probe
FlashA Itention
Inpul lokens
Output
M Memory=O((-)
Memory=O(()
Regular takens
More memory & slower
memory access & faster
(a) Efficient Saliency Metric with Probe Tokens
(b) Standard A Aention
(c) FlashA Uention
Figure 4: (a): Efficient saliency metric only requires attention scores of probe tokens through
standard attention, enabling fast computation for the majority of tokens through FlashAttention. (b):
In standard attention, full attention scores are computed before deriving the attention output. (c):
FlashAttention avoids large attention matrix memory transfers by partitioning input matrices into blocks for incremental computation.
probe tokens, and compute their attention scores Aprobe follows:
Qprote KT
A probe Softmax
vdg
By substituting Aprobe into Eq. 8, we can approximate the saliency of all tokens. For the remaining non-probe tokens, their attention scores do have to be computed explicitly, enabling the integration of fast attention implementations to expedite the generation process, as illustrated in Figure 4(a).
However, the positions of the probe tokens will un- doubtedly affects the accuracy of the approximated token saliency and the selection of probe tokens is under explored. In this study, we suggest four strategies for sampling probe tokens:
Table Performance comparisons of various probe strategies. Data is col- lected from LLaMA3-8B model on GSM8K dataset. We quantize 40% salient tokens to 4-bit and the remaining 60% tokens to 2-bit. The proportion of probe tokens 10%.
Random tokens. The probe tokens are randomly sam- pled from all positions.
Special tokens. The special tokens and punctuation tokens will be treated as probe tokens.
Prohe Strategy
Acc.(%)
All tokens
52.54
Recent tokens. The most recent tokens are selected as probe tokens.
Random tokens
47.46
Special tokens
46.78
Recent tokens
51.10
Random+recent tokens. The probe tokens will be di- vided into two parts, one using recent tokens and the other randomly selecting from the remaining tokens.
Random--cccett tokens
52.08
It should be emphasized that study diverges from prior research [16] in that, rather than directly choosing special or recent tokens as salient tokens, we opt to sample subset of tokens as "probes" detect the salient ones. As depicted in Table 2, present comprehensive comparison of the performance among four distinct sampling strategies. Among the four strategies examined, hybrid approach that combines recent tokens with randomly selected tokens emerges the most effective. Unless otherwise specified, this hybrid strategy with 5% recent tokens and 5% random tokens will be employed in our method.
5 Experiment
5.1
Implementation Details
Models and datasets. To validate the efficacy of our proposed method, we conduct experiments with three open-source LLMS:: Mistral [20], LLaMA2 [37] and LLaMA3. These models are evaluated on three challenging benchmarks: GSM8k [6] for math problem solving, HumanEval [4] for code generation, and Line Retrieval [25] for data retrieval. ensure reproducibility. the reported results obtained using the Language Model Evaluation Harness [15] and LongEval [24].

---

Quantization and generation settings. We employ mixed precision quantization for KV cache where salient tokens will be quantized 4-bit while the remaining will quantized 2-bit. For both subsets, we apply channelwise quantization for the key cache and channel channel-separable tokenwise quantization for the value cache. The proportion of salient tokens will denoted by "Saliency Ratio" in the experimental results. During the decoding process, ZipCache adopts streaming strategy [21] and repeats the compression process for the KV cache whenever 100 new tokens are generated.
5.2 Comparison with SOTA methods
5.2.1 Evaluation on GSM8k
We begin our evaluation on GSM8K dataset with chain-of-thoughts (CoT) prompting, and the results are presented in Table 3. This task requires LLM to solve mathematical problems and return the final answer without multiple options. This task poses considerable challenges and previous KV cache compression methods manifest notable declines in accuracy. For instance, KIVI [32] shows an accuracy drop of 7.89% on LLaMA3-88 model, indicating the suboptimality of preserving recent tokens in full precision instead of identifying salient ones. Moreover, there a substantial decrease in accuracy, amounting to 20.4%, for MiKV [43] under the high compression ratio. This suggests that accumulated attention scores mistakenly identify salient tokens, resulting the loss of vital information during compression. By contrast, the proposed normalized attention scores accurately measure token saliency, leading to a substantial enhancement in accuracy by 18.27% for LLaMA3-8B models in comparison to MiKV. In comparison to GEAR [21], which quantizes the entire KV cache to 4-b our approach additionally quantizes 40% tokens to 2-bit with enhanced performance on Mistral-7B model. This underscores the superiority of accurate adaptive compression of KV cache.
Table 3:
Performance comparisons on GSM8k with CoT prompts. Here, "H/L" denotes the bit-width for salient tokens (high-preet prees-preiiionn and regular tokens (low- www-precisioo), respectively. The compression ratio is calculated with an average input length of = 840.
Bit-width (H/L.
Model
Method
Saliency Ratio
Compression Ratio
Acc. (%)
FP16
16/16
100%
1x
41.62
H2O [46]
40.0%
2.50
1,67
GEAR [21]
4/4
100%
3.00
39.42
Mistral-7B
KIVI [32]
16/2
15.2%
3,46x
39.04
MiKV [43]
4/2
60.0%
4.98 x
36.32
ZipCache
4/2
60.0%
4.98
41.24
FP16
16/16
100%
1x
H20 [46]
16/0
40.0%
2.50x
13.50
GEAR [21]
4/4
100%
3.00x
12.96
LLaMA2-7B
KTVI [32]
16/2
15.2%
3.46x
13.19
MiKV [43]
4/2
60.0%
4.98
9.02
ZipCache
4/2
60.0%
4.98x
13.50
HP16
16/16
100%
1x
28.05
H2O [46]
16/0
40.0%
2.50x
26.00
GEAR [21]
4/4
100%
3.00x
25.40
LLaMA2-13B
KIVI [32]
16/2
15.2%
3.46x
27.29
MiKV [43]
4/2
60.0%
4.98x
23.65
ZipCache
60.0%
4.98
27.85
FP16
16/16
100%
1x
55.88
H2O [46]
16/0
40.0%
2.50x
27.82
GEAR [21]
4/4
100%
3.00x
49.43
LLaMA3-SB
KTVT [32]
16/2
15.2%
3.46x
47.99
MiKV [43]
70.0%
4.69x
35.48
ZipCache
4/2
70.0%
4.69x
53.75
5.3 Evaluation on HumanEval
In this subsection, we assess the performance of code generation across various KV cache compression methods, as summarized in Table Remarkably, ZipCache attains compression ratio of 4.94x without sacrificing performance when tested with the Mistral-7B model, outperforming predecessor methods. Moreover, when evaluating on L.aMM3-8B model, our approach outperforms KIVI-2 [32] by 7.32% with significantly higher compression ratio x vs. 2.55x), It should be noted that the

---

average input length for this task is only 119, while KIVI retains the recent 32 tokens in full- cill--reiision. thereby considerably diminishing its overall compression ratio. This underscores the advantage of ZipCache over methods that consistently retain information of recent tokens.
Table 4: Performance comparisons on HvmanEval for code generation. Here, "H/L" denotes the bit-width for salient tokens denotes the tokens are evicted. The compression ratio is calculated with an average input length of
and regular tokens (low-precision), respectively. 0-bit
1 1= 120.
Bit-width
Model
Method
Saliency Ratio
Compression Ratio
Acc.(%)
(ILL)
FP16
16/16
100%
1x
29.27
H2O [46]
16/0
40.0%
2.50
14.63
GEAR [21]
100%
3.00
28.05
Mistral-7B
KIVI [32]
16/2
26.7%
2.55x
28.05
MiKV [43]
4/2
60.0%
4.94x
27.44
ZipCache
4/2
60.0%
4.94
29.27
FPT6
16/16
100%
1x
14.02
H2O [46]
16/0
40.0%
2.50x
11.59
GEAR [21]
4/4
100%
3.00x
13.02
LLMMA2-7B
KIVI [32]
16/2
26.7%
2.55x
11.59
MiKV [43]
4/2
80.0%
4.39x
10.37
ZipCache
4/2
80.0%
4.39
12.80
FPI6
16/16
100%
Tx
33.54
H2O [46]
16/0
40.0%
2.50x
15.85
GEAR [21]
4/4
100%
3.00x
28.66
LLaMA3-8B
KIVI [32]
16/2
26.7%
2.55x
25.61
MiKV [43]
4/2
80.0%
4.39x
29.88
ZipCache
4/2
80.0%
4.39x
32.93
5.3.1 Evaluation Line Retrival
We further evaluate the data retrieval performance of various KV cache compression methods on Line Retrieval [25] dataset, where LLMs required to retrieve specific content from record of lines using a corresponding line index. The accuracy results under various number of lines depicted in Figure 5. Notably, all nuantization-basee compression methods exhibit superior performance compared to the eviction- based approach H2O [46]. For eviction-based methods, information is permanently discarded upon eviction. whereas quantization introduces only minor errors while preserving the integrity of the data. Additionally, in comparison to KIVI [32], which always maintains recent caches at full precision, accuracy. This can be attributed to the nature of retrieval tasks, where salient tokens may appear any position within the context, rather than being confined to the most recent caches. Moreover, when compared to MiKV [43], which employs accumulated attention scores as a saliency metric, our method yields remarkable 42% accuracy improvement when evaluated using 200 lines on the Mistral-7b model. This substantial enhancement once more highlights the effectiveness of normalized attention scores identifying salient tokens.
approach consistently achieves better retrieval
Additional experimental results on HumanEval [4] can be found in the supplementary material.
LLaMA2-138
LLaMA3-BB
Mistral-78
100
100
100
Full Cache
ZipCache
Cache
Accuracy (%)
Accuracy
60
60
ZipCache
60
ZipCarhe
H20
KIV1-2
KM1-2
MiKV
20
20
20.
100
150
200
200
100
Figure 5: Performance comparisons of various KV cache compression methods on Line Retrieval.

---

5.4 Generation Efficiency
In this subsection, we compare the latency and memory consumption of ZipCache and MiKV [43] under various input lengths, as depicted in Figure 6. Data is collected by serving LLaMA3-8B model on a Nvidía A100 GPU. MiKV employs accumulated attention scores to estimate token saliency, necessitating the use of standard attention for both prefill and decoding phases. Conversely, through an efficient approximate saliency metric, ZipCache requires only the calculation of the attention matrix for 10% of the tokens, while the remaining 90% tokens can be computed using either FlashAttention [7] or FlashDecoding [9]. Consequently, ZipCache achieves faster inference speed and lower memory usage, boasting 37. .3% reduction in prefill-phase latency, 56.9% reduction in decoding phase latency, and a 19.8% reduction in GPU memory usage when the input length scales to 4096.
Decoding phase latency (ms)
Prefill phase latancy ms))
MIKV
MIKV
Max GPU Memory (MB)
50000
MIKV
ZipCache
ZipCache
ZipCache
40000
-.0%%
4000
30000
3000
6.00%
2000
SU.1%
20000
3000
-14.76
30000
2040
1024
20400
0096
Input length (batch size=8)
Input length (batch size=8)
Input length (batch size=8)
Prefill phase latency
Decoding phase latency
(c) GPU memory
Figure 6: Comparisons of prefill-phase, decoding-phase latency and memory consumption between MiKV and ZipCache.
Conclusion and Future Work
In this paper, have proposed ZipCache, accurate and efficient mixed-precision quantization framework for compressing KV cache. To commence, we introduce a channel-separable quantization scheme for KV cache, effectively reducing the overhead of storing quantization parameters compared to traditional fine-grained quantization schemes without performance degradation. Additionally, present novel metric for accurately assessing token saliency based normalized attention scores. This metric enables adaptive quantization of all tokens according to their saliency, leading improved compression ratios without sacrificing model performance. Moreover, introduce an efficient approximation method for the token saliency metric, seamlessly integrating with fast attention implementations such as FlashAttention and FlashDecoding. This enhancement signifi- cantly boosts generation speed and reduces GPU memory requirements. Our extensive experiments have demonstrated that ZipCache achieves state-of-the-ar compression performance in terms of compression ratio, accuracy and generation speed. We believe that ZipCache will pave the way for more practical and scalable deployment of LLMs in various real-world applications.
Limitations and Broader Impacts. While ZipCache presents promising advancements in KV cache mixxe-quantization frameworks for LLMs, the saliency ratio manually specified before evaluation and cannot be automatically adjusted based on task datasets. Moreover, similar to other generative models, ZipCache can potentially be used generate malicious content.
Acknowledgemmnnt This work was supported by National Key Research and Development Program of China (2022YFC3602601)
References
[1] T. Brown, B. Mann, Ryder, Subbiah, Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-199 1901, 2020.
[2] A. Chauhan, U. Tiwari, et al. Post training mixed precision quantization of neural networks using first- order information. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1343-1352, 2023.
10

---

[3] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph.
G. Brockman, et al. Evaluating large language models trained code. arXiv preprini arXiv:2107.03374. 2021.
[4] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph,
G. Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:207..03374
2021.
[5] W.-L. Chiang, 7. Li, 7. Lin, Y. Sheng, Z. Wu, H. Zhang, I.. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, march 2023. URI. https://lmsss. org/hlog/2033-0339000 -iccvvu 3(5), 2023.
[6] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton,
R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
[7] T. Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. 2023.
[8] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. FlashAttention: Fast and memory-efficiem exact attention with 10-awareness. In Advances in Neural Information Processing Systems, 2022.
[9] T. Dao, Haziza, F. Massa, and G. Sizov. Flash-decoding for long-context inference, 2023.
[10] J. C. de Winter. Can chatgpt pass high school exams english language comprehension? International Journal of Artificial Intelligence in Education, pages 1-16, 2023.
[11] T. Dettmers, M. Lewis. Y. Belkada, and L. Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers scale. Advances in Neural Information Processing Systems, 35:30318 30332. 2022.
[12] Z. Dong, 7. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer. Hawq: Hessian aware quantization of neural networks with muxxddpprecisio Computer Vision, pages 293-302, 2019.
In Proceedings of the IEEE/CVF International Conference on
[13] M. Du, F. He, N. Zou, D. Tao, and X. Hu. Shortcut learning of large language models in natural language understanding. Communications of the ACM, 67(1): 11110--10,,00 2023.
[14] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.1732. 2022.
[15] I.. Gao, J. Tow, B. Abhasi, S. Biderman, S. Black, A, DiPofi, C. Foster, I., Golding, Hsu, A. Le Noac'h h,, H. Li. K. McDonell, N. Muennighoff, C. Ociepa, J. Phang. L. Reynolds, H. Schoelkopf. A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation, 12 2023.
[16] S. Ge, Y. Zhang, L. Liu, M. Zhang, J. Han, and J. Gao. Model tells you what to discard: Adaptive kv cache compression for llms. In The Twelfth International Conference on Learning Representations, 2024.
[17] Y. He, L. Liu, J. Liu, W. Wu, H. Zhou, and B. Zhuang. Ptqd: Accurate post post-training quantization for diffusion models. Advances in Neural Information Processing Systems, 36, 2023.
[18] L. Hou, R. Pang. T. Zhou, Y. Wu, X. Song, X. Song, and Zhou. Token dropping for efficient bert pretraining. arXiv preprint arXiv:2203.13244 2022.
[19] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint
10Xiv:1704.046// 2017.
[20] A. Jiang A. Sablayrolles, A. Mensch, C. Bamford, Chaplot, D. d. Casas, F. Bressand, Lengyel. G. Lample. L. Saulnier, et al. Mistral 7b. arXiv preprint 2XXi:2331000825. 2023.
[21] H. Kang, Q. Zhang, S. Kundu, G. Jeong, Z. Liu, T. Krishna, and T. Zhao. Gear: An efficient kv cache compression recipefor near-lossless generative inference Ilm. arXiv preprint urXiv:2403.05522 2024.
[22] Y. J. Kim, R. Henry, R. Fahim, and H. H. Awadalla. Finequant: Unlocking efficiency with fine-grained weight-only quantization for llms. arXiv preprint 20X.::300800722 2023.
[23] C. Li, W. Wang, J. Hu, Y. Wei, N. Zheng, H. Hu, Z. Zhang, and H. Peng. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.0 0400004466. 2024.
[24] D. Li, R. Shao, A. Xie, Y. Sheng, I.. Zheng, J. Gonzalez, I. Stoica, X. Ma,, and H. Zhang. How long can open-source Ilms truly promise on context length?, June 2023.
[25] D. Li, R. Shao, A. Xie, Y. Sheng, L. Zheng, J. Gonzalez, I. Stoica, X. Ma, and H. Zhang. How long can context length of open-source uims truly promise? In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
[26] Y. Li, R. Gong, X. Tan, Y. Yang. P. Hu, Q. Zhang, F. Yu, W. Wang, and S. Gu. Breeq: Pushing the limit of post-training training quantization by block reconstruction. In International Conference on Learning Representations, 2020.
11

---

[27] J. Lin. J. Tang, H. Tang, S. Yang, X. Dang, and S. Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.0097 2023.
[28] J. Liu, R. Gong, X. Wei, Z. Dong, J. Cai, and B. Zhuang. Qllm: Accurate and efficient low-bitwidth quanti- zation for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[29] J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024.
[30] Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyrillidis, and A. Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for Ilm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024.
[31] Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Mehdad, Y. Shi, R. Krishnamoorthi,, and V. Chandra. LJm- qat: Data-free quantization aware training for large language models. arXiv preprint arXiv 2023.
[32] Z. Liu, J. Yuan, H. Jin, S. Zhong. Xu, Braverman, B. Chen, and X. Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024.
[33] H. Luo. Q. Sun, C. Xu. P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308. 09583, 2023.
[34] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, al. Improving language understanding by generative pre-training. 2018.
[35] M. Tan, L. Wang. L. Jiang, and J. Jiang. Investigating math word problems using pretrained multilingual language models. arXiv preprint arXiv:2105.066 08928, 2021.
[36] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, F. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[37] H. Touvron, L.. Martin, Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
[38] A. Vaswani. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[39] K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han. Haq: Hardware- automated quantization with mixed precision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2019.
[40] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099. PMLR, 2023.
[41] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2023.
[42] F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoom. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Sympostum on Machine Programming. pages 1-10, 2022.
[43] J. Y. Yang, B. Kim, J. Bac, B. Kwon, G. Park, F. Yang, S. J. Kwon, and D. Lee. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402.180% 2024.
[44] Z. Yao, Z. Dong, Z. Zheng, A. Gholami, J. Yu, E. Tan, L. Wang, Q. Huang, Y. Wang, M. Mahoney, et al. v3: Dyadic neural network quantization. In International Conference on Machine Learning, pages 11875 11886. PMLR, 2021.
[45] Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168-277 111668228883 27122,1122228 2022.
[46] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, et al. H2o: Hcavy-hitter hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2023.
12

---

Appendix
A Calculation of Overhead for Different Quantization Schemes
Assuming 8, hd | = 4096, and that the KV cache is quantized to 4-bit, we proceed to calculate the actual compression ratio for different quantization granularities. For groupwise quantization with a group size of n 32, the compression ratio Rgroup is given by:
2 bhld 16
R Ryroup
3.200
(A)
2 bhld 4 46hld 16
For tokenwise quantization, the compression ratio Rtoken be calculated as:
2 x bhld x 16
Rioken 2 x bhld + 4 bl x 16
33992
(B)
For our proposed quantization baseline, the compression ratio Reaseline is determined by:
2 x bhld x 16
Reaseline bhld x x hd 16 + 2 6z 16
=3.995
(C)
B Implementation Details of ZipCache
In this section, we provide an overview of the channel-separable tokenwise quantization scheme in Algorithm 1. Additionally,
present the process of ZipCache's prefill phase as described in Algorithm 2, as well as its decoding phase detailed Algorithm It worth mentioning that during both the prefill and decoding phases, rather than calculating attention outputs separately for probe tokens and regular tokens followed by merging, FlashAttention [7] is utilized to compute the attention output for all tokens simultaneously. Additionally, attention scores of probe tokens calculated. By bypassing the substantial memory accesses associated with matrix splitting and merging, this strategy enhances generation speed.
Algorithm 1: Channel-separable Tokenwise Quantization (CSTQuant)
procedure CSTQuant:
Input: data X E RRixdd target bit-width
for i +- 0 to hdd do
C
(X,x
X
// Normalizing each channel of X
X TookenQuan(X,k) // Do tokenwise quantization
for i +- 0 to hd do
X, X, x c // Rescale each channel of X
return X
C Additional Experimental Results
C.1 Effeet of Saliency Metric
In this section, we conduct ablation studies on our saliency metric, shown in Table A. It demon- strates the superiority of our saliency metric over using accumulated attention scores or consistently prioritizing the latest tokens.
C.2 Accuracy and Efficiency Comparisons of various KV cache compression methods
In this section, we present the accuracy and efficiency comparisons of various KV cache compression methods, presented in Table B. Data is collected by evaluating LLaMA3-8B model on 200-line
13

---

Algorithm 2: ZipCache for Prefill Phase
procedure 2ipCachePrefill:
Input:
Query states Q, key states K, value states V, saliency ratio r%, bit-width for salient tokens k,, bit-width for regular tokens ky
// Salient Token Identification
Select probe tokens and compute their attention scores Aproe by Eq.
Measure the token saliency p with Aprobe by Eq. 8
// Computing Attenti inttioonn Output with FlashAttention
O FlashAttention Q K. V)
// Compressing KV Cache
Partition key states: Kaalient Kregutar Split(K,, p. r%)
Partition value states: V salient, V Vregular Split V P. r%)
K salient ChannelQuant( Kaliintt; kn), V salient CSTQuant( V salients kR)
regular = Channel (hhaneelQuantii Kregular, k1), regulaar = CSTQuant regular, ki)
K Concat( K salient; K Kregutar)
V Concat( satients V regular)
// Return Attention Output and Compressed KV Cache
return O, (K, V)
Algorithm 3: ZipCache for Decoding Phase
procedure ZipCacheDecoding:
Input: Query vector q. key vector k, value vector v, KV cache (K, V), saliency ratio r%, bit-widthh for salient tokens kn, bit-width for regular tokens ly, decoding token index
i, probe attention score Aprobe
K Concat(k, K) // Concatenate key cache
V Concat(v, V) // Concatenate value cache
O = FlashAttention(q. K, V) // Compute attention output
i
if == 100 then
// Re-compress every 100 tokens
Extract -[[]-]00] and V[: -100] and adaptively compress them with Ayrobe
Reset i 0, A probe None
else if 95 or randin/(O, 100) < then
// probe tokens consists of 5% recent and 5% random tokens. Compute attention scores of current token by Eq. 4
Aprobe = Concat(a, Aprobe)
// Return Attention Output, KV Cache and Attention Scores from Probe Tokens
return O, (K, V), Aprobe
retrieval task with a Nvidia A 100 GPU. We use a batch size of 8 and an average input length of 3072. To ensure fair comparison, implement GEAR and KIVI with FlashAttention integration.
Notably, the latency of ZipCache is lower than that of GEAR, which can be attributed to our efficient quantization scheme, whereas GEAR has high overhead duc outlier extraction. Compared to KIVI, ZipCache's 's latency is slightly higher (2584.01 ms vs. 2482.26 ms), but ZipCache achieves a higher compression ratio and better performance. This difference is due KIVI's fixed compression strategy, while we adaptively compress the KV cache based on the saliency. In comparison to MiKV [43], which identifies salient tokens through accumulated attention scores, method achieves notable 10.0% accuracy improvement by accurately pinpointing salient tokens and a substantial 38.0% decrease in prefill latency by integrating FlashAttention [7].
Moreover, to demonstrate the efficacy of proposed efficient quantization scheme, we also imple- ment ZipCache with groupwise quantization. The results show that using groupwise quantization
14

---

Table A: The effect of various saliency metric GSM8k with CoT prompts. Here, "H/L" denotes the bit-width for salient tokens (high-precision) and regular tokens (low-prees now-precision), respectively. "Locality" means the recent tokens identified as salient tokens. The compression ratio is calculated with an average input length of 840.
Bit-width (H/L)
Model
Metric
Saliency Ratio
Compression Ratio
Acc.(%b
FP16
16/16
100%
1x
41.62
Locality
4/2
60.0%
4.98
25.40
Mistral-7B
Accumulated Attention Scores
4/2
60.0%
4.98
38.20
Normalized Attention Scores
60.0%
4.98
41.24
increases inference speed (2664.05 ms vs. 2584.01 ms) and reduces the compression rate (3.81x vs. 4.43x) ) due to massive quantization overhead.
Table Accuracy and efficiency comparisons over LLaMA3-8B on the 200-line retrieval task. Here, "H/L" denotes the bit-width for salient tokens -high-precision) and regular tokens (lowprrccision). respectively. 0-bit denotes the tokens are evicted. Saliency ratio denotes the proportion of salient. tokens. The compression ratio is calculated with an average input length of = 3072.
Bit-width
Prefill-phase
Method
Saliency Ratio
Compression Ratio Acc.(%)
(H/L.)
Latency (ms)
FP16
16/16
100%
Tx
100
2340.11
H2O
16/0
40.0%
2.50x
4335.01
GEAR
4/4
100%
3.00x
2968.47
KIVI
16/2
8.33%
4,36x
96
2482.26
MiKV
4/2
80.0%
4.43x
90
4170.61
ZipCache
4/2
80.0%
4,43x
100
2584.01
ZipCache (Groupwise Quantization)
4/2
80.0%
3.81x
100
2664.05
C.3 Evaluation on LongBench
In this subsection, we evaluate the performance of ZipCache on LongBench using the longchat-7b- v1.5-32k model, shown Table The results show that ZipCache outperforms the previous state-of-the-ae method, KIVI on long context scenario.
Table C: Performance comparisons on LongBench.
Model
Method
Qasper
QMSum
MukiNNws
TREC
TriviaQA
SAMSum
LCC
RepaBench-P
FP16
21.92
21.01
26.09
64.00
83.51
41.5
58.39
52.26
-lamm-2-7b-chat
KIVI-2
14.31
20.76
25.75
64.00
83.38
39.14
56.17
50.12
ZipCache
20.93
20.69
26.12
64.50
83.38
40.11
56.60
52.00
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The authors have discussed the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoritical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The authors have disclosed all the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
The code is open sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
oniiiiinpccc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececceccccccct /ttpa///giiieeeeeeeuuubiiiiiii ssisnnoillllo onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The authors have included experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because of limited computing resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The authors have provided information on computing resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttp///eers//////ee//eee//e/eeee cc
Answer: [Yes]
Justification:
The authos confirm that the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The authors have discussed broader impacts of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper has no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The authors have cited the related papers.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
20

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "OCFFmgUZpE", "reviewer": "Reviewer_ozSB", "summary": "The paper introduces a channel-separable quantization scheme that decouples the quantization along channel and token dimensions. This method significantly reduces the quantization overhead without compromising performance. To accurately recognize salient tokens, the paper introduces a new token saliency metric based on normalized attention scores, which alleviates the bias towards earlier tokens that accumulate more values. The authors have demonstrated on different LLM models with different tasks.", "questions": "refer to weakness.", "limitations": "The system benefits are not clear or thorough for the reviewer to clearly appreciate. Also more fair system eval is needed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The idea of channel separable token saliency quantization seems to work better than standard groupwise quantization.\n\nThe results are strong and beats the SoTA!\n\nThe demonstration with flashattention integration is useful", "weaknesses": "The paper needs to be proof read by a native English speaker, to improve its readability.\n\nThe contribution is incremental, and the idea of group quantization (different for different K and V) are not new as already noted by the authors.\n\nThe idea of speeding up through flashattention and flashdecoding while the baselines not leveraging that is a bit of unfair comparison. Both GEAR and KIVI can exploit similar speed up benefit, thus Fig 1 comparison and later in table speed comparison is not fair.\n\nThe details of system level implementation is missing, it will be good to help the reviewer provide how the speed up can be demonstrated on a single GPU inference system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: The writing can be more native.**\n\nThanks for your valuable comment. We will carefully proofread the paper in the final version.\n\n**Q2: The idea of group quantization (different for different K and V) is not new.**\n\nThere might be some misunderstandings regarding groupwise quantization. Groupwise quantization is a scheme built upon tokenwise quantization, further processing outlier channels in distinct groups, as illustrated in Figure 2(c) of the paper. Additionally, applying different quantization schemes for K and V respectively (channelwise quantization for K and channel-separable tokenwise quantization for V) is not the key contribution of our paper. In terms of the quantization scheme, our main contribution lies in the introduction of channel-separable quantization to the KV cache compression, significantly reducing the overhead of quantization parameters without sacrificing performance, as detailed in General Response Q1.\n\n**Q3: The contribution is incremental.**\n\nPlease refer to General Response Q1 for the detailed contribution of our channel-separable quantization scheme and to General Response Q2 for an elaboration on the overall contribution of our paper. \n\n**Q4: The speed comparison is not fair.**\n\nThank you for your valuable comments. Firstly, we highlight that we are the pioneering work that enables adaptive KV cache compression to be compatible with FlashAttention, greatly enhancing the generation speed compared to previous approaches like H2O [i] and MiKV [ii]. Moreover, the reported speed results of GEAR [iii] and KIVI [iv] were obtained with their official implementations, which did not integrate FlashAttention at the time of our paper submission. To ensure a fair comparison, we implement GEAR and KIVI with FlashAttention integration, and the results are shown in Table A in the rebuttal PDF. Notably, the latency of ZipCache is lower than that of GEAR, which can be attributed to our efficient quantization scheme, whereas GEAR has a high overhead due to outlier extraction. Compared to KIVI, ZipCache's latency is slightly higher (2584.01 ms vs. 2482.26 ms), but ZipCache achieves a higher compression ratio and better performance. This difference is due to KIVI's fixed compression strategy, while we adaptively compress the KV cache based on the saliency. These results will be revised in the final version.\n\n\n**Q5: The details of system level implementation are missing.**\n\nThe detailed processes of ZipCache for both prefill and decoding phases are summarized in Algorithms 2 and 3 in the Appendix of our paper. Currently, we implement ZipCache based on the Hugging Face Transformers library, with specific modifications to the KV cache module. Our method is also orthogonal to other system level frameworks such as vLLM [v]. It should be noted that we utilize FlashAttention to maximize computational efficiency for both the prefill and decoding phases, eliminating the need to customize additional matrix multiplication kernels. We will release the source code upon acceptance.\n\n[i] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.\n\n[ii] No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization.\n\n[iii] GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM.\n\n[iv] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache.\n\n[v] Efficient Memory Management for Large Language Model Serving with PagedAttention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nW0RIrQKbc", "reviewer": "Reviewer_Nc8R", "summary": "This paper proposes a post training quantization framework named ZipCache for quantizing the key and value cache of LLMs. The authors introduce a channel-separable token-wise quantization scheme, which consumes less memory than the group quantization in terms of the quantization parameters. They also select saliency tokens based on a normalized attention matrix to avoid the accumulated bias. Additionally, the authors introduce an approximation method to apply it to FlashAttention.", "questions": "- The authors have compared the channel-separable quantization with many dynamic methods, but lacks comparison of accuracy with static methods, e.g., WKVQuant [1]. It is better to further explore whether the $c_i$ in equation (6) can outperform well-trained parameters, since they consumes the same memory space.\n- How does the calculation of $c_i$ in equation (6) effect the inference latency? It is better to compare the channel-separable quantization with group quantization and static methods in terms of the inference latency.\n\n[1]. WKVQuant: Quantizing weight and key/value cache for large language models gains more.", "limitations": "This paper contains the describtion of limitations. For suggestions, please refer to the Weakness and Questions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Overall, the writing of the paper is clear and easy to follow. Especially, the figures are drawn well.\n- By proposing the ZipCache, this work promotes the PTQ quality of LLMs on three chanllenging datasets.\n- The methods are well-integrated into popular framwork like FlashAttention.", "weaknesses": "- The idea of channel-separable quantization is not novel. It is a common practice to smooth the outliers of activations in the channel dimension, e.g., SmoothQuant [1], OmniQuant [2].\n- As shown in Table 3 (a), the values on the diagonal of the matrix are relatively large, while the values at other positions are relatively small. Based on this circumstance,  if the mean operation is performed for none-zero positions, earlier tokens may have smaller $p_i$ for they have more tiny values counted. Therefore, it may cause a prefer to the latest tokens. I think this is a issue for the proposed technique in Section 4.2, which needs more analysis.\n- Ablation studies are not provided. \n- As stated in Line 268 in the paper, new tokens remain un-quantized unless they reach 100. As far as I know, keeping part of tokens in full precision can improve the accuracy [3, 4]. Therefore, it is better to distinguish the contributions of this implementation and the proposed techniques. \n- I think the value of 'H/L' should be carefully considered. This is because both KIVI and ZipCache keep new tokens in FP16 at first and then perform quantization. Since the 'H' value of KIVI is 16, why it becomes 4 for ZipCache?\n- The equation (5) is not accurate for not subtracting the zero-point 'z' in the de-quantization.\n\n[1]. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.\n\n[2]. Omniquant: Omnidirectionally calibrated quantization for large language models.\n\n[3]. KIVI: ATuning-Free Asymmetric 2bit Quantization for KV Cache.\n\n[4]. SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: The idea of channel-separable quantization is not novel.**\n\nPlease refer to General Response Q1.\n\n**Q2: Performing the mean operation may cause a prefer to the latest tokens.**\n\nIndeed, there might be a prefer to the latest tokens since the values on the diagonal of the matrix are relatively large. However, the overall token saliency is adaptively determined in our method. As shown in Figure A in the rebuttal PDF, we present the token saliency with a 100-line retrieval sample as input. Besides the latest tokens, the system prompts in the front and related context in the middle are also assigned high saliency by our method. Moreover, Table B in the rebuttal PDF shows that our method achieves superior performance compared to using accumulated attention scores or a fixed prefer to the latest tokens.\n\n**Q3: Ablation studies are not provided.**\n\nAs referred to Table 1 and Table 2 in the paper, we have conducted ablation experiments to demonstrate the effectiveness of channel-separable quantization scheme and the efficient approximation of the saliency metric, respectively. We further conduct ablation studies on our saliency metric, as shown in Table B in the rebuttal PDF. It demonstrates the superiority of our saliency metric over using accumulated attention scores or consistently prioritizing the latest tokens.\n\n**Q4: Remaining new tokens un-quantized can improve accuracy.**\n\nDuring the prefill phase, KIVI [i] maintains full-precision KV caches for recent tokens to ensure accuracy, while our approach **adaptively quantizes all KV caches**. During the decoding phase, we quantize all KV caches with mixed-precision every 100 tokens, while KIVI adopts a sliding window and always keeps KV caches for latest tokens (128 tokens by default) in full-precision. This implies that KIVI always retains more KV cache at full precision compared to our method. Moreover, the streaming strategy we adopted aligns with GEAR [ii] and **is aimed at enhancing decoding speed**. Therefore, the comparison with KIVI and GEAR fairly demonstrates the efficacy of our method.\n\n**Q5: The value of `H/L' should be carefully considered.**\n\nAs referred to Q4, during the prefill phase, we quantize all KV caches, with the bit-width for salient tokens set to 4. This approach contrasts with KIVI's method of maintaining full-precision (FP16) for the recent tokens.\n\n**Q6: The equation (5) is not accurate.**\n\nThanks for your valuable comment. Equation (5) has been revised as follows:\n\n$\\hat{\\mathbf{x}}=\\mathcal{Q}_U(\\mathbf{x},k)=(\\mathrm{clip}(\\lfloor \\frac{\\mathbf{x}}{s}\\rceil +z, 0, 2^{k}-1) - z) \\cdot s.$\n\n\n**Q7: Comparisons between channel-separable quantization and static methods like WKVQuant.**\n\nDue to different settings, ZipCache and WKVQuant [iii] are not directly comparable. WKVQuant quantizes model weights and KV cache together, optimizing parameters through cross-block reconstruction regularization and a gradient descent algorithm. In contrast, our method focuses solely on KV cache compression and is entirely training-free.\n\n**Q8: Compare the channel-separable quantization with group quantization and static methods in terms of the inference latency.** \n\nInference latency comparisons among channel-separable quantization, groupwise quantization, and static quantization methods are detailed in Table A of the rebuttal PDF. Using static quantization can slightly reduce latency (2556.03 ms vs. 2584.01 ms). Conversely, groupwise quantization increases inference speed (2664.05 ms vs. 2584.01 ms) and reduces the compression rate (3.81$\\times$ vs. 4.43$\\times$) due to massive quantization overhead.\n\n[i] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache.\n\n[ii] GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM.\n\n[iii] WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fOxlHgnYLh", "reviewer": "Reviewer_S3gk", "summary": "The paper introduces ZipCache, an adaptive KV cache compression method for LLMs by accurately identifying salient tokens. It first presents a channel-separable tokenwise quantization scheme that reduces the overhead of quantization parameters. Next, it proposes a metric for identifying salient tokens based on normalized attention scores. By efficiently approximating the saliency metric, the method integrates seamlessly with fast attention mechanisms like FlashAttention. The efficacy of ZipCache is demonstrated through extensive experiments, showing superior performance in terms of compression ratio, speed, and minimal accuracy loss compared to existing methods.", "questions": "See weaknesses", "limitations": "See weaknesses", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.The introduction of normalized attention scores as a metric for token saliency is significant and promising for adaptive KV cache compression. It can accurately identify salient tokens, marking a substantial improvement over prior methods.\n\n2.The approximated saliency metric can be integrated with FlashAttention, making it practical for real-world applications.\n\n3.The idea of the channel-separable tokenwise quantization approach is novel and well-motivated, reducing memory overhead compared to groupwise counterparts.\n\n4.The experimental results are promising, clearly demonstrating the efficacy of ZipCache in terms of compression ratio, speed, and minimal accuracy loss.\n\n5.The paper is clearly written and easy to follow.", "weaknesses": "1.The experiments should encompass a broader diversity of tasks to comprehensively assess the method's effectiveness, such as LongBench.\n\n2.The font size in Figures 5 and 6 is too small, making them difficult to read. It is recommended to increase the font size for better clarity and accessibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: Evaluate ZipCache over LongBench.**\n\nAs shown in Table C in the rebuttal PDF, we evaluate the performance of ZipCache on LongBench. The results show that ZipCache outperforms the previous state-of-the-art method, KIVI. Due to the limited time slot of the rebuttal, we will conduct experiments on more models and add the results to the revised version.\n\n**Q2: The font size in Figures 5 and 6 is too small.**\n\nThanks for your valuable comment. We will revise it in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0GHLJ20aKQ", "reviewer": "Reviewer_E63Z", "summary": "The paper presents an adaptive, mixed-precision quantization method for compressing KV cache in LLMs. It proposes a channel-separable tokenwise quantization scheme to establish a robust quantization baseline, reducing the memory overhead of quantization parameters. A saliency metric, based on normalized attention scores, is employed to identify salient tokens accurately. This approach preserves essential information while aggressively quantizing less salient data, allowing for higher compression ratios with minimal impact on model accuracy. The method can be integrated with FlashAttention with an efficient approximation of the proposed metric. Experimental results validate the superiority of ZipCache on speed and accuracy over existing methods.", "questions": "1. Have you considered using only the attention scores from the most recent token to determine token saliency? What impact might this have on the overall performance?\n2. What does saliency mean in Figure 3c? What is the probability of each token being selected as a salient token, as mentioned in the caption of Figure 3?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of preserving salient tokens while compressing less critical tokens is both intuitive and effective, with the proposed accurate saliency metric being crucial for achieving this.\n2. The paper presents a novel quantization scheme tailored for channel-wise outliers in LLMs, providing a notable reduction in memory overhead compared to group-wise quantization schemes.\n3. The paper demonstrates strong experimental results on GSM8k and HumanEval, convincingly showcasing the benefits of adaptive KV cache compression.\n4. The integration with FlashAttention greatly enhances overall generation speed.\n5. The paper is well-organized.", "weaknesses": "1. Evaluating ZipCache on other generation or comprehending tasks will make the findings more robust.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1:  Evaluating ZipCache on other generation or comprehending tasks.**\n\nAs shown in Table C in the rebuttal PDF, we evaluate the performance of ZipCache on LongBench. The results show that ZipCache outperforms the previous state-of-the-art method, KIVI. Due to the limited time slot of the rebuttal, we will conduct experiments on more models and add the results to the revised version.\n\n**Q2: Using only the most recent token to determine token saliency.** \n\nAs shown below, determining token saliency with only the most recent token leads to a 3.2% accuracy drop compared to ZipCache.\n\n Table: The effect of different saliency metric on GSM8k with CoT prompts. Here, \"H/L\" denotes the bit-width for salient tokens (high-precision) and regular tokens (low-precision), respectively. The compression ratio is calculated with an average input length of $l=840$.  \n  \n | Model       | Metric                     | Bit-width (H/L) | Saliency Ratio | Compression Ratio | Acc. (%) |  \n |-------------|----------------------------|-----------------|----------------|-------------------|----------|  \n | Mistral-7B  | FP16                       | 16/16           | 100%           | 1$\\times$         | 41.62    |  \n | Mistral-7B  | Last Token's Attention Scores | 4/2             | 60.0%          | 4.98$\\times$      | 38.04    |  \n | Mistral-7B  | Normalized Attention Scores | 4/2             | 60.0%          | 4.98$\\times$      | **41.24** |\n\n**Q3: What does saliency mean in Figure 3c?**\n\nThere are many attention layers and heads in a model. For each token, this value is derived by counting the number of times this token is selected as a salient token among all attention heads, then normalizing this count by the total number of attention heads.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable feedback. Overall, our work has been well recognized as
- "It is novel and well-motivated" (Reviewers S3gk and E63Z)
- "It is well-integrated into FlashAttention" (All Reviewers)
- "It is clear and easy to follow" (Reviewers Nc8R, S3gk, and E63Z)
- "It demonstrates strong performance" (Reviewers ozSB, S3gk, and E63Z).

We have summarized and addressed the main concerns as follows:

**Q1: The proposed channel-separable quantization is not novel.**  

We highlight a significant challenge in KV cache groupwise quantization: the number of quantization parameters grows linearly with the product of sequence length and hidden dimension, as referred to Table 1 in the paper, which greatly impacts the KV cache compression ratio. This crucial issue is overlooked in previous literature and motivates us to disentangle quantization along channel and token dimensions to reduce quantization overhead. Therefore, **our motivation is fundamentally different from model quantization such as SmoothQuant [i] and OmniQuant [ii]**, which focus on migrating quantization difficulties between activations/weights or query/key states before matrix multiplications. As referred to Table 1 in the paper, our scheme significantly reduces the overhead of quantization parameters and achieves superior performance compared to groupwise quantization.

**Q2: The contribution of our paper.** 

Our contributions are summarized as follows:
1) We propose a channel-separable quantization scheme to reduce the overhead of quantization parameters. It brings a higher compression ratio and achieves superior performance compared to groupwise quantization.
2) We introduce an accurate metric to identify salient tokens and adaptively quantize all KV caches based on their saliency, thereby improving the overall compression ratio.
3) ZipCache is the pioneering work that enables adaptive KV cache compression to be compatible with FlashAttention, significantly enhancing the generation speed and practicality. Overall, as referred to Table 3 in the paper, ZipCache achieves the **highest KV cache compression ratio** and the **highest accuracy** compared to predecessor methods, demonstrating the efficacy and contribution of our method.

**Q3: Additional experiments on LongBench.** 

As shown in Table C in the rebuttal PDF, we evaluate the performance of ZipCache on LongBench. The results show that ZipCache outperforms the previous state-of-the-art method, KIVI [iii]. Due to the limited time slot of the rebuttal, we will conduct experiments on more models and add the results to the revised version.

[i] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.

[ii] Omniquant: Omnidirectionally Calibrated Quantization for Large Language Models.

[iii] KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache.

Meta-review: This paper presents ZipCache for KV cache quantization with salient token identification. After the rebuttal, it receives one strong accept, one accept, and two borderline accept. Its merits, including clear motivation, good organization, interesting idea, and good results, are well recognized by the reviewers. The authors' response well resolves the concerns of unsatisfying writing, insufficient implementation details, and incorrect formulation. Based on the enclosed reviews, I think this work should be accepted. Please also incorporate these details in the revised manuscript.
Decision: Accept (poster)