Title: ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Model Compression

Large Language Models

Model Quantization

Efficient Models
Abstract: KV cache stores key and value states from previous tokens to avoid re-computation, yet it demands substantial storage space, especially for long sequences. 
  Adaptive KV cache compression seeks to discern the saliency of tokens, preserving vital information while aggressively compressing those of less importance.
  However, previous methods of this approach exhibit significant performance degradation at high compression ratios due to inaccuracies in identifying salient tokens. 
  Additionally, the compression process introduces excessive overhead, substantially increasing memory burdens and the generation latency.
  In this paper, we present ZipCache, an accurate and efficient KV cache quantization method for large language models (LLMs). 
  First, we construct a strong baseline for quantizing KV cache. Through the proposed channel-separable tokenwise quantization scheme, the memory overhead of quantization parameters are substantially reduced compared to fine-grained groupwise quantization.
  To enhance the compression ratio, we propose normalized attention score as an effective metric for identifying salient tokens by considering the lower triangle characteristics of the attention matrix. The quantization bit-width for each token is then adaptively assigned based on their saliency.
  Moreover, we develop an efficient approximation method that decouples the saliency metric from full attention scores, enabling compatibility with fast attention implementations like FlashAttention.
  Extensive experiments demonstrate that ZipCache achieves superior compression ratios, fast generation speed and minimal performance losses compared with previous KV cache compression methods. For instance, when evaluating Mistral-7B model on GSM8k dataset, ZipCache is capable of compressing the KV cache by $4.98\times$, with only a 0.38% drop in accuracy. In terms of efficiency, ZipCache also showcases a 37.3% reduction in prefill-phase latency, a 56.9% reduction in decoding-phase latency, and a 19.8% reduction in GPU memory usage when evaluating LLaMA3-8B model with a input length of 4096. Code is available at https://github.com/ThisisBillhe/ZipCache/.

ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification

Abstract

KVcache stores key and value states from previous tokens to avoid re-computation, yet it demands substantial storage space, especially for long sequences. Adaptive KV cache compression seeks to discern the saliency of tokens, preserving vital information while aggressively compressing those of less importance. However, previous methods of this approach exhibit significant performance degradation at high compression ratios due to inaccuracies in identifying salient tokens. Additionally, the compression process introduces excessive overhead, substantially increasing memory burdens and the generation latency. In this paper, we present ZipCache, an accurate and efficient KV cache quantization method for large language models (LLMs). First, we construct a strong baseline for quantizing KV cache. Through the proposed channel-separable tokenwise quantization scheme, the memory overhead of quantization parameters are substantially reduced compared to fine-grained groupwise quantization. To enhance the compression ratio, we propose normalized attention score as an effective metric for identifying salient tokens by considering the lower triangle characteristics of the attention matrix. The quantization bit-width for each token is then adaptively assigned based on their saliency. Moreover, we develop an efficient approximation method that decouples the saliency metric from full attention scores, enabling compatibility with fast attention implementations like FlashAttention. Extensive experiments demonstrate that ZipCache achieves superior compression ratios, fast generation speed and minimal performance losses compared with previous KV cache compression methods. For instance, when evaluating Mistral-7B model on GSM8k dataset, ZipCache is capable of compressing the KV cache by 4 . 98 × , with only a 0 . 38% drop in accuracy. In terms of efficiency, ZipCache also showcases a 37 . 3% reduction in prefill-phase latency, a 56 . 9% reduction in decoding-phase latency, and a 19 . 8% reduction in GPU memory usage when evaluating LLaMA3-8B model with a input length of 4096 . Code is available at https://github.com/ThisisBillhe/ZipCache/.

1 Introduction

LLMs with the next-token-prediction scheme have achieved remarkable advancements in various text-related tasks, such as language understanding [13, 34, 10], content creation [1, 5, 36], coding [3, 29, 42] and mathematics [33, 23, 35]. In this generation scheme, the forthcoming token interacts with all previous tokens via the attention mechanism [38], where the query, key and value states will be

calculated for each token. As the past tokens will not be altered, previously computed key and value states can be stored as KV cache to prevent re-computations, significantly improving the generation speed. However, as the batch size and the input context length grows, the stored KV cache emerges as a new memory bottleneck for LLMs. For example, when serving a 175 B-parameter LLM [1] with a batch size of 64 and a context length of 4096 , the KV cache can occupy 1 . 2 TB of memory space, while the model weights only require 350 GB. Meanwhile, the size of KV cache will continue to increase as decoding progresses. Therefore, the compression of KV cache is crucial for the efficient deployment of LLMs.

Recent compression methods for KV cache can be broadly categorized into two types. The first type of methods compresses the KV cache uniformly, without considering the significance of individual tokens. To preserve performance, these methods often rely on either high-precision quantization [21] or maintaining recent tokens in full-precision [32], which undoubtedly compromise the compression ratio. Additionally, if salient tokens are not among the most recent ones, such as in information retrieval tasks, it may result in degraded performance. The other type of methods [46, 43, 16] compress KV cache adaptively by identifying salient tokens and compresses them separately. This approach aligns with the observation that a minority of tokens contribute the majority of attention scores [41], potentially achieving higher compression ratios than non-adaptive methods. However, current adaptive KV cache compression methods [46, 43] use accumulated attention scores as a metric of token saliency, which is insufficient in two aspects. First, accumulated attention scores is inaccurate in identifying important tokens. Due to the presence of attention masks, the attention matrix is a lower triangular matrix. Earlier tokens tend to have larger softmax attention values and more attention scores to be accumulated, as illustrated in Figure 3. Under this metric, the saliency of the most recent tokens can never surpass that of the first token, thereby introducing a bias in determining token saliency. Additionally, to obtain accumulated attention scores, full attention matrices must be explicitly computed and stored, which can be inefficient for serving LLMs. Given an input context length of l , fast attention implementations such as FlashAttention [8, 7] only require O ( l ) memory by computing attention output in blocks without retaining complete attention matrices. By contrast, storing full attention matrices requires O ( l 2 ) memory, and the large number of memory accesses significantly slows down the inference speed, as depicted in Figure 4.

To address these challenges, we introduce ZipCache, an efficient KV cache compression method that attains exceptionally high compression ratios by accurate salient token identification. Figure 1 presents an overview of latency-accuracy comparisons among ZipCache and diverse KV cache compression methods. We start by designing an efficient quantization baseline for compressing the KV cache. To preserve performance, predecessor methods [32, 21] employ fine-grained groupwise quantization, which involves independent quantization for a small channel group within each token. However, this method necessitates storing extensive quantization parameters and results in significant memory overhead. By contrast, we introduce a channel-separable quantization scheme that decouples the quantization along channel and token dimensions. This method significantly reduces the quantization overhead without compromising performance. To accurately recognize salient tokens, we introduce a new token saliency metric based on normalized attention scores, which alleviates the bias towards earlier tokens that accumulate more values. All tokens, without exception, will be quantized to the target bit-width based on their estimated saliency, boosting the overall compression ratio. Moreover, to

88

91

94

97

100

ZipCache

MiKV

KIVI

GEAR

H2O

0

Latency (s)

Accuracy (%)

2

3

4

5

ease integration with fast attention implementations, we introduce an efficient approximation of the token saliency metric. This approximation only relies on computing and storing attention scores from a few number of tokens, which we refer to as probe tokens. An effective probe token selection strategy is then introduced to minimize performance loss. As a result, the majority of tokens can benefit from fast attention implementations, significantly enhancing the generation speed.

In summary, our contributions are as follows:

We establish an efficient channel-separable quantization scheme for KV cache, which significantly reduces the overhead of quantization parameters without compromising performance compared to fine-grained groupwise quantization approach.

We propose an accurate metric for assessing token saliency based on normalized attention scores. All tokens are adaptively quantized according to their assessed saliency, thereby improving the overall compression ratio.

We further develop an efficient approximation method for the token saliency metric that integrates seamlessly with fast attention implementations, enhancing generation speed.

By integrating these three techniques, we present ZipCache, an accurate and efficient framework for KV cache compression. Extensive experiments demonstrate that ZipCache reaches a new state-of-the-art performance for KV cache compression in terms of compression ratio, accuracy and generation efficiency.

2 Related Work

2.1 Model Quantization

Quantization is a prevalent technique for compressing deep neural networks by representing model weights and activations with lower numerical bit-widths. This technique can be categorized into two primary approaches based on the necessity of fine-tuning: post-training quantization (PTQ) [26, 17, 14] and quantization-aware training (QAT) [28, 31]. For large language models (LLMs), where fine-tuning can be data- and computation-intensive, PTQ is often the preferred method [40, 11, 45, 27]. In this paper, we also quantize KV cache in a post-training manner. For both approaches, quantization can be implemented at various levels of granularity, including channelwise, tokenwise, and groupwise approach. Typically, a finer quantization granularity involves the independent quantization of smaller parameter groups, which often results in improved performance albeit at the cost of more quantization parameters and increased memory overhead. In the context of LLMs, fine-grained quantization is frequently utilized due to the presence of outliers [22, 45]. However, for KV cache compression, this will greatly reduce the overall compression ratio.

Mixed precision quantization [39, 44, 12, 2] allocates varying bit-widths to distinct parts of a model or tensor, enabling a more compact compression. This approach originates from the observation that model components exhibit differing sensitivities to quantization. Consequently, components with low sensitivity can utilize reduced bit-widths without impairing performance. For LLMs, previous studies [46, 43, 30, 18] have shown significant disparities in the importance of tokens, indicating that heavy compression of non-critical tokens has minimal impact on overall performance. This insight highlights the applicability of mixed precision quantization for compressing the KV cache.

2.2 KV Cache Compression

While KV cache effectively prevents re-computation and significantly enhances generation speed, its memory footprint is notably substantial with long-context input. To alleviate this, many efforts have been made to reduce the KV cache size. Based on the compression method, these methods can be categorized into two groups: token dropping [46, 16, 30] and KV cache quantization [43, 21, 32]. The former identifies and drops unimportant tokens in the KV cache. For example, H2O [46] only maintain 20% heavy-hitted tokens and 20% recent tokens while evicting the rest. However, discarding tokens permanently erases their information, which proves to be suboptimal for tasks such as retrieval [43]. Conversely, the latter category employs quantization on the cached key and value states, and mixed precision quantization can further be applied once token importance is identified [43]. To tackle the outliers present in the KV cache, these methods extract the outlier as full precision [21] or use finer-grained quantization scheme [32], which increases the quantization overhead. In this study, we propose an efficient channel-separable quantization scheme with reduced quantization overhead and strong performance. Additionally, both categories of methods commonly adopt accumulated attention scores as the metric for token importance [46, 43]. However, we observe that this criterion is inaccurate and can result in significant performance deterioration at low bit-widths. In contrast, we achieve superior compression performance by utilizing a more accurate metric for identifying salient tokens.

3 Preliminary

3.1 Attention Block in LLMs

Given an input prompt, the generation process of LLMs can be broadly categorized into two distinct phases: the prefill phase, which computes and stores the KV cache for input tokens, and the decoding phase, where new tokens are generated through a next-token-prediction scheme. Given input data X and an attention block with its weight matrices W Q , W K and W V , the prefill phase can be formulated as:

Here, d k is the dimension of the key, and A refers to the attention scores. K and V will be stored as KV cache. For clarity, we have omitted the output projection.

For the decoding phase, given x as the embedding vector of the current token, the query q becomes a vector and the KV cache matrices will be updated as follow:

The attention output are then computed as follows:

To ensure clarity and consistency, we introduce notation to define the hyper-parameters used in the paper. Specifically, we denote the batch size as b , the number of attention heads as h , the sequence length as l , and the head dimension as d .

3.2 Model Quantization

Uniform quantization is adopted in our study and all experiments. Given a floating-point vector x , it can be uniformly quantized to k -bit as follows:

Here, ⌊·⌉ denotes the round operation, s = max( x ) -min( x ) 2 k -1 and z = -⌊ min( x ) s ⌉ are quantization parameters. It should be noted that the quantization parameters are stored in full-precision, which can lead to significant overhead if the quantization is fine-grained.

4 Method

4.1 A Strong Baseline for KV Cache Quantization

Tokenwise quantization, as depicted in Figure 2(b) is prevalent in quantizing large language models (LLMs) due to the distinct representations of individual tokens. However, it has been widely observed, as illustrated in Figure 2(a), that outliers emerge within the channel dimensions of key and value matrices [43, 32], posing challenges for tokenwise quantization. To address this, recent work [32] resorts to groupwise quantization, where outlier channels are processed in distinct groups, as illustrated in Figure 2(c). However, this fine-grained quantization approach introduces excessive memory overhead, thereby significantly impacting the compression ratio. For instance, considering X ∈ R b × h × l × d as the data to be quantized and a group size of n , tokenwise quantization only results in 2 bl quantization parameters, while groupwise quantization would yield 2 bhld n quantization parameters. Since these parameters are usually stored in full precision, this overhead would constitute a substantial portion of the storage cost for quantized data.

Motivated by depthwise separable convolution [19], we introduce an efficient channel-separable tokenwise quantization scheme, which disentangles the channel and token dimensions. As shown in

𝒔, 𝒛 ∈ 𝑅

𝑙

(b) Tokenwise

Quantization

(c) Groupwise

Quantization

(d) Channel-separable

Tokenwise Quantization

𝒄 ∈ 𝑅 ℎ𝑑

(a) Visualization of

Key and Value States

𝐗 ∈ 𝑅 𝑙∗ℎ𝑑

𝐗 ∈ 𝑅 𝑙∗ℎ𝑑

𝐗 ∈ 𝑅 𝑙∗ℎ𝑑

𝒔, 𝒛 ∈ 𝑅

𝑙∗ℎ𝑑/𝑛

𝒔, 𝒛 ∈ 𝑅

𝑙

Figure 2(d), our approach initiates by normalizing each channel of data X with a scaling factor c . For the i -th channel in X , the normalization process can be formulated as:

After normalization, each channel is scaled to a closed magnitude, mitigating the influence of outliers during tokenwise quantization. Subsequently, tokenwise quantization can be reliably applied and the scales c are multiplied back to restore the magnitude of each channel. The process of channel-separable tokenwise quantization is summarized in the supplementary material. Within this quantization scheme, the total number of quantization parameters amounts to hd +2 bl , representing a notable reduction compared to groupwise quantization, while effectively balancing the outlier channels and the representation of each token.

As referred to Figure 2(a), since the differences in token representations are small in key cache, we employ channelwise quantization for the key cache to further reduce overhead and employ channelseparable tokenwise quantization for the value cache. As depicted in Table 1, this configuration yields superior performance with reduced quantization overhead compared with groupwise quantization, thereby establishing a robust baseline for KV cache quantization.

4.2 Accurate Salient Token Identification

Adaptive KV cache compression [46, 43, 16] aims to discern the saliency of each token, keeping the information of salient tokens while evicting or aggressively compressing the rest, to achieve a higher compression ratio. These salient tokens, also referred to as "Heavy Hitters" [46], are often identified based on accumulated attention scores. Given attention score matrix A ∈ R l × l , the saliency of token i is estimated by:

So   there  are   five   eggs

1.00

0.00

0.00

0.00

0.00

0.33

0.67

0.00

0.00

0.00

0.35

0.30

0.35

0.00

0.00

0.04

0.06

0.14

0.76

0.00

0.02

0.02

0.04

0.06

0.86

egs

five   are   there   So

: 40% salient tokens

1.74

1.05

0.53

0.82

0.86

0.35

0.26

0.18

0.41

0.86

Question: There are 15 trees in the

grove…

Let's think step by step…

Question: If there are 3 cars…

Let's think step by step…

Question: Leah had 32 chocolates…

Let's think step by step…

…

Question: Olivia has $23…

Let's think step by step…

Question: Janet's ducks lay 16 eggs per

day…How much in dollars does she

make every day at the farmers' market?

(a)

(b)

(c)

𝒑𝒊

෥

𝒑𝒊

Tokens with large saliency values are then considered salient tokens. However, this approach has inherent limitations due to the lower triangular nature of the attention score matrix, as illustrated in Figure 3(a). There are two primary issues. Firstly , earlier tokens benefit from having more values accumulated since the elements above the diagonal are all zero. For instance, in a sequence of length l , the initial token accumulates l positive values, whereas the final token only accumulates one. Secondly , Softmax function converts real numbers into probabilities, so that the earlier rows of the attention matrix tending to have higher values, as fewer numbers are involved in the Softmax calculation. Consequently, the accumulated attention score of the final token will always be smaller than that of the first, which exceeds 1 . To address this, previous works, such as H2O [46], always maintain recent caches in full precision. Nevertheless, this solution is suboptimal since recent tokens are not necessarily the most significant ones.

To enhance the evaluation of each token's saliency, we introduce an accurate token saliency metric based on normalized attention scores ˜ p i :

Here, nnz ( A : ,i ) denotes the number of non-zero elements in the i -th column of A . As evidenced in Figure 3(a), normalizing the accumulated attention scores mitigates the influence of excessively large values in the initial rows of the attention score matrix, thereby delivering a more precise assessment. To validate the efficacy of our new metric, we input a sample from GSM8k dataset with chain-ofthoughts (CoT) prompting to the LLaMA3-8B model and identify saliency of each token by Eq. 7 and Eq. 8, respectively. As depicted in Figure 3(b) and (c), the salient tokens are at the end of the prompt, which correspond to the question for LLM to answer. However, these tokens are identified as low saliency by accumulated attention scores. Under the KV cache compression framework, these tokens would either be discarded or quantized to extremely low bit-width, resulting in a significant performance deterioration. In contrast, our method accurately identifies the salient tokens. Additional experimental results regarding the accuracy of our method will be detailed in Section 5.2.

4.3 Efficient Approximation of Saliency Metric

As analyzed in Section 4.2, adaptive KV cache compression requires the explicit computation of full attention scores, as referred to Figure 4(b), which clashes with fast attention implementations like FlashAttention [8, 7, 9]. As shown in Figure 4(c), FlashAttention computes attention outputs in tiles without storing the intermediate attention scores. To reconcile the efficiency of FlashAttention with the substantial compression offered by adaptive KV caching, we devise an effective approximation for Eq. 8 as a measure of token saliency. Specifically, we sample a small group of tokens, designated

𝐀

=

(b) Standard Attention

𝐀

(c) FlashAttention

𝑸

𝑽

𝑶

𝑽

𝐊 𝑻

𝑸

𝐊 𝑻

𝑶

Memory=

𝑶(𝒍 𝟐

)

More memory access & slower

Memory=

𝑶(𝒍)

Less memory access & faster

(a) Efficient Saliency Metric with Probe Tokens

Input tokens

𝐀𝑝𝑟𝑜𝑏𝑒

Probe tokens

(b) Standard Attention

(c) FlashAttention

Regular tokens

Output

: Intermediate attention scores

as probe tokens , and compute their attention scores A probe as follows:

By substituting A probe into Eq. 8, we can approximate the saliency of all tokens. For the remaining non-probe tokens, their attention scores do not have to be computed explicitly, enabling the integration of fast attention implementations to expedite the generation process, as illustrated in Figure 4(a).

However, the positions of the probe tokens will undoubtedly affects the accuracy of the approximated token saliency and the selection of probe tokens is under explored. In this study, we suggest four strategies for sampling probe tokens:

Random tokens . The probe tokens are randomly sampled from all positions.

Special tokens . The special tokens and punctuation tokens will be treated as probe tokens.

Recent tokens . The most recent tokens are selected as probe tokens.

Random+recent tokens . The probe tokens will be divided into two parts, one using recent tokens and the other randomly selecting from the remaining tokens.

Table 2: Performance comparisons of various probe strategies. Data is collected from LLaMA3-8B model on GSM8k dataset. We quantize 40% salient tokens to 4-bit and the remaining 60% tokens to 2-bit. The proportion of probe tokens is 10%.

It should be emphasized that our study diverges from prior research [16] in that, rather than directly choosing special or recent tokens as salient tokens, we opt to sample a subset of tokens as "probes" to detect the salient ones. As depicted in Table 2, we present a comprehensive comparison of the performance among four distinct sampling strategies. Among the four strategies examined, a hybrid approach that combines recent tokens with randomly selected tokens emerges as the most effective. Unless otherwise specified, this hybrid strategy with 5% recent tokens and 5% random tokens will be employed in our method.

5 Experiment

5.1 Implementation Details

Models and datasets. To validate the efficacy of our proposed method, we conduct experiments with three open-source LLMs: Mistral [20], LLaMA2 [37] and LLaMA3. These models are evaluated on three challenging benchmarks: GSM8k [6] for math problem solving, HumanEval [4] for code generation, and Line Retrieval [25] for data retrieval. To ensure reproducibility, the reported results are obtained using the Language Model Evaluation Harness [15] and LongEval [24].

Quantization and generation settings. We employ mixed precision quantization for KV cache where salient tokens will be quantized to 4-bit while the remaining will be quantized to 2-bit. For both subsets, we apply channelwise quantization for the key cache and channel-separable tokenwise quantization for the value cache. The proportion of salient tokens will be denoted by "Saliency Ratio" in the experimental results. During the decoding process, ZipCache adopts a streaming strategy [21] and repeats the compression process for the KV cache whenever 100 new tokens are generated.

5.2 Comparison with SOTA methods

5.2.1 Evaluation on GSM8k

We begin our evaluation on GSM8k dataset with chain-of-thoughts (CoT) prompting, and the results are presented in Table 3. This task requires LLM to solve mathematical problems and return the final answer without multiple options. This task poses considerable challenges and previous KV cache compression methods manifest notable declines in accuracy. For instance, KIVI [32] shows an accuracy drop of 7.89% on LLaMA3-8B model, indicating the suboptimality of preserving recent tokens in full precision instead of identifying salient ones. Moreover, there is a substantial decrease in accuracy, amounting to 20 . 4% , for MiKV [43] under the high compression ratio. This suggests that accumulated attention scores mistakenly identify salient tokens, resulting in the loss of vital information during compression. By contrast, the proposed normalized attention scores can accurately measure token saliency, leading to a substantial enhancement in accuracy by 18.27% for LLaMA3-8B models in comparison to MiKV. In comparison to GEAR [21], which quantizes the entire KV cache to 4-bit, our approach additionally quantizes 40% tokens to 2-bit with enhanced performance on Mistral-7B model. This underscores the superiority of accurate adaptive compression of KV cache.

5.3 Evaluation on HumanEval

In this subsection, we assess the performance of code generation across various KV cache compression methods, as summarized in Table 4. Remarkably, ZipCache attains a compression ratio of 4 . 94 × without sacrificing performance when tested with the Mistral-7B model, outperforming predecessor methods. Moreover, when evaluating on LLaMA3-8B model, our approach outperforms KIVI-2 [32] by 7 . 32% with a significantly higher compression ratio ( 4 . 39 × vs. 2 . 55 × ). It should be noted that the

average input length for this task is only 119 , while KIVI retains the recent 32 tokens in full-precision, thereby considerably diminishing its overall compression ratio. This underscores the advantage of ZipCache over methods that consistently retain information of recent tokens.

5.3.1 Evaluation on Line Retrival

We further evaluate the data retrieval performance of various KV cache compression methods on Line Retrieval [25] dataset, where LLMs are required to retrieve specific content from a record of lines using a corresponding line index. The accuracy results under various number of lines are depicted in Figure 5. Notably, all quantization-based compression methods exhibit superior performance compared to the eviction-based approach H2O [46]. For eviction-based methods, information is permanently discarded upon eviction, whereas quantization introduces only minor errors while preserving the integrity of the data. Additionally, in comparison to KIVI [32], which always maintains recent caches at full precision, our approach consistently achieves better retrieval accuracy. This can be attributed to the nature of retrieval tasks, where salient tokens may appear at any position within the context, rather than being confined to the most recent caches. Moreover, when compared to MiKV [43], which employs accumulated attention scores as a saliency metric, our method yields a remarkable 42% accuracy improvement when evaluated using 200 lines on the Mistral-7b model. This substantial enhancement once more highlights the effectiveness of normalized attention scores in identifying salient tokens.

Additional experimental results on HumanEval [4] can be found in the supplementary material.

50

100

150

200

Number of Lines

0

20

40

60

80

100

Accuracy (%)

LLaMA2-13B

Full Cache

ZipCache

KIVI-2

MiKV

H2O

100

200

300

400

Number of Lines

0

20

40

60

80

100

Accuracy (%)

LLaMA3-8B

Full Cache

ZipCache

KIVI-2

MiKV

H2O

50

100

150

200

Number of Lines

0

20

40

60

80

100

Accuracy (%)

Mistral-7B

Full Cache

ZipCache

KIVI-2

MiKV

H2O

5.4 Generation Efficiency

In this subsection, we compare the latency and memory consumption of ZipCache and MiKV [43] under various input lengths, as depicted in Figure 6. Data is collected by serving LLaMA3-8B model on a Nvidia A100 GPU. MiKV employs accumulated attention scores to estimate token saliency, necessitating the use of standard attention for both prefill and decoding phases. Conversely, through an efficient approximate saliency metric, ZipCache requires only the calculation of the attention matrix for 10% of the tokens, while the remaining 90% tokens can be computed using either FlashAttention [7] or FlashDecoding [9]. Consequently, ZipCache achieves faster inference speed and lower memory usage, boasting a 37 . 3% reduction in prefill-phase latency, a 56 . 9% reduction in decoding-phase latency, and a 19 . 8% reduction in GPU memory usage when the input length scales to 4096 .

(a) Prefill phase latency

(b) Decoding phase latency

(c) GPU memory

(ms)

100

(m

S

6000

MiKV

latency

MiKV

（MB）

50000

MiKV

latency

5000

ZipCache

80

ZipCache

MaxGPUMemory

40000

ZipCache

-19.8%

-4.05%

4000

-37.3%

phase

60

30000

-0.05%

Prefill phase

3000

-38.0%

-56.9%

-0.02%

40

-58.4%

20000

2000

-27.9%

Decoding

-42.4%

-58.1%

1000

-14.1%

20

10000

0

1024

2048

3072

4096

0

1024

2048

3072

4096

0

1024

2048

3072

4096

Inputlength (batchsize=8)

Inputlength(batchsize=8)

Inputlength(batchsize=8)

6 Conclusion and Future Work

In this paper, we have proposed ZipCache, an accurate and efficient mixed-precision quantization framework for compressing KV cache. To commence, we introduce a channel-separable quantization scheme for KV cache, effectively reducing the overhead of storing quantization parameters compared to traditional fine-grained quantization schemes without performance degradation. Additionally, we present a novel metric for accurately assessing token saliency based on normalized attention scores. This metric enables adaptive quantization of all tokens according to their saliency, leading to improved compression ratios without sacrificing model performance. Moreover, we introduce an efficient approximation method for the token saliency metric, seamlessly integrating with fast attention implementations such as FlashAttention and FlashDecoding. This enhancement significantly boosts generation speed and reduces GPU memory requirements. Our extensive experiments have demonstrated that ZipCache achieves state-of-the-art compression performance in terms of compression ratio, accuracy and generation speed. We believe that ZipCache will pave the way for more practical and scalable deployment of LLMs in various real-world applications.

Limitations and Broader Impacts. While ZipCache presents promising advancements in KV cache mixed-quantization frameworks for LLMs, the saliency ratio is manually specified before evaluation and cannot be automatically adjusted based on task datasets. Moreover, similar to other generative models, ZipCache can potentially be used to generate malicious content.

Acknowledgement This work was supported by National Key Research and Development Program of China (2022YFC3602601).

References

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

A. Chauhan, U. Tiwari, et al. Post training mixed precision quantization of neural networks using firstorder information. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1343-1352, 2023.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://lmsys. org/blog/2023-03-30-vicuna , 3(5), 2023.

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

T. Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. 2023.

T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems , 2022.

T. Dao, D. Haziza, F. Massa, and G. Sizov. Flash-decoding for long-context inference, 2023.

J. C. de Winter. Can chatgpt pass high school exams on english language comprehension? International Journal of Artificial Intelligence in Education , pages 1-16, 2023.

T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems , 35:30318-30332, 2022.

Z. Dong, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 293-302, 2019.

M. Du, F. He, N. Zou, D. Tao, and X. Hu. Shortcut learning of large language models in natural language understanding. Communications of the ACM , 67(1):110-120, 2023.

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 , 2022.

L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac'h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation, 12 2023.

S. Ge, Y. Zhang, L. Liu, M. Zhang, J. Han, and J. Gao. Model tells you what to discard: Adaptive kv cache compression for llms. In The Twelfth International Conference on Learning Representations , 2024.

Y. He, L. Liu, J. Liu, W. Wu, H. Zhou, and B. Zhuang. Ptqd: Accurate post-training quantization for diffusion models. Advances in Neural Information Processing Systems , 36, 2023.

L. Hou, R. Y. Pang, T. Zhou, Y. Wu, X. Song, X. Song, and D. Zhou. Token dropping for efficient bert pretraining. arXiv preprint arXiv:2203.13240 , 2022.

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

H. Kang, Q. Zhang, S. Kundu, G. Jeong, Z. Liu, T. Krishna, and T. Zhao. Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm. arXiv preprint arXiv:2403.05527 , 2024.

Y. J. Kim, R. Henry, R. Fahim, and H. H. Awadalla. Finequant: Unlocking efficiency with fine-grained weight-only quantization for llms. arXiv preprint arXiv:2308.09723 , 2023.

C. Li, W. Wang, J. Hu, Y. Wei, N. Zheng, H. Hu, Z. Zhang, and H. Peng. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706 , 2024.

D. Li, R. Shao, A. Xie, Y. Sheng, L. Zheng, J. Gonzalez, I. Stoica, X. Ma, , and H. Zhang. How long can open-source llms truly promise on context length?, June 2023.

D. Li, R. Shao, A. Xie, Y. Sheng, L. Zheng, J. Gonzalez, I. Stoica, X. Ma, and H. Zhang. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following , 2023.

Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang, and S. Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations , 2020.

J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978 , 2023.

J. Liu, R. Gong, X. Wei, Z. Dong, J. Cai, and B. Zhuang. Qllm: Accurate and efficient low-bitwidth quantization for large language models. In The Twelfth International Conference on Learning Representations , 2024.

J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems , 36, 2024.

Z. Liu, A. Desai, F. Liao, W. Wang, V. Xie, Z. Xu, A. Kyrillidis, and A. Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems , 36, 2024.

Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Y. Mehdad, Y. Shi, R. Krishnamoorthi, and V. Chandra. Llmqat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888 , 2023.

Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V . Braverman, B. Chen, and X. Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750 , 2024.

H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583 , 2023.

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.

M. Tan, L. Wang, L. Jiang, and J. Jiang. Investigating math word problems using pretrained multilingual language models. arXiv preprint arXiv:2105.08928 , 2021.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8612-8620, 2019.

G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning , pages 38087-38099. PMLR, 2023.

G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations , 2023.

F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , pages 1-10, 2022.

J. Y. Yang, B. Kim, J. Bae, B. Kwon, G. Park, E. Yang, S. J. Kwon, and D. Lee. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402.18096 , 2024.

Z. Yao, Z. Dong, Z. Zheng, A. Gholami, J. Yu, E. Tan, L. Wang, Q. Huang, Y. Wang, M. Mahoney, et al. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning , pages 11875-11886. PMLR, 2021.

Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems , 35:27168-27183, 2022.

Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Ré, C. Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems , 36, 2023.

Appendix

A Calculation of Overhead for Different Quantization Schemes

Assuming b = 8 , hd = l = 4096 , and that the KV cache is quantized to 4 -bit, we proceed to calculate the actual compression ratio for different quantization granularities. For groupwise quantization with a group size of n = 32 , the compression ratio R group is given by:

For tokenwise quantization, the compression ratio R token can be calculated as:

For our proposed quantization baseline, the compression ratio R baseline is determined by:

B Implementation Details of ZipCache

In this section, we provide an overview of the channel-separable tokenwise quantization scheme in Algorithm 1. Additionally, we present the process of ZipCache's prefill phase as described in Algorithm 2, as well as its decoding phase detailed in Algorithm 3. It is worth mentioning that during both the prefill and decoding phases, rather than calculating attention outputs separately for probe tokens and regular tokens followed by merging, FlashAttention [7] is utilized to compute the attention output for all tokens simultaneously. Additionally, attention scores of probe tokens are calculated. By bypassing the substantial memory accesses associated with matrix splitting and merging, this strategy enhances generation speed.

C Additional Experimental Results

C.1 Effect of Saliency Metric

In this section, we conduct ablation studies on our saliency metric, as shown in Table A. It demonstrates the superiority of our saliency metric over using accumulated attention scores or consistently prioritizing the latest tokens.

C.2 Accuracy and Efficiency Comparisons of various KV cache compression methods

In this section, we present the accuracy and efficiency comparisons of various KV cache compression methods, as presented in Table B. Data is collected by evaluating LLaMA3-8B model on 200 -line

retrieval task with a Nvidia A100 GPU. We use a batch size of 8 and an average input length of 3072 . To ensure a fair comparison, we implement GEAR and KIVI with FlashAttention integration.

Notably, the latency of ZipCache is lower than that of GEAR, which can be attributed to our efficient quantization scheme, whereas GEAR has a high overhead due to outlier extraction. Compared to KIVI, ZipCache's latency is slightly higher (2584.01 ms vs. 2482.26 ms), but ZipCache achieves a higher compression ratio and better performance. This difference is due to KIVI's fixed compression strategy, while we adaptively compress the KV cache based on the saliency. In comparison to MiKV [43], which identifies salient tokens through accumulated attention scores, our method achieves a notable 10 . 0% accuracy improvement by accurately pinpointing salient tokens and a substantial 38 . 0% decrease in prefill latency by integrating FlashAttention [7].

Moreover, to demonstrate the efficacy of the proposed efficient quantization scheme, we also implement ZipCache with groupwise quantization. The results show that using groupwise quantization

increases inference speed (2664.05 ms vs. 2584.01 ms) and reduces the compression rate ( 3 . 81 × vs. 4 . 43 × ) due to massive quantization overhead.

C.3 Evaluation on LongBench

In this subsection, we evaluate the performance of ZipCache on LongBench using the longchat-7bv1.5-32k model, as shown in Table C. The results show that ZipCache outperforms the previous state-of-the-art method, KIVI on long context scenario.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The authors have discussed the limitations of the work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoritical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The authors have disclosed all the information needed to reproduce the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is open sourced.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The authors have included experimental details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because of limited computing resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The authors have provided information on computing resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authos confirm that the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The authors have discussed broader impacts of the paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper has no such risks

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The authors have cited the related papers.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an adaptive, mixed-precision quantization method for compressing KV cache in LLMs. It proposes a channel-separable tokenwise quantization scheme to establish a robust quantization baseline, reducing the memory overhead of quantization parameters. A saliency metric, based on normalized attention scores, is employed to identify salient tokens accurately. This approach preserves essential information while aggressively quantizing less salient data, allowing for higher compression ratios with minimal impact on model accuracy. The method can be integrated with FlashAttention with an efficient approximation of the proposed metric. Experimental results validate the superiority of ZipCache on speed and accuracy over existing methods.\n\nSTRENGTHS:\n1. The idea of preserving salient tokens while compressing less critical tokens is both intuitive and effective, with the proposed accurate saliency metric being crucial for achieving this.\n2. The paper presents a novel quantization scheme tailored for channel-wise outliers in LLMs, providing a notable reduction in memory overhead compared to group-wise quantization schemes.\n3. The paper demonstrates strong experimental results on GSM8k and HumanEval, convincingly showcasing the benefits of adaptive KV cache compression.\n4. The integration with FlashAttention greatly enhances overall generation speed.\n5. The paper is well-organized.\n\nWEAKNESSES:\n1. Evaluating ZipCache on other generation or comprehending tasks will make the findings more robust.\n\nQUESTIONS:\n1. Have you considered using only the attention scores from the most recent token to determine token saliency? What impact might this have on the overall performance?\n2. What does saliency mean in Figure 3c? What is the probability of each token being selected as a salient token, as mentioned in the caption of Figure 3?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
The paper introduces ZipCache, an adaptive KV cache compression method for LLMs by accurately identifying salient tokens. It first presents a channel-separable tokenwise quantization scheme that reduces the overhead of quantization parameters. Next, it proposes a metric for identifying salient tokens based on normalized attention scores. By efficiently approximating the saliency metric, the method integrates seamlessly with fast attention mechanisms like FlashAttention. The efficacy of ZipCache is demonstrated through extensive experiments, showing superior performance in terms of compression ratio, speed, and minimal accuracy loss compared to existing methods.

STRENGTHS:
1.The introduction of normalized attention scores as a metric for token saliency is significant and promising for adaptive KV cache compression. It can accurately identify salient tokens, marking a substantial improvement over prior methods.

2.The approximated saliency metric can be integrated with FlashAttention, making it practical for real-world applications.

3.The idea of the channel-separable tokenwise quantization approach is novel and well-motivated, reducing memory overhead compared to groupwise counterparts.

4.The experimental results are promising, clearly demonstrating the efficacy of ZipCache in terms of compression ratio, speed, and minimal accuracy loss.

5.The paper is clearly written and easy to follow.

WEAKNESSES:
1.The experiments should encompass a broader diversity of tasks to comprehensively assess the method's effectiveness, such as LongBench.

2.The font size in Figures 5 and 6 is too small, making them difficult to read. It is recommended to increase the font size for better clarity and accessibility.

QUESTIONS:
See weaknesses

LIMITATIONS:
See weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a post training quantization framework named ZipCache for quantizing the key and value cache of LLMs. The authors introduce a channel-separable token-wise quantization scheme, which consumes less memory than the group quantization in terms of the quantization parameters. They also select saliency tokens based on a normalized attention matrix to avoid the accumulated bias. Additionally, the authors introduce an approximation method to apply it to FlashAttention.\n\nSTRENGTHS:\n- Overall, the writing of the paper is clear and easy to follow. Especially, the figures are drawn well.\n- By proposing the ZipCache, this work promotes the PTQ quality of LLMs on three chanllenging datasets.\n- The methods are well-integrated into popular framwork like FlashAttention.\n\nWEAKNESSES:\n- The idea of channel-separable quantization is not novel. It is a common practice to smooth the outliers of activations in the channel dimension, e.g., SmoothQuant [1], OmniQuant [2].\n- As shown in Table 3 (a), the values on the diagonal of the matrix are relatively large, while the values at other positions are relatively small. Based on this circumstance,  if the mean operation is performed for none-zero positions, earlier tokens may have smaller $p_i$ for they have more tiny values counted. Therefore, it may cause a prefer to the latest tokens. I think this is a issue for the proposed technique in Section 4.2, which needs more analysis.\n- Ablation studies are not provided. \n- As stated in Line 268 in the paper, new tokens remain un-quantized unless they reach 100. As far as I know, keeping part of tokens in full precision can improve the accuracy [3, 4]. Therefore, it is better to distinguish the contributions of this implementation and the proposed techniques. \n- I think the value of 'H/L' should be carefully considered. This is because both KIVI and ZipCache keep new tokens in FP16 at first and then perform quantization. Since the 'H' value of KIVI is 16, why it becomes 4 for ZipCache?\n- The equation (5) is not accurate for not subtracting the zero-point 'z' in the de-quantization.\n\n[1]. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.\n\n[2]. Omniquant: Omnidirectionally calibrated quantization for large language models.\n\n[3]. KIVI: ATuning-Free Asymmetric 2bit Quantization for KV Cache.\n\n[4]. SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models.\n\nQUESTIONS:\n- The authors have compared the channel-separable quantization with many dynamic methods, but lacks comparison of accuracy with static methods, e.g., WKVQuant [1]. It is better to further explore whether the $c_i$ in equation (6) can outperform well-trained parameters, since they consumes the same memory space.\n- How does the calculation of $c_i$ in equation (6) effect the inference latency? It is better to compare the channel-separable quantization with group quantization and static methods in terms of the inference latency.\n\n[1]. WKVQuant: Quantizing weight and key/value cache for large language models gains more.\n\nLIMITATIONS:\nThis paper contains the describtion of limitations. For suggestions, please refer to the Weakness and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

comments: SUMMARY:
The paper introduces a channel-separable quantization scheme that decouples the quantization along channel and token dimensions. This method significantly reduces the quantization overhead without compromising performance. To accurately recognize salient tokens, the paper introduces a new token saliency metric based on normalized attention scores, which alleviates the bias towards earlier tokens that accumulate more values. The authors have demonstrated on different LLM models with different tasks.

STRENGTHS:
The idea of channel separable token saliency quantization seems to work better than standard groupwise quantization.

The results are strong and beats the SoTA!

The demonstration with flashattention integration is useful

WEAKNESSES:
The paper needs to be proof read by a native English speaker, to improve its readability.

The contribution is incremental, and the idea of group quantization (different for different K and V) are not new as already noted by the authors.

The idea of speeding up through flashattention and flashdecoding while the baselines not leveraging that is a bit of unfair comparison. Both GEAR and KIVI can exploit similar speed up benefit, thus Fig 1 comparison and later in table speed comparison is not fair.

The details of system level implementation is missing, it will be good to help the reviewer provide how the speed up can be demonstrated on a single GPU inference system.

QUESTIONS:
refer to weakness.

LIMITATIONS:
The system benefits are not clear or thorough for the reviewer to clearly appreciate. Also more fair system eval is needed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17