KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
Authors: Tianyi Zhang
Jonah Wonkyu Yi
Zhaozhuo Xu
Anshumali Shrivastava
Venue: NeurIPS 2024 poster (2024)
Keywords: large language model
efficiency
kv cache
quantization
Primary area: other

Abstract: Efficient deployment of Large Language Models (LLMs) requires batching multiple requests together to improve throughput. As batch size, context length, or model size increases, the size of key and value (KV) cache quickly becomes the main contributor to GPU memory usage and the bottleneck of inference latency and throughput. Quantization has emerged as an effective technique for KV cache compression, but existing methods still fail at very low bit widths. Currently, KV cache quantization is performed per-channel or per-token independently. Our analysis shows that distinct channels of a key/value activation embedding are highly interdependent, and the joint entropy of multiple channels grows at a slower rate than the sum of their marginal entropy, which implies that per-channel independent quantization is sub-optimal. To mitigate this sub-optimality, we propose Coupled Quantization (CQ), which couples multiple key/value channels together for quantization to exploit their interdependence and encode the activations in a more information-efficient manner. Extensive experiments reveal that CQ compares favorably with existing baselines in preserving model quality, and improves inference throughput by 1.4–3.5$\times$ relative to the uncompressed baseline. Furthermore, we demonstrate that CQ can preserve model quality reasonably with KV cache quantized down to 1 bit.

KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
Tianyi Zhang
Jonah Yi
Dept. of Computer Science, Rice University xMAD.at
Dept. of Computer Science, Rice University xMAD.ai Houston, TX juy4drice edu
Houston, TX tz21@rice.edu
Zhaozhuo Xu
Anshumali Shrivastava
Dept. of Computer Science, Stevens Institute of Technology
Dept. of Computer Science, Rice University Ken Kennedy Institute
xMAD.ai Hoboken, NJ zxu79@stewens.co edu
ThirdAI Corp.
xMAD.ai Houston, TX
anshumal i@ricee......o edu
Abstract
Efficient deployment of Large Language Models (LLMs) requires batching mul- tiple requests together improve throughput. As batch size, context length, or model size increases, the size of key and value (KV) cache quickly becomes the main contributor to GPU memory usage and the bottleneck of inference latency and throughput. Quantization has emerged as an effective technique for KV cache compression, but existing methods still fail very low bit widths. Currently, KV cache quantization is performed per-channel or per-token independently. Our anal- ysis shows that distinct channels of key/value activation embedding are highly interdependent, and the joint entropy of multiple channels grows at a slower rate than the sum of their marginal entropy, which implies that per-channel independent quantization is sub-optimal. To mitigate this sub-optimality, we propose Cou- pled Quantization (CQ), which couples multiple key/value channels together for quantization to exploit their interdependence and encode the activations in more inermatiio-efficicen manner. Extensive experiments reveal that CQ compares favorably with existing baselines in preserving model quality, and improves infer- ence throughput by 1.4-3.5x relative to the uncompressed baseline. Furthermore, demonstrate that CQ preserve model quality reasonably with KV cache quantized down to bit.
Introduction
Large Language Models (LLMs) have showcased remarkable generalization abilities across various tasks without needing specific fine-tuning [30] These impressive capabilities have empowered
LLMs to find applications in numerous domains [9). However, high computational demands and prohibitive deployment costs of LLMs create significant barriers, hindering their widespread adoption [19 4 Particularly, as LLMs move towards larger model size [[4] and longer context
length 44), they require faster hardware accelerators such as graphics processing units (GPUs) with higher memory capacity for efficient inference. Hence it is crucial to develop approaches for reducing the computational costs and memory requirement of LLMss.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Key and value (KV) caching [47 has proven to be an effective technique for accelerating LLM inference without affecting model quality. In autoregressive LLMs, KV caching works through trading off memory to save computations: the key and value activations of all previous tokens in the current sequence are cached in memory to avoid their recomputation for generating the next token. However, KV cache can quickly overwhelm the memory capacity of GPUs as context length or batch size increases, since its storage scales linearly with these two factors. Consider the OPT- 175b model 43 storing its KV cache for 128 sequences of 2048 tokens requires 1.2 terabytes of memory, which is around 3 the storage of its weights. Because inference throughput scales with batch size, the substantial memory demands of KV caching can become a significant bottleneck to throughput. In addition, as KV cache is not shared across sequences within a batch, it has a low compute-to to memory ratio, making reading the KV cache from GPU memory the primary source of latency opposed to the attention computation [5] KV cache compression can bring the following benefits: 1. speeding up LLM inference by improving the compute-to-memon ratio, 2. improving the serving throughput by fitting more sequences into memory and hence enabling larger batch sizes, 3. lowering the GPU requirements for inference for given batch size, context length, and model size. Existing approaches typically achieve compression of KV cache through token eviction [47 24 or activation quantization 22 15). While these methods can preserve model quality at moderate compression rates (4x compression or 4 bits per activation), model quality quickly deteriorates at high compression rates (16x compression or bit per activation).. In this work, we leverage the interdependency between key/value channels, an insight overlooked by existing approaches, to achieve higher compression rates of KV cache while maintaining model quality.
Our approach is motivated by the obser- vation that distinct channels within the same key/value activation embedding are
LLaMA-7b w/ 1-bit KV Cache 620.08
LLaMA-2-133 w/ 1-bit KV Cache
highly interdependent and correlated (see
our analysis in Section 3.1. It is hence
Perplexity on WikText-2
2064.46
10a
more information-effiiiint to encode mul-
tiple channels of KV cache at once, which
10a
we call channel coupling. Existing solu-
10a
28. 39
tions, in contrast, only employ per-channel
20 90.64
or per-token quantization strategies 22 15]
10'
10:47
8.09
10
8.73
6.56
26.01
4.87
for compressing KV cache, which is not
optimal for exploiting the dependency be-
2
4
2
Num, of Coupled K/V Channels
Num. of Coupled K/V Channels
tween channels. As result, we observe significant model quality degradation at the extreme compression rate of 1 bit per ac- tivation. By leveraging channel coupling, enable compression at the level of 1-bit quantization of KV cache while preserving model quality. In Figure [ we show the perplexity of two models from the LLaMA family [3 37 WikiText-2 (27] under b-bit quantization with varying numbers of coupled channels. The full experimental setup is presented in Section 4 Coupling more channels significantly enhances model quality, as the perplexity quickly approaches the performance using uncompressed FP16 KV cache. By further combining KV cache quantization with sliding window of 128 recent tokens cached in full precision, we achieve a negligible 0.3-0.33 increase in perplexity with 1-bit KV cache.
* 1-bit KV cache
1-bit KV cache w/ sliding window of 128 recont tokens
FP16 cache
Figure 1: Perplexity of LLMs with 1-bit quantized KV cache approaches the uncompressed FP16 performance as the number of coupled K/V channels increases.
We summarize our contributions as follows.
1. We observe the phenomenon that distinet channels within the same key/value activation embedding share high amount of dependency or mutual information, which has not been leveraged by existing approaches.
2. We propose Coupled Quantization (CQ), novel KV cache quantization method that jointly encodes multiple key/value channels to exploit the dependency across channels.
3. Through extensive experiments, we demonstrate the effectiveness of CQ at preserving model quality and speeding up L.M inference against competitive baselines. Furthermore, we demonstrate CQ reasonably preserves model quality at an extreme level of 1-bit quantization.

---

2
Background
This section introduces the relevant background information including the KV caching technique and per-channel quantization.
2.1 LLM Attention and KV Cache
Decoder-only transformer-bue based LLMs employ masked self-attention [8]. in which activations of the current token only depend on the previous tokens and are unaffected by future ones. This property enables training parallelism for the next-token prediction objective, and gives rise to the KV caching technique for efficient decoding during inference. Consider the decoding step for the (-th token in single head of attention in an LLM. The input embedding of the (-th token (a column vector), et, goes through three distinct transformations to become key, query, and value activation embeddings fK (et), (e(ee) ). fv (Ce), where the transformations fK. for J' are composed of linear projection and positional encoding methods such as RoPE [35]. is computed as
The output embedding of attention for the t-th token
attention(e.) = (v(ei) fv(e)) softmax (I ([x(ee) ... SK((e) I' (elaa) /va
(1)
where d is the dimensionalityy of fk(e)). Computing the output embedding of the current token requires the key and value activation embeddings of all previous tokens, fe;): and fv(4)) where E (I....,t 13. These embeddings are cached in memory from previous decoding steps to eliminate redundant computations, process known as KV caching. The size of KV cache can x 2
be calculated as the number of tokens in each sequence, I is the number of layers in the model, 2 is for key and value, h is the number of key/value attention heads, and d the dimensionality of single head of key/value activation embedding. As batch size, context length, model size increases, the size of the KV cache can quickly overwhelm the limited GPU memory. KV cache bottlenecks inference throughput since it limits the maximum batch size, and it is major contributor to latency due to the low cempuut-tt-memory ratio [1].
floating-pons
numbers, where the batch size, n is
2.2 Per-Channel Quantization
employ per-channel quantization for keys and per-token quantization for values. Per-channel and per-token quantization are similar, except the direction along which the quantization centroids are learned (or the direction along which the scaling factor and zero-point are determined for uniform quantization). Keys are quantized per-channel based on the observation that certain key channels have significantly higher magnitudes than others, while values are quantized per-token because value channels have no such outliers. In non-uniform per-channel quantization, set of centroids is learned for each channel. Suppose A is key value activation matrix, and let A;, denote the i-th channel of A. Then, non-uniform b-bit per-channel quantization aims learn set of centroids C; R for each channel of through the objective
Existing KV cache quantization methods [15] 25
C arg min A ((Ai))
(2)
CCB
IC
where q quantizes each value in Ar; to the nearest centroid in C.
3
Methodology
In this section, we motivate our proposal using information theory and introduce the Coupled Quantization (CQ) approach for KV cache compression.
3.1 Motivations
Our proposed approach is inspired by concepts in information theory 33 We consider channels in
a key/value activation embedding as random variables X1, X2,.... The amount information (or
uncertainty) in channel X can be measured by entropy, defined as ((X) = -Ix p(x) loga p(z) dz,
3

---

Sum of Marginal Entropies
Joint Entropy
LLaM4-7 Roys 1LMM4 Valuee
Loer Chimel
Layer
Layer
Layer 3
Layer 4
Entropy of Valuus (bin)) Entroyy of Keys (bits)
15
10
-
Laper Chrooil
05
Liger 24 Chrree
10
10
10
Layr 28 Channil
0.5
5-
5-
Lover 32 Chinel
Num. of Joint Channels
(a)
(b)
Figure 2: (a) Growth rate of joint entropy versus sum of marginal entropies of the key/value activation
embeddings of LLaMA-7b 7b 262k tokens of WikiText-2. Entropy is estimated using Equation
4 The slower growth rate of joint entropy implies that quantizing more channels together is more information-efficiie
than quantizing fewer channels. (b) Correlation matrices of the first 32 channels of layers of LLaMA-7b key and value activation embeddings on WikiText-2. Channel pairs exhibit high levels of linear dependency, shown by high magnitudes of the correlation coefficients.
where p(.) is the probability density function and X is the support of X. ((X) measures the theoretical number of bits needed for losslessly encoding the channel X, hence be used to gauge how "quantizable" a channel is: if H (Xi) < (X2), then channel X may be quantized to fewer bits than channel X₂ with the same quantization error.
Our insight is that different channels from the same key/value activation embedding may be in- terdependent, which would reduce the number of bits required for jointly encoding multiple chan- nels together compared to encoding them independently. The total amount of information (or uncertainty) in two channels X1, X2 is measured by joint entropy, defined as (X,,, X2).
-Jx, Jz, p(#1, 222 log, p( 222) drz dr,, where p(:.)) :) is the joint probability density function. Equivalently, the joint entropy of two channels is the difference between the sum of their marginal entropies and their mutual information, i.e., H HX1,, X2) H(X)) H(x)) I(Xi, X2), where I(.,)) is a non-negative quantity for measuring the mutual dependency of two random variables.
Thus, have
H(X,,X,) H(x)) + H(X2)
which implies the number of bits needed for jointly encoding two channels is no more than the total number of bits needed for encoding them independently. Previous works have demonstrated that deep neural networks [16] and attention-based networks 9 tend produce low-rank embeddings, which suggests that channels of key/value embedding in LLM may exhibit high amount of mutual dependency.
is hence beneficial to measure the difference between the sum marginal entropies of multiple key/value channels and their joint entropy. A significant difference would suggest that encoding these channels together is more information-eeffiiinn than encoding them independently. However, it is difficult derive the exact entropy joint entropy of channels, since their probability density functions are not known. Therefore, we employ the "binning" trick [21] to estimate entropy. We first observe an empirical distribution of key and value channels by saving the KV cache on dataset, and partition the support of each channel into equally sized bins. Then, values of each channel are discretized the index of the bin they fall into. Finally, the joint entropy of n channels Xi...., X,, is estimated with the Riemann sum,
H(X,, X,)
Σ EB
Σ Sh E3,
Mar,..., ........, Fa) logz p f(11,.... Fn)
(4)

---

Per-Channel Quantization
orginnll value quarrized value
Coupled Quantization (ours)
origina value quantized
||A s(A) 72 601.4
controid
quant.
||4 2(.4)
P 250.6
key or value activation embedding
per-channel centroids
0.0
key or value
[..1]
0.1
activation embedding
channel-coupled centroids
-.2
-0.-
channel
(9:9)
0,00
coupled channets 0&1
channel
0.4-
4.33
-0.4
-0.4 0.0
-0.5
-0.4
0.0
0.4
channell 0
channel
Figure 3: Per-channel quantization (left) and our proposed Coupled Quantization (right). The 1-bit quantization results on the first two channels of the first-layer key activation embeddings of LLaMA- 7b on the WikiText-2 2 dataset are shown. CQ leverages the dependency between channels achieve lower quantization errors than per-channel quantization.
where IB; is the support of the binned or discretized X; and p(.) is the empirical probability mass function. Specifically, we divide the channels of key and value embeddings of LLaMA-7b (36] into non-overlapping groups each containing e contiguous channels, where c E {1, 2, 4], and estimate the joint entropy and the sum of marginal entropies of each group. The support of each channel is partitioned into 16 equally sized bins. Figure 2a shows the mean and standard deviation of the estimated joint entropy and sum of marginal entropies of four layers of LLaMA-7b on 262k tokens of the WikiText-2 dataset [27], averaged over groups. We only show a maximum group size of 4, since increasing the group size requires saving exponentially more key and value embeddings to avoid empty bins and maintain estimation quality. As shown in Figure 2a the sum of marginal entropies grows a linear rate while the joint entropy increases slower at sub- linear rate. This implies that as the number of jointly quantized channels increases, the total amount of information needed for encoding them decreases. This phenomenon is the foundation that motivates our proposed approach.
In addition to studying the marginal and joint entropy, we also analyze the linear relationships between channels of a key/value activation embedding using Pearson correlation coefficient. Figure presents the correlation matrices for the first 32 channels of 5 layers of LLaMA-7b keys and values on WikiText-2. The key and value channels exhibit high levels of linear dependency, and clearly not independently distributed, as shown by high magnitudes of the correlation coefficients. In Section M of of the appendix, we include the correlation matrices of all layers of LLaMA-7b, and present scatter plots to visualize the patterns in key and value activations.
3.2 Coupled Quantization
Motivated by the finding that distinct key/value channels exhibit high amounts of dependency, propose Coupled Quantization (CQ), an informatiin--effcciet KV cache quantization approach that couples multiple key/value channels for quantization. More concretely, channels of key or value activation embedding are divided into equally sized, non-overlapping groups of contiguous channels. The channels in each group are coupled, as they are jointly quantized and share a single quantization code. For each group of coupled channels, distinct set of multi-channel centroids are learned, where each centroid has dimensionality equal the number of channels in that group. When quantizing a key or value activation embedding, each channel group is quantized to the nearest centroid in terms of L2 distance. We use the CQ-<c>e<b>b notation to denote the configuration of channel coupling and quantization bit width, where <c> is the number of channels in each group and <b> indicates the number of bits quantized code for a group. For example, CQ-4c8b means that every 4 contiguous channels are coupled together and each coupled group shares an 8-bit code, which is equivalent to 2-bit per-channel quantization in terms of storage overhead of quantized codes. An illustrative comparison of per-channel quantization and CQ is shown in Figure 3 Although previous works 15| 25 opt to quantize keys per-channel and values per-token, we adopt channel-coupled quantization for both keys and values, which we empirically show is effective for both in Section 4.3 CQ quantizes keys before the positional encoding such as RoPE (35) applied, which increases the quantization difficulty by introducing more outliers in key activations [15 25
5

---

3.2.1
Centroid Learning
In CQ. the multi-channel centroids for each channel group are learned offline on calibration dataset by leveraging uniform clustering or second-order-inffe order-information- ionommttto--ifommmed clustering. Specifically, for uniform centroid learning of the CQ- bbbb configuration, a set of centroids C₁ *R is learned independently for each channel group through the objective
C = arg min Atte-ct1jde - oq (Aq(AA(caa (A(cc-Ct)xee.)) 1
(5)
CCR |||=2*
where A(ic-c+1)iie Irice, is the sub-matrix of A containing all coupled channels from the i-th group, and eq quantizes each column vector to the nearest centroid in C' in terms of L.2 distance. We use the k-means algorithm [26] with k-means++ initialization [ to optimize the objective.
LLMs are more sensitive the quantized precision of certain weights than others (20), hence centroids of CQ should be learned to bias towards preserving the precision of more important activations. To this end, we leverage an approximation to the Hessian perform centroid learning. More concretely, we use the diagonals of the Fisher information matrix JF to identify the more influential key/value activations and guide the centroid learning process. Using the diagonal Fisher information matrix for quantization was proposed in [22], and we extend it to the multi-channe case. For performing fisher--uidees centroid learning, we first save a key/value activation matrix A and its gradient g(A) _ ((A) on calibration dataset, where is the training loss function. We approximate the Hessian matrix using the diagonals of the Fisher information matrix, diag(F) = g (A) @ g(4), which the element-wise square of the gradient matrix. We use the sum of diagonal entries of the Fisher information matrix as a measure of importance for each group of activations, and obtain the centroid set O for the i-th channel group using the objective
order-inforrmaioon
OF arg min Σ [g(Atiectt)sse.) g ((A(ic-ct))ic.))
| A(ie-ct1)ac, - cq(AA
CCk
IC|=2*
partial sum dliaggF)
(6) which we leverage weighted k-means to optimize. We discuss the overhead of centroid learning and centroid storage in Section E in the appendix.
3.3 Efficient Inference Through Kernel Fusion
We design fused GPU kernels to enable efficient inference of CQ. During inference, dequantiz- ing couple-quantized KV cache requires many random accesses for lookups of multi-dimensional centroids. the centroids reside in GPU global memory, these random accesses would greatly hinder the inference efficiency. We cireumvent this issue by caching centroids in the shared memory, which has significantly lower latency and higher bandwidth than global memory. Due to limited size of the shared memory for each thread block, assign the work of a single channel group to each thread block, which only requires loading a single group of centroids into a block of shared memory. We perform kernel fusion merge dequantization of key cache, positional encoding and KQ multiplication, as well as to merge dequantization of value cache and its multiplication with attention scores. We validate the inference efficiency of CQ empirically Section
4 Experiments
In this section, we perform extensive experiments to validate the effectiveness of our proposed CQ approach for KV cache compression. We first introduce the experimental setups including hardware. software, datasets, metrics, and basclines used. Then, we present the detailed empirical results and provide discussions. Finally, we perform an ablation study to validate the efffcttivnness of each component of our proposal.
Hardware and Software Experiments are performed on a Linux server equipped with 4 NVIDIA A100 40GB GPUs. Our software implementation of CQ is based on PyTorch (29 and the Hugging- Face Transformers library (39)
Evaluation Metries and Datasets We compare different KV cache quantization by evaluating the quality of 5 LLMs on various benchmarks. The 5 LLMs considered are I. LLaMA-7b. 2. LLaMA-

---

Table I: Perplexity of LLMs on WikiText-2 under different KV cache quantization methods at varying bit widths. The results of INT. NF, and KVQuant (except -Ib and -1b+1% sparse) are from [[5] "NaN" means Not Number, which is caused by quantization numerical instability. Our proposed method CQ outperforms baselines under the same bit width.
Bits Per Activation
LLaMA-7b
LLaMA-13b
LLaMA-2-7b
LLaMA-2-13b
Mistral-7b
FP16
16
5.68
5.09
5.12
4.57
4.76
INT4
4.00
5.98
5.32
5,66
5,01
4.97
INT4-g128
4.16
5.77
5.16
5.32
4.71
4.82
NF4
4.00
5.87
5.23
5.47
4.90
4.91
NF4-g128
4.25
5.77
5.17
5.30
4.71
4.83
KVQuant-4b
4.00
5.73
5.15
5.18
4.63
4.81
K KVQuant-4b+IB 1 sparse
4.32
5.70
5.11
5.14
4.59
4.78
CQ-2c8b
4.00
5.70
5.11
5.14
4.59
4.79
INT2
2.00
11779
69965
4708
3942
573
INT2-g128
2.14
37.37
41.77
117.88
93.09
51.96
NF2
2.00
3210.5
$785.6
13601
4035.6
902.51
NF2-g128
2.25
351.23
141.19
634.59
642.44
252.85
KVQuant-2h
2.00
8.17
7.29
9.75
29.25
7.33
KVQuant-2b+IB sparse
6.06
5.40
5.50
4.92
5,16
CQ-4c8b
2.00
5.97
5.32
5.42
4.81
5.11
CQ-4cbb
2.26
5.88
5.26
5,32
4.74
4.98
K KVQuant-1h
1.00
321.58
1617.40
NaN
4709.83
203.73
KV KVQuant-Iie 1b+ 1 sparse
1.32
9.93
7.97
9.50
13.76
10.07
CQ-8c8b
1.00
8.09
7.02
7.75
6.55
7.25
CQ-8c10b
1.27
6.78
6.00
6.25
5.47
5.90
361. 3. LLaMA-2-76 4. LLaMA-2-13b 37 Mistral-7b [18]. We evaluate the quality of LLMs
using the metric perplexity on 2 datasets: WikiText-2 (27] and C4 [I], and zero-shot accuracy on
3 benchmarks: WinoGrande (32, PIQA 3. and ARC Challenge (Arc-C) (5). Furthermore,
evaluate on long-context benchmarks GSM8K (6] with chain-of-thought (CoT), and few-shot MMLU
13 with COT. For perplexity and accuracy evaluations, the KV cache of all tokens in all layers is quantized, during both the prefill and decoding stages. The experimental details, including the procedures for perplexity and benchmark evaluations, are presented in Section A the Appendix. More experimental results, including a comparison between CQ and KIVI (25) on LongBench (2). and results on more models and passkey retrieval 28 be found in the Appendix.
Baselines We compare our proposed approach with uncompressed FP16 KV cache and competitive KV cache quantization methods, including 1. uniform integer (INT) quantization (without grouping and with group size of 128), 2. NormalFloat (NF) quantization 88 (without grouping and with group size of 128), 3. KVQuant [155 (dense-only and with 1 % outliers stored in sparse format). K KVQuant-<b>b+ % sparse is dense-and-spars method that stores outlier activations in sparse matrix and requires additional sparse matrix multiplication during inference, which introduces extra computational overhead. For calibration. we use the same set of 16 sequences of WikiText-2, each with 2048 tokens, for KVQuant and CQ. Other methods do not require calibration. Calibration is performed only once and the learned centroids are used for all downstream evaluations. Calibration done on the training set of WikiText-2. 2, while perplexity and accuracy are evaluated on the test sets of different datasets and benchmarks. For 1-bit and 2- KVQuant, we employ Q-Norm to mitigate distribution shift, recommended by [5]]. We report Bits Per Activation (BPA) to measure the compression rate of each method, where each activation in the uncompressed KV cache is a 16-bit float. Detailed calculations of bits per activation for CQ are presented in Section in the appendix.
4.1 Results
Table I presents the perplexity of LLMs on WikiText-2 under different KV quantization methods. CQ consistently outperforms baselines under the same quantization bit width. In low bit width regions of 1-bit and 2-bit quantization, dense-only quantization baselines quickly deteriorates in quality while CQ preserves quality well. We highlight that CQ-8c8b (1 bit per activation) outperforms KVQuant-20 (2 bits per activation) with only half the memory. CQ also compares favorably against
7

---

Table 2: Accuracy of LLMs on benchmarks under different KV cache quantization methods varying bit widths.
Bits Per Activation
Benchmark
I.aMM-7hh
I.aMA-13h
L1aMA-2-7h
LLaMA-2-
Mistral-7h
Average
WinoGrande
09.93
72.69
68.90
71.98
73.88
FP16
PIQA
78.67
79.16
78.07
79.16
80.58
65.55
ARC-C
41.72
46.42
48.29
WinoGrande
71.59
KVQuani-4b
4.00
PIQA
78.62
79.22
77.86
78.94
80.58
65.17
ARC-C
42.32
45.99
42.75
46.67
49.06
WinoGrande
70.72
73.40
68.67
72.30
73.72
KVQuant-4b- sparse
ARC-C
41.38
46.76
43.17
47.87
49.91
WinoGrande
70.40
72.45
68 08.27
72.53
73.48
CQ-2e8b
4.00
PIQA
78.61
79.11
77.91
78.62
80.52
65.31
ARC-C
41.55
47.78
WinoGrande
70.17
70.80
KVQuant- 2h-1 S sparse
2.32
PIQA
77.69
78.51
76.60
78.51
79.65
63.79
ARC C
38.74
45.14
41.47
44.97
47.53
WinoGrande
68.51
69.93
67.40
71.67
70.71
CQ-4c96
2.26
PIQA
76.82
78.51
77.09
77.31
79.48
63.75
ARC C
39.16
45.14
41.64
44.97
47.95
WinoGrande
$3.59
59.35
$1.70
$1.30
63.46
KVQuant-2b
2,00
PIQA
72.47
74,81
63.38
65.40
75.46
$2.20
ARC-C
32.00
34.47
38.57
WinoGrande
CQ-428b
2.00
PIQA
76.11
78.29
76.12
77.42
79.49
62.85
ARC-C
38.48
44.03
39.93
44.11
45.65
WinoGrande
56.67
61.01
57.77
57.30
58.17
KVQuant- 1b-1 spanse
1.32
PIQA
71.38
73.83
54.31
ARC-C
29.69
35.32
31.48
32,59
33.19
WinoGrande
60.46
65.27
59.19
62.9%
63.93
CQ-8:10b
1.27
PIQA
73.45
75.90
73.07
74.37
77.31
ARC-C
33.28
37.12
34.64
38.74
39.59
WinoGrande
50.51
49.41
KVQuant. lh
1.00
PIQA
53.26
53.54
$3.37
50.92
54.73
41,35
ARC C
21.76
21.33
20.65
21.67
19.88
WinoGrande
56.51
61.56
55.01
57.14
58.25
CQ-Se8b
1.00
PIQA
71.16
73.99
71.22
73.01
75.24
$4.36
ARC-C
30.20
33.79
30.20
34.30
33.79
Table 3: Accuracy of LLaMA-2-70 2-7b on 5 long-context benchmarks under different KV cache quanti- zation methods at varying bit widths.
MMLUU, CoT Fewshor
Activation
GSMBK, CoT
STEM
Humanitics
Social
Other
FP16
13:57
33.43
41.12
56.60
KVQuant-lb=11% sparse
4.32
14.35
31.04
41.12
48,37
55.43
CQ
4.00
33.73
KVQuant- -he 56 sparse
2.32
10.3
28.06
35.64
42.43
46.39
CQ-429b
2.26
10.31
27.76
35.91
44.51
45.75
KVQuant-25
2,00
CQ-4c8b
2,00
8.04
28.67
30.89
45.40
41.94
KVQuant-Ib+13 spanse
1.32
10.75
14.09
19.94
CO 10-8b10b
1.27
2.35
13.13
21.81
28.19
26.98
KVQuant- Ib
1.00
0.68
0.00
0.00
0.00
0.00
CQ-8cRh
1.00
1.74
5.37
11.39
20.77
16.72
dense-and-spant baselines by outperforming KVQuant+1 % sparse in most cases despite using less bits. We present the perplexity results on C4 in Table in the appendix. present the accuracy results of KVQuamt and CQ on different benchmarks. CQ consistently outperforms dense-only KVQuant at I-bit and 2-bit, and performs better or on par with the dense-and sanse-and-sparse KVQuant under the same bit width. Jableeppresete the accuracy comparison of KVQuant and CQ long-context benchmarks, with CoT or sho- COT. CQ mostly outperforms K KVQuant under similar bit widths.

---

Table 4: Accuracy of LLaMA-2 models with couple-quantized KV cache and a sliding window of 32
recent tokens cached in FP16. CQ achieves minimal accuracy degradation compared to the FP16
baseline.
Quant.
BPA
WinoGrande
PIQA
Are-C
Arc-E
Hellaswag
Average
FP16
16
68.90
78.07
43.43
76.30
57.14
64.768
CQ-2c8b
4.00
69.14
78.18
43.34
76.52
57.12
64.860 (+0.092)
LLaMA-2-7b
CQ-4c8b
2.00
69.06
77.86
42.83
76.01
56.79
64.510 (-0.258)
CQ-8c8b
1.00
69.14
77.91
42.92
75.67
55.11
64.150 (-0.618)
FP16
16
71.98
79.16
48.29
79.42
60.04
67.778
CQ-2e8b
4.00
72.30
78.94
47.95
79.55
60.18
67.784 (+0.006)
LLaMA-2-13b
CQ-4c8b
2.00
72.30
78.89
47.61
79.21
59.84
67.570 (-0.208)
CQ-8c8b
1.00
72.22
78.84
47.78
79.12
58.55
67.302 (-0.476)
4.2 Near-native Performance with Sliding Window Full-pun prreisieiinnnn Cache
CQ mostly outperforms competitive baselines under the same bit width for fully quantized KV cache, and we further investigate the performance of CQ when combined with sliding window of recent tokens cached in FP16 precision. This sliding window of full-precision cache only introduces a small constant memory overhead for each sequence. In Table 4 present the accuracy of LLaMA-2 models on benchmarks (WinoGrande, PIQA, Are-C, Are Easy (Are-E) E) [5], and Hellaswag [42)) with sliding window of 32 recent tokens in FP16 and the rest of the tokens coupled coupled-quantize.. quantized. CQ
preserves the accuracy well, achieving a mere 0.476-0. 0.618% loss in average accuracy over 5 tasks with 1-bit quantization. CQ also preserves perplexity well at 1-bit with sliding window of 128 tokens in FP16, as shown in Figure 1
Table 5: Ablative study on the effects of channel cou- pling for quantizing keys only, values only, and both keys and values, using I-bit CQ. Perplexity of LLaMA- 7b on WikiText-2 is reported.
Table 6: Ablative study on the effects of Fisher-guided centroid learning for CQ. Perplexity LLaMA-7b on WikiText-2 reported.
Perplexity Values Only
Centroids
Perplexity
BPA
Keys Only
Keys & Values
Uniform
5,77
CQ-2c8b
CQ-1clh
1.00
17.17
177.13
620.08
Fisher-guided
5.70
CQ-2c2b
1.00
8.29
9.49
28.39
Uniform Fisher guided
6.86
CQ-4ebb
7.11
CQ:4cbb
5.97
CQ 8c8h
1.00
6.53
6.54
8.09
Uniform Fisher-guided
FP16
5.68
CQ-&c8b
8.09
4.3 Ablation Study
We perform a set of ablation experiments answer the following questions. The experimental results are presented in Tables &
Q1: Under the same quantization bit width, does coupling more channels lead to better model quality? Yes, the model performance approaches the FP16 baseline level as the number of coupled channels increases, shown in Table This also holds true for different models and bit widths, as shown in Figure and Table in the appendix.
Q2: Is channel coupling effective for quantizing both keys and values? Yes, channel coupling highly effective for both keys and values. As shown Table perplexity improves significantly under the same bit width the number of coupled channels increases, and holds true for both key-only and value-only quantization.
Q3: Does Fisher-guided learning of centroids produce better model quality than uniform clustering? Yes, Fisher-guided centroid learning improves the model perplexity over uniform clustering, as shown in T Table 8 in the appendix further shows it is effective for different models.
4.4 Efficiency of CQ

---

We study the efficiency of CQ by comparing its decoding throughput with uncompressed FP16 KV cache using the
Throughpus (tokers ssoc)
HuggingFace Transformers implementation (39 We mea-
750
sure the decoding throughput by running ILaMMA2-77b on an A A100-40GBB GPU with CQ and FP16 KV cache
CQ-8edh (1-be)
to process a 100-token prompt and generate 1000 tokens.
CQ-4:8l (2-bi)
We increase the inference batch size until the GPU runs
200
CQ (4
FPP16
out of memory. Results are presented in Figure 4 CQ achieves 3.75 x, 7.5x, and 15x larger batch size than the FP16 baseline at 4-bit, 2-bit, 1-bit quantization, respec- tively. Moreover, CQ improves the decoding throughput by 1.4-3.5x relative to the FP16 baseline. Additional la- tency measurements for CQ are presented in Section K in the Appendix.
100
200
300
Batch Size
Figure 4: Inference throughput of CQ versus FP16 KV cache for LLaMA-2- 7b.
Related Works
Existing works mitigate the high memory overhead of KV cache through token eviction, quantization, or tensor offloading. Scissorhands (24) and H2O [47] achieve KV cache compression while preserving model quality by storing only the pivotal tokens and evicting the unimportant ones. KIVI [25] proposes to quantize keys per-channel and values per-token using group-wise integer quantization. KVQuant [5] proposes sensitivity- based and dense-and Sonse-and-sparse quantization for KV cache to reduce quantization errors. Flexgen 34 proposes offload KV cache instead of weights enable high- throughput inference on a single GPU. Weight quantization [1] 23 is an orthogonal line of work that reduce GPU memory requirements and improve effciency of LLM inference and fine-tuning 7 and NoMAD-Attention [46] are system optimizations for speeding up LLM inference GPUs and CPUs, respectively. Product quantization (17] is method for nearest. neighhor search that compresses vectors by decomposing the vector space into a Cartesian product of low-dimensional subspaces, and quantizing each subspace independently.
45 FlashAttention
Conclusion
We propose Coupled Quantization (CQ) for mitigating the latency and throughput bottleneck of LLM inference by quantizing KV cache. We discover that channels of KV cache are highly interdependent, which implies the existing approach of per-channel quantization approach is sub-optimal. We propose channel coupling to exploit the interdependency across channels to achieve more information-effieien encoding of key/value activations. Extensive experiments demonstrate that method outperforms competitive baselines in model quality in most cases, and can reasonably preserve model quality with KV cache quantized down bit.
Acknowledgements
This work was supported by National Science Foundation SHF-2211815, Ken Kennedy Institute, and grants from Adobe and VMware.
Limitations & Broader Impacts
KV cache quantization is a form of lossy compression which inevitably affects model quality. Although we study its effects on perplexity and accuracy, it remains unclear how it affects other aspects of the model such hallucination and adversarial robustness. Making LLM inference more efficient contributes to the democratization of artificial intelligence and the reduction of carbon footprints. We expect no additional negative societal impacts other than the ones already posed by LLMs.
10

---

References
[1] David Arthur, Sergei Vassilvitskii, et al. k-means++: The advantages of careful seeding. In Soda, volume 7, pages 1027-1035, 2007.
[2] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng. Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14500, 2023.
[3] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1432-7439, 2020.
[4] Lingjia Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05/ 2023.
[5] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. 2018.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2 14168, 2021.
[7] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359 2022.
[8] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. Advances in Neural Information Processing Systems, 36, 2024.
[9] Yihe Dong, Jean- Caan-Bapisste Cordonnier, and Andreas Loukas. Attention not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793-2803. PMLR, 2021.
[10] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[11] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:22 17323, 2022.
[12] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023.
[13] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
[14] Jordan Hoffmann. Sebastian Borgeaud. Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022.
[15] Coleman Hooper, Schoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Imm inference with kv cache quantization. arXiv preprint wwX.v:2401.18077 2024.
[16] Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The rank simplicity bias in deep networks. arXiv preprint arXiv:2103.1042 2021.

---

[17] Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117-128, 2010.
[18] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprini arXiv:23 2023.
[19] Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307. 10169, 2023.
[20] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: bense-and-spars quantization. arXiv preprint arXiv:2306.07629, 2023.
[21] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 69(6):066138 2004.
[22] Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint NKI::2102.05426,, 2021.
[23] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-awaae weight quantization for IIm compression and acceleration. arXiv preprint arXiv:2306.00978 2023.
[24] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Seissorhands: Exploiting the persistence of impor- tance hypothesis for IIm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024.
[25] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint 200.:2240..02750 2024.
[26] Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 2892):129-137, 1982.
[27] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprini (oXXvv:609.0700 2016.
[28] Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. In Workshop on Efficient Systems for Foundation Models@ ICML2023.
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, figh-performance deep learning library. Advances in neural information processing systems. 32, 2019.
[30] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog. 188)9 2019.
[31] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-63 2020.
[32] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106, 2021.
[33] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-44) 1948.
12

---

[34] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: thighhhrouuhhut generative inference of large language models with single gpu. In International Conference on Machine Learning, pages 31094-31116 PMLR, 2023.
[35] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568:127063, 2024.
[36] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977, 2023.
[37] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092865 2023.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[39] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38 45, 2020.
[40] Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309. 16039, 2023.
[41] Zhichao Xu, Ashim Gupta, Tao Li, Oliver Bentham, and Vivek Srikumar. Beyond perplexity: dimeiiadieniiooal safety evaluation of Ilm compression. arXiv preprint arXiv:2407.04965. 2024.
[42] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.0788 2019.
[43] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.011 01068, 2022.
[44] Tianyi Zhang and Anshumali Shrivastava. Leanquant: Accurate and scalable large language model quantization with loss- loss-error-aware grid. arXiv preprint arXiv:2407.10032, 2024.
[45] Tianyi Zhang, Junda Su, Oscar Wu, Zhaozhuo Xu, and Anshumali Shrivastava. Spallm: Unified compressive adaptation of large language models with sketching. arXiv preprint 2401::2410.06364, 2024.
[46] Tianyi Zhang, Jonah Wonkyu Yi, Bowen Yao, Zhaozhuo Xu, and Anshumali Shrivastava. Nomad-attention: Efficient IIm inference on cpus through multiply-add-fre attention. arXiv preprint arXiv:2403.01273. 2024.
[47] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Appendix Supplemeenal Material
A Experimental Details
Perplexity is evaluated on the test set of the datasets, WikiText-2 and C4, at the maximum context length of the LLMs (2048 for LLaMA, 4096 for LLaMA-2, and 8192 for Mistral), following the setup in [15].
Perplexity Evaluations
Benchmark Evaluations We use In-evaluation-hareen for performing benchmark evaluations. We use the following task names: winogrande, piqa, arc_ arc_challenge, gsmåk_cot, mmlu fouu_flan_cool www.ffewttwtaaaleleeeeeeeeeeee mmlu_flan fell__flan_cotfffeeeeeeeee hot___oot__ewws__sttt__to mmlu mmlu_flan_cee fttt_ffllanootfteaaaaete wtw.coooeeetwtttteeeeeeeeeeee mmlu cotaanot___teeeeeeeeeeoeee lolaa_ccotfewsht_otteeeee
12 (package version 0.4.2)
B Perplexity Results on C4
Table presents the perplexity results of different quantization algorithms the test set of C4 dataset. Our proposed method CQ performs better than or on par with baselines under the same bit width.
Table 7: Perplexity of LLMs C4 under different KV cache quantization methods at varying bit widths. The results of INT, NE, and KVQuant (except -lb and -1b+ I % sparse) are from [15.
Bits Per Activation
LLaMA-7b
LLaMA-13b I.amma2-77b
ILaMA-2-13b
Mistral-7b
FP16
16
7.08
6.61
6.63
6.05
5.71
INT4
4.00
7.40
6.82
7.31
6.59
5.91
INT4-g128
4.16
7.16
6.67
6.87
6,20
5.76
NF4
4.00
7.27
6.74
7.09
6,45
5.85
NF4-g128
4.25
7.16
6.66
6.86
6,20
5.77
KVQuant-4b
4.00
7.13
6.65
6.70
6.11
5.75
KV KVQuant-4b+135 sparse
4.32
7.09
6.62
6.65
6.06
5.72
CQ-2c8b
4.00
7.11
6.64
6.67
6.09
5.74
INT2
2.00
10892
100870
4708
4220
477
INT2-g128
2.14
43.49
56.25
113.49
97.04
50.73
NF2
2.00
2850.1
4680.3
13081.2
4175.6
1102.3
NF2-g128
2.25
248.32
118.18
420.05
499.82
191.73
KVQuant-2b
2.00
10.28
9.05
15.16
43.77
8.40
KVQuant-2b+13 sparse
2.32
7.38
6,83
7.06
6.38
6.08
CQ-4c8b
7.52
6.96
7.23
6.52
6.17
CQ-4c9b
2.26
7.37
6.84
7.02
6,36
5.99
KVQuant- lb
1.00
168,90
1316.41
362,94
4223.37
127.07
KVQuant-1b++ sparse
1.32
11.18
9.56
16.04
22.87
10.53
CQ-8c8b
1.00
12.13
10.53
12.49
10.53
9.89
CQ-8c10b
1.27
9.12
8.23
9.03
8.01
7.46
C Effects of Channel-coupling and Fisher-guided Centroid Learning
We validate the effectiveness of our proposed channel-coupling and Fisher- Fisher-guided centroid learning by compressing LLaMA-7b KV cache to 1-bit and 2-bit, and present the perplexity results and quanti- zation errors A| - [ averaged over layers) on WikiTexx-22 under different CQ configurations Figure 5) As the number of coupled channels increases, perplexity improves significantly and approaches the FP16 performance. The quantization errors of keys and values also decrease as the number of coupled channels increase. Although Fisher-guided centroid learning increases the quantization error, it better preserves the salient activations and achieves lower perplexity.
14

---

LLaMA-7b 1-bit CQ
LLaMA-7o 2-bit CQ
LLaMA-7b 1-bit CQ
LLaMA-75 2-bit CQ
Papplsity WilTess2
2042
C1634
Purpesity on WiniTee-22
204
Qiamt Ener on WilTeeo 2
2000
Quant Enor WilTod-22
1000
302
1600
900
106
Lo1
6:37
4.80
750
CQ Corfig
CD Condg
Conti
Conti
Uniorm CQ
Unform CQ Key
Fishe-guiee
Unform CQ Valie
Fötcergided
grided
Figure 5: Perplexity and key/value quantization emror (averaged over all layers) of LLaMA-7b on WikiText-2. Channels coupling and Fisher-guided centroid learning are effective for improving perplexity.
D Ablation Study on More Models
study the effectiveness each component our proposed approach in Table perplexity of 2 models Mistral-7b and LLMMA-2-13b on WikiText-2 using CQ at 2 bits per activation, with varying number of channels coupled and comparing uniform centroid learning and Fisher- Fisher-guided centroid learning. Fisher- sesher-guieed centroids significantly improve model quality demonstrated by lower perplexity. With either uniform centroids or Fisher-guides guided centroids, perplexity improves as the number of coupled channels increases. Hence, proposed techniques of channel coupling and Fisher-guided centroid learning are both effective for maintaining model quality.
evaluate
Table 8: Perplexity of different models on WikiText-2 using CQ with varying number of coupled channels and fisher-guided centroids, Perplexity consistently improves the number of coupled channels increases.
Mistral-7b
LLaMA-2-13b
Rits Per Activation Num, of Chunnels Coupled
2
2
1
4
4
1
4
1
2
Fisher-guided Centroids
/
Perplexity L
97.76
16.29
5.42
5.34
5.20
5.11
890.42
171.96
6.62
6.06
4.91
4.81
E
Overhead of Centroid Learning and Storage
We present the computational overhead of centroid learning and the memory overhead centroid storage for CQ in Table 9 The centroid learning process of CQ consists of many independent k-means runs, which can be time-consuming on CPUs. Hence, we leverage a GPU implementation to accelerate the learning process. In all our experiments, iterations of k-means on a single GPU to obtain the centroids. The memory overhead of storing the centroids can be calculated for keys and values, / is the number of key/value attention heads, is the number of channels in single-head key/value activation embedding, and b is the bit width of quantized codes. As shown in Table 9 CQ can easily scale to large model sizes with low learning and memory overheads.
use k-means++ initialization and run 100
d 2 FP16 numbers, where the number of layers, is
Bits Per Activation Calculations for CQ
To calculate bits per activation, assume the batch size is 16 and the sequence length is 65,536. For CQ- bb,, the storage of quantized codes requires the following number of bits
16 x 65536 x x 2 x h x d x b/c
where is the number of layers, 2 is for key and value, h is the number of key/value attention heads, is the number of channels for single head of attention, and b/c the average number of bits per token per channel.
15

---

Table Learning and memory overhead of different CQ configurations and models. The number of centroid parameters are shown in millions, and the percentage to the model weights is shown in brackets.
Centroud Learning Time
Parameter Count in Centroids 4,96
CQ Config.
2c8b
4c8bb
4696
Rc8b
Sc106
2clbb
4c80
8:8b
8c10b
MaMAA-7
53 mins. 28 mins 62 mins 14 mins 104 mins
67. 1M 9
67.1 (0:99696)
134,22M (I (9025)
67. 11M (1 (..966))
268. 44M (1495)
LLaMA-13h
mins mins 96 mins 22 mins 160 mins
104 86M (0.8069))
104 BGM 80098)
209.72M (1 (.61256)
104 SCM (0,005))
419 A3M 2249)
1LMMA-2-33b
mins mins
104 8068))
86M 8099)
(6125))
104 (0.06)))
(349))))
Mistra-7h
mins mins 15 mins mins mins
16.78M (0.2522%)
16.78M (0.25556)
33.50M (0146456))
16.78M (0.2325)
67. 67.11M (0.9239))
The learned centroids of CQ are stored in FP16 format. Hence the storage of CQ centroids requires the following number bits
1 x x x 2 16
CQ has no other storage overhead besides quantized codes and centroids. per activation for CQ- ec b is calculated
Therefore, the average bits
16 65536
2
b/ h x x 2 16 x 16
16 x 65536 x
x 2 X
16
65536 b/c 2
65536
G
Generalizability of Learned Centroids
Centroids for CQ are learned once on a calibration dataset and can be used for different downstream tasks. We perform ablation study examine the effects of calibration data the quality of learned CQ centroids. We use 16 sequences of 2048 tokens from different datasets as the calibration set for CQ and evaluate 4 downstream tasks, and the results presented in Table TO Despite using different calibration datasets, CQ performs similarly in various downstream tasks. The results suggest that calibration language modeling datasets, such WikiTexx-2 and C4, provides transferable performance on downstream tasks.
Table 10: Zero-shot accuracy of LLaMA-2-7b with couple-quantized KV cache, calibrated using two different datasets WikiText-2 and C4. CQ displays similar accuracy in various downstream tasks, despite using different calibration datasets.
Downstream Task
Calibration Dataset
WinoGrande
PIQA
Arc-C
GSM8K CoT
WikiText-2 C4
68.27
77.91
43.34
14.71
CQ-2c8b
68.35
77.86
43.16
14.71
WikiText-2 C4
66.45
76.12
39.93
8,04
CQ-4c8b
66.22
76.61
39.93
8.34
WikiText-2 C4
55.01
71.22
30.20
1.74
CQ-8c8b
56.27
71.55
30.52
1.90
Table 11: Accuracy comparison of CQ and KIVI LongBench for LLaMA-2-7b. The results of KIVI are from [25].
Sliding Window Size
Qasper
QMSum
MultiNews
TREC
TriviaQA
SAMSam
LCC
RepoBench-P
FP16
9,52
21.28
3.51
66.00
87.72
41.69
66.66
59.82
KIVI-2h
32
9.26
20.53
0.97
66.00
87.42
42.61
66.22
59.67
CQ-4e80
32
87.72
$9.75
16

---

H
Comparison with KIVI
We perform empirical comparison of CQ with KIVI 22 using LLaMA-2-7b on LongBench (2) For both methods, we use 2-bit quantization and a Wldingg-wnnoon We compare CQ-4c8b against KIVI with group size of 32. The aceuracy results are presented in CQ mostly outperforms KIVI in preserving model accuracy.
full-precision cache of 32 tokens.
Table 11
I Passkey Retrieval
We compare the quantization quality of CQ and KVQuant by evaluating them the passkey retrieval task [28]. We follow the setup in [15. and measure the success rate of passkey retrieval for LLaMA- 2-7b its maximum context length of 4096. Table 12 presents the passkey retrieval results, where CQ outperforms or performs the same as KVQuant at various bit widths.
Table 12:
The passkey retrieval success rate of CQ and KVQuant at different quantization bit widths, for LLaMA-2-7b its maximum context length of 4096.
Bit Width
Success Rate
KVQuant-4b+I sparse
4.32
100%
KVQuant-4b
4.00
100
CQ-2c8b
4.00
100% %
KVQuant-2b+11 % sparse
2.32
94%
CQ-4c9b
2.26
98%
KVQuant-2b CQ-4c8b
2.00
0%
2.00
96 %
KVQuant- lb+1 % sparse CQ-8c10b
1.32
2%
1.27
78%
KVQuant-Ib CQ-8c8b
1.00
0%
1.00
12%
J Results on LLaMA 3
We evaluate the effectiveness of CQ and KVQuant for quantizing the KV cache of the LLaMA model [IO]. For both methods, quantize the KV cache of all tokens of the LLaMA-3-8) model, and perform the calibration on 16 sequences from the training set of WikiText-2 and evaluate perplexity on the test set, and benchmark on 3 downstream tasks: WinoGrande, PIQA and Arc Challenge. The results are presented Table [0]. CQ mostly outperforms KVQuant, especially lower bit widths.
Table 13: The effectiveness of CQ and KVQuant on LLaMA-3-888
Bits Per Activation
WikiText-2 PPL.
WinoGrande
PIQA t Arc-C 1
FP16
16
5.54
72.69
79.71
50.51
K KVQuant-4b
4.00
5.66
72.77
79.98
47.44
CQ-2c8b
4.00
5.58
73.16
78.84
49.83
KVQuant-2b
2.00
18.96
56.27
63.49
24.40
CQ-4c8b
2.00
6.09
69.22
78.62
44.03
KVQuant-Ib CQ-8c8b
1.00
22238.91
50.04
53.05
22.35
1.00
9.56
56.04
72.58
32.51
K Latency Measurements
We measure the prefill and decoding latency CQ, with and without a sliding window of full- precision cache, and compare against KIVI [25] and full full-precision FP16 cache. The latency mea-
17

---

surements are conducted on single A100-40GB GPU for LLaMA-2-75 with a batch size of 1 and prompt length of 2000 tokens The results are presented in Table CQ achieves prefill and decoding latency on with KIVI.
Table 14: Latency measurements of decoding LLaMA-2-7b with batch size of 1 and a prompt of 2000 tokens, averaged over 100 tokens, using FP16 KV cache, and CQ and KIVI quantized KV cache.
Full-precision Sliding-window Length
Prefill Time
Decoding Time (s)
FP16
0.853
0.0559 + 0.0044
KIVI-4b
32
1.483
0.0693 0.0212
KIVI-2b
1.291
0.0684 = 0.0213
CQ-2c8b
0
1.820
0.0695 + 0.0056
CQ-2c8b
32
1.926
0,0701 + 0.0057
CQ-4c8b
0
1.684
0.0704 + 0.0056
CQ-4c8b
32
1.790
0.0706 0.0058
CQ-8c8b
0
1.726
0.0670 + 0.0070
CQ-8c8b
32
1.857
0.0799 + 0.0066
L Extreme Sub-1-bit Quantization
We demonstrate the effectiveness of CQ by performing extreme sub-1-bit quantization with LLaMA- 7b. We evaluate the perplexity of CQ-16c which 12-bit code for every group of 16 coupled channels, averaging 0.81 bits per activation. Table 13 presents the results with comparison FP16 cache and KVQuant- lb. CQ preserves model quality reasonably under extreme KV cache compression.
Table 15: Perplexity of LLaMA-7b under extreme KV cache compression with CQ- 16c 12b (12 bits 16 coupled channels, averaging 0.81 bits per activation).
BPA
WikiText-2 1
C4
FP16
16
5.68
7.08
KVQuant-1b
1.00
321.58
168.90
CQ- 16c1bb
0.81
8.71
14.40
M Correlation Matrices and Scatter Plots
Figure and and show the correlation matrices for key and value channels of each layer of the LLaMA- 7b model on 262k tokens WikiText-2. Figure and d present the 2D scatter plots of key and value channel pairs of 4 layers of the LLaMA-7b model on 262k tokens of W
N Variations in Channel Correlation
As shown in Figure Gand 7 the amount of correlation between channels varies depending the layer. In Table 16 we study this variation by presenting the mean absolute correlation (MAC), excluding the diagonals, for different layers of LLaMA-7b 262K tokens of WikiText-2. Although the key correlation of the first few layers are significantly higher than the later layers, the key/value correlation of any layer is never very close zero, meaning channel coupling can be effective for any layer. Furthermore, the ablation study presented in Table also demonstrates the effectivenss of channel coupling for capturing the channel correlation.
Limitations
Low-bit quantization is type lossy model compression, which affects model quality including metrics such as perplexity and accuracy, and also impacts model safety including toxicity and bias ДП]. This work does not study the effects of extreme low-bit quantization of KV cache on the safety aspects of the model.
18

---

Iayer
Layer
Iayer
Layers
Isyer
Layer
Layer
15
Layers
Layer
Layer
Layer
Layer
Layer
Layer
Layer
15
Channel
Layer
Layer
Layer
Layer
Layer
Layer
Iayer
Layer
15
-0.
Layer
Layer
laper
Layer
Iayer
Layer
Iayer
Laye
15
15
310
15
310
15
31.0
15
310
15
31.0
15
310
15
310
15
31
Channel
Figure 6: Correlation matrix for the first 32 channels of pre-RoPE key activation embeddings of all L. LLaMA-7b layers on WikiText-2.
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
15
Layer
Layer
Laper
Layer
Layer
Layer
Layer
Layer
15
Channel
31
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
15
005
Layer
Layer
Layer
Layer
Layer
Layer
Layar
Laysr
15
15
310
15
310
15
31.0
15
310
15
310
15
310
15
310
15
31
Channel
Figure 7: Correlation matrix for the first 32 channels of value activation embeddings of all LLaMA-7b layers on 2
19

---

Layer
Carroll
Clamal
CamuLi
Layer
Charne
Chance
Chanra
Layer 16
Caanel
Chairel 15
Layer 32
Clossal
DhammelD
Charal2
Charred 4
Charme b
Chamal B
Chammel 30
Chamnel22
Chanrel 14
Figure 8: Scatter plots of pairs of channels in pre-RoPE key activation embeddings of LLaMA-7b layers WikiText-2.
Layer
Clanet
Cunnal
Cramel
Charcal 11
Layer
Charoe
Oharre
Camal
Camuts
Laver 15
Chaooe
Ohnrra
Ohine 15
Layer 32
Opaital 11
Channel
Chanid
Charnel
Charne 6
Chamat 5
Charnel 10
Chamid12
Channel
Figure 9: Scatter plots of pairs of channels value activation embeddings of LLaMA-7b layers on WikiText-2.
20

---

Table 16: The mean absolute correlation (MAC), excluding the diagonals, of key/value channels in different layers of LLaMA-7b, computed on 262K tokens of WikiText-2.
Layer
2
5
6
7
8
Key
0.407
0.212
0.193
0.178
0.114
0.113
0.115
0.122
Value
0.071
0.084
0.061
0.073
0.055
0.056
0.056
0.057
Layer
9
10
11
12
13
14
15
16
Key
0.131
0.138
0.090
0.098
0.094
0.141
0.109
0.114
Value
0.061
0.067
0.047
0.042
0.065
0.065
0.062
0.035
Layer
17
18
19
20
21
22
23
24
Key
0.136
0.111
0.091
0.101
0.102
0.156
0.071
0.094
Value
0.039
0.038
0.052
0.070
0.031
0.038
0.061
0.074
Layer
25
30
31
32
Key
0.107
0.069
0.057
0.103
0.095
0.097
0.100
0.090
Value
0.027
0.044
0.038
0.072
0.070
0.035
0.105
0.090
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All of our claims made in the abstract and the introduction are supported by the results presented in "Experiments" section, and they accurately reflect the paper's contributions and scope as shown in the "Background" and "Methodology" sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have made clear the limitations of our proposal, and included a separate "Limitations & Broader Impacts" section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
22

---

Answer:
[NA]
Justification:
We present no new theoretical results this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper fully discloses all information for reproducing the experimental
results, including algorithmic details, parameters, experimental setup, and baselines in the "Methodology" and "Experiments" sections.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code is proprietary xMAD.ai. However, anyone is able reproduce our results using the algorithm and procedure we describe the paper. We will provide a mechanism reproduce numbers in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips..om cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /tips.cc////p/ http:/pblccggiideeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We specify all relevant experimental details in the paper for understanding the results the "Experiments" section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars where applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information the resources needed for reproducing the experiments in the "Experiments" section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and can confirm that paper conforms with it in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We include an "Limitations & Broader Impacts" section to discuss our potential societal impacts.
Guidelines:
25

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We release no new models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit and respect the licenses and terms of the assets we use in our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets are documented where applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "HW2s1hULLz", "reviewer": "Reviewer_p3Eo", "summary": "LLM inference typically involves Key-Value (KV) caching to avoid recomputation. However, the KV cache size grows with batch and context length, imposing bottlenecks in memory footprint and inference speed. Quantization can be employed to reduce the size of the KV cache. Instead of channel-wise independent quantization, this work proposes Coupled Quantization to quantize the KV cache by grouping the channels and performing vector quantization by learning a joint codebook with a calibration corpus. This enables additional compression as the entropy of each channel is distributed over the code bits.", "questions": "- How are shared memory bank conflicts handled during centroid lookup?", "limitations": "The authors have included a separate Limitations & Broader Impacts section in the manuscript.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- With the proposed method, more aggressive compression of the KV cache is enabled.\n- System support is provided.", "weaknesses": "- Table 2 shows that CQ only outperforms the baselines in the 1bit regime, which already suffers great accuracy loss. For relatively “safer” bitwidths, it is on-par or sub-par to baselines.\n- As far as I know, the common sense reasoning tasks utilized in the experiments(Windogrande, PIQA, ARC, etc.) are usually multiple choice and do not require long generation (also, they are relatively easy tasks). It could be hard to verify the effectiveness of the proposed method for long generation setting(for instance, experiments on benchmarks such as GSM8k with chain-of-thought, which is a relatively hard task), where the incoming KVs are also quantized.\n- Baselines are absent in Table 3.\n- Testing the proposed method on long context benchmarks or multi-hop retrieval tasks, such as the RULER[1] dataset will improve the quality of the paper.\n- The proposed method involves offline learning of centroids using a calibration dataset. The quality of the centroids may not hold under distribution shifts.\n\nReference:\n\n[1] Hsieh et al, “RULER: What's the Real Context Size of Your Long-Context Language Models?”, arXiv 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their careful consideration of our work and for providing valuable feedback. We have addressed their comments in detail below.\n\n**[W1] Table 2 shows that CQ only outperforms the baselines in the 1bit regime, which already suffers great accuracy loss.**\n\n- It is essential to consider the bits per activation metric when comparing quantization methods in Table 2. Under identical whole-number bit widths (4.00 bits, 2.00 bits, and 1.00 bit), CQ consistently surpasses KVQuant in average accuracy, particularly in the 2-bit and 1-bit regimes where the improvement exceeds 10%. When considering similar bit widths, CQ outperforms or matches the performance of sparsity-based KVQuant.\n- We highlight that, in Table 2, CQ-8c8b (with a bit width of 1) outperforms KVQuant-2b (with a bit width of 2) in average accuracy despite using half the memory, demonstrating the efficacy of our proposed approach.\n- Additionally, it's important to note that KVQuant+1% sparse is a sparsity-based method that relies on storing outlier activations in a sparse matrix. This approach can potentially introduce latency overhead and deployment complexities. In contrast, our CQ method maintains a fully dense representation, avoiding these potential issues.\n\n**[W2] Evaluations with long generation.**\n\n- We present additional experimental results below with long-context datasets comparing CQ and KVQuant using Llama-2-7b on GSM8K with chain-of-thought (CoT) and MMLU with CoT Fewshot (gsm8k_cot, mmlu_flan_cot_fewshot_humanities, mmlu_flan_cot_fewshot_stem, mmlu_flan_cot_fewshot_social_sciences, mmlu_flan_cot_fewshot_other from lm-evaluation-harness). In long-context settings, CQ mostly outperforms or is comparable to KVQuant under the same bit width.\n\n|  | BPA | GSM8K CoT | MMLU (STEM) CoT Fewshot | MMLU (Humanities) CoT Fewshot | MMLU (Social Sciences) CoT Fewshot | MMLU (Other) CoT Fewshot |\n|---|---|---|---|---|---|---|\n| KVQuant-4b+1% sparse | 4.32 | 14.33 | 31.04 | 41.12 | **48.37** | 55.43 |\n| CQ-2c8b | 4.00 | **14.71** | **33.73** | **43.44** | 47.77 | **56.01** |\n| KVQuant-2b+1% sparse | 2.32 | **10.31** | **28.06** | 35.64 | 42.43 | **46.39** |\n| CQ-4c9b | 2.26 | **10.31** | 27.76 | **35.91** | **44.51** | 45.75 |\n| KVQuant-2b | 2.00 | 2.27 | 9.85 | 12.55 | 20.18 | 19.94 |\n| CQ-4c8b | 2.00 | **8.04** | **25.67** | **30.89** | **45.4** | **41.94** |\n| KVQuant-1b+1% sparse | 1.32 | 2.27 | 10.75 | 14.09 | 20.77 | 19.94 |\n| CQ-8c10b | 1.27 | **2.35** | **13.13** | **21.81** | **28.19** | **26.98** |\n| KVQuant-1b | 1.00 | 0.68 | 0 | 0 | 0 | 0 |\n| CQ-8c8b | 1.00 | **1.74** | **5.37** | **11.39** | **20.77** | **16.72** |\n\n**[W3] Baselines are absent in Table 3.**\n\n- To add more baselines to Table 3, we present additional experiments on LongBench below comparing CQ and KIVI with sliding window full-precision cache using Llama-2-7b. For KIVI, we use 2-bit quantization, a full-precision sliding window (residual length) of 32 tokens, and a group size of 32. For CQ, we use 2-bit (4c8b) quantization and a sliding window size of 32. CQ mostly outperforms KIVI across different tasks.\n\n|  | Sliding Window Size | Qasper | QMSum | MultiNews | TREC | TriviaQA | SAMSum | LCC | RepoBench-P |\n|---|---|---|---|---|---|---|---|---|---|\n| FP16 | - | 9.52 | 21.28 | 3.51 | 66.00 | 87.72 | 41.69 | 66.66 | 59.82 |\n| KIVI-2b | 32 | 9.26 | 20.53 | 0.97 | **66.00** | 87.42 | **42.61** | 66.22 | 59.67 |\n| CQ-4c8b | 32 | **9.58** | **20.87** | **1.93** | **66.00** | **87.72** | 41.13 | **66.57** | **59.75** |\n\n**[W4] Testing the proposed method on long context benchmarks or multi-hop retrieval tasks.**\n\n- Please see the first table above for evaluations on long-context benchmarks including GSM8k with CoT and MMLU with CoT Fewshot.\n- We tried our best to evaluate our method on the RULER dataset in the limited timeframe of the rebuttal. However, we could not run RULER on our servers due to compatibility issues with docker. We will keep trying our method on RULER during the discussion period.\n\n**[W5] The quality of the centroids may not hold under distribution shifts.**\n\n- We have added an ablation study as follows to suggest that calibration on language modeling datasets provides transferable performance on downstream tasks. We use 16 sequences of 2048 tokens from WikiText-2 and C4 as the calibration set and evaluate CQ on 4 downstream tasks. \n- Despite using different calibration datasets, CQ performs similarly in various downstream tasks.\n\n|  | Calibration Dataset | WinoGrande | PIQA | Arc-C | GSM8K CoT |\n|---|---|---|---|---|---|\n| CQ-2c8b | WikiText-2 | 68.27 | 77.91 | 43.34 | 14.71 |\n|  | C4 | 68.35 | 77.86 | 43.16 | 14.71 |\n| CQ-4c8b | WikiText-2 | 66.45 | 76.12 | 39.93 | 8.04 |\n|  | C4 | 66.22 | 76.61 | 39.93 | 8.34 |\n| CQ-8c8b | WikiText-2 | 55.01 | 71.22 | 30.2 | 1.74 |\n|  | C4 | 56.27 | 71.55 | 30.52 | 1.9 |\n\n**[Q1] How are shared memory bank conflicts handled during centroid lookup?**\n\n- Due to a relatively high number of centroids (256 centroids for 8-bit codes and 1024 centroids for 10-bit codes), we did not notice shared memory bank conflicts to significantly impact system performance. We thank the reviewer for raising this insightful question and will continue to explore potential optimizations in this area.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vadn2W9weu", "reviewer": "Reviewer_KwWk", "summary": "The authors have addressed the KV-cache compression problem by providing a finer quantization level. The KV-cache can pose a significant barrier to the inference of most autoregressive language models, a challenge that has been well studied in recent publications at ICML and NeurIPS. This paper introduces a novel approach by coupling multiple key/value channels together for quantization, exploiting their interdependence to encode the activations in a more information-efficient manner.", "questions": "- There is additional overhead regarding storing the centroid for each coupled key/value pair, making it difficult to track. I would like you to mention this overhead and explain how you set those values.", "limitations": "- There is additional overhead regarding storing the centroid for each coupled key/value pair, which can complicate tracking and management. It would be beneficial to address this overhead and provide details on how these values are set.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The method is novel and demonstrates comparable accuracy to KVQuant, one of the pivotal approaches in this field.\n- It includes a substantial number of experiments to validate accuracy.\n- The quantization method introduced here is novel compared to other approaches. Additionally, the implementation in PyTorch represents a significant contribution.", "weaknesses": "- The code is not available. For research on KV cache, it is important to have the code available.\n\n- The section describing the random variables and entropy, specifically line 120, does not explicitly describe the random variables in mathematical notation. This should be revised for clarity. I would like to see this section more polished.\n\n- I believe the LLAMA3 model was available before the NeurIPS submission. Since that time, the authors may have extended their findings to these models. I would like to see the performance of your model in that setting.\n\n- I want to see how the runtime of your method compares to other methods. Recent works, like **QJL**, include good plots for token-wise generation time or end-to-end timing. Since you compare with KVQuant, it is also good to compare with **KIVI**, as it is one of the best methods. I recommend comparing with QJL and **KIVI**, and plotting the runtime alongside these methods.\n\n- I highly recommend the authors run their code on longer context datasets. LongBench could be a great example to evaluate its performance compared to other methods. I would suggest that perplexity is not the best metric for comparison.\n\nRelevant papers: [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache], [QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead]\n\n**If you address the concerns regarding the experiments and provide a broader comparison to the other methods, I would increase my score.**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude to the reviewer for their thoughtful comments and suggestions. We address the reviewer's concerns as follows.\n\n**[W1] The code is not available.**\n\n- We will open source our code during the camera-ready phase. To provide additional context, we have included expanded experimental results and implementation profiling details below.\n\n**[W2] Section 3 should be revised for clarity.**\n\n- We will revise and polish section 3 to describe the random variables and entropy in mathematical notations. We appreciate the reviewer's careful reading of our paper and their valuable feedback.\n\n**[W3] Extension to LLAMA3.**\n\n- We present additional experimental results comparing CQ and KVQuant on LLaMA-3-8b in the table below. CQ mostly outperforms KVQuant, especially in lower bit widths. We would like to gently remind that LLaMA-3 was released only one month before the NeurIPS deadline.\n\n|  | BPA | WikiText-2 PPL | WinoGrande | PIQA | Arc-C |\n|---|---|---|---|---|---|\n| FP16 | 16 | 5.54 | 72.69 | 79.71 | 50.51 |\n| KVQuant-4b | 4 | 5.66 | 72.77 | **79.98** | 47.44 |\n| CQ-2c8b | 4 | **5.58** | **73.16** | 78.84 | **49.83** |\n| KVQuant-2b | 2 | 18.96 | 56.27 | 63.49 | 24.4 |\n| CQ-4c8b | 2 | **6.09** | **69.22** | **78.62** | **44.03** |\n| KVQuant-1b | 1 | 22238.91 | 50.04 | 53.05 | 22.35 |\n| CQ-8c8b | 1 | **9.56** | **56.04** | **72.58** | **32.51** |\n\n**[W4] Comparison with QJL and KIVI.**\n\n- We tried our best to compare with QJL in the limited timeframe of the rebuttal. However, we ran into some issues due to incompatibilities of the QJL codebase with our hardware. Specifically, we ran into the following error: `File \".../QJL/models/llama3_utils_qjl.py\", line 138, in build_sketch\n    self.key_states_norm = torch.norm(key_states, dim=-1)\nRuntimeError: CUDA error: limit is not supported on this architecture`. This could be caused by incompatibilities of our V100 GPUs with the code. We will try to resolve this issue during the discussion period. We would like to kindly remind that QJL was first published online in June, which is after the NeurIPS deadline.\n- We present additional experiments in the table below comparing CQ with KIVI on accuracy of LongBench with Llama-2-7b. For KIVI, we use 2-bit quantization, a full-precision sliding window (residual length) of 32 tokens, and a group size of 32. For CQ, we use 2-bit quantization and a sliding window size of 32. CQ mostly outperforms KIVI across different tasks.\n\n|  | Sliding Window Size | Qasper | QMSum | MultiNews | TREC | TriviaQA | SAMSum | LCC | RepoBench-P |\n|---|---|---|---|---|---|---|---|---|---|\n| FP16 | - | 9.52 | 21.28 | 3.51 | 66.00 | 87.72 | 41.69 | 66.66 | 59.82 |\n| KIVI-2 | 32 | 9.26 | 20.53 | 0.97 | **66.00** | 87.42 | **42.61** | 66.22 | 59.67 |\n| CQ-4c8b | 32 | **9.58** | **20.87** | **1.93** | **66.00** | **87.72** | 41.13 | **66.57** | **59.75** |\n\n- In the table below, we present additional latency measurements of CQ with comparison to KIVI and FP16 cache. We use Llama-2-7b with a batch size of 1 and a prompt of 2000 tokens to decode 100 tokens.\n\n|  | Full-precision Sliding Window Length | Prefill Time (s) | Decoding Time (s) |\n|---|---|---|---|\n| FP16 | - | 0.853 | 0.0559 +/- 0.0044 |\n| KIVI-4b | 32 | 1.483 | 0.0693 +/- 0.0212 |\n| KIVI-2b | 32 | 1.291 | 0.0684 +/- 0.0213 |\n| CQ-2c8b | 0 | 1.820 | 0.0695 +/- 0.0056 |\n| CQ-2c8b | 32 | 1.926 | 0.0701 +/- 0.0057 |\n| CQ-4c8b | 0 | 1.684 | 0.0704 +/- 0.0056 |\n| CQ-4c8b | 32 | 1.790 | 0.0706 +/- 0.0058 |\n| CQ-8c8b | 0 | 1.726 | 0.0670 +/- 0.0070 |\n| CQ-8c8b | 32 | 1.857 | 0.0799 +/- 0.0066 |\n\n**[W5] Longer context datasets such as LongBench.**\n\n- Please see the table above for CQ's results on LongBench with comparison to KIVI.\n- We present additional experiments with longer context datasets comparing CQ and KVQuant using Llama-2-7b on GSM8K with chain-of-thought (CoT) and MMLU with CoT Fewshot (gsm8k_cot, mmlu_flan_cot_fewshot_humanities, mmlu_flan_cot_fewshot_stem, mmlu_flan_cot_fewshot_social_sciences, mmlu_flan_cot_fewshot_other from lm-evaluation-harness). In long-context settings, CQ mostly outperforms or performs similarly to KVQuant under the same bit width. \n\n|  | BPA | GSM8K CoT | MMLU (STEM) CoT Fewshot | MMLU (Humanities) CoT Fewshot | MMLU (Social Sciences) CoT Fewshot | MMLU (Other) CoT Fewshot |\n|---|---|---|---|---|---|---|\n| KVQuant-4b+1% sparse | 4.32 | 14.33 | 31.04 | 41.12 | **48.37** | 55.43 |\n| CQ-2c8b | 4.00 | **14.71** | **33.73** | **43.44** | 47.77 | **56.01** |\n| KVQuant-2b+1% sparse | 2.32 | **10.31** | **28.06** | 35.64 | 42.43 | **46.39** |\n| CQ-4c9b | 2.26 | **10.31** | 27.76 | **35.91** | **44.51** | 45.75 |\n| KVQuant-2b | 2.00 | 2.27 | 9.85 | 12.55 | 20.18 | 19.94 |\n| CQ-4c8b | 2.00 | **8.04** | **25.67** | **30.89** | **45.4** | **41.94** |\n| KVQuant-1b+1% sparse | 1.32 | 2.27 | 10.75 | 14.09 | 20.77 | 19.94 |\n| CQ-8c10b | 1.27 | **2.35** | **13.13** | **21.81** | **28.19** | **26.98** |\n| KVQuant-1b | 1.00 | 0.68 | 0 | 0 | 0 | 0 |\n| CQ-8c8b | 1.00 | **1.74** | **5.37** | **11.39** | **20.77** | **16.72** |\n\n**[Q1, L1] Overhead regarding storing the centroids. I would like you to mention this overhead and explain how you set those values.**\n\n- We kindly refer the reviewer to Section A.4 for a discussion on the overhead of storing and learning the centroids. We have reported the number of centroid parameters and the learning time for each CQ configuration and model, and described how the overhead is calculated in detail.\n- CQ only has two hyperparameters: the number of channels and the number of bits in a code. These parameters are explicitly reported in our experimental results. There is no need of extensive hyperparameter tuning for our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uqcMZoSvhk", "reviewer": "Reviewer_qfND", "summary": "The paper explores the idea of compressing the KV cache in Transformer models through quantization; specifically, the authors propose Coupled Quantization, which quantizes multiple KV channels together in order to exploit their interdependency. The gains are guaranteed by the fact that the joint entropy is smaller than or equal to the sum of marginal entropies of the channels. The channels are coupled in continuous groups. The experimental results show that CQ maintains the model quality and improves the inference throughput, even with high compression rates.", "questions": "Does CQ affect the training process or is it purely an inference-time optimization?\n\nDo you envision a way of using non-contiguous channels in the coupling, perhaps with some limited search of finding groups of channels with higher interdependency?\n\nCan you clarify the meaning of bold and underline in Tables?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper uses known information theoretic techniques and applies them to KV cache quantization.\nThe method is well founded in information theory.\nThe experimental evaluation is comprehensive, showing significant improvements in inference throughput.\nThe method is relevant for a real bottleneck in deploying LLMs, which is the GPU memory usage due to KV caching.", "weaknesses": "While the experiments are extensive, they are focused on the Llama family of models. \nThe novelty of the paper is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their careful review of our paper and the insightful suggestions. We address your concerns as follows.\n\n**[W1] Focused on the Llama family of models. -- We have added Mistral model results!**\n\n- We would like to draw the reviewer's attention to the results on Mistral in Tables 1, 2, and 6.\n\n**[W2] The novelty of the paper is limited.**\n\nAlthough novelty is a multifaceted concept in research, we believe it can be viewed roughly from two aspects: ***empirical novelty***, which involves the discovery of properties unknown to the community (e.g. the lottery ticket hypothesis [1]), and ***technical novelty***, which refers to the development of new solutions (e.g. Attention [2]). We argue that our work exhibits both forms of novelty, as detailed below.\n\n- **Empirically Novel Observation that KV Channels are Highly Interdependent:** To the best of our knowledge, we are the first to highlight the phenomenon that channels of KV cache exhibit high amounts of mutual information or correlation. This discovery opens up new avenues for KV cache compression.\n- **Technically Novel Approach that Enables Extreme KV Cache Compression:** Based on our novel observation and concepts from Information Theory, we propose a new approach for KV cache quantization by coupling multiple KV channels. Existing approaches such as KVQuant and KIVI quantize KV cache channel-wise or token-wise independently, which cannot take advantage of the high mutual information shared between channels. By exploiting the interdependency among channels, we enable KV cache compression rates previously difficult or impossible to achieve, i.e., 1-bit quantization of KV cache.\n\n\n**[Q1] Does CQ affect the training process or is it purely an inference-time optimization? --- It does not!**\n\n- CQ is a post-training quantization (PTQ) approach that does not affect the training process of the models. CQ can improve the inference efficiency by saving memory and increasing batch size.\n\n**[Q2] Do you envision a way of using non-contiguous channels in the coupling, perhaps with some limited search of finding groups of channels with higher interdependency? --- Yes!**\n\n- Coupling non-contiguous, highly interdependent channels will likely further improve the quantization accuracy. One potential way of achieving that is to place channels with the highest correlation into the same channel group through a greedy search. We leave this for future work due to the difficulty in system implementation and optimizations.\n- We thank the reviewer for pointing us in this direction.\n\n**[Q3] Can you clarify the meaning of bold and underline in Tables? --- Sure!**\n\n- The bolded number is the best perplexity/accuracy achieved under the same bit-width, while the underlined number is the second best. We will clarify this in the final paper.\n\n**References**\n\n[1] Frankle, Jonathan, and Michael Carbin. \"The lottery ticket hypothesis: Finding sparse, trainable neural networks.\" arXiv preprint arXiv:1803.03635 (2018).\n\n[2] Vaswani, Ashish, et al. \"Attention is all you need.\" Advances in neural information processing systems 30 (2017).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "119Qxt6OmP", "reviewer": "Reviewer_dwgV", "summary": "This paper identifies a significant interdependency among distinct channels of key/value activation tensors in Transformer models. By quantizing multiple key/value channels together using joint entropy, the authors achieve high inference throughput while maintaining model quality. In extreme cases, the KV cache can be quantized down to 1 bit.", "questions": "This paper is an enjoyable read, and I believe grouping K/V channels is an effective way to reduce memory storage burden and improves inference performance. Nevertheless, I have some questions listed below:\n\n1. This work primarily focuses on quantizing the KV cache. Can it be combined with other quantization methods such as weight/activation quantization like SmoothQuant and AWQ? The authors mention that these methods are orthogonal to this work; could you elaborate on how they might be incorporated together?\n2. \"We employ the 'binning' trick [17] to estimate entropy. We first observe an empirical distribution of key and value channels by saving the KV cache on a dataset\". Does this mean that the estimation needs to be done for each dataset and cannot be reused? Which dataset was used for the centroid learning in Table 8?\n3. Section 3.3 seems to be a standard practice for fusing (de)quantization operations with other operations. The lookup table also does not seem efficient. Is it possible to conduct an ablation study or profiling to understand the overhead of centroid loading? Can these centroids be reused instead of being frequently loaded from global memory?\n4. The experiment in Section 4.4 with large batch sizes is more relevant for online serving scenarios. It would be beneficial to integrate your method with [vllm](https://blog.vllm.ai/2023/06/20/vllm.html) or other serving frameworks and use real traces to evaluate performance. From Figure 4, it seems all CQ methods perform worse than the fp16 version with small batch sizes. Can you explain why? Is it due to the overhead of (de)quantization?\n5. \"This sliding window of full-precision cache only introduces a small constant memory overhead for each sequence.\" What about the latency overhead? How does the sliding window affect inference latency?\n6. Based on Figure 1, it seems that only by combining the sliding window and CQ can a similar perplexity to the fp16 version be achieved. How do you determine the best sliding window size to achieve the optimal tradeoff? Also, what happens if the number of coupled channels exceeds 8?", "limitations": "Refer to the Weaknesses and Questions sections. Additionally, here are some minor issues:\n\n* It would be useful to include latency experiments in Section 4.4 in addition to throughput, which can help readers better understand the benefits of the proposed method.\n* It would be more helpful to show the actual memory usage in GB instead of the parameter count in Table 8.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The observation regarding the interdependency of KV channels is actually interesting and insightful.\n* The proposed joint entropy-based quantization is intuitive and effective.\n* The presentation is easy to follow and supported by comprehensive experiments.", "weaknesses": "* The improvement in inference throughput over the fp16 version is primarily observed with large batch sizes due to the high (de)quantization overhead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the support of our paper and the insightful suggestions. We address the reviewer's concerns as follows.\n\n**[W1, L1] Include latency experiments.**\n\n- In the table below, we present additional latency measurements of CQ with comparison to FP16 cache and KIVI. We use Llama-2-7b with a batch size of 1 and a prompt of 2000 tokens to decode 100 tokens.\n- We observe that CQ achieves comparable efficiency with KIVI. We plan to further optimize CQ at system level to reduce latency and improve throughput.\n\n|  | Full-precision Sliding Window Length | Prefill Time (s) | Decoding Time (s) |\n|---|---|---|---|\n| FP16 | - | 0.853 | 0.0559 +/- 0.0044 |\n| KIVI-4b | 32 | 1.483 | 0.0693 +/- 0.0212 |\n| KIVI-2b | 32 | 1.291 | 0.0684 +/- 0.0213 |\n| CQ-2c8b | 0 | 1.820 | 0.0695 +/- 0.0056 |\n| CQ-2c8b | 32 | 1.926 | 0.0701 +/- 0.0057 |\n| CQ-4c8b | 0 | 1.684 | 0.0704 +/- 0.0056 |\n| CQ-4c8b | 32 | 1.790 | 0.0706 +/- 0.0058 |\n| CQ-8c8b | 0 | 1.726 | 0.0670 +/- 0.0070 |\n| CQ-8c8b | 32 | 1.857 | 0.0799 +/- 0.0066 |\n\n**[Q1] Combining with other weight/activation quantization methods.**\n\n- Our KV cache quantization method can be combined with other weight/activation quantization methods. In the standard CQ calibration process, the centroids are learned using the KV cache of the full-precision model based on a calibration dataset. \n- To combine CQ with other weight/activation quantization methods, we use the KV cache produced by the quantized model for centroid learning to minimize the distortions introduced by CQ.\n\n**[Q2] Does this mean that the estimation needs to be done for each dataset and cannot be reused? Which dataset was used for the centroid learning in Table 8? --- No, centroid learning only needs to be done once.**\n\n- Centroids of CQ are learned once during the calibration phase, and can be used for different downstream tasks.\n- For Table 8, the centroids of CQ are learned on a set of 16 sequences from WikiText-2, each with 2048 tokens. We have specified this on line 243 in the paper. The centroids are learned only once and used for all perplexity and accuracy experiments in the paper.\n- We present additional experiments below on the accuracy of CQ with different calibration datasets. We use 16 sequences of 2048 tokens from WikiText-2 and C4 as calibration set and evaluate CQ on 4 downstream tasks. \n- Despite using different calibration datasets, CQ performs similarly in various downstream tasks.\n\n|  | Calibration Dataset | WinoGrande | PIQA | Arc-C | GSM8K CoT |\n|---|---|---|---|---|---|\n| CQ-2c8b | WikiText-2 | 68.27 | 77.91 | 43.34 | 14.71 |\n|  | C4 | 68.35 | 77.86 | 43.16 | 14.71 |\n| CQ-4c8b | WikiText-2 | 66.45 | 76.12 | 39.93 | 8.04 |\n|  | C4 | 66.22 | 76.61 | 39.93 | 8.34 |\n| CQ-8c8b | WikiText-2 | 55.01 | 71.22 | 30.2 | 1.74 |\n|  | C4 | 56.27 | 71.55 | 30.52 | 1.9 |\n\n**[Q3] Ablation study on the overhead of centroid loading.**\n\n- We perform an additional ablation study to understand the overhead of centroid loading. We profile the KQ multiplication kernel for CQ-2c8b with a single query and 4K or 16K keys using Llama-2-7b hidden dimensions. We enable and disable loading centroids from global memory to shared memory. The results shown in the table below are average over 1000 kernel runs.\n- As shown in the table, centroid loading does not significantly contribute to the overall latency. The latency primarily comes from reading quantized KV cache and queries from global memory, and writing the results to global memory. Hence reusing the centroids will not significantly reduce the latency.\n\n|  | Sequence Length = 4000 | Sequence Length = 16000 |\n|---|---|---|\n| Centroid Loading | 214.549 us +/- 12.353 | 556.184 us +/- 10.970 |\n| No Centroid Loading | 207.896 us +/- 11.544 | 548.111 us +/- 7.917 |\n\n**[Q4] Integration with vllm or other serving frameworks. From Figure 4, it seems all CQ methods perform worse than the fp16 version with small batch sizes. Can you explain why? Is it due to the overhead of (de)quantization?**\n\n- We anticipate compatibility between our proposed CQ and PagedAttention in vLLM, and believe their integration is a promising avenue for future exploration. Given the implementation difficulty, we leave this to future investigations.\n- CQ has higher latency than FP16 in small batch sizes due to the overhead of (de)quantization. As suggested by Reviewer rmbd, the efficiency of our kernel can be further optimized by loading centroids of more channel groups at once into shared memory, and adjusting the thread block assignment and reduction strategies. We will incorporate these improvements after the rebuttal period.\n\n**[Q5] How does the sliding window affect inference latency?**\n\n- Please see the first table for a latency comparison between sliding window and no sliding window. Sliding window full-precision cache does not significantly contribute to the overall latency in the prefill or the decoding stage.\n\n\n**[Q6] How do you determine the best sliding window size to achieve the optimal tradeoff? Also, what happens if the number of coupled channels exceeds 8?**\n\n- The sliding window size needs to be larger for lower bit widths to compensate for the precision loss. For CQ-2c8b (4-bit quantization), a small window size of 16 tokens or no window may suffice, but for CQ-8c8b (1-bit quantization), a larger window of 128 tokens may be necessary for preserving quality.\n- The number of coupled channels can exceed 8. We present additional experimental results below using LLaMA-7b with CQ-16c12b (12 bits per 16 coupled channels, averaging 0.81 bits per activation).\n\n|  | BPA | WikiText-2 PPL | C4 PPL |\n|---|---|---|---|\n| CQ-16c12b | 0.81 bits | 8.71 | 14.40 |\n\n**[L2] It would be more helpful to show the actual memory usage in GB instead of the parameter count in Table 8.**\n\n- We will include the actual memory usage in the final paper. The memory overhead of centroids can be calculated as (parameter count x 2) bytes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g1qRnzsG4G", "reviewer": "Reviewer_rmbd", "summary": "This paper presents a novel approach to KV Cache compression in Large Language Models (LLMs) called Coupled Quantization (CQ). The authors analyze the correlation between different channels in the KV Cache from an information entropy perspective, revealing significant interdependencies. Leveraging this insight, they propose a multi-channel joint non-uniform quantization method that achieves superior compression performance compared to previous per-channel quantization approaches. Experimental results demonstrate that by combining CQ with a sliding window approach, the KV Cache can potentially be compressed to 1 bit while maintaining model quality.", "questions": "1. Quantization Setting: Is the KV Cache quantized during the prefill stage, or only during the decoding stage (after full-precision prefill computation), similar to the KIVI approach?\n2. Channel Grouping Strategy: The paper chooses to group adjacent continuous channel groups together. Would coupling non-adjacent channels with high mutual information potentially bring more benefits?\n3. CUDA Kernel Implementation Questions: The compute pattern described for kernel fusion maybe potentially inefficient. Given the relatively small size of centroids for each channel group (e.g., 256 bytes for 4-bit quantization), is shared memory size truly a limiting factor? Could the efficiency be improved by adjusting the thread block assignments and reduction strategies for operations like $QK^T$ and $SV$?", "limitations": "Based on my review of the paper, I believe the authors have partially addressed limitations and potential negative societal impacts.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Thorough Analysis and Clear Motivation: The paper provides a detailed and intuitive explanation of the inter-channel correlation using information entropy, solidly justifying the design of Coupled Quantization. The motivation and effectiveness of the proposed method are well-articulated and supported by comprehensive experimental results.\n﻿2. Good Performance at Low Bit-widths: CQ demonstratesadvantages over previous per-channel quantization methods without introducing additional overhead.\n﻿3. Practical Efficiency Gains: The authors demonstrate substantial improvements in inference throughput and batch size compared to the FP16 baseline, highlighting the practical benefits of their approach.", "weaknesses": "1. Limited Evaluation Tasks: The main experiments focus on datasets (WinoGrande, PIQA, and ARC-C) that primarily consist of short-context QA or multiple-choice questions. Testing KV Cache quantization on these tasks may not fully demonstrate its benefits, as KV Cache compression is most relevant in long-text scenarios.\n﻿2. Incomplete Explanation of Channel Correlation: While the visualizations in Figure 2 and the appendix show some layers have significant channel correlation, this is mainly evident in the first few layers. The paper doesn't fully explain the existence and variability of channel correlation across different layers, which may raise questions about the method's general applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful review and invaluable feedback. We address the reviewer's concerns as follows.\n\n**[W1] Limited Evaluation Tasks: KV Cache compression is most relevant in long-text scenarios.**\n- We appreciate the reviewer's suggestion to test KV cache quantization in long-text scenarios. We present additional experiments in long-text settings below. Specifically, we test CQ and KVQuant using Llama-2-7b on GSM8K with chain-of-thought (CoT) and MMLU with CoT Fewshot (gsm8k_cot, mmlu_flan_cot_fewshot_humanities, mmlu_flan_cot_fewshot_stem, mmlu_flan_cot_fewshot_social_sciences, mmlu_flan_cot_fewshot_other from lm-evaluation-harness).\n- In long-text settings, CQ mostly outperforms or performs similarly to KVQuant under the same bit width. \n\n|  | BPA | GSM8K CoT | MMLU (STEM) CoT Fewshot | MMLU (Humanities) CoT Fewshot | MMLU (Social Sciences) CoT Fewshot | MMLU (Other) CoT Fewshot |\n|---|---|---|---|---|---|---|\n| KVQuant-4b+1% sparse | 4.32 | 14.33 | 31.04 | 41.12 | **48.37** | 55.43 |\n| CQ-2c8b | 4.00 | **14.71** | **33.73** | **43.44** | 47.77 | **56.01** |\n| KVQuant-2b+1% sparse | 2.32 | **10.31** | **28.06** | 35.64 | 42.43 | **46.39** |\n| CQ-4c9b | 2.26 | **10.31** | 27.76 | **35.91** | **44.51** | 45.75 |\n| KVQuant-2b | 2.00 | 2.27 | 9.85 | 12.55 | 20.18 | 19.94 |\n| CQ-4c8b | 2.00 | **8.04** | **25.67** | **30.89** | **45.4** | **41.94** |\n| KVQuant-1b+1% sparse | 1.32 | 2.27 | 10.75 | 14.09 | 20.77 | 19.94 |\n| CQ-8c10b | 1.27 | **2.35** | **13.13** | **21.81** | **28.19** | **26.98** |\n| KVQuant-1b | 1.00 | 0.68 | 0 | 0 | 0 | 0 |\n| CQ-8c8b | 1.00 | **1.74** | **5.37** | **11.39** | **20.77** | **16.72** |\n\n**[W2] Incomplete Explanation of Channel Correlation: The paper doesn't fully explain the existence and variability of channel correlation across different layers, which may raise questions about the method's general applicability.**\n\n- The amount of correlation between channels does vary across layers. In the table below, we present the mean absolute correlation (MAC), excluding the diagonals, of different layers of LLaMA-7b on 262K tokens of WikiText-2.\n- Although the key correlation of the first few layers are higher than the later layers, the key/value correlation of any layer is never very close to zero, meaning channel coupling can be effective for any layer.\n- Correlation as a metric only captures the linear dependency between a pair of channels. In practice, we couple up to 8 channels together to leverage the higher order dependency between multiple channels.\n- We thank the reviewer for the careful reading of our paper. We will include a discussion on the existence and variability of channel correlation across different layers in the final paper.\n\n| Mean Absolute Correlation |  |  |  |  |  |  |  |  |\n|---|---|---|---|---|---|---|---|---|\n| Layer | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |\n| Key | 0.407 | 0.212 | 0.193 | 0.178 | 0.114 | 0.113 | 0.115 | 0.122 |\n| Value | 0.071 | 0.084 | 0.061 | 0.073 | 0.055 | 0.056 | 0.056 | 0.057 |\n| Layer | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 |\n| Key | 0.131 | 0.138 | 0.090 | 0.098 | 0.094 | 0.141 | 0.109 | 0.114 |\n| Value | 0.061 | 0.067 | 0.047 | 0.042 | 0.065 | 0.065 | 0.062 | 0.035 |\n| Layer | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 |\n| Key | 0.136 | 0.111 | 0.091 | 0.101 | 0.102 | 0.156 | 0.071 | 0.094 |\n| Value | 0.039 | 0.038 | 0.052 | 0.070 | 0.031 | 0.038 | 0.061 | 0.074 |\n| Layer | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |\n| Key | 0.107 | 0.069 | 0.057 | 0.103 | 0.095 | 0.097 | 0.100 | 0.090 |\n| Value | 0.027 | 0.044 | 0.038 | 0.072 | 0.070 | 0.035 | 0.105 | 0.090 |\n\n**[Q1] Quantization Setting: Is the KV Cache quantized during the prefill stage, or only during the decoding stage (after full-precision prefill computation), similar to the KIVI approach?**\n\n- For the majority of the experiments presented in the paper, including Table 1,2,4,5,6, all tokens are quantized in both the prefill and the decoding stage. We have specified this on line 247 of the paper.\n- For experiments with sliding window full-precision cache, including Table 3 and parts of Figure 1, we keep a constant number of the most recent tokens in full precision and quantize the rest of the tokens, and this holds true with respect to any token in the prefill and the decoding stage. This is slightly different from the KIVI implementation, which computes prefill in full precision and adopts a sliding window approach during decoding. We adopt this approach since perplexity testing and some tasks are log-likelihood-based and do not have a decoding stage.\n\n**[Q2] Channel Grouping Strategy: Would coupling non-adjacent channels with high mutual information potentially bring more benefits? --- Yes, very likely!**\n\n- Coupling channels with high mutual information will highly likely bring benefits. As shown in Figure 6 and 7, certain pairs of non-adjacent channels have higher correlations. By coupling them into the same channel group, we further reduce the joint entropy of channel groups, leading to better quantization accuracy.\n- We thank the reviewer for pointing us in this direction, and leave this for future work due to the difficulty of system implementation and optimizations.\n\n**[Q3] CUDA Kernel Implementation Questions**\n\n- Yes, the efficiency may be improved by adjusting the thread block assignments and reduction strategies.\n- First, it is important to note that the centroid size of each channel group is greater than 256 bytes. For example, for CQ-8c8b, the centroid size of each channel group is (num_centroids x num_channels x 2 bytes) $2^8 \\times 8 \\times 2=4096$ bytes. Assuming 100KB of shared memory per thread block, we can fit at most 25 groups of centroids into shared memory.\n- Fitting more channel groups into the same thread block reduces the number of concurrent writes to the HBM, hence speeding up the computation. \n- We thank the reviewer for this valuable suggestion. We will incorporate this kernel improvement into our implementation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers' careful evaluation of our paper and their valuable feedback. In the following section, we address some common concerns raised by multiple reviewers. We are happy to provide further clarification during the discussion period.

**1. Evaluations with long-context benchmarks.**

- We present additional experimental results below with long-context datasets comparing CQ and KVQuant using Llama-2-7b, with all tokens quantized, on GSM8K with chain-of-thought (CoT) and MMLU with CoT Fewshot (gsm8k_cot, mmlu_flan_cot_fewshot_humanities, mmlu_flan_cot_fewshot_stem, mmlu_flan_cot_fewshot_social_sciences, mmlu_flan_cot_fewshot_other from lm-evaluation-harness). In long-context settings, CQ mostly outperforms or is comparable to KVQuant under the same bit width.

|  | BPA | GSM8K CoT | MMLU (STEM) CoT Fewshot | MMLU (Humanities) CoT Fewshot | MMLU (Social Sciences) CoT Fewshot | MMLU (Other) CoT Fewshot |
|---|---|---|---|---|---|---|
| KVQuant-4b+1% sparse | 4.32 | 14.33 | 31.04 | 41.12 | **48.37** | 55.43 |
| CQ-2c8b | 4.00 | **14.71** | **33.73** | **43.44** | 47.77 | **56.01** |
| KVQuant-2b+1% sparse | 2.32 | **10.31** | **28.06** | 35.64 | 42.43 | **46.39** |
| CQ-4c9b | 2.26 | **10.31** | 27.76 | **35.91** | **44.51** | 45.75 |
| KVQuant-2b | 2.00 | 2.27 | 9.85 | 12.55 | 20.18 | 19.94 |
| CQ-4c8b | 2.00 | **8.04** | **25.67** | **30.89** | **45.4** | **41.94** |
| KVQuant-1b+1% sparse | 1.32 | 2.27 | 10.75 | 14.09 | 20.77 | 19.94 |
| CQ-8c10b | 1.27 | **2.35** | **13.13** | **21.81** | **28.19** | **26.98** |
| KVQuant-1b | 1.00 | 0.68 | 0 | 0 | 0 | 0 |
| CQ-8c8b | 1.00 | **1.74** | **5.37** | **11.39** | **20.77** | **16.72** |

**2. Regarding Centroid Learning.**

- Centroids for CQ are learned once on a calibration dataset and can be used for different downstream tasks.
- We have added an ablation study as follows to suggest that calibration on language modeling datasets provides transferable performance on downstream tasks. We use 16 sequences of 2048 tokens from WikiText-2 and C4 as the calibration set and evaluate CQ on 4 downstream tasks. 
- Despite using different calibration datasets, CQ performs similar in various downstream tasks.

|  | Calibration Dataset | WinoGrande | PIQA | Arc-C | GSM8K CoT |
|---|---|---|---|---|---|
| CQ-2c8b | WikiText-2 | 68.27 | 77.91 | 43.34 | 14.71 |
|  | C4 | 68.35 | 77.86 | 43.16 | 14.71 |
| CQ-4c8b | WikiText-2 | 66.45 | 76.12 | 39.93 | 8.04 |
|  | C4 | 66.22 | 76.61 | 39.93 | 8.34 |
| CQ-8c8b | WikiText-2 | 55.01 | 71.22 | 30.2 | 1.74 |
|  | C4 | 56.27 | 71.55 | 30.52 | 1.9 |

Meta-review: The paper presents a novel approach to compressing the key-value (KV) cache in transformer models through a method called Coupled Quantization (CQ). This technique aims to enhance inference efficiency in large language models while maintaining model quality, addressing significant GPU memory constraints. The authors provide compelling experimental evidence that demonstrates substantial throughput improvements, which could be impactful for various applications in machine learning.
Overall, the reviewers provided a mix of support and concerns regarding the submission. While some reviewers agreed unanimously on the solid contributions of the paper, others expressed reservations about the novelty of the method and the limited scope of experiments primarily focused on certain transformer models, particularly those in the Llama family. 
Despite the authors' thorough rebuttals, certain reviewer concerns remain unresolved, particularly regarding the generalizability of the findings across different model architectures and tasks. Reviewers noted the necessity for further validation in long-context settings and multi-hop retrieval tasks, as the current evaluations may not fully demonstrate CQ’s benefits. Additionally, the quality of the centroids under distribution shifts was questioned, signaling a need for more robust testing in diverse contexts.
It is vital that the authors address the reviewers' suggestions in the final version of the paper. These improvements, including expanded evaluation metrics and clarifications regarding the method's application, will significantly enhance the overall quality and impact of the research. I encourage the authors to implement these changes diligently to meet the expectations established through the review process.
Decision: Accept (poster)