CommVQ: Commutative Vector Quantization for KV Cache Compression
Authors: Junyan Li
Yang Zhang
Muhammad Yusuf Hassan
Talha Chafekar
Tianle Cai
Zhile Ren
Pengsheng Guo
Foroozan Karimzadeh
Colorado Reed
Chong Wang
Chuang Gan
Venue: ICML 2025 poster (2025)
Keywords: KV Cache
LLM
Primary area: deep_learning->large_language_models
TL;DR: We propose CommVQ, a novel KV cache quantization method that reduces FP16 KV cache size by 87.5% while maintaining high accuracy through additive quantization and a RoPE-commutative codebook.

Abstract: Large Language Models (LLMs) are increasingly used in applications requiring long context lengths, but the key-value (KV) cache often becomes a memory bottleneck on GPUs as context grows. To address this, we propose Commutative Vector Quantization (CommVQ) to significantly reduce memory usage for long-context LLM inference. We first introduce additive quantization with a lightweight encoder and codebook to compress the KV cache, which can be decoded via simple matrix multiplication. To further reduce computational costs during decoding, we design the codebook to be commutative with Rotary Position Embedding (RoPE) and train it using an Expectation-Maximization (EM) algorithm. This enables efficient integration of decoding into the self-attention mechanism. Our approach achieves high accuracy with additive quantization and low overhead via the RoPE-commutative codebook. Experiments on long-context benchmarks and GSM8K show that our method reduces FP16 KV cache size by 87.5% with 2-bit quantization, while outperforming state-of-the-art KV cache quantization methods. Notably, it enables 1-bit KV cache quantization with minimal accuracy loss, allowing a LLaMA-3.1 8B model to run with a 128K context length on a single RTX 4090 GPU. The source code is available at: https://github.com/UMass-Embodied-AGI/CommVQ.

Comm Commutative Vector Quantization for KV Cache Compression
Junyan Li Yang Zhang² Muhammad Yusuf Hassan'
Talha Chafekar' Tianle Cai Zhile Ren"
Pengsheng Guo Binazir Karimzadeh" Colorado J Reed Chong Wang* Chuang Gan
Abstract
2024; Jin et al., 2024) has managed to extend this even further, with some models achieving context lengths of over million. Supporting longer contexts enables LLMs to process richer inputs and generate more tokens, improving their capacity for more complex tasks and reasoning (Wei et al., 2022).
Large Language Models (LLMs) are increas- ingly used in applications requiring long context lengths, but the key-value (KV) cache often be- comes a memory bottleneck on GPUs as context grows. To address this, we propose Commutative Vector Quantization (CommVO) to significantly reduce memory usage for long- long-context LLM in- ference. We first introduce additive quantization with a lightweight encoder and codebook to com- press the KV cache, which can be decoded via simple matrix multiplication. To further reduce computational costs during decoding, we design the codebook to be commutative with Rotary Po- sition Embedding (RoPE) and train it using an Expectation- MatiattonnMMximzzztionn (EM) algorithm. This enables efficient integration of decoding into the self-attention mechanism. Our approach achieves high accuracy with additive quantization and low overhead via the NoPE-commutative codebook. Experiments on long-context benchmarks and GSM8K show that our method reduces FP16 KV cache size by 87.5% with 2-bit quantization, while outperforming ttate-of-the-aaa art KV cache quantiza- tion methods. Notably, it enables 1-bit KV cache quantization with minimal accuracy loss, allow- ing a LLaMA-3.1 8B model to run with a 128K context length on single RTX 4090 GPU. The source code is available at: https://githu/t com/UMas SettkaassEEmbodee ConnnoV
However, this increased context length presents a signifi- cant challenge GPU memory usage. The causal attention mechanism used in LLMs relies on Key-Value (KV) cache to speed up inference. This cache stores the keys and values of all previous tokens, eliminating the need to recompute them when generating the next token. As context lengths increase, the size of the KV cache grows proportionally, eventually becoming the primary bottleneck for memory usage often far exceeding the memory required for the model itself. For instance, LLaMA 3.1 8B model requires 16 GB of memory to store the model weight in FP16 preci- sion. If the context length is set to its maximum of 128K with a batch size of 2, the KV cache alone would require 88 GB of memory. This makes it impossible to run inference a single GPU without KV cache offloading, even for H100-80GB.
Efforts to reduce KV cache size are ongoing (Shi et al., 2024; Yuan et al., 2024), with KV cache quantization (Liu et al., 2024b; Hooper et al., 2024) being a key approach. Quantization lowers the memory footprint by reducing the bit width used represent each FP16 scalar in the KV cache. For instance, INT4 quantization cuts memory usage by factor of four compared FP16. However, this comes cost: aggressive quantization, such as 2-bit even 1-bit quantization. results in significant information loss, severely degrading model performance.
1. Introduction
We are witnessing a growing trend in inereasing the context length of large language models (LLMs). For instance, the latest LLaMA 3.1 models (Dubey et al., 2024) support up to 1 128K context length, and recent research (Ding et al.,
To address these challenges, we propose Commutative Vector Quantization (CommVQ), novel method for ef- ficient and accurate KV cache quantization tailored long- context LLMs. Unlike existing quantization techniques that treat each scalar in the KV cache independently. Comm performs quantization at the vector level. Specifically, we treat the key/value vector for each token as a single unit rather than processing scalars individually. To achieve this,
University of Massachusetts Amherst z Massaachusett Institute of Technology Priicetonn University * Apple Inc. Correspondence to: Junyan Li <junyanli @ uuuass.cdu>..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
experiments were conducted by Junyan at UMass Amherst.

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
leverage additive quantization (Babenko & Lempitsky, 2014), variant of vector quantization, encode each vec- tor into a low-bitwidth representation utilizing learned codebook, minimizing quantization error while significantly reducing memory usage.
Zhang et al., 2023) aim reduce KV cache size by evicting less important tokens, storing only the keys and values for the most relevant tokens. These approaches are orthogonal to our method and could potentially be combined with it to achieve even higher compression rates.
More importantly to integrate additive quantization into the self-attention mechanism in computationally efficient manner, the codebook is innovatively designed be commu- tative with the Rotary Position Embedding (RoPE) matrix (Su et al., 2024). This allows a drastic reduction in the computational overhead of KV decoding, where interme- diate results can be pre-computed against each code in the codebook, and are then efficiently reused in computing the lengthy key-query products.
Another important approach to KV cache compression is quantization (Liu et al., 2024b; Hooper et al.. 2024; Zhang et al., 2024a; Kumar, 2024), which reduces the bit width of the KV cache, thereby lowering its overall storage require- ments. While prior works have successfully demonstrated 4-bit and 2-bit quantization for KV caches, few have ex- plored the feasibility of achieving 1-bit quantization for longecontext LLMs. Our method addresses this gap by in- troducing novel vector based KV cache quantization tech- nique that enables I-bit quantization with minimal accuracy loss. Furthermore, offer new perspective on how to reformulate self-attention mechanisms and our quantization method for much more efficient integration.
By combining these innovations, CommVQ achieves a su- perior trade-off between memory savings and accuracy. Ex- tensive evaluation on two long-context benchmarks, Long- Bench (Bai et al., 2023) and InfiniteBench (Zhang et al., 2024b), as well as GSM8K (Cobbe et al., 2021), a bench- mark designed for complex reasoning, shows that com- pared to state-of-the-ar KV cache quantization baselines, we achieve nearly lossless KV cache compression with 2-bit quantization, outperforming other methods. Furthermore, we achieve 1-bit quantization with significantly better accu- racy than existing baselines, demonstrating the effectiveness of our method in pushing the compression limits of KV cache. We summarize our contributions as follows:
Vector Quantization. Vector quantization (Gray, 1984) is a widely studied technique in signal processing and ma- chine learning that represents high high-dimensional data using smaller set of representative vectors, known codebooks. Variants of VQ, such product quantization (PQ) (Jegou et al., 2010) and additive quantization (AQ) (Babenko & Lempitsky, 2014), have been introduced to improve its effi- ciency and capacity.
In machine learning, vector quantization has been success- fully applied areas such as generative modeling (Van Den Oord et al., 2017; Esser al., 2021). However, its po- tential extension in the context of KV cache compression, re- mains largely unexplored. A recent study, VQLLM (Kumar, 2024), introduced residual vector quantization (RVQ) for compressing the KV cache. However, their achieved com- pression rate is relatively modest, and their basic decode- then-self self-attttttnnno process introduces significant computa- tional overhead, limiting its practicality for serving long- context LLMs. There remains a lack of in-depth research on how to optimize vector quantization and better integrate it with self-attention for both more efficient and effective KV cache compression.
Unlike prior work that quantizes each scalar individ- ually in the KV cache, we quantize each vector as a whole in the KV cache into a low bit-width representa- tion using a learned codebook.
By leveraging the commutative property of the RoPE matrix and the characteristics of self-attention, we re- fine our codebook be RoPE-commutative. This refinement enables us to reformulate the self-attention computation to incorporate the decoding process more efficiently. Additionally, we provide a Triton imple- mentation of our method to demonstrate real memory savings.
Extensive experiments demonstrate the superiority of our method, particularly in altra-low-bitwidur KV cache quantization scenarios (e.g.. 1-bit quantization). This opens up the possibility of serving long-context LLMs under limited GPU memory constraints.
3. Preliminaries
3.1. Self-Attention and KV Cache
Self-attention is the fundamental building block of LLMs. It takes query (Q), key (K), and value (V) matrices inputs and produces an output (O) with the same shape as Q. The causal attention mask used in self-attention allows us to cache the key and value matrices, significantly speeding up token generation. During LLM inference, the process is divided into two stages:
2. Related Works
KV Cache Compression. Several prior works have ad- dressed KV cache compression, which generally fall into two categories: token eviction and quantization. Methods fo- cused on token eviction (Xiao et al., 2024; Liu et al., 2024a;
2

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
Prefilling Stage. At this stage, given the input prompt, the attention output is computed while simultaneously generat- ing the KV cache. Given the hidden states of the prompt, X E I. Nxd, where N is the number of tokens, and d is the hidden size, the Q K, and V matrices, as well as the self-attention output, are computed as follows:
where each 2-dimensional sub-vector km (kise: : Key con-
sists of two scalars within the km vector. The corresponding 2 2 diagonal sub-matrix of Rm. denoted as Ri E IR is then applied to each sub-vector kn
R2 = =(
COS mb; - sin md; sin sin cos m0;
)
(6)
Q =WQ: K == XWK,
=
ki + k Ry
(OK)
QKT
(I)
Self-Attm Softmax
where 0, = 10000-2(6-1)4 The diagonal sub-matrix R is, fact, a rotation matrix, which satisfies the follow- ing property:
Here, Wq. WK. and Wy E Rdxd are the projection matri- for the query, key, and value, respectively. The computed K and V matrices are then cached for use in the subsequent decoding stage.
Property 1 (Commutativitty): Let C' 1R2x2 defined as
Decoding Stage. During this stage, the KV cache is reused for self-attention computations. Given the current input hidden state IR xxd the KV cache is updated as follows:
y C
- x
The following commulativity holds:
V K <- Concat( K, .WK),, V <- Concat(V,
(2)
I2m C C
(8)
The updated KV cache then used to compute the self- attention output, following the same equation as in Eqn.
This property shows that C' and Ri, commutative under matrix multiplication. We leverage this key property to optimize our method in Sec. 4.2.
Q zWQ
QKT Vd
(3)
Self-Attm Softmax
V
4. Method
The prefilling stage occurs once to process the input tokens and generate the first output token. This is followed by multiple iterations of the decoding stage, which produces all subsequent output tokens.
The KV cache is dominant factor in long-context LLM inference scenarios, storing and loading the large KV cache becomes significant memory and latency bottleneck. Therefore, compressing the KV cache, even at the cost of ad- ditional encoding and decoding processes, is advantageous. Motivated by vector quantization, a classical quantization technique from signal processing. and particularly inspired by its recent variant, additive quantization (Babenko & Lem- pitsky, 2014), we adopt similar approach to quantize each vector in the KV cache into compressed representation using a learned encoder and codebook, as described in Sec. 4.1.
3.2. Rotary Position Embedding
Positional encoding is added to the query (Q) and key (K) matrices to encode token position information during self- attention computation. Recent open-source LLMs, such as LLaMA (Dubey et al., 2024), Mistral (Jiang et al., 2023), and Qwen (Yang et al., 2024), commonly utilize Rotary Po- sition Embedding (RoPE) (Su al., 2024) for this purpose:
To efficiently integrate vector quantization into self-attention while minimizing the computational overhead introduced by the additional decoding process, we innovatively refor- mulate the self-attention computation by designing a RoPE- commutative codebook and reordering the matrix multipli- cations involved in self-attention. This refinement enables a large portion of computation reuse and significantly re- duces the computational cost of our method, as detailed in Sec. 4.2.
Qm. 4m Rm:
ki f- Rm
(4)
where qm, km E 1xd represent the query and key vector for the th token, respectively, while Rm, E Raxd denotes the RoPE matrix applied the th token. RoPE matrix is sparse matrix with nonzero values only in its 2 x 2 diagonal blocks. Therefore, the application of RoPE can be refor- mulated by first dividing the km vector (and similarly qm here, we use km as an example) into multiple 2-dimensional vectors:
4.1. Learning Additive Quantization for KV Cache
km [ki,; kir; kig...,, k(a//)= k(a/2)))
Additive quantization (Babenko & Lempitsky, 2014) aims to represent a given d-dimensional vector as the element-wise
(5)
[k],,... ..., 1,/121
3

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
K/V Cache
sequent computations in the self-attention mechanism.
This
The encoder E' and codebook C' are optimized via gradient descent to minimize the MSE loss between the original tensor &t and its decoded counterpart E..
is
apple
encode
KV Cache Reduction Analysis. The size of the original FP16 KV cache for each layer can be calculated as B x N x d 2 x 16 bits, where B is the batch size, N is the number of tokens and d is the hidden size. After quantization, the size of the quantized KV cache for each layer becomes B x N x N₂ x 2 x 1 bits. Therefore, the reduction rate RR can be expressed as 1 led
01001100
8
select
Codebook
decode
Σ
.N_.. Subsequently, the average bit
width used to represent each scalar in the KV cache is:
f
N d
Avg. bit 16 16 RR
(10)
Figure 1. An illustration of vector quantization for KV cache compression. Each vector in the KV cache, corresponding to token, is first encoded into a low-bitwidth representation, signifi- cantly reducing storage size. This process is applied to all vectors in the KV cache individually. When needed, the compressed repre- sentations loaded and decoded back the original KV cache using a codehook.
For instance, the hidden size d for the key and value for ----M-..-----nsstteeee model is 1024, If we N₂ to
1024, the reduction rate will be 15 1-bit quantization. We will use Avg. bit as the metric to measure the compression rate in our experiment section. A lower Avg. bit indicates a higher compression rate and more reduced GPU memory usage compared to FP16.
T6 which is equivalent to
summation of several vectors from a learned codebook C. For simplicity, we adopt a per-token quantization scheme for both the key and value matrices, meaning that the d- dimensional key and value vectors are quantized individu- ally for each token.
Computation Complexity. The computational overhead arises from both encoding and decoding the KV cache. Since the KV cache only needs to be encoded once, the primary source of computational overhead comes from de- coding the full KV cache each time it is used during the generation of the next token.
Inspired by additive quantization, method incorporates learned encoder E' encode the given d-dimensional vector into a binary sequence, consisting of Os and Is, of length No, and a codebook C E Red to decode the original vector back using a simple matrix multiplication, as illustrated in Figure! 1.
The computation for self-attention with KV cache decoding during inference step can be summarized as:
q x qR,
(11)
Sy SuC Ro 81 Ck Ri
Encoding. Given a key or value vector for the 7th token, E R4, we use encoder E to encode this vector into quantized vector 8; E {0. 1).,, namely our implementation, the encoder E consists of simple linear layer followed by an activation function, and then another linear layer for output. Gumbel-softmax (Jang et al., 2016) is used to make the encoder end-to-end differentiable. The encoding step takes place after the generation of the KV cache, and the quantized vector s; for each token is concatenated and stored together to form the quantized KV cache S.
K
(12)
E(4). In
SN-ICK
SN- Cx RN-L
V + SyCv
(13)
9KT
Self-Attn = Softmax
(14)
where RIxd is the current query, N is the number of tokens generated so far, Sy E {0, 1 are the quantized value, 8; E {0, 1]. No is the quantized vector for to key, CK. Cv E R N. are the codebook for the key and value, respectively, and R, denote the RoPE matrix for 7th token. The computational complexity can be expressed as:
Decoding. When loading the KV cache, the quantized KV cache retrieved. For each token's 's 8; we perform a sim- ple matrix multiplication with the codebook C' to reconstruct the decoded tensor Z;
O ((2d 1)N + 2dN.N)
(15)
s.C
(9)
where d is the hidden dimension, is the number of tokens, and N₂ the number of rows in the codebook. The first
The decoded key and value can then participate in the sub-
4

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
term in Eqn. 15 corresponds to the computation for Eqn. 14, which represents vanilla self-attention. The second term, 2dN,N, accounts for the KV cache decoding process for the key and value, where each contributes dN....
the key vector, i.e.,
C₁ = CC CR.CC,K......... ... D
(17)
where No is the number of quantization levels, and each CK is a 2 2 matrix that satisfies the form defined in Eqn. 7, and thus has the commutative property Ri C = C R..
Comparing the first and second terms in Eqn. 15, we see that the computational overhead from the additional vector quantization decoding process Ne times higher than that of vanilla self-attention. This increase arises from the need to first sum a large number of rows in the codebook to reconstruct the decoded vector, after which the decoded vector can participate in the self-attention computation. The overhead is particularly significant because N₂ is typically large (e. 1024 for 1-bit quantization in LLaMA-3.1 8B model).
The quantized vector for ky. denoted as s/ is 2- dimensional vector taking the values from {O..... No - 1}. The decoded key is represented as
N
R = Σ 1
|s -
(18)
where [s] = 1] is a 2-dimensional boolean indicator vec- tor, with each dimension equal I if the corresponding dimension of sy is equal to I, and 0 otherwise.
In the next section, we introduce our novel redesign, which leverages commutative codebooks to more efficiently inte- grate vector quantization with self-attention. This method will significantly reduce the overall computational overhead.
Notice that the decoding scheme in Eqn. 18 is more compli- cated than previously discussed (Eqn. 9). This is because previously we used the same codebook for all the dimen- sions, but now the codebook for different dimensions is different. Please refer to Appendix A.1 for more explana- tions.
4.2. Commutative Codebook for Efficiency
To simplify notation, we focus on the calculation within the softmax and ignore the vd constant: a = qk",, where the output a is an N-dimensional vector. Each scalar entry ag in a is computed as:
With the new decoding scheme, the benefit of commutativity remains. To see this, notice that Eqn. 16 should now be rewritten as
a-
= (q T
(16)
where R₂ denotes the RoPE matrix for the query. R; denotes the RoPE matrix for the 2t key, and 8; is the quantized vector for the 2th key. Notice that qR and Ck remain unchanged as varies. If R3 were independent of as well, then the computation of (q C% would only need to be performed once across different significantly reducing the computational cost. Unfortunately, since R2 varies with i, such reduction is not possible.
ai = EUPRERRR
EGPRR =
(19)
EGPRDR [ 117
*f RT = 1 5d
However, if we could design the codebook Cx to be com- mutative with R2, then the right-hand-side of Eqn. 16 could be rewritten as (q K RT sT. which makes the bulk of computation, Ck. independent of and thus reusable, leading to substantial computational savings.
The first equality decompose the inner products into those of the sub-vectors; the last equality applies the commutativity property, making (g' RR)CK" reusable across i.
Learning the Codebook. The codebook is learned by mini- mizing the reconstruction error:
Next, we discuss how to define our new commutative code- book Cx as well as how to learn it.
Designing a Commutative Codebook. As explained in Sec. 3.2, since the RoPE matrix is block-diagonal, we can break the problem into 2-dimensional subspaces and design the commutative codebook within each block. Formally, as defined in Eqns. 5 and 6, k: and R.. represent the j-th 2-dimensional sub- caab--eeoor/submmatri of k; and R,, respec- tively. Extending Property 1, we can obtain a design for a sub-space codebook.
min U.(C) .
Σ - MIP S.t. Eqn. 18
(20)
This is a canonical clustering objective and can be efficiently solved via an EM-like algorithm, where the E-step mini- mizes over sf holding C% fixed, and the M-step minimizes over C% holding sf fixed, as described in Algorithm 1. More details can be found in Appendix A.2, including the actual update formula and the techniques used to stabilize the optimization process.
Specifically, let C%2 be set of codebooks for subspace j of
5

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
Algorithm EM Algorithm for Learning RoPE Commutative Codebook.
By integrating these adjustments into the original self- attention mechanism, the optimized computation complexity is now:
1: Input: A calibration set K E RRNx2
2: Parameters: codebook C¹K
3: Goal: Optimize C₂K to minimize the clustering error
over calibration set K.
O(RR Ne 1)N + d(Ne+ RN2))
(23)
4: while Cj converges do
5: E Step: Fix CkK- update the assignment S such that each & E K is assigned to its nearest clustering cen-
This optimization effectively reduces the computational cost compared to the unoptimized decode-then-sell tnee--sef--attenton cal- culation (Eqn. 15). Previously, the complexity was No. times higher than that of the original self-attention; now, it is ap- proximately R+1 times higher. Since It is relatively small hyperparameter (e.g., R = 11 for 1-bit quantization), the overhead remains minimal.
ter.
6: M Step: Fix S. update Cj such that the MSE loss between each k and its assigned clustering center is minimized.
7: end while
2
Notice that of requires 2 logz (Nz) bits, which limits its ability to achieve a high compression rate relative to ki. To improve compression, we group consecutive sub-vectors into one group and share the quantized value within the group, i.e., sP. = s = = - This allows the entire vector be represented using significantly fewer bits.
5. Experiments
5.1. Settings
Models. We evaluate Comm VQ using the latest LLaMA- 3. I--t--untttrctt model (Dubey et al., 2024), which supports a context length of up to 128K tokens. A subset of the Web-EEu dataset (Lozhkov et al., 2024) is used to learn the encoder and codebooks. We present evaluation results for two quantization levels: 2-bit and 1-bit quantization (see Appendix A.4 for codebook configuration). To demon- strate the generalizability of our method, we also conduct additional experiments on the LLaMA-2-8B (Touvron et al., 2023) and Mistral-8B (Jiang et al., 2023) models.
To further improve the quantization accuracy, we iteratively apply the clustering algorithm on the quantization error ten- sors with new codebook for each time, repeating the process until the error is sufficiently minimized. Specifically, we run the clustering algorithm R times, resulting in codebooks C% for subspace j. R is a hyperparameter that balances quantization accuracy and compression rate. This iterative refinement is conceptually similar to the residual approach in residual vector quantization (Barnes et al., 1996), where subsequent iterations focus on reducing the remaining error. As result, total of π instances of 8 are required to quan- tize 2g FP16 scalars, and the total number of bits needed to quantize the full d-dimensional vector is 2R loga (Ner) 2g Accordingly, the average quantization bit can be computed
Baselines. We compare our method to three recent KV cache quantization techniques: KIVI (Liu et al., 2024b), K KVuuant (Hooper al., 2024), and VQLLM (Kumar, 2024). KIVI employs asymmetric quantization, KVQuant uses non-uniform quantization, and VQLLM applies resid- ual vector quantization. For faimess, we reproduced their results using their official open-source implementations on the same models. We denote quantization versions as <method><<>>, where <n>> represents bits per scalar in KV cache. For VQLLM, we set C 256, K = 8 for 2-bit and C = 256, K for 1-bit quantization.
as:
R log, (Ne)
Avg. bit
(21)
g
For example, to achieve 1-bit quantization, we set N₂r = 64, R- = 11 and 64. We provide an ablation study on how to choose No., R and in Appendix A.4.
Tasks. To evaluate the effectiveness of our method for long- context LLMs, we test it alongside the baselines on two long-context benchmarks: LongBench (Bai et al., 2023) and InfiniteBench (Zhang et al., 2024b), Additionally, to assess the model's 's ability to perform complex reasoning, we evalu- ate it on GSMSK (Cobbe al., 2021). Apart from the task score, we also report the average quantization bit, denoted as Avg. bit, for each method to quantify the actual KV cache size reduction. A lower Avg. bit indicates less storage re- quired for the KV cache, leading to greater memory savings. For baseline methods, we follow the calculations provided in their respective papers to determine the Avg. bit.
Reduced Computational Complexity. For value quantiza- tion, we retain our original method but reorder the matrix multiplication process. Specifically, we first multiply the softmaxed attention weights by Sv, followed by multiplying the result by Cv. as illustrated below:
Self-Attn = Softmax(A)Sy
(22)
This simple reordering of matrix multiplication reduces the computational complexity from OdNNNN + dN) in Eqns. 13,14 to ONN.N + ddNe) in Eqn. 22, which is nearly times lower, assuming d and N₂ are of a similar scale.
6

---

Comm VQ: Commutative Vector Quantization for KV Cache Compression
Method
Avg. bit
Qasper
QMSum
MultiNews
TREC
TriviaQA
SAMSum
LCC
RepoBench-P
Average (1)
FPI6 Bascline
16
25.19
23.31
26.82
72.50
91.65
43.49
52.47
49.01
48.05
KIVI-2
3,00
22.71
24.33
27.29
72.50
92.06
43.26
51.32
47.53
47.62
KVQuant-2
2.33
41.86
22.37
26.76
69.00
89.00
42.09
36.22
36.51
45.35
VQLLM-2
2,00
32.39
25.20
26.22
69.95
92.01
41.03
40.58
36.19
45.45
CQ2
2.00
24.67
24.36
26.48
72.50
91.92
43.98
53.02
46.92
47.98
KIVI-1
2,00
4.99
9.20
38.75
25.07
11.93
17.67
16.40
16.70
KVQuant-1
1.33
8.71
6.06
6.64
11.09
5.88
VQLLM-1
1.00
11.92
17.91
13.12
47.98
03.34
23.72
18.92
22.44
27.42
CommVQ-4
1.03
18.86
23.02
24.34
09.00
91.61
41.83
48.78
42.08
44.94
Table 1. LongBench evaluation for the LLaMA-3.1-8B-Insttee model.
Method
Avg. bit (1)
R.PK
R.Num
R.KV
En.Sum
En.QA
En.MC
En.
Code.D
Math.F
Average (1)
FP16 Baseline
100.00
99.49
55.20
26.74
14.28
66.81
20.00
22.08
33.43
48.67
KIVI-2
3,00
100.00
97.
0.60
25.41
13.90
66.81
22.50
23.35
33.71
42,68
KVQuant-2
2.33
98.81
88.81
0.00
25.02
7.77
35.81
8.00
25.63
10.29
33.34
VQLLM-2
2.00
100.00
97.96
0.00
18.27
8.09
44.54
9.50
21.83
29.71
36.66
CommVQ-2
2.00
100.00
93.39
12.20
24.14
14.37
67.25
18.00
22.08
33.14
42.75
KIVI-1
2,00
1.86
0.00
0.00
12.44
3.24
55.46
4.50
23.86
34.00
15.04
KVQuant-1
1.33
0.00
0.00
0.00
17.83
1.36
0.44
2.00
0.25
1.14
2.56
VQLLM-I
1.00
82.37
14.75
10.46
2.69
25.33
1.50
21.83
7.43
18.48
Comm VQ-1
1.03
99.15
62.37
0.00
19.34
11.30
65.50
18.00
22.08
33.14
36.76
Table 2. InfiniteBench evaluation for the LLaMA-3.1-88 Instruct model.
5.2. Long Context Benehmarks Evaluation
sequence length is set to 128K.
LongBench Evaluation. LongBench (Bai et al., 2023) is a benchmark for evaluating models on long-context tasks like multi-doc QA, summarization, and code completion. Following KIVI (Liu al., 2024b), we assess performance four subgroups and report
Experiment results are shown in Table Our method con- tinues to excel ultra-long context scenarios, with even greater superiority
1-bit quantization. For challenging tasks that require accurate information, such retrieval (R.PK, R.Num, and R.K.), other methods fail to produce accurate results, whereas ours retains some capacity even at low quantization levels. This is due to our codebook design, which effectively preserves information in the KV cache, as confirmed by our method's lower quantization error (mea- sured in MSE) shown in Table 11 in Appendix A.S.
on the same eight tasks both individual and average scores. The maximum sequence length is set to 128K.
Experiment results are presented in Table 1. Our 2-bit quantization model achieves lossless accuracy on most tasks, maintaining almost the same average score as FP16 model while outperforming other baselines as well as of- fering greater memory savings. Compared KIVI, our method provides 33% more memory savings while achiev- ing higher average score. Among baselines with a sim- ilar average quantization bit, our approach performs sig- nificantly better, with scores 2.63% higher than KVQuant and 2.53% higher than VQLLM. For 1-bit quantization, our method substantially outperforms other baselines, achiev- ing 17.52% higher average score than VQLLM while maintaining comparable memory savings. This minimizes accuracy degradation compared to the FP16 model.
Needle-in-a- Haystaakk Evaluation. We further evaluate our method using LLaMA-3.1 8B model on the Needle-in- a- Haystack (NIAH) benchmark, which specifically targets retrieval capabilities in long-context settings by requiring the model to identify a small piece of information embedded within a large amount of irrelevant text. This test serves as strong indicator of how well KV cache and attention mechanisms preserved under quantization.
Experimental results in Figure demonstrate that our 2-bit quantization model successfully retains full retrieval capa- bility, matching the performance of the FP16 baseline. This confirms that our approach introduces negligible degrada- tion even under aggressive compression. More notably, our 1-bit CommVo variant achieves stronger retrieval accuracy than KIVT's 1-bit counterpart, highlighting the effective- ness of our codebook design in preserving critical attention
InfiniteBench Evaluation. InfiniteBench (Zhang et al., 2024b) evaluates models on ultra-long contexts, mimicking real-world scenarios with near-infinite input lengths. tasks include multiple types of retrieval, QA, summarization, and code debugging. In our experiment, the maximum

---

Comm Commutative Vector Quantization for KV Cache Compression
1.1-ttt-nstruct FP16
1lama-3.1-88-Imas. CommVQ-2
10
11.0
11.0
Depth Percent
Depth Percent
44.0
44.0
67.0
67.0
78.0
78.0
100.0
100.0
18k
32K
64k
s6k
128K
18k
32k
64k
96k
128K
Token Number
Token Number
Llama-3 1-888---8
1--8-Innsten Instruct KIVi-1
Llame-3.1-88- CommVQ-1
0.0
10
0.0
10
Depth Percent
22.0
Degth Percent
22.0
Score
56.0
56.0
78.0
89.0
89.0
100.0
100.0
18k
sex
128K
18k
32K
128k
Taken Number
Figure 2. Ncedle-in-a Haabt--aayytakk test using the LLaMA-3.1 8B model. We present the test result for FP16 baseline (top-left), KIVI-1 (bottom-left), CommVQ-2 (top-right) and CommVQ-1 (bottom-right), method's version matches FP16 performance, while the 1-bit CommVQ variant outperforms KIVI, demonstrating strong retrieval fidelity under extreme compression.
Method
Avg. bit (4)
GSM8K (1)
Model
Avg. (L) LongBench (f)
FP16 Baseline
16
76.27
FP16
48.43
Llama-2-7B
KIVI
3.00
47.14
KIVI-2
3.00
73.69
Comm
2.00
47.27
VQLLM-2
2.00
52.69
FP16
16
53.40
Comm VQ-2
2.00
76.04
Mistral-7B
KIVI
3.00
52.78
KIVI-1
2.00
2.20
Comm VQ
2.00
53.04
VQLLM-I
1.00
1.67
CommVQ-1
1.03
66.57
Table 4. Performance comparison of full precision (FP16), KIVI, and Comm applied two additional LLMs for model ablation.
Table GSM8K benchmarks evaluation for the LLaMA-3.1-8B Instruct model. Exact match accuracy is used as the metric.
only minimal accuracy degradation of 0.23% compared to FP16. Furthermore, while our baselines struggle to gen- erate accurate results under 1-bit quantization, our method continues to demonstrate strong reasoning capabilities even under this extreme compression.
signals. These findings reinforce that CommVQ maintains high fidelity in information-denss retrieval tasks, even under extreme compression ratios.
5.4. Model Ablation
5.3. GSM8K Evaluation
We also apply our method to two additional long-context L.Mss: LLaMA-2-7B (the 32K context length version from Together.ai) and Mistral-7B-v0.3 (which natively supports 32K context length). We evaluate both on LongBench, com- paring their average scores the FP16 baseline and KIVI.
To demonstrate the effectiveness of our approach on chal- lenging and complex tasks, we conduct experiments on GSM8K (Cobbe et al., 2021), a rigorous benchmark com- prising high-quality, linguistically diverse math problems meticulously crafted by human experts. These problems require intricate multi-step reasoning and involve complex arithmetic operations, making GSM8K an ideal benchmark for evaluating complex reasoning capabilities, As shown in Table 3, our 2-bit quantization model outperforms other baselines, maintaining significantly higher accuracy, 2.35% above KIVI and 23.35% above VQ-LLM, while exhibiting
The experimental results summarized in Table 4. For both the LLaMA-2 and Mistral models, our method con- sistently preserves the FP16 bascline's average score while achieving a better compression rate aeeeaccrrcyy trade-off than KIVI. These results highlight the broad applicability and effectiveness of our approach.
8

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
Latency (ms)
8K
32K
128K
70
FP16
Commvo
Naive Impl.
2.4
9.2
36.6
60
Optimized Impl.
0.4
1.1
3.8
37.5
Speedup
6.0
8.4
9.6
Memory
30.0
22.5
19.2
19.4
19.6
19.8
20.0
Table Lateney comparison between the naive implementation and the optimized implementation utilizing the commutative code- book. Latency per layer per token is measured for context lengths of 8K, 32K, and 128K, reported in milliseconds (ms).
10
20K
40K
60K
BOK
100K
120K
Length
(a) Memory usage vs. context length (batch size = 1).
Method
FineWeb-Edu
Repobench-p
KV_ RV_Retrieval
FP16
FP16
CommvQ
59.8
CQ2
11.54
6.14
2.78
32.72
PPL Diff
+1.37
+0.47
+0.58
+0.79
Memory (GB)
35.0
Table Perplexity (PPL) comparison between FP16 baseline and Com across different domains.
30
23.9
19.4
20.1
21.3
20
0OM
OOM
00M
0OM
00M
5.5. Robustness Analysis Under Domain Shift
16
128
Batch Size
Our codebook and encoder trained the FineWeb- Edu (Lozhkov et al., 2024) pre-training dataset. To evalu- ate their robustness when the testing domain differs from the training domain, we conducted an analysis using the LLaMA-3.1 8B model. Specifically, we compared the per- plexity of CommVQQ against the FP16 baseline across four distinct datasets: Fine Web-Edu, general text dataset, also the training set; GSM-8K (Cobbe al., 2021), math dataset; Repobench-p (Bai et al., 2023), code retrieval and completion dataset; and KV Retrieval in In- finiteBench (Zhang et al., 2024b), synthetic UUID key- value retrieval dataset. The first dataset represents in-domain evaluation, while the last three represent evaluations with domain shifts, i.e., the codebooks and encoder are trained on general text and tested on math, code, and synthetic UUID data. The results summarized in Table 6.
(b) Memory usage batch size (context length 32K).
Figure 3. Per-token decoding memory usage of Comm VQ (1-bit) compared FP16 model. Experiments are conduct on I.L.MA- Instrut model.
validating the commutative codebook's effectiveness in re- ducing computational overhead.
We also implement Triton kernels to achieve real memory savings. Figure 3 highlights the real per-token decoding memory savings using the LLaMA-3.1-8BB In8t-Instruct model, measured on an H100-80GB GPU. A 120K context length requires 60GB in FP16, while our method reduces it to 20GB, enabling inference on single consumer GPU such as RTX 4090. For 32K context length, FP16 runs into OOM at batch size of 8, but our method scales up to 128. This improves long- long-context and large-batch serving, benefiting lot of applications such as long document QA. We also provide additional analysis of codebook size in Appendix A.3, demonstrating that its size is negligible, es- pecially compared the large KV cache for long contexts.
We find no significant increase in perplexity (PPI.) due to domain shifts when compared to in-domain evaluations. This suggests that our method performs consistently well across domains that differ from the calibration data, includ- ing synthetic UUID data, which is unlikely appear in the calibration set. Overall, conclude that our method is robust and generalizable under domain shifts.
6. Conclusion
5.6. Efficiency Results
We introduce CommVQ, a novel KV cache quantization approach for long-context LLMs. By leveraging vector quantization and a RoPE-commutative codebook, Com- mVQ significantly reduces KV cache size while maintain- ing high computational efficiency. Evaluations on long- context benchmarks show that Comm outperforms exist- ing KV cache quantization methods, enabling more memory- efficient and scalable long-context LLM inference.
In Table 5, we present a latency comparison demonstrate the impact of our method's 's reduced computation (denoted as the optimized implementation) compared to the naive implementation, which does not utilize the commutative codebook to reduce the computation. For context lengths of 8K, 32K, and 128K, the optimized implementation con- sistently achieves speedups over the naive implementation,

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
Impact Statement
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, E., Lengyel, G., Lample, G., Saulnier, L.,, et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Jin, H., Han, X., Yang, J., Jiang, Z., Liu, Z., Chang, C.- Y.. Chen, H., and Hu, X. Lim maybe longlm: Self- extend Ilm context window without tuning. arXiv preprint arXiv:2401.01325. 2024.
References
Babenko, A. and Lempitsky, V. Additive quantization for extreme vector compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 931-938, 2014.
Kumar, A. Residual vector quantization for kv cache compression in large language model. arXiv preprint arXiv:2410.1570% 2024.
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference Machine Learning (ICML, 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Bai, Y.. Lv, X., Zhang, J.. Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context under- standing. arXiv preprint arXiv:2308. 14508, 2023.
Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z.,, Kyril- lidis, A., and Shrivastava, A. Scissorhands: Exploiting the persistence of importance hypothesis for Ilm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024a.
Barnes, C. F., Rizvi, S. A., and Nasrabadi, N. M. Advances in residual vector quantization: review. IEEE transac- tions on image processing, 5(2):226-262, 1996.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. Kivi: A tuning-free asym- metrie 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024b.
Ding, Y., Zhang, L. L., Zhang, C., Xu, Y., Shang, N., Xu, J. Yang, F., and Yang, M. Longrope: Extending Ilm context window beyond 2 million tokens. arXiv preprint arXiv:2402. 13753, 2024.
Lozhkov, A., Ben Allal, L., von Werra, L., and Wolf, T. Fineweb-edu: the finest collection of educational content, 2024. URL https: https::/huqgiingface. co/ datasets /Hugg
Dubey, A.. Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Shi, L., Zhang, H., Yao, Y., Li, Z., and Zhao, H. Keep the cost down: A review on methods to optimize 11m's kv-cache consumption. arXiv preprint arXiv:2407.18003 2024.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873 -12833, 2021.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 167633, 2024.
Gray, R. Vector quantization. IEEE Assp Magazine, 1(2): 29, 1984.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Hooper, C., Kim, S., Mohammadzadeh, H. Mahoney, M. W. Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length 11m in- ference with kv cache quantization. arXiv preprint arXiv:2401.1807,, 2024.
Van Den Oord, A., Vinyals, O., et al. Neural discrete rep- resentation learning. Advances in neural information processing systems, 30, 2017.
Jang, E., Gu, S., and Poole, B. Categorical repa- rameterization with gunbel-softmax. arXiv preprint arXiv:1611.01144 2016.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thoughh prompting elicits reasoning in large language models. Advances in neural information processing systems, 0124824-24831 2022.
Jegou, H., Douze, M., and Schmid, C. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117-12, 2010.
10

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
Xiao, G., Tang, J., Zuo, J., Guo, Yang, S., Tang, H., Fu, Y., and Han, S. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. arXiv preprint
2024.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F,, Wei, H., al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15113 2024.
Yuan, J., Liu, H., Zhong, S., Chuang, Y.-N., Li, S., Wang, G., Le, D., Jin, H., Chaudhary, V., Xu, Z., et Kv cache compression, but what must we give in return? compre- hensive benchmark of long context capable approaches. arXiv preprint arXiv: 240701577 2024.
Zhang, T., Yi, J., Xu, Z., and Shrivastava, A. Kv cache is 1 bit per channel: Efficient large language model inference with coupled quantization. arXiv preprint arXiv:2405.03917, 2024a.
Zhang. X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z., et al. oeBench: Extending long context evaluation beyond 100k tokens. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15262-15277, 2024b.
Zhang, Z., Sheng. Y., Zhou, T., Chen, T., Zheng, L.., Cai, R., Song, Z., Tian, Y. Ré, C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-3411 -37100, 2023.

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
A. Appendix
A.1. Explanation of Encoding and Decoding Using Commutative Codebooks
In this section, we provide an additional explanation of how the 2-dimensional sub-vector kg is represented using the quantized vector sf and the codebook Cj as defined in Eqn. 18.
We first formulate the quantization process as a clustering problem, where the cluster centers are defined as follows:
Ca.b [ Ck(la
[[] C6
(24)
Here, Ck(a] and Cj [b] represent the th and boh x 2 sub-codebooks in Cj-- Consequently, the codebook Cj forms a total of N2 clustering centers. We can then quantize Kū into its nearest clustering center, and use {a, b} as the quantized representation to represent ky. When decoding, we use the assigned clustering center to approximate Ag. so
k Ca.b
0 [
(25)
= 0
k
[a]
+
And Eqn. 25 is doing exactly the same thing Eqn. 18.
A.2. EM Algorithm Implementation Details
As mentioned in Sec 4.2, we apply a EM-like algorithm to learn the codebook. In this section we provide the algorithm detail. We a subset of FineWeb-Edu (Lozhkov et al., 2024) the calibration K and optimize the codebook over this calibration set. The E Step is straightforward as it simply assign each vector k in K to its nearest clustering center. For M Step, we derive a closed form formula to update C₁K given the current assignment S. Recall the definition of C₁
C = {OK
C
C 2
(26)
C# =
(27)
I
We first define some useful terms:
Φ = Iza Vili-0 N2-1
m [m: k.j= -
(28)
S diag (N N N1p)
N
where Φ is a dimmensnao vector that represent the codebook Cj& and we are going to optimize it. m is a N3- dimensional vector that represents the mean for the data points assigned to each cluster center. is 2N x 2N matrix that has the total number data points assigned to each cluster center in its diagonal.
Next, we define an auxiliary constant matrix E (11,0,1) (2N, ()x(2 (x(22)) where its entries are given by:
T2(oNo491.22 1, T2) +B)1,y 1, TeloN
(29)
(30)
(31)
for v, y E 10.1...., N₂r - 1}. and all other elements 7 are zero
We can then rewrite Eqn. 18 into the matrix form:
min TT S(T m)
(32)
12

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
The closed form solution given by
(T ST) Sm
(33)
We iteratively update & until it converges. During our experiments, we found that the learning process was not stable and would frequently fail to optimize since the number of the clustering centers is large in case (e.g., there 4096 clustering centers forN, 64) and the clustering centers needs to satisfy Eqn. 24. In order to stabilize the optimization process, we employ two techniques. The first technique is soft clustering center assignment, where instead of hard-assigning a data point to its nearest clustering center, we distribute it among all clustering centers with different weights. The weight assigned to each center depends on its proximity to that data point, with the closest center receiving the highest weight. To be specific, let N be the number of data points in the calibration set K, and Nec be the number of clustering centers, weight matrix W E xx is calculated as
e Du
Wi
(34)
Exe-DA D&s
where D IR NxNee is the L2 distance matrix measuring the L2 distance between each data point and each clustering center. We then use W instead of the hard assignment to calculate m and S in Eqn. 28.
Furthermore, we empirically observe that in the early iterations, the distribution of W needs to be smoother to prevent dead clustering centers. In contrast, in later iterations, a sharper distribution of W helps achieve better convergence. Therefore, we introduce temperature annealing in Eqn. 35 to regulate the distribution:
Wi
(35)
Σn e
where 7 is the temperature parameter, which exponentially decays over the iterations.
A.3. Codebook Size Analysis
Our method uses vector quantization with a codebook to compress the KV cache, which requires additional GPU memory to store the codebook. The codebook configuration is shown in Table and stored FP16. The total codebook size is calculated as:
Value Codebook Size (MB) N. x d x 2
(36)
x Key Codebook Size (MB) = 2 x 2 x No x R x 2
(37)
where d is the hidden size of the LLM. For LLaMA-3.1 1...AA-..--BB-ine 1-8B-Innttruct model, d = 1024. N. is the number of rows in the value codebook. For the key codebook, Ne. means the number of quantization level, means the number of the residual quantization.
We analyze the codebook size for both 2-bit and 1-bit quantization for LLaMA-3.1 1-B--B.--BBBB--- -nstrrutt model, shown in Table 7. For comparison, KV cache with a 128K context length requires 256 MB each for values and keys. Notably, the codebook size remains constant regardless of the number of tokens, making its memory overhead negligible for long-context LLM inference.
A.4. Ablation Study on Commutative Codebook Configuration
We utilize specially designed commutative codebook to quantize key cache as described in Sec. 4.2. There are three hyper-parameters that control the level of compression: No, the number of quantization levels, Rt, the number of the residual quantization, and the number of sub-ve in group that share the same quantization vector s. Revisit that the formula to compute the average quantization bit is:
Avg. bit Rlog,
(38)
All experiments in this section are conducted using LLaMA-3.1 1-8---Insttttttt model and the quantization error, measured in MSE, is calculated using the key cache for the first layer of the LLaMA model on a subset the Web-Web (Lozhkov et al., 2024) dataset.
13

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
Avg. Bit
bit
2 bit
Value Codebook
2.00 MB
4.00 MB
Key Codehook
2.75 MB
5.25 MB
Table Analysis on Codebook Size. We calculate the codebook size based on
model.
bit
2 bit
N:
1024
2048
R
11
21
Ner
64
64
Table 8. Codebook Configuration. N₂ used for value codebook, and R, N2 used for key codebook.
In our first ablation study, we examine the effect of when we make R and keep the average quantization bit to be the same, as shown in Table 9. We also vary R for different g to keep the average quantization bit to be as shown in Table 10.
Ne
R
MSE
g
Ne
R
Avg. bit
MSE
8
1
0.2699
8
2
8
bit 0.0798
16
1
0.2011
16
4
8
bit 0.0790
32
16
1
0.1265
32
16
8
bit 0.0254
64
64
1
0.0906
64
64
11
bit
0.0095
Table Ablation study on y with 1, while main- taining consistent Avg. bit.
Table 10. Ablation study on g and while keeping the Avg. bit to be 1.
From both Table and we can conclude that when keeping the average quantization bit the same, larger g will result in a lower quantization error, though in the cost of an increased number of Ne which will induce an increase in the computation complexity stated in Eqn. 23. Comparing the same row in Table and 10 that has the same g and Ng, further conclude that larger R will bring lower quantization error when keeping g and No unchanged.
0.02
1bb
2-bit
0.00
Figure 4. Ablation study on while keeping g = and Nor = 64.
Next, we keep 64 and No = 64, and vary R to get the quantization error for different compression rate. The result is shown in Figure 4. The R value for 1-bit quantization and 2-bit quantization is labeled in the figure by blue dot. As can see, increasing A will continuously decrease the quantization error, in the cost of a higher average quantization bit.
In conclusion, a larger and larger R will lead to better quantization accuracy, but also lead to higher computation and lower compression rate. As a result, to achieve a good trade-off between quantization accuracy, computation cost and
14

---

CommVQ: Commutative Vector Quantization for KV Cache Compression
compression rate, set y = 64, Nº 64 for all main experiments, and 11 for 1-bit quantization and 21 for 2-bit quantization respectively.
A.5. Quantization Error Comparison
We conduct experiments compare the quantization error of our method, measure in MSE between the original KV cache and the decoded KV cache, with our baseline method. To be specific, use the value cache from the first layer of LLaMA-3.1 1-8------ntt model for MSE calculation. Asymmetrie quantization serves the baseline for comparison. From Table 1, we observe that our additive quantization- based method outperforms asymmetric quantization, particularly when the quantization bit is low, such as in the case of 1-bit quantization.
Method
Avg. bit
MSE
Asymmetric quant.
0.00030
bit 0.00014
CommVQ
Asymmetric quant.
0.00380
bit
CommVQ
0.00027
Table Comparison of MSE between Comm VQ asymmetric quantization used (Liu et al., 2024b), calculated using the cached value matrix from the first layer of LLaMA-3. 1.8---Inssrre.. The MSE is evaluated on small subset of the FeneWeb-Edu dataset (Lozhkov al., 2024).
15


Reviews:
[{"review_id": "X5wM22oJls", "reviewer": "Reviewer_86AS", "summary": "This paper introduces CommVQ, a novel approach to compress the KV cache during inference, particularly when processing long contexts. Unlike existing scalar-based quantization methods, CommVQ employs vector quantization at the token level using a learned encoder and codebook approach. CommVQ makes two key innovations: (1) leveraging additive quantization to compress each vector in the KV cache into a low-bit representation and (2) designing a codebook that commutes with RoPE, allowing for efficient integration with the self-attention mechanism. This approach achieves impressive compression rates while maintaining high accuracy compared to baseline methods, even enabling effective 1-bit quantization with minimal performance degradation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback and insightful suggestions. Below, we address your concerns.\n\n**1. CommVQ applied to latest models such as Qwen-2.5**\n\nTo evaluate our method's generalization to the latest models, we applied CommVQ to the **Qwen-2.5 7B** model and evaluated it on LongBench. Due to the limited time for rebuttal and compatibility issues (e.g., KIVI and KVQuant do not officially support Qwen in their open-sourced code), we used **KV-4**, the built-in 4-bit KV cache quantization method (HQQ) in HuggingFace Transformers v4.46.2, as our baseline.\n\n| Method      | Avg. bit | Qasper | QMSum | MultiNews | TREC | TriviaQA | SAMSum | LCC   | RepoBench-P | Average  |\n|:-------------:|:--------:|:--------:|:--------:|:------------:|:------:|:-----------:|:---------:|:--------:|:----------------:|:--------:|\n| FP16        | 16 | 13.04  | 20.70  | 22.47      | 72.50 | 89.47     | 46.16   | 58.97 | 64.51         | 48.48  |\n| KV-4 (HQQ)  | 4 | 4.59   | 12.58  | 7.85       | 35.92 | 25.70     | 10.63   | 15.00 | 15.12         | 15.92  |\n| **CommVQ-2** | 2 | 14.58  | 22.57  | 24.05      | 68.00 | 87.04     | 45.34   | 55.46 | 61.48         | **47.31**  |\n\nCommVQ achieves strong performance while compressing the KV cache to an average of 2 bits, significantly outperforming the default 4-bit quantization baseline. This indicates that our method can be effectively applied to the latest models. We also demonstrate that CommVQ can be applied to much larger models, such as the **LLaMA-3.1 70B** model, in our response to **Reviewer yREq (1. Experiments on larger models)**. All these results show the broad applicability and effectiveness of CommVQ.\n\n\n**2. Latency comparison with the FP16 baseline and prior methods such as KIVI**\n\nWe report latency per generated token (in seconds) using the **LLaMA-3.1 8B** model on a single NVIDIA H100 80G GPU. Both KIVI and our method are optimized with Triton kernels. We set the batch size to 1 and vary the context length. As shown below, CommVQ-1 is consistently faster than KIVI-1, especially as the context length increases.\n\n| Context Length | FP16   | KIVI-1 | CommVQ-1 |\n|:---------:|:--------:|:--------:|:-----------:|\n| 8K      | 0.024  | 0.045  | **0.031** |\n| 16K     | 0.026  | 0.058  | **0.033** |\n| 32K     | 0.031  | 0.102  | **0.051** |\n| 64K     | 0.037  | 0.190  | **0.085** |\n| 128K    | 0.051  | 0.297  | **0.152** |\n\nWhile both CommVQ and KIVI appear slower than the FP16 baseline in our current measurements, we believe this is primarily due to practical factors such as the usage of flash attention (the FP16 model uses flash attention by default, while KIVI and our method currently do not support flash attention during the generation stage), Triton kernel launch overhead, and the hardware and model we chose. Importantly, as the context length grows, CommVQ achieves better efficiency than KIVI. We are actively optimizing our Triton implementation and expect further latency improvements in future versions.\n\n\n**3. Lack of some detailed proofs, such as why RoPE embedding is commutative in Property 1**\n\nWe thank the reviewer for pointing this out. Due to the page limit, we have omitted some proofs in the main paper. We will add detailed proofs to the main paper in our future revisions.\n\nAs for why RoPE embedding is commutative in Property 1, since\n\n$$\nR_m^iC =\n\\\\left(\\\\begin{array}{cc}\n\\\\cos m \\\\theta_i & -\\\\sin m \\\\theta_i \\\\\\\\\n\\\\sin m \\\\theta_i & \\\\cos m \\\\theta_i \\\\\\\\\n\\\\end{array}\\\\right)\n\\\\begin{pmatrix}\n    x & y \\\\\\\\\n    -y & x \\\\\\\\\n\\\\end{pmatrix}=\n\\\\begin{pmatrix}\nx\\\\cos m\\\\theta_i + y\\\\sin m\\\\theta_i & y\\\\cos m\\\\theta_i - x\\\\sin m\\\\theta_i \\\\\\\\\nx\\\\sin m\\\\theta_i - y\\\\cos m\\\\theta_i & y\\\\sin m\\\\theta_i + x\\\\cos m\\\\theta_i\n\\\\end{pmatrix}\n$$\n\nand\n\n$$\nCR_m^i =\n\\\\begin{pmatrix}\n    x & y \\\\\\\\\n    -y & x \\\\\\\\\n\\\\end{pmatrix}\n\\\\left(\\\\begin{array}{cc}\n\\\\cos m \\\\theta_i & -\\\\sin m \\\\theta_i \\\\\\\\\n\\\\sin m \\\\theta_i & \\\\cos m \\\\theta_i \\\\\\\\\n\\\\end{array}\\\\right)=\\\\begin{pmatrix}\nx\\\\cos m\\\\theta_i + y\\\\sin m\\\\theta_i & y\\\\cos m\\\\theta_i - x\\\\sin m\\\\theta_i \\\\\\\\\nx\\\\sin m\\\\theta_i - y\\\\cos m\\\\theta_i & y\\\\sin m\\\\theta_i + x\\\\cos m\\\\theta_i\n\\\\end{pmatrix}\n$$\n\nWe can see that they are equivalent, so we can confirm Property 1.", "claims_and_evidence": "The motivation for using vector quantization is clear, while the proposed method further addresses the computational cost of naive VQ.", "methods_and_evaluation_criteria": "The design of commVQ makes sense, which addresses the critical efficiency bottleneck when combining VQ with RoPE embedding. Moreover, experiment results show the effectiveness of proposed method.", "theoretical_claims": "The paper's claims are right. However, it lacks some detailed proof, such as why RoPE embedding is communicative in Property 1. Therefore, I would expect the authors to focus more on this part, as understanding it is very important for the design of CommVQ.", "experimental_designs_or_analyses": "The experimental designs include various long-context benchmarks; however, only LLaMa-3.1-8B-Instruct, LLaMA-2-7B, and Mistral-7B are included. Therefore, I would expect more experiments to be conducted on the latest models, such as Qwen-2.5.\n\nMoreover, I would expect some latency comparison with FP16 attention and prior quantization methods such as KIVI. The current results show the speedup of the new codebook, however, it is still unclear when comparing to other methods.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zgi1EBlHEO", "reviewer": "Reviewer_Vmsc", "summary": "In general, this paper quantizes the KV cache into a 1-bit representation and then uses the 1-bit representation to combine some basis vectors for the attention process. Intuitively, it is equivalent to decomposing the KV cache into a combination of a finite number of basis vectors to speed up the overall calculation process. It performs well on long sequence processing benchmarks such as Long-Bench and Infinite-Bench, proving the effectiveness and efficiency of the solution.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and helpful suggestions. Below, we address your concerns.\n\n\n**1. Applicability to retrieval tasks such as RULER**\n\nWe appreciate the suggestion to evaluate additional retrieval-specific tasks. We have included results on the **RULER** benchmark using the **LLaMA-3.1 8B** model. We set the context length to 128K. As shown below, CommVQ-2 achieves the highest average score among the methods that can achieve an average quantization bit of 2. CommVQ-1 also retains competitive retrieval ability under the extreme 1-bit quantization.\n\n\n| Method | Avg. bit | Niah1 | Niah2 | Niah3 | MKey1 | MKey2 | MKey3 | MValue | MQuery |   VT   | CWE  |  FWE  | QA1  | QA2  | Avg  |\n|:-----------:|:----------:|:-------:|:-------:|:-------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:------:|:-------:|:------:|:------:|:------:|\n| FP16      | 16       | 99.4  | 96.6  | 99.6  | 97.4   | 68.9    | 55.7    | 89.3   | 97.3    |  59.0   | 0.1   |  75.0  | 71.4 | 41.4 | 73.2 |\n| KVQuant | 2.33     | 89.0  | 36.0  | 40.8  | 37.0   | 1.0    | 0.0    | 24.3  | 26.4  | 36.7  | 0.3 | 66.0  | 25.4 | 24.4 | 31.3 |\n| KIVI    | 2.00     | 31.0  | 0.6   | 0.0   | 1.0    | 0.0    | 0.0    | 0.0    | 0.0    | 0.16   | 0.4  | 30.8  | 12.2 | 8.0  | 6.5  |\n| **CommVQ-2**  | 2.00     | 97.2  | 91.4  | 92.4  | 95.0   | 61.0   | 4.8    | 78.0  | 88.4   | 49.1  | 0.26 | 72.9 | 68.4 | 39.6 | **64.5** |\n| **CommVQ-1**  | 1.03     | 63.0  | 53.6  | 11.6  | 64.4   | 18.8   | 0.0    | 23.6   | 24.0   | 28.6   | 0.18 | 67.7 | 63.4 | 37.0 | **35.1** |\n\n\nThese results demonstrate that CommVQ can preserve retrieval capability even under aggressive compression while performing better than other methods under the same compression rate.\n\n\nApart from RULER, we also conducted experiments on the **Needle-in-a-Haystack test**, which also focuses on retrieval; please refer to our response to **Reviewer 9BoD (1. Full LongBench and Needle-in-a-Haystack results)** for details. The NIAH result figures are presented in this [link](https://github.com/commvq/CommVQ/blob/main/fig1.png). \n\nIn summary, CommVQ under 2-bit quantization can preserve the full retrieval capability of the FP16 model, while our 1-bit quantization version performs better on the NIAH test than KIVI's 1-bit quantization version. Moreover, from Table 2 in our main paper, CommVQ achieves good performance on retrieval tasks (namely R.PK, R.Num, and R.KV), especially in the extreme 1-bit quantization case where CommVQ significantly outperforms the baselines. Our comprehensive results demonstrate our method's effectiveness on retrieval tasks.\n\n\n**2. Comparison with low-rank decomposition**\n\nThanks for the suggestion, and we chose to compare CommVQ with **Palu**, a state-of-the-art low-rank decomposition KV cache compression method published at ICLR 2025. We conducted experiments on LongBench using the **LLaMA-3.1 8B** model. We set the context length to 128K. As shown below, CommVQ consistently outperforms Palu at both 2-bit and 1-bit quantization levels. This shows that our method is **more effective** than low-rank decomposition methods under various compression rates.\n\n\n| Method | Avg. bit | Qasper | QMSum | MultiNews | TREC  | TriviaQA | SAMSum | LCC   | RepoBench-P | Average |\n|:----------------:|:----------:|:--------:|:-------:|:-----------:|:-------:|:----------:|:--------:|:-------:|:--------------:|:---------:|\n| FP16           | 16       | 25.19  | 23.31 | 26.82     | 72.50 | 91.65    | 43.49  | 52.47 | 49.01        | 48.05   |\n| Palu-30% (3 bits) | 2.10   | 11.71  | 24.25 | 26.16     | 66.50  | 86.73    | 42.99  | 50.14 | 51.13        | 44.95   |\n| **CommVQ-2**       | 2.00     | 24.67  | 24.36 | 26.48     | 72.50 | 91.92    | 43.98  | 53.02 | 46.92        | **47.98**   |\n| Palu-60% (3 bits) | 1.20   | 2.80   | 16.37 | 11.06     | 51.08 | 3.43     | 6.56   | 18.66 | 21.36        | 16.42   |\n| **CommVQ-1**       | 1.03     | 18.86  | 23.02 | 24.34     | 69.00 | 91.61    | 41.83  | 48.78 | 42.08        | **44.94**   |", "claims_and_evidence": "The paper well supports the authors' claim.", "methods_and_evaluation_criteria": "This paper does not build its benchmarks and mainly uses the typical benchmarks (LongBench and InfiniteBench) to evaluate the effectiveness and efficiency of models. In addition, we argue that the benchmark RULER should be included in this paper.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "The typical benchmarks (LongBench and InfiniteBench) are used to evaluate the effectiveness and efficiency of models. We argue that this paper should add the benchmark RULER to evaluate the context retrieval ability.", "relation_to_broader_scientific_literature": "This paper is related to the current sparse attention mechanism including KV cache compression and eviction.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "This method can compress KV-cache into a 1-bit form and performs well on typical long-sequence benchmarks. However, I still have some concerns.\n\nAt the experimental level, the paper's work performs poorly on retrieval tasks. However, the RULER and other retrieval tasks are now the core evaluation benchmarks for evaluating long-sequence processing. It is recommended that the quantization method in this paper should be analyzed to determine whether it can be used for retrieval tasks. \n\nAt the model level, decomposing the KV cache into a combination of basis vectors can achieve 1-bit cache quantization, but I think it is better to perform low-rank decomposition directly. Low-rank decomposition can guarantee the model's performance and be easily accelerated on the underlying hardware. It is recommended that low-rank decomposition be included as a baseline to prove the necessity and advantages of 1-bit quantization.", "other_comments_or_suggestions": "Please refer to the 'Other Strengths And Weaknesses' part.", "questions_for_authors": "Please refer to the 'Other Strengths And Weaknesses' part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GaSebVPC8x", "reviewer": "Reviewer_yREq", "summary": "* This paper leverages additive quantization by introducing a lightweight encoder and codebook to compress the KV cache, which can then be decoded with a simple matrix multiplication.\n* The authors design a  commutative with Rotary Position Embedding (RoPE), and utilize an ExpectationMaximization (EM) algorithm to learn the codebook which allows for efficient integration of decoding into the self-attention mechanism to reduce computation overhead.\n* Paper shows an impressive 87.5% reduction in FP16 KVCache size while maintaining accuracy on standard long context datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and positive assessment of our direction and formulation. Below, we address the key concerns raised.\n\n\n**1. Experiments on larger models (e.g., 70B)**\n\nWe focused on 8B models (LLaMA-2, Mistral, and LLaMA-3.1) due to their popularity and our resource constraints. However, to address your suggestion, we have now evaluated CommVQ on the **LLaMA-3.1 70B** model under 1-bit quantization using LongBench. Results are shown below:\n\n\n| Method      | Avg. bit | Qasper | QMSum | MultiNews | TREC  | TriviaQA | SAMSum | LCC   | RepoBench-P | Average |\n|:------------:|:--------:|:--------:|:--------:|:------------:|:--------:|:-----------:|:---------:|:--------:|:----------------:|:---------:|\n| FP16       | 16 | 45.46  | 23.75  | 27.76      | 75.50  | 93.18     | 46.23   | 37.62 | 55.24         | 50.59   |\n| KIVI-1     | 2.00 | 7.90   | 6.92   | 11.13      | 37.50  | 52.21     | 8.99    | 24.02 | 21.26         | 21.24   |\n| **CommVQ-1** | 1.03 | 37.96  | 22.03  | 22.17      | 58.50  | 92.41     | 38.63   | 33.16 | 44.93         | **43.72**   |\n\nThese results show that CommVQ generalizes well to the 70B model, achieving strong accuracy using a 1-bit KV cache. This confirms CommVQ’s scalability and practical utility in larger models.\n\n\n**2. Actual practical benefits for 1-bit quantization**\n\nCompressing the KV cache to 1-bit offers several practical benefits. The most significant benefit is a 16× memory reduction compared to the standard FP16 KV cache. These memory savings are applicable regardless of the model's precision, meaning the model doesn't need to operate at 1-bit to take advantage of 1-bit KV cache quantization.\n\nSuch a substantial memory reduction not only allows a single GPU to handle longer context lengths and larger batch sizes (as shown in Figure 2 of our main paper), but it also significantly reduces data transfer time when offloading the KV cache to non-GPU memory. This benefit is especially notable in offloading scenarios, which are common when serving large models or deploying on edge devices. In these cases, limited PCIe bandwidth often makes KV cache transfer a major bottleneck. Reducing the size of the KV cache helps mitigate this issue, leading to faster overall inference.\n\n\n**3. Clarification on \"87.5% reduction\"**\n\nWe clarify that the 87.5% reduction refers to the reduction from **FP16 (16-bit) KV cache to 2-bit**, i.e., (1 - 2 / 16) = 87.5%. This applies to CommVQ-2. For CommVQ-1 (1-bit), the reduction reaches **93.75%**, enabling even more aggressive compression.\n\n\n**4. More insights behind the codebook, encoder and EM algorithm**\n\nThank you for your suggestions. We chose to use a simple neural network as the encoder, as it proved to be simple yet effective in our preliminary experiments. We selected the EM algorithm because it offers two advantages:\n\n1. The EM algorithm **converges very quickly**, requiring far fewer iterations than gradient-based approaches.  \n2. The EM algorithm incorporates principled techniques—such as **soft assignment** and **annealing**—to prevent **mode collapse**, where some centroids become obsolete after a few iterations. This has been one of the major challenges in quantization algorithms.\n\nRegarding the codebook configurations, we included the ablation study in the Appendix primarily due to page limitations. As suggested, we will provide more insights in the main paper in future revisions.", "claims_and_evidence": "* \"Existing quantization techniques treat each scalar in the KV cache independently, CommVQ performs quantization at the vector level. The method uses additive quantization\" This seems like a very promising insight.\n* \"We refine our codebook to be RoPE-commutative. This refinement enables us to reformulate the self-attention computation to incorporate the decoding process more efficiently. \" Great to see focus on efficiently achieving the required quantization, making deployment easier.\n* Codebook is learnt using a simple neural network by minimizing the reconstruction error and the technique used is EM. Would have been great so see some more insights behind the choice of this technique.", "methods_and_evaluation_criteria": "* Neural network-based learnt codebook with EM algorithm to minimize the reconstruction error. Overall this approach makes sense and is intuitive.\n* Datasets used are standard for long-context evaluation for LLMs.\n* We provide an ablation study on how to choose Nc′, R and g in Appendix A.4. These seem to be crucial for CommVQs performance and it would have been good to have at least the insights behind the choices in the main body of the paper.", "theoretical_claims": "* Equations in Section 4.2 seem convincing and make sense. However, a lot of the key details seem to be moved to the appendix. It would be helpful to at least have the main insights from them in the main body of the paper.", "experimental_designs_or_analyses": "* The experimental section is well set and baselines are appropriate. I was surprised to see the evaluation with only models of 8B parameters, where the KV Cache is still relatively small. Given that KV Caches of larger models would greatly benefit from quantization, it would be helpful to see CommQV's performance on 70B or larger models.\n* Performance gains of CommQV-2 seem to be marginal. This seems to suggest that CommQV only performs well only for 1-bit quantization. My concern here is how useful gains on 1-bit quantization are, do operators typically use 1-bit quantized models? Would be great to see some evidence for this.\n* I am missing where the initial claim of 87.5% reduction is coming from. Is this for reducing to 2-bit from 16-bit? In that case, the 2-bit quantization seems to have marginal gains. Most of the gains seem to be in 1-bit, but then the  reduction should be \n* Figure 2 has impressive results, I liked the focus on reduced cost and overhead and the extensive demonstration of the same.", "relation_to_broader_scientific_literature": "* This is a very promising direction and quantization of KV Cache has extremely practical consequences. I liked the way the problem was structured for both maintaining quality and having efficient inference.", "essential_references_not_discussed": "References are adequate.", "other_strengths_and_weaknesses": "* Promising overall direction with a dual focus on quality and cost of compression.\n* Insights on commutativity of ROPE is very useful.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "* I'd encourage the author to provide more insights into that 1-bit quantization has actual practical benefits. At first glance, it seems a little strange why 16 bits can be compressed to 1-bit with a win on both efficiency and quality.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dNbTvhNp5L", "reviewer": "Reviewer_2m1K", "summary": "This paper proposes a novel method, CommVQ, for compressing the KV cache in LLMs. The core innovation lies in using additive vector quantization—treating each token’s key/value vector as a unit rather than quantizing individual scalars—and designing a “commutative” codebook that allows efficient integration with RoPE. Experimental results on multiple long-context benchmarks and reasoning benchmarks show that CommVQ reduces memory usage while maintaining high accuracy relative to other KV cache compression baselines. The authors also provide an implementation that demonstrates real memory savings, enabling longer context sizes and larger batch sizes on a single GPU.\n\n## update after rebuttal\n\nThe authors have addressed my questions. I keep my opinion that this paper leans toward being accepted.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and constructive feedback. Below, we address the key concerns raised.\n\n**1. Latency comparison**\n\nPlease see our response to **Reviewer 86AS (2. Latency comparison with the FP16 baseline and prior methods such as KIVI)** for a detailed latency comparison. In summary, CommVQ-1 is consistently faster than KIVI-1, especially as context length increases, as measured on the **LLaMA-3.1 8B** model using a single NVIDIA H100 80G GPU.\n\n**2. Compatibility with other compression methods**\n\nCommVQ is compatible with other KV cache compression techniques like token eviction and quantization.\n\nFor **token eviction or retrieval-based methods**, which retain only key tokens during decoding, CommVQ integrates naturally by quantizing just those essential tokens. This allows further compression as only selected tokens are encoded and decoded using the learned codebook.\n\nCommVQ can also benefit from **codebook quantization**, reducing storage and speeding up decoding via low-bit matrix operations.\n\nIn short, CommVQ is **orthogonal** to other methods and can be combined with them for greater compression. Exploring such combinations is a promising direction for future work.\n\n**3. Robustness under domain shift**\n\nThanks for pointing this out. In practice, we find that our codebooks and encoder trained on general pre-training datasets (e.g., FineWeb-Edu) transfer reasonably well across tasks and domains. To validate this point, we conducted an ablation study on the **LLaMA-3.1 8B** model to show CommVQ-2's perplexity changes compared to the FP16 baseline on 4 datasets:\n1. **FineWeb-Edu**: a **general** dataset.\n2. **GSM-8K**: a **math** benchmark.\n3. **Repobench-p**: a **code** retrieval and completion benchmark.\n4. **KV_Retrieval in InfiniteBench**: a **synthetic** UUID key-value retrieval benchmark.\n\nThe first dataset represents in-domain evaluation, while the last three represent evaluations with domain shifts—i.e., the codebooks and encoder are trained on **general text** and tested on **math**, **code**, and **synthetic UUID data**.\n\n| | FineWeb-Edu | GSM-8K | Repobench-p | KV_Retrieval |\n|:---:|:---:|:---:|:---:|:---:|\n| FP16 | 10.17 | 5.67 | 2.20 | 31.93 |\n| CommVQ-2 | 11.54 | 6.14 | 2.78 | 32.72 |\n| PPL Diff | 1.37 | 0.47 | 0.58 | 0.79 |\n\nWe find no significant increase in perplexity (PPL) due to domain shifts when compared to in-domain evaluations. This suggests that our method performs consistently well across domains that differ from the calibration data, including synthetic UUID data, which is unlikely to appear in the calibration set. Overall, we conclude that our method is robust and generalizable under domain shifts.\n\nFinally, if a significant domain shift is encountered, we recommend further fine-tuning the encoder and codebook on domain-specific data—similar to best practices in other calibration-based quantization methods.\n\n**4. Additional benchmark (e.g., LongBench v2)**\n\nThank you for the suggestion. Due to the limited time available during the rebuttal phase, we conducted additional evaluations on **LongBench v2** using the **LLaMA-3.1 8B** model and compared them against **KIVI**, **KVQuant**, and **VQLLM**. KIVI and KVQuant fail to produce meaningful output with 1-bit quantization, so their results are omitted from the table. As shown below, CommVQ continues to outperform the baseline methods at comparable average quantization bit levels.\n\n| Method | Avg. bit | Easy | Hard | Short | Medium | Long | Overall |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| FP16 | 16 | 27.1 | 25.4 | 30.6 | 24.2 | 22.2 | 26.0 |\n| KIVI-2 | 3.00 | 25.7 | 24.8 | 25.6 | 25.7 | 23.1 | 25.1 |\n| KVQuant-2 | 2.33 | 26.2 | 21.0 | 26.7 | 20.1 | 22.4 | 23.0 |\n| VQLLM-2 | 2.00 | 15.6 | 17.7 | 21.1 | 15.3 | 13.0 | 16.9 |\n| **CommVQ-2** | 2.00 | 24.5 | 26.0 | 28.3 | 23.7 | 24.1 | **25.4** |\n| VQLLM-1 | 1.00 | 7.8 | 6.8 | 8.9 | 8.4 | 1.9 | 7.2 |\n| **CommVQ-1** | 1.03 | 25.5 | 22.2 | 26.7 | 21.4 | 22.2 | **23.5** |\n\n**5. Explanation on how to incorporate the commutative constraints during codebook training**\n\nThank you for pointing this out. The commutativity constraint is enforced by restricting each 2D subvector in the codebook to the form `[[x, y], [-y, x]]`, which ensures commutativity with RoPE rotations. In other words, for each 2D subvector, we only learn two scalars—x and y—and use them to construct the subvector for computation. We will provide a more detailed explanation in our future revision.\n\n**6. Code and Implementation**\n\nUpon acceptance, we will open-source our training pipeline, model weights, and optimized Triton kernels to facilitate reproducibility and further research. Please refer to the response to **Reviewer 9BoD (2. Triton kernel implementation details)** for explanations of the Triton kernel implementation.", "claims_and_evidence": "The major claims are supported by experiments (e.g. reduced KV size and better performance)", "methods_and_evaluation_criteria": "Overall make sense.\n\nCan consider more dataset with long generation like MATH/AIME.", "theoretical_claims": "The complexity computation part is correct.\n\nFor the RoPE part, it do not contain rigorous proof but no issues stand out.", "experimental_designs_or_analyses": "Experiment design is valid.\n\nNeed to report throughput/latency to show efficiency", "relation_to_broader_scientific_literature": "This work is about KV cache compression. Related to literatures like quantization, token eviction, etc.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The commutative codebook idea is elegant and likely generalizable to many LLMs using RoPE-based positional encoding.\n- The experiments show strong performance on multiple tasks at extremely low bit-rates (1–2 bits).\n\nWeaknesses:\n- No throughput or latency comparison to existing compression baselines is provided. Demonstrating real-world decode speed on multiple tasks or system loads would bolster confidence in the claimed advantages.\n- The method seems does not combine well with other KV cache compression or retrieval-based methods.\n- Lack of exploration of how domain shifts (in the data used to learn the dictionary vs. real downstream data) might affect compression quality.\n- Benchmark dataset, consider LongBench v2, MATH or AIME.", "other_comments_or_suggestions": "- Some details on how to incorporate the commutative constraints during codebook training could be expanded.\n- The paper might include best practices for calibrating the dictionary if the user’s data distribution significantly shifts from the calibration domain.\n- It would be good to release code/implement details in supple/appendix.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QrdFV8uEBk", "reviewer": "Reviewer_9BoD", "summary": "This paper introduces CommVQ, which sigificantly reduces KV cache memory in long-context LLMs while preserving accuracy. It uses additive quantization with a lightweight encoder and a RoPE-commutative codebook for efficient self-attention integration. In practice, CommVQ enables 1-bit KV cache quantization with minimal accuracy loss.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and thoughtful suggestions.\n\n**1. Full Longbench and Needle-in-a-Haystack results**\n\nThe full LongBench consists of 21 tasks in total. Following prior works such as KIVI and KVQuant, we report results on the same eight representative tasks for fair comparison and consistency. To address your comment, we now provide results on the full LongBench benchmark, including all 21 tasks, using the **LLaMA-3.1 8B** model. \n\nThe results below show that CommVQ-2 maintains accuracy across nearly all subtasks, while CommVQ-1 achieves competitive performance even under extreme 1-bit compression.\n\n|              | FP16  | KIVI-2 | CommVQ-2 | KIVI-1 | CommVQ-1 |\n|:----------------------:|:-------:|:--------:|:----------:|:--------:|:----------:|\n| Avg. bit             | 16    | 3.00   | 2.00     | 2.00   | 1.03     |\n| Qasper               | 25.19 | 22.71  | 24.67    | 4.99   | 18.86    |\n| QMSum                | 23.31 | 24.33  | 24.36    | 9.57   | 23.02    |\n| MultiNews           | 26.82 | 27.29  | 26.48    | 9.20   | 24.34    |\n| TREC                 | 72.50 | 72.50  | 72.50    | 38.75  | 69.00    |\n| TriviaQA             | 91.65 | 92.06  | 91.92    | 25.07  | 91.61    |\n| SAMSum               | 43.49 | 43.26  | 43.98    | 11.93  | 41.83    |\n| LCC                  | 52.47 | 51.32  | 53.02    | 17.67  | 48.78    |\n| RepoBench-P          | 49.01 | 47.53  | 46.92    | 16.40  | 42.08    |\n| NarrativeQA          | 31.69 | 31.47  | 32.20    | 2.87   | 29.80    |\n| MultifieldqaEN       | 29.16 | 27.50  | 29.47    | 7.11   | 24.93    |\n| MultifieldqaZH       | 19.95 | 19.92  | 19.47    | 4.83   | 18.97    |\n| HotpotQA             | 17.18 | 20.14  | 19.97    | 7.88   | 16.48    |\n| 2wikimQA             | 16.36 | 17.14  | 16.98    | 5.85   | 14.11    |\n| Musique              | 11.64 | 11.99  | 12.43    | 3.77   | 9.40     |\n| Dureader             | 29.66 | 27.76  | 26.59    | 7.79   | 22.06    |\n| GovReport            | 34.54 | 34.12  | 31.87    | 9.16   | 26.41    |\n| Vcsum                | 16.15 | 15.94  | 16.39    | 9.40   | 15.45    |\n| Lsht                 | 46.00 | 45.00  | 45.50    | 17.25  | 31.50    |\n| PassageCount         | 6.02  | 8.19   | 8.04     | 5.83   | 8.29     |\n| PassageRetrievalEN   | 98.45 | 97.25  | 98.55    | 23.02  | 97.28    |\n| PassageRetrievalZH   | 77.72 | 67.45  | 84.25    | 2.76   | 89.00    |\n| **Average**          | 39.00 | 38.33  | **39.31**| 11.48  | **36.34**|\n\n\nThese results further validate CommVQ’s generalization across a wide range of long-context tasks and domains.\n\nWe also provide the results for the **Needle-in-a-Haystack** test using the **LLaMA-3.1 8B** model. The NIAH result figures are included in this link: https://github.com/commvq/CommVQ/blob/main/fig1.png\n\nWe can see that CommVQ-2 preserves the full retrieval capability of the FP16 baseline, while our 1-bit quantized version, CommVQ-1, significantly outperforms its counterpart, KIVI-1.\n\n**2. Triton kernel implementation details**\n\nOur Triton kernels implement the techniques described in Section 4 of the main paper. Specifically, they include:\n\n1. A kernel that fuses RoPE application with commutative codebook decoding, reducing intermediate memory operations;  \n2. A mixed-precision batched matrix multiplication for efficient computation;  \n3. Loading low-bit representations on the fly, avoiding the need to upcast them to higher precision as required in native PyTorch implementations.\n\nTogether, these optimizations reduce memory access overhead and improve compute utilization by fully leveraging Triton’s capabilities. This implementation is under active development and will continue to be optimized. We will release the Triton kernels, along with code and implementation details, upon acceptance.", "claims_and_evidence": "As listed at the end of Introduction section, the main claims made the submission are \n\n(1) maintain performance with a  per-token quantization without using small group\n\n(2) cash out real world efficiency through the commutative property of the RoPE matrix and the characteristics of self-attention\n\n(3) enable 1-bit KV Cache quantization\n\nAll these three points are justified in the experimental section", "methods_and_evaluation_criteria": "This paper follows the common practice of KV Cache compression work. The selected benchmark, i.e., longbench and InfiniteBench, and evaluation criteria make sense for the long context senarios.\n\nHowever, I believe LongBench has much more subtasks besides the reported 8 tasks. Also, the it would be great to see the needle in a haystack test", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes, it follows default setting of LongBench and InfiniteBench", "relation_to_broader_scientific_literature": "It is nice to see 1-bit KV Cache quantization can still maintain the performance in the long context scenario!", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "There is no details on the Triton implementation of the proposed method. How the kernel is implemented and why it is faster than the baseline?", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers are consistent in their opinion that this is a nice result that could be accepted.  
Overall, the reviewers view the paper positively while indicating several possible improvements, including additional proofs and adding low-rank decomposition as a baseline.
Decision: Accept (poster)