SQuat: Subspace-orthogonal KV Cache Quantization
Authors: Hao Wang
Ligong Han
Kai Xu
Akash Srivastava
Venue: COLM 2025 (2025)
Keywords: KV cache quantization
LLMs
TL;DR: We propose a KV cache quantization method that preserves task-critical information throughout the quantization process.

Abstract: The key-value (KV) cache accelerates LLMs decoding by storing KV tensors from previously generated tokens. It reduces redundant computation at the cost of increased memory usage. To mitigate this overhead, existing approaches compress KV tensors into lower-bit representations; however, quantization errors can accumulate as more tokens are generated, potentially resulting in undesired outputs. In this paper, we introduce SQuat (Subspace-orthogonal KV cache quantization). It first constructs a subspace spanned by query tensors to capture the most critical task-related information. During key tensor quantization, it enforces that the difference between the (de)quantized and original keys remains orthogonal to this subspace, minimizing the impact of quantization errors on the attention mechanism’s outputs. SQuat requires no model fine-tuning, no additional calibration dataset for offline learning, and is grounded in a theoretical framework we develop. Through numerical experiments, we show that our method reduces peak memory by $2.17\times \sim 2.82\times$, improves throughput by $2.45\times \sim 3.60 \times$, and achieves more favorable benchmark scores than existing KV cache quantization algorithms.

Published as a conference paper at COLM 2025
SQuat: Subspace-orthogonal KV Cache Quantization
Hao Wang", Ligong Han; Kai Xu, Akash Srivastava Red Hat Al Innovation
Abstract
The key-value (KV) cache accelerates LLMs decoding by storing KV tensors from previously generated tokens. It reduces redundant computation the cost of increased memory usage. To mitigate this overhead, existing approaches compress KV tensors into lower-bit representations; however, quantization errors can accumulate more tokens are generated, poten- tially resulting in undesired outputs. In this paper, we introduce SQuat Luusspace-rttogonn KV cache quantization). It first constructs sub- space spanned by query tensors to capture the most critical task-related information. During key tensor quantization, it enforces that the difference between the (de)quantized and original keys remains orthogonal to this subspace, minimizing the impact of quantization errors the attention mechanism's 's outputs. SQuat requires no model fine-tuning, no additional theoretical
calibration dataset for offline learning, and is grounded framework we develop. Through numerical experiments, we show that our method reduces peak memory by 2.17x 2.82x, improves through- put by 2.45x ~ 3.60x, and achieves more favorable benchmark scores than existing KV cache quantization algorithms. Our code is available at: https:/ httpgb..tthho///gog Red-HHa-All-nnoye Teaonn-TTaam/-aae (SQut
Introduction
The future of AI technology should be accessible to everyone and operate efficiently, Just as computers evolved from bulky, expensive machines into widely available, affordable tools, Al undergoing a similar transformation. Today, large language models (LLMs) are increasingly used in many applications, yet challenges remain. Unlike traditional software applications that execute tasks almost instantly, LLMs even during diference-requeee significant computational resources to process user queries. To reduce compute costs in ruto-regressive decoding, causal language models often use the key-value (KV) cache. This cache stores previously computed keys and values for the attention mechanism, allowing the model to avoid redundant calculations. In essence, it is just like how humans remember the context of conversation-we do not start from scratch with every reply, but build on what was said before.
major bottleneck introduced by the KV cache the increased GPU memory consumption and the latency from frequent data transfers between memory and compute units (Lienhart, 2024; Verma & Vaidya, 2023). The challenge amplified in modern reasoning models (OpenAT, 2024; Guo et al., 2025), which often require long responses to "think" through problems before arriving at final answer. Moreover, inference-time scaling methods further exacerbate these issues by increasing context lengths or the number of concurrent decoding paths (Lightman et al., 2023; Rush & Ritter, 2024). To address this challenge, a burgeoning line of research investigates strategies to optimize KV cache management, mitigate memory constraints, and accelerate inference speed (see e.g., Kwon et al., 2023; Dubey et al., 2024; Liu et al., 2024a). Within this effort, KV cache quantization is crucial approach that compresses stored keys and values into lower-precision formats. eliminates the need for re-training or fine-tuning I.MMs, preserves full-context information, and remains compatible with methods like token pruning and model compression.
"qqull contribution. Correspondence to: (hao-wang, lihan hee.lliieaannatettee @eredhat.com

---

Published as a conference paper at COLM 2025
Existing approaches to KV cache quantization typically treat it as a lossy data compression problem (see e.g., Yang et al., 2024; Liu et al., 2024e; Kang et al., 2024; Hooper et al., 2024b). They aim to minimize the discrepancy between full-precision KV tensors (e.g., FP16) and their low-bit representations (e.g, INT2) without explicitly accounting for their effect on the model's next-token prediction. Consequently, while they preserve the numerical fidelity of KV tensors to some extent, task-critical
for accurately responding to user que be lost. Moreover, quantization errors from early tokens can accumulate and amplify during decoding, causing later tokens to deviate from expected outputs and eventually leading degraded response quality or misaligmment with user queries. This motivates a fundamental question:
How we preserve task-releoant information during KV cache quantization to ensure that the LLM generates outputs as if using full-precision KV tensors?
In this paper, we introduce SQuat (Subspace-orthogonal KV cache quantization), new approach for KV cache quantization. motivated by our two observations:
Obseroation 1. The attention scores in transformer architectures are calculated by taking the inner product between the query tensors of the new token and the key tensors of all past tokens. Thus, when quantizing key tensors, the primary objective should be to preserve their inner products with future tokens' query tensors, rather than merely minimizing the difference between the original and (de)quantized key tensors.
Observation 2. Query tensors tend to lie within subspace whose dimension significantly smaller than the hidden dimension. Hence, even without knowing future tokens' query tensors at the time of quantizing the current token's key tensors, we can still leverage this subspace to guide the quantization process.
Building on these insights, SQuat first constructs task-relevant subspace using the query tensors from all tokens in user-provided prompts. It then quantizes each token's key tensors, ensuring that the residuals (i.e., differences between the (de)quantized and original key tensors) stays as orthogonal to this subspace as possible, reducing the quantization error's s effect on critical task information (see Figure for our pipeline). SQuaf requires no model training, no additional calibration data for offline learning, and is grounded in a theoretical framework we develop.
SQuat is based on an optimization that balances the trade-off between maintaining quanti- zation fidelity and keeping residuals orthogonal to the query subspace. We introduce an iterative algorithm that approximately solves this optimization problem. At each step, element (or block of elements) of the key tensors is quantized, followed by an update to the remaining elements. This update minimizes the quantization error's 's impact on inner products with vectors in the query subspace. We prove that the optimal update rule has closed-form expression and present an efficient algorithm for computing it. This effort significantly reduces computational complexity and enables SQuat to run on-the-fly.
We conduct extensive numerical experiments to evaluate SQuat and compare it against tuning-fret baselines. Specifically, we apply these methods to quantize the KV cache of four LLMs: 11ama-2-78 (Touvron et al., 2023), Llama-3. 1- -IB--rrruttt (Dubey al., 2024), Mistral-78-Instrucct
(Jiang et al., 2023), and DeepSeek-R1-Distille et al., 2025). We evaluate their performance across two benchmark families: (1) set of challenging reasoning tasks that often need long responses before reaching the final answer, and (2) LongBench tasks that focus on long-context understanding, such as document QA and summarization, which require long input windows (Bai et al., 2023). Together, these benchmarks cover 14 tasks. SQuat achieve more favorable performance than existing tuning-fres baselines. When applied to the Llama-2- 7- model, SQuat reduces peak memory usage by 2.17x ~ 2.82x compared to the default FP16 format, resulting in a throughput improvement of x.45x x.60
(Guo
2

---

Published as a conference paper at COLM 2025
1.1 Related Work
KV cache management. Recent research has explored advanced KV cache management techniques to optimize inference efficiency and reduce memory usage in LLMs. One strategy focuses on token pruning, eviction, or merging (Xiao al., 2023b; Beltagy et al., 2020; Kim et al., 2022; Ge et al., 2023; Liu et al., 2024d; Han et al., 2023; Zhang et al., 2023; Lee et al., 2025; Wang et al., 2024; Wan et al., 2024; Zhang et al., 2024b; Jiang et al., 2024), where older or less relevant tokens are removed from the KV cache, or similar and redundant tokens are merged. Some methods (Cai et al., 2024; Li et al., 2024) allow for selective eviction of KV tensors across different attention layers rather than uniformly pruning KV tensors for the same tokens at all layers. A related technique (Xu et al., 2024) prunes specific key tensor channels instead of entire tokens, leveraging query information to guide the pruning process. Beyond pruning-base bbaen methods, there are other techniques to reduce the memory overhead of storing the KV cache. For example, Tang et al. (2024); Hooper et al. (2024a); Ribar et al. (2023) propose storing the full KV cache but dynamically load only the relevant aen keys and values during inference. These approaches, along with advancements in system and architecture design, are orthogonal to our method on KV cache quantization and can potentially be combined together to enhance GPU utilization.
KV cache compression. There has been active research on KV cache quantization (Tao et al., 2024; Yang et al., 2024; Byun et al., 2024; Zhang et al., 2024a; Liu et al., 2024e; Kang et al., 2024; Hooper et al., 2024b; Sheng et al., 2023; Zandieh et al., 2024; He et al., 2024; Yue et al., 2024; Liu et al., 2024c; Chang et al., 2024; Zhang & Shen, 2024; Duanmu et al., 2024; Zhao et al., 2024). For example, Xiao et al. (2023a) Sheng et al. (2023) introduce 8-bit and 4-bit group-wise quantization for both model weights and KV cache. More recently, Liu et al. (2024e); Kang et al. (2024); Hooper et al. (2024b) explore per-channel quantization for key tensors and per-token quantization for value tensors, as key tensors show significant variation across channels, whereas values do not. Similarly, Duanmu et al. (2024) propose rearranging KV cache channels to enhance similarity within quantization groups. Hooper et al. (2024b) further introduce Pre-RoPE quantization, outlier isolation, and non-uniform quantization. Kim et al. (2024) develop an input-independent dictionary and represent KV tensors sparse linear combinations of its elements. These advancements enable 2-bit quantization while maintaining performance comparable to non-quantized baselines. Among these approaches, Liu et al. (2024e); Kang et al. (2024); He et al. (2024), like our method, do not require fine-tuning or a calibration dataset. Liu et al. (2024e) present a hardware-friente GPU implementation, inspiring new quantization method integrated into Hugging Face Transformers (Turganbay, 2024). Kang et al. (2024) reduce quantization error by storing a low-rank matrix and sparse matrix. He et al. (2024) consider first identifying salient tokens and then applying mixed-precision quantization to compress the KV cache. As we will discuss in Section 3.1, these compression- compression-baseed based approaches do not explicitly account for quantization errors' impact on attention outputs. In contrast, our method preserves the inner product between key tensors and future tokens' query tensors, minimizing guantization-induced deviations in LLM outputs. A comprehensive comparison with these methods will be provided in our numerical experiments (Section 4).
Efficient system and architecture designs. Optimizing computer systems and transformer architectures for efficient LLM inference has been an active research area. One line of research focuses on system design for optimizing KV cache management on GPUs (Jin et al., 2023; Yu et al., 2022; NVIDIA, 2023). Among them, Kwon et al. (2023) propose paged attention, a block-based dynamic memory allocation system that serves as the foundation of vL..M, a widely adopted library for LLM inference and serving. Another research direction explores modifications to attention mechanisms in transformer-based LLMs to reduce KV cache size (Child et al., 2019; Katharopoulos et al., 2020; Choromanski et al., 2020; Wu & Tu, 2024; Liu et al., 2024b;a; Yuan et al., 2025; Yu et al., 2024). For example, multi-query and grouped-query attention mechanisms maintain multiple query heads while sharing single set of KV pairs across all heads within groups of heads (Ainslie et al., 2023; Shazeer, 2019). This architecture has been incorporated into many ooun-source LLMs, including Llama (Dubey et al., 2024) and PaLM (Chowdhery et al., 2023).
3

---

Published as a conference paper at COLM 2025
Model compression. The study of pruning, quantization, and sparsification in neural networks has a long history (Gupta et al., 2015; Frankle & Carbin, 2018; Frantar & Alistarh, 2022), originating from early efforts (LeCun et al., 1989; Hassibi et al., 1993) to optimize computational efficiency and memory usage while preserving accuracy. Recently, these techniques have gained increasing attention in the context of L.MMs, which have grown to hundreds of billions of parameters, demanding substantial compute resources for both training and inference (Frantar & Alistarh, 2023; Sun al., 2023; Frantar al., 2022; Xiao et al., 2023a; Yao et al., 2022; Dettmers et al., 2022; Kurtić et al., 2023; Malinovskii et al., 2024; Ashkboos et al., 2024). Our work focuses on quantizing the KV cache rather than model weights. Unlike static weight matrices, which can be pre-quantized and stored before inference, KV tensors are generated dynamically as new tokens arrive. This streaming nature demands quantization to be performed on-the-fly, requiring methods that minimize computational overhead and latency to ensure efficient decoding.
Background
(Masked) Attention mechanism. For the sake of illustration, we assume batch size is and the attention mechanism has single head. A transformer consists of L layers and each layer contains a multi-head attention and a feed-forward network. Suppose the input token embeddings are: X E Raxd
Query matrix: Q = XW Key matrix: K = XW Value matrix: V = XWV,
where Wº, w WV E R are the projection matrices. The attention mechanism outputs:
[OK" +M) V, where the mask M is strictly upper triangular
Attention Q K, V) = softmax
Vi
matrix, with zeros on and below the diagonal and oo in every element above the diagonal.
Inference-time workflow. In the prefill phase, let Xo E R'prompe Xi represent the input tensor, where /prompt the length of the user's 's input prompt and d denotes the model's hidden size. During this phase, the key (K) and value (V) tensors are computed and cached. This is followed by the decoding phase, in which the LLM predicts the next token sequentially, conditioned on the tokens generated so far. Note that LLMs only compute the query, key, and value for the newly generated token. The new key and value are then concatenated with the previously cached tensors. This caching mechanism significantly reduces computational overhead, as avoids recomputing the key and value tensors for all prior tokens at every decoding step, resulting in faster inference. However, the KV cache may become a memory bottleneck when handling long contexts or large batch sizes (see Appendix B). Compressing the KV cache- tache-through challenge.
through methods such quantization miantizationooffers a natural solution to this
Quantization. In practice, quantization is often applied to group of numbers ",. Xn simultaneously. In the context of KV cache quantization, these numbers can be elements of the KV cache for the same token across hidden dimensions (per-token quantization) ele- ments from multiple tokens within the same hidden dimension (per-channel quantization). Quantization begins by determining m min(x;) and M max | and then maps each number to a b-bit integer using:
x; A
M
qtz(x;) =
with
2
Here in and A are referred to as the zero-point and scaling factor, respectively. The operator 1-1 rounds to the nearest integer. After quantization, the b-bit integer falls within the range [0,2" - 1]. The quantized values, along with the zero-point and scaling factor, are stored to enable dequantization: (ēq(ī,) f x A + m.
4

---

Published as a conference paper at COLM 2025
orthogonal to the subspace
Step 1. (Section 3.2)
construct a
query subspace Q
Quantized keys
deq (kgg²
k
INT2 FP16
query salepace
Kn
4 n-
quantize it
Step 2. (Section 3.3)
update rem aining elements using Eq. (3)
run an iterative algorithm to quantize key tensors
Figure 1: The pipeline of SQuat for quantizing key tensors. We first identify subspace Q spanned by query tensors that carry the most relevant task information necessary for answering the user question. Then we quantize each key tensor kn element by element (or block by block) and update the remaining elements using Eq. (3) at each iteration. This update ensures that the difference between the (de)quantized and original key remains as orthogonal to the query subspace as possible, preserving critical information for responding to the user question.
3 Main Results
3.1 Motivation and Problem Formulation
Motivation. We first recall how existing methods (e.g,, Liu et al., 2024e; Kang et al., 2024; Hooper et al., 2024b) quantize the KV cache. When new token produces its key tensor, it is not immediately quantized but instead stored in residual buffer. Once this buffer accumulates R key tensors, they are quantized together. This process involves grouping the key tensors and computing a zero-point and scaling factor for each group. Existing methods explore different strategies to reduce quantization error, such per- channel grouping and pre-RoPE quantization. Despite these differences, they share the same fundamental goal: minimizing the difference between the (de)quantized and original key tensors: | deq [ ll (with the same principle applied to value tensors). Based on this criterion, we refer to them as compression- based quantization methods.
Compression-based methods overlook the impact of quantization on the attention mecha- nism itself. Since attention scores are determined by the inner products between query and key vectors, even small perturbations in key tensors can lead to disproportionate shifts in attention outputs. The following theorem formalizes this intuition.
Theorem 1.
For a new token with query tensor On E R2, we can upper bound the output of the attention mechanism when using original keys&roalues compared to quantized keys&roalues:
Attention qu, {k: (oi) - Attention(q.. qn. (K." | {0t
11=1
E
10;
deq o0t Ill
H
(ki(k; deq(k) 7) +
Σ
||
deq deq (ottt)))) ||2
2
Theorem indicates that the outputs of the attention mechanism remain unchanged when using quantized tensors, provided two conditions are satisfied: (i) the quantized value tensors closely approximate the original ones, and (ii) the key residuals-thh differences between the (de)quantized and original key tensors-an Accordingly, we quantize value tensors on per-token basis to minimize their approximation errors relative to the original value tensors. For key tensors, we quantize them while ensuring that key residuals remain orthogonal to the query tensors of future tokens.
orthogonal to the query tensor.
Problem formulation. We formalize the above intuition for key tensor quantization as optimization problem. Given key tensor our goal is to find its b-bit representation
5

---

Published as a conference paper at COLM 2025
k- such that the quantization error is minimized, while the residual k deq(k remains orthogonal to the subspace spanned by the query tensors of future likens-denoted by å to preserve task-relevant information. 'This leads to the following constrained optimization:
min | deq (kq (K 1l2 k4
Q(k -eeq(A)))))) 0, k- is vector of b-bit integers.
(1)
We face two challenges to solve this optimization. First, the query tensors of future tokens are unavailable when quantizing the current token's key tensor. In Section 3.2, we explore how to estimate Q without knowing future tokens. Second, the optimization in (1) is combinatorial and generally intractable. In Section 3.3, we introduce an efficient iterative algorithm to obtain an approximate solution.
and thus the subspace Q --are
3.2 Exploring Query Subspace
Our key observation is that query tensors typically lie within subspace (up to certain error) whose dimension is significantly smaller than the hidden dimension d. Importantly, this subspace can be derived directly from the prompt tokens, without requiring access to the response. In other words, even when future tokens are unknown at the time of quantizing the key tensors for the current token, this prompt-derived subspace can still guide the quantization process.
We illustrate this observation using a sequence from the math-500 dataset and generate its response with the Llama-3. 1-8BIInstrue model. To precisely define what it means for query tensors to lie within a subspace, we introduce the following definition.
Definition 1. For a query tensor 9, we define its deviation from given subspace P as ll Projp (g) 22, where Pr)jp(q) denotes the projection of onto P. When the query tensor normalized, its deviation from any subspace stays within [0.
We apply singular value decomposition (SVD) to query tensors from: (1) all tokens (prompt + response), (2) only prompt tokens, (3) all tokens from a different ma math-500 sequence, and (4) all tokens from a sequence in QuALITY (Pang et al., 2021), a multiple-choice QA dataset. In each case, we select the top singular vectors to form the subspace.
Figure 2 (left) illustrates the deviation of (normalized) query tensors of tokens in the math-500 sequence from the four subspaces. As shown, a 30-dimensional subspace can capture the entire sequence's query tensors with ~20% deviation error. Moreover, using prompt-only query tensors iitioors-vviilbllle before the decoding phase begins query tensors from a different math-500 sequence does not increase the deviation error. The deviation increases slightly, however, when the subspace is spanned using sequence from QuALITY, a different dataset. Based on these observations, we apply SVD to the query tensors of all prompt tokens. Then we define the task-relevant query subspace as Q E R constructed from the top singular vectors scaled by their corresponding singular values.
3.3 Key Tensors Quantization
We propose an iterative algorithm to solve (1). In each iteration, we selects a coordinate of k, quantize it, and update the remaining coordinates ensure the residual is orthogonal to the query subspace as much as possible. The process repeats for the remaining coordinates until all coordinates of k are quantized.
Specifically, we use k E Rd to represent the (de)quantized value of k at the 1-th iteration and set kg = k. At iteration t, the first - elements of k remain unchanged from k, as they have already been quantized. We then quantize the f-th element and update the remaining d f elements using (3) in the following proposition. The hyperparameter À E [0, 00) controls the trade-off between making the (de)quantized key close to its original values and
In Appendix A.2, we extend our algorithm to do block-by-block quantization of key tensors, rather than deement-by-elemen In our experiments, each key tensor is partitioned into no more than 4 blocks, leading to at most iterations.

---

Published as a conference paper at COLM 2025
Compression-bast methods
SQuat (ours)
Deviation subspace
0..
different dataset
0.6
different sequence
prompt only
0.5
prompt response
0.2
0.1
0.0
subspace Dimension
Figure 2: Left: query tensors tend to lie in a low-dimensional subspace. This subspace can be identified using query tensors from prompt tokens, which are available before decoding begins. Middle and Right: the absolute difference in attention scores between using original and quantized key tensors. As shown, our method better preserves attention scores. See Appendix D.1 for experimental setup.
enforcing orthogonality of its residuals to the query subspace. When À O, the algorithm reduces to compression-baseer quantization method. For any vector x, use [x]] to denote its i-th coordinate.
Proposition Consider the optimization problem
min EER" ||k - k: + Q(; - k
(2)
s.t. [kr]: = [k for i 1, , , and [kr] = (eq(qtz((((-1))))
We denote Pine (Q Suppose At Rext is the top-left block of Pino and B, (d-++]xt is the bottom left block of PinD- Let hy the last column of A, Then the optimal solution to (2) has its last d elements given by
[k, 1le + (dea(qtz((k, i)))) - [k, 1) Bih
(3)
The main computational cost in Proposition 1 arises from computing A, at each iteration, which has a complexity of O(E"_, f3) O(d*). The following proposition introduces an iterative algorithm to compute A, 1 It begins with A = I + ΛÂ Q by definition and subsequently derives A, from A A This way, reduce the complexity of the iterative algorithm in Proposition 1 from O(d)) to O(d)).
Proposition 2. Recall from Proposition 1 that A1+1 E (((++)x((++) is the top-left block of P Pixa- For f = d - -.. 1, if we write A
A4t1 a
where Arti E Rixt then we have
a++1
A, A+t1
1
a T
1++1-
ā++1
We show how our method preserves attention scores compared to compression--asee methods in Figure 2 (middle and right) and defer more comprehensive evaluation to the next section. A pseudo-code of our algorithm is provided in Algorithm 1, with an illustration shown Figure 5.
4 Numerical Experiments
We evaluate SQuat and SQuatPre for 2-bit quantization of the KV cache using four LLMs across 14 benchmark tasks, comparing them with four baseline quantization methods.
7

---

Published as a conference paper at COLM 2025
Method
KV size
GSMSk
MMLU_Pro ProLUPPro_aae Math
MMLU PMMLUPProJaA Law_Law
IFEval
GPQA
BBH
Average
Llana-2-78-hff
100%
14.03%
39.53%
KIVI
19.7%
11.98%
8,51%
12.53%
26.29%
9.60%
35.92%
18.86%
GEAR
22.5%
11.60%
9.25%
13.72%
27.94%
9.60%
33.77%
19.28%
ZipCache
21.6%
6.37%
7.48%
15.80%
25.71%
8.48%
24.73%
15.39%
SQuat
19.7%
13.42%
7.62%
15.35%
27.44%
9.82%
33.76%
19.59%
SQuat
19.7%
12.21%
6.66%
13.72%
25.62%
11.83%
36.85%
19.34%
Llana-3. 1..1 3.1--8-111111 88-Instruct
FP16
100%
77.56%
39.38%
26.70%
52.96%
16.74%
71.06%
50.27%
KIVI
21.4%
71.49%
30.72%
27.25%
50.80%
13.39%
59.62%
44.86%
GEAR
25.8%
69.37%
30.35%
27.79%
54.32%
15.40%
57.70%
45.17%
ZipCache
22.1%
72.02%
32.72%
27.43%
51.83%
11.16%
55.32%
44.08%
SQuat
21.4%
72.71%
33.83%
27.07%
54.14%
16.96%
60.55%
46.97%
SQuatPie
21.4%
22.56%
32.94%
27.34%
55.91%
17.86%
62.85%
47.86%
Mistral-78 78 78-Instruct-v0.
FP16
100%
50.49%
23.05%
24.34%
54.21%
18.30%
56.23%
40.59%
KIVI
20.7%
42.61%
20.21%
23.07%
54.01%
18.53%
47.77%
36.91%
SQuat
20.7%
45.26%
21.24%
23.52%
53.82%
19.87%
50.27%
38.32%
SQuatPre
20.7%
44.50%
22.95%
23.98%
52.28%
18.53%
50.79%
37.91%
Table 1: We compare SQuat and SQuatPre against existing tuning-free KV cache quantization baselines and FP16 (no quantization) across multiple tasks from LM-Eval. KV size is the average ratio of the compressed KV cache to its FP16 counterpart. For each task, the best result is shown in bold, and the second-best is underlined.
The key difference between SQuat and SQuatPre that SQuatPre quantizes and stores the KV cache before applying RoPE (Su et al., 2024), as recommended by Hooper et al. (2024b). Because SQuatPre applies positional embeddings on-the-fly after dequantization, it operates slightly slower than SQuat.
Baselines and LLMs. We consider FP16 (no quantization) and existing KV cache quanti- zation algorithms-Kot KIVI (Liu et al., 2024e), GEAR (Kang et al., 2024), ZipCache (He et al., 2024) which,, like our method, do not require fine-tuning or calibration dataset. We evaluate these methods on four open-source LLMs: Llama-2-78, Llama-3. 1--8-Innstruc DeepSeek-R1-Disttll---eae
We assess performance across several reasoning and long-context benchmarks, and report the KV size as the average percentage of the compressed cache relative to the FP16 cache at the end of generation. We present the experimental setup, including the choice of hapaeepassmeters for SQuat, SQuatPre, and other baseline methods in Appendix D.2.
Mistral-7B-Instrrut--
Reasoning tasks. We evaluate SQuat and baseline methods on range of reasoning tasks from LM-Eval (Gao al., 2024), including GSMBk, #MLU_Pro Pro_Math, MMLU_Pro_Lav IFEval, GPQA, and BBH. These benchmarks are widely adopted in prior work on KV cache compression and are known for their difficulty, often requiring multi-step reasoning (e.g,, thain-of-thoughh) before arriving at the final answer. In such settings, KV cache quantization plays a critical role by substantially reducing memory consumption. We present the results Table 1. As shown, SQuat achieves higher average scores across all models. Additionally, GSM8k, GPQA, and BBH-three particularly challenging reasoning tasks-out method consistently outperforms all baselines by significant margin.
Lench tasks. We evaluate SQuat and baseline methods on long-context benchmarks, following the recommendations of Liu et al. (2024e) and using the same set of tasks (Qasper, QMSum, MultiNews, TREC, TriviaQA, SAMSum, LCC, and RepoBench-P) from LongBench (Bai et al., 2023). These tasks evaluate the ability of LLMs to process and reason over long contexts, such as document-level question answering and summarization, and require deep understanding across diverse real-world scenarios. As shown in Table 2, SQuat achieves higher average
8

---

Published as a conference paper at COLM 2025
Method
KV size
Qasper
QMuu
MultiNews
TREC
TriviQQA
SAMSum
LCC
RepoBench.PP
Average
Llama-2-78-hr
FP16
100%
9.61%
21.15%
3.51%
66.00%
87.72%
41.53%
66.66%
59.81%
44.50%
KIVI
19.1%
9.43%
20.71%
0.92%
66.00%
67.42%
42.69%
66.47%
59.83%
44.18%
GEAR
20.7%
9.77%
20.47%
0.96%
66.00%
87 42%
42.42%
66.82%
59.48%
44.17%
ZipCache
19.9%
9.22%
19.63%
3.97%
64.50%
87.48%
40.21%
63.44%
55,99%
43.06%
SQuat
19.1%
10.32%
66.00%
87.28%
42.27%
59.59%
SQuat
88.09%
66.29%
59.27%
Llama-3.1
1-88-Instruct
IP16
100%
45.00%
23.40%
27.21%
69.50%
91.20%
44.01%
63.45%
55.25%
52.38%
KIVI
19.6%
44.03%
23.86%
26.79%
69.50%
91.88%
43.47%
62.48%
53.0%%
51.87%
GEAR
21.4%
43.13%
23.26%
26.79%
69.00%
90.74%
43.21%
61.82%
52.67%
51.35%
ZipCache
21.4%
42.87%
23.24%
91.66%
51.48%
SQuat
19.6%
42%
23.55%
92.19%
53.09%
52.06%
SQualPee
19.6%
44.25%
23.87%
26.78%
69.50%
92.02%
43.88%
63.00%
53.53%
52.10%
Mistral-78-
FP16
100%
40.54%
24.34%
27 23.83%
74.00%
88.39%
47.74%
58.39%
56.89%
52.27%
KIVI
19.2%
38.87%
25. 23.90%
26.94%
74.00%
58.34%
47.74%
37.64%
55.81%
51.66%
SQuat
19.2%
27.01%
88.89%
55.77%
51.7%%
SQuatFee
19.2%
23.80%
27.31%
87.94%
46.93%
56.42%
51.64%
Table 2: Comparison of different KV cache quantization methods on tasks from LongBench. For each task, the best result is shown in bold, and the second-best is underlined.
0.85
0.80
0.784
0.770
0.770
0.748
0.75
0.730
Score
0.70
0.65
0.60
0.55
0.516
0.50
FP 16
KIVI
GEAR
ZipCache
SQuatere
SQuat
Figure 3: Experiments using a reasoning model (Depseek-R--Disiil--- -ii---Dilllllaaae on the math-500 dataset. We compare SQuat and SQuatPre with different KV cache quantization baselines and FP16 (no quantization).
scores across all models than other quantization baselines and are nearly lossless compared with FP16.
Reasoning model. We evaluate - -Distill-Laama on the math-500 dataset (Lightman et al., 2023). We set the maximum number of new tokens 8k to ensure the model has sufficient window for multi-stage reasoning. This setup highlights key use case for KV cache quantization: modern reasoning models often require long responses to "think" through problems-e.g. by generating chain-of-thought at a final answer. In such scenarios, the KV cache becomes a memory bottleneck, and quantization proves especially helpful when compute resources are limited. As shown in Figure 3, both SQuat and SQuatPre outperform all existing KV cache quantization baselines.
before arriving
Efficiency comparisons. We apply SQuat to quantize the KV cache to 2-bit and evaluate its efficiency in terms of memory usage, latency, and throughput the Llama-2-78 model. Empirically, GEAR and ZipCache run much slower than KIVI (Liu et al., 2024e) and SQuat, compare SQuat only against KIVI and an FP16 baseline (without quantization) in

---

Published as a conference paper at COLM 2025
40
70
Wall clock time (sec.)
6k
Peak memory (GB)
35
Tokens per second
60
30
3k
baseline
30
20
(9738)
(residual length
6&1 your sighh
tresidual length 128)
20
50kel pooid lnigh
lingth 1281
50kel imoid wiigh 300
Souar lingth
SQuar (reviuual length 128)
4Quu (revituoa leagth 92)
100
200
30D
400
500
100
200
30D
430
500
100
200
300
400
500
Batch
Batch
Batch
Figure 4: We evaluate SQuat's efficiency by measuring peak memory usage (left), latency (middle), and throughput (right), comparing against KIVI (Liu et al., 2024e) and an FP16 baseline. As shown, SQuat requires the same peak GPU memory KIVI. In terms of throughput, SQuat achieves a 2.45x 3.60x improvement over FP16, which is comparable to KIVI's 2.63x
3.82x improvement.
Figure 4. To ensure fair comparison, we adopt the same experimental setup as KIVI, which synthesizes workloads based on ShareGPT (2023). We evaluate two residual buffer lengths (i.e., the number of most recent tokens whose KV cache is stored in full precision), 32 and 128, for both our algorithm and KTVI. We gradually increase the batch size until the GPU runs out of memory. All experiments are conducted on single Nvidia H100 GPU (80GB). As shown in Figure 4, SQuat reduces peak GPU memory by 2.17X 2.82x. This memory efficiency enables up to a 4x increase in batch size compared to the FP16 baseline, leading to throughput improvement of 2.45X 3.60x, which is comparable KIVI's throughput improvement of 2 ~ 3.82x.
5 Conclusion, Future work, and Limitations
The past years have seen rapid advancements in techniques for developing LLMs. As SOTA models continue to grow in size to achieve higher performance (Zhao et al., 2023), ensuring accessibility to these
sisqques--specialllly for those with limited computational resources- has become increasingly important. This paper responds to this call by exploring methods to make LLM inference-
decoding more efficient. We focus specific aspect of this challenge: KV cache quantization. Unlike existing compression-basee algorithms, our method quantizes key tensors with the objective of preserving their inner product with future tokens' query tensors. This design choice ensures that quantization errors minimally affect model outputs. We establish rigorous theoretical foundations for our algorithm and hope our work inspires new research that applies theoretical insights to practical LLM development, promoting efficient and accessible inference techniques.
There are several promising directions for further exploration. First, some recent LLMs do not store the KV cache directly. For instance, multi-head latent attention (Liu et al., 2024a) caches compressed latent vectors and applies projection matrices to generate key and value tensors during inference. Investigating whether these latent vectors can be further quantized, and understanding the impact of quantization errors model performance, is an interesting avenue for research. From a theoretical standpoint, also valuable to examine how reducing KV cache size- through quantization memory usage, and throughput. Identifying the optimal trade-offs between compression rate and model performance for specific tasks could offer insights into more efficient model deployment strategies.
-fffentss latency,
10

---

Published as a conference paper at COLM 2025
Acknowledgment
We would like to thank Shivchander Sudalairaj, Abhishek Bhandwaldar, Mustafa Eyceoz, Aldo Pareja from the Red Hat AI Innovation Team for their help in setting up some of the evaluation benchmarks.
References
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint Xrro:2305.13222 2023.
Wael Alghamdi, Hsiang Hsu, Haewon Jeong, Hao Wang, P Winston Michalak, Shahab Asoodeh, and Flavio P Calmon. Beyond adult and compas: Fairness in multi-class prediction. ar preprint XrXiv:2206.07801, 2022.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37: 100213-100240, 2024.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arv preprint ar Stt::2308.140000 2023.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document trans- former. arv preprint ar arXiv22004.0511 2020.
AH Bentbib and A Kanber. Block power method for svd decomposition. Analele Stüintifice ale Unioersitatii Ovidius Constanta, Seria Matematica, 23(2):45-58, 2015.
Woohong Byun, Jongseok Woo, and Saibal Mukhopadhyay. Hessian-aware kv cache quanti- zation for llms. In 2024 IEEE 67th International Midwest Symposium on Circuits and Systems (MWSCAS), pp. 243 247. IEEE, 2024.
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic ky cache compression based on pyramidal information funneling. Xiv preprint ar orXiv24406.2069) 2024.
Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Yu Lin, Chong- Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning- Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprint ar drXiv:2407.21118 2024.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint Xiv::1904 10509, 2019.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, al. Rethinking attention with performers. arXiv preprint ar Xiv::22009.14799 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318 (000008118333222 2022.
Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. Skvq: Sliding-window key and value cache quantization for large language models. Xiv preprint ar xrX:::240..06219 2024.
11

---

Published as a conference paper at COLM 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arxiv preprint or Xrx::24407. 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, train- able neural networks. arXiv preprint xrXi:11833.0365, 2018.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-4488, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference Machine Learning, pp. 10323-10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. ar preprint a<Xiv:2200.17323, 2022.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2024. URL https://zenodo org/records//2ee00 26686602.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for 1lms. arXiv preprint axXio:2310.01801, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint ar #xXx:25001112443 2025.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference on machine learning, pp. 1737-1746. PMLR, 2015.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.1611 2023.
Babak Hassibi, David Stork, and Gregory Wolff. Optimal brain surgeon: Extensions and performance comparisons. Advances in neural information processing systems, 6, 1993.
Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification. Advances in Neural Information Processing Systems, 37:68287-68307, 2024.
Coleman Hooper, Schoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, June Paik, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Accel- erating long context length Ilm inference. arXiv preprint ar 7Xiv:2411.09633 2024a.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length llm inference with kv cache quantization. ar Xiv preprint ar Xr1::2200.11807, 2024b.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
12

---

Published as a conference paper at COLM 2025
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin- Yew Lin, et al. Minference 1.0: Acceler- ating pre-filling for long-context Ilms via dynamic sparse attention. Advances in Neural Information Processing Systems, 37:52481 77-52581-52515, 2024.
Yunho Jin, Chun-Feng Wu, David Brooks, and Gu- GeoY Wei. S3: tion during generative inference for higher throughput. Advances in Neural Information Processing Systems, 36: 36:18015-80027 2023.
Increasing gpu utiliza-
Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipefor near-lossless generative inference of Ilm. Xiv preprint arXiv:2403.0552, 2024.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mnns: Fast autoregressive transformers with linear attention. In International conference machine learning, pp. 1166-5165. PMLR, 2020.
Junhyuck Kim, Jongho Park, Jaewoong Cho, and Dimitris Papailiopoulos. Lexico: Ex- treme kv cache compression via sparse coding over universal dictionaries. arXiv preprint ar arXiv:2412.08890 2024.
Schoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 784-794, 2022.
Eldar Kurtić, Elias Frantar, and Dan Alistarh. Ziplm: Inference-aware structured pruning of language models. Advances in Neural Information Processing Systems, 36:65397-65617, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-626, 2023.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Heejun Lee, Geon Park, Jaduk Suh, and Sung Ju Hwang. Infinitehip: Extending language model context up to 3 million tokens single gpu. arXiv preprint arXio:2502.088 089910 2025.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. arXiv preprini ar Xioo:240..14469 2024.
Pierre Lienhart. Llm inference series: kv caching, a deeper look. https::/medium. com/ 2024.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arxiv preprint ar Xiv:2305.20054 2023.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint ar arXin:2405.0443 2024a.
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: Ky cache compression in depth dimension for large language models. arv preprint ar Xiv:2205. 14366, 2024b.
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. ar Xiv preprint wrXin:2403.01241, 2024c.
13

---

Published as a conference paper at COLM 2025
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anas- tasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for Ilm kv cache compression at test time. Adoancess in Neural Information Processing Systems, 36, 2024d.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free free asymmetric 2bit quantization for kv cache. Xiv preprint ar XxX::22402.02750, 2024e.
Vladimir Malinovskii, Andrei Panferov, Ivan Ilin, Han Guo, Peter Richtárik, and Dan Alistarh. Pushing the limits of large language model quantization via the linearity theorem. arXiv preprint arXie:2411.17525 2024.
NVIDIA, Fastertransformer. https://github... http:///ggttt..cooo//////aaaaaaa 2023.
OpenAl. Introducing openai ol. https:://opemai.com// 2024.
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. Quality: Question answering with long input texts, yes! arXiv preprint Xt:::1112.08800 2021.
Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth-efficient 1lm inference. arXiv preprint arXie:2312.04995, 2023.
Sasha Rush and Daniel Ritter. Speculations on test-time scaling, https://github.. com/ srush/awesome--11 woweessmeeolttbbreeaeee htt.aa/rrddairroo----eeeeeeaaaaa 20224
ShareGPT. Sharegpt. https://sharegot. com, 2023,
Noam Shazeer. Fast transformer decoding: One write-head is all you need, arviv preprint arXiv:1911.02150, 2019.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with single gpu. In International Conference Machine Learning, pp. 31094-31116. PMLR, 2023.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. ar Xrv preprint ar Xiv:2306.116699 2023.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context I1m inference. arXiv preprint arXiv:2406.1077 10774, 2024.
Qian Tao, Wenyuan Yu, and Jingren Zhou. Asymkv: Enabling 1-bit quantization of kv cache with layer-wise asymmetric quantization configurations. arXiv preprint arXto:241011321, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arviv preprint arXiv:2307.092 2023.
Raushan Turganbay. Unlocking longer generation with key-value cache quantization. https://huggingfaae coo/blog/yy-ccch-qquuee quantizatinn, 2024.
Shashank Verma and Neal Vaidya. Mastering 1lm techniques: Inference optimization. https: .ww.ee...ee..iiiaaaa....a...a.ia nvidia.
//developer. 2023.
wtnttterrinnnmmmmmeeeeeeeee -nnattrrrinn-n---teeiiiuuussss cnnferrnnce-opimmizaee
14

---

Published as a conference paper at COLM 2025
Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, and Mi Zhang. D2o: Dynamic discriminative operations for efficient generative inference of large language models. arXiv preprint Xrv:24406 13035, 2024.
Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for Ilms on long-context tasks. arXiv preprint ar arXiv:2407.0844 08844,, 2024.
Haoyi Wu and Kewei Tu. Layer-condensed kv cache for efficient inference of large language models. arxiv preprint ar arXio:2405.10637, 2024.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks. arXiv preprint arX:::230 17453, 2023b.
Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. Think: Thinner key cache by query-driven pruning. Xrv preprint ar Xixv:2407.21012 2024.
Zhen Yang, JN Han, Kan Wu, Ruobing Xie, An Wang, Xingwu Sun, and Zhanhui Kang. Lossless kv cache compression to 2%. Xrviv preprint arXiv:2410.15252, 2024.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yux- iong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168-27118 2022.
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for ('Transformer-Bee B generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538, 2022.
Hao Yu, Zelan Yang, Shen Li, Yong Li, and Jianxin Wu. Effectively compress kv heads for 11m. Xiv preprint arXiv:2406.07055 2024.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-alignes and natively trainable sparse attention. ar arv preprint arXiv:2502.11089 2025.
Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quantizing weight and key/ /value cache for large language models gains more. Xrv preprint ar arXio:2402.12005
2024.
Amir Zandieh, Majid Daliri, and Insu Han. Qjl: 1-bit quantized jl transform for kv cache quantization with zero overhead. Xrv preprint arXiv:2406.0344 03482, 2024.
Hongxuan Zhang, Yao Zhao, Jiaqi Zheng, Chenyi Zhuang, Jinjie Gu, and Guihai Chen. Csr: Achieving bit key-value cache via sparse representation. arXio Xiv preprint ar Xr1::22412.11741, 2024a.
Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. Cam: Cache merging for efforryefficient lims inference. In Forty-first International Conference on Machine Learning, 2024b.
Zeyu Zhang and Haiying Shen. Zero-delay qkv compression for mitigating kv cache and network bottlenecks in 1lm inference. arviv preprint arXiv:2408.04100; 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-3477 2023.
15

---

Published as a conference paper at COLM 2025
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint ar XrX::2233..1822, 1(2), 2023.
Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate IIm serving. Proceedings of Machine Learning and Systems, 6:196-200 209, 2024.
A Omitted Proofs
A.1 Proof of Theorem 1
Proof. We denote deq (k? and v; deg a deq(v; qi io 1, 1, n. We introduce two vectors:
(qn, ky deeq,
(mmck1)
(qn, kn)
9nx kn
wodg
w
Vd
dd
Vd
Vd
Let ydeq Ruxd where the rows consist of oy and vi deg
respectively. Then we have
Attention(qu, - {=;) Attention(q., (K90))
1i=1 |-
10,100
1i=1
|softmax w)V - softmax (w vdeg 2
|softmax(w)) (V vdde) + (softmax(w)) softmax ( ydea|12 v
softmax w))( (V vdeq) + (softmax( w) softmax (eetaa (ee) Vdde]]l2-
Note that
(V- (VVVVVe))L
l|2 < Σ ||0; 2
Similarly, we have
|| (softmax(w) - softmax(wdeg)) ydee < |sofmax(w)) softmax(w0 ||2 Vdeq ||..
Note that softmax is 1 1/2-Lipschitz continuous (see Appendix A.4 in Alghamdi al., 2022). Therefore,
1 || softmax(w) - softmax(20d0))12 S _11 wolall22
(9n. (k1 k 2 + + (gn, (k,, kn deo
2vd
1
1n (k; - T.
2vd
A.2 Block-Wise Quantization
The algorithm presented in the main body quantizes the key vectors element by element. In this section, we extend it to perform block-wise quantization, improving efficiency by quan- tizing multiple elements of the key vectors simultaneously while updating the remaining elements. As special cases of this approach, we prove Proposition 1 and Proposition 2.
Before diving into the details, we first recall some standard results from linear algebra that will be used in proofs.
16

---

Published as a conference paper at COLM 2025
Lemma 1. Consider block matrix:
A C
B D
M
where A and are square matrices. We denote M/D - BD 'C M/A - Suppose D and M/D are both invertible. Then
CA IB.
(M/D) M -D-IC(M/D)- D DT++D'C(M/D)IBB
(M/D) BD
Similarly, suppose A and M/A A are both invertible. Then
A B
A A B(MA)
M
(M/A)
(M/A)
Using this lemma, we immediately obtain the following corollary, which is also a standard result in linear algebra. Corollary 1.
We denote
B D
P Q R
M
M
C
Suppose A and M/A A are both invertible. Then we have -1
P QS QS- RR.
Proof. Lemma 1 implies that
P A + A B(M/A)
Q A - B(M/A)
R (M/A) 'CA
S = (M/A)-1.
Substituting the last three equations into the first one yields
A-MAA-- QS-.
Now, we consider quantizing a block of g elements per iteration in sequential manner. Suppose the hidden dimension d of the key tensors is divisible by g. At iteration the first (t- 1)g elements have already been quantized and are kept fixed. During this step, we quantize the elements indexed from (f 1f- + to ig and update the rest of the elements by using the following lemma.
Lemma 2. Consider the optimization problem
min ||k k.-1ll2 + A 1 R,
s.t. [kr]; = [k(i): for = 1, (1-1)
(5)
[k]]: deq(qtz((kk for = (t- (t-1)g 1)g + 1, ,g.
We denote
P Pime (1+ 1 A B
C.
where A E R is the lop-left block of Pino- in (5) Jus a closed-form optimal solution:
Let H2 be the last g columms of A, The optimization
[i]] = { [k, ili i i- 1, ,,. (t-1)g 1)g
( ( ili)) i= (t- 1)g + +1, +,,...,
17

---

Published as a conference paper at COLM 2025
and the remaining elements of k are given by 11g- + B,H,d,
(6)
where d R8 has its i-th element corresponding to the (t 1)g i-th element of dfqq(tt((k(-1)) 1
Proof. We denote - and let d E R8 has its i-th element corresponding to the (t- (1- + i-th element of (et((tz(k; 1)) k, Moreover, we denote
I, where the identity matrix has dimension ig x ig and b E R's. The optimization problem in (5) can be rewritten as
= b. min 8 + 15 RC
This expression can be simplified to
min 8 P8, óER
Tg
where we denote P A Q Note that P is positive definite so it is invertible. We denote Pinv p-1 The Lagrangian of (7) is:
L(6,A) 8' A (Tō b).
The KKT conditions give
2P8 A
(8)
T8 = b.
(9)
Hence, we have
1
2 Piny T X,
1 TP inv A b.
TP
2
Suppose TP T is invertible. Then we have
A 2 inv ) b.
Substituting the above equation into (8) leads to
8 Pimy T
(TP inv T") -
(10)
A B:
B C
Recall that T = [I and Piny
Hence, we have
A B:
Pinv T
TP TP T = Ar.
=
As result, we have
P Pinv T (TP T²)
A B
(A+)
B((
Substituting the above equation and
into (10) leads
d
8
d
B, (A,)
-
18

---

Published as a conference paper at COLM 2025
Recall that H, is the last g columns of (A+)-1 Then
8
d
H,H,d
Now we are in position to prove Proposition 1.
Proof of Proposition 1. Proposition can be viewed as special case of Lemma 2 by setting 8 1. In this case, the vector becomes scalar deq deq(qtz((ke (eq(q(((kk))))) [k-1]; and the matrix H, becomes a vector ht
Lemma 3. Suppose we can write
M
N 1+1
N++1
0.41
where M Rixxig No41 x Rexig O.41 E R888, Then we have
= M,41 NZ,,
Proof. This lemma can obtained directly by applying Corollary
Proof of Proposition 2. This proposition is a special case of Lemma 3 and can alternatively be derived using Lemma from Frantar & Alistarh (2022).
Remark 1. Assume T = d/g is an integer. By definition, Ay P Pinv (I+ which implies that A, = 1+ A Then, by using Lemma 3, can iteratively compute
A
Proposition 3. The iterative algorithm based on Proposition 1 has computational complexity of Odt). By applying Proposition 2 to update A, this complexity reduced to (dd))
Proof. The matrix + AQT has dimensions d x d, and computing its inverse to obtain Piny has complexity? of (dd)) For A; R directly computing its inverse requires O(P) operations. Additionally, computing B,h; involves O((d f)f) operations. Since the iterative algorithm processes = 1,...,, d, the total computational cost is:
0 Σ p +
(d t)} O(d))
By leveraging Proposition the cost of updating A, reduces 0(P) per iteration, de- creasing the overall computational complexity O(d3).
B Memory Requirements for KV Cache Management
Maintaining a KV cache requires substantial memory. Specifically, for a batch of b queries, each with sequence length of (/prompt + fresponse), an LLM with L layers, h attention heads per layer, hidden dimension d, and precision p (e.g., 2 bytes per parameter for FP16) requires memory 2 x b (pprompp + (response) x L x h x d p bytes for storing the KV cache. In the case of an LLaMA-2 7B model with batch of 4 queries and a context length of 2048, the memory requirement is x 4 2048 32 32 128 x 2 bytes 4 GB.
19

---

Published as a conference paper at COLM 2025
tokens
INT2
FP16
attention
iteration
quantize
update
quantize
via Eq. (6)
each channel within this group shares the same zero point and scaling factor
residual buffer
iteration 2
Our algorithm
Figure 5: We illustrate how our iterative algorithm quantizes key tensors. We maintain residual buffer to store the most recent tokens' key tensors in their original precision. Once the buffer reaches a predefined size (e.g,, 32 in our experiments), we apply our quantization algorithm to all stored tensors. Specifically, we first quantize a block of elements (e.g., the first half of the hidden dimensions), then update the remaining elements using Eq. (6). This process continues iteratively until all elements are quantized.
C Details of Our Main Algorithm
We present the pseudocode of our main algorithm in Algorithm 1. As our main focus is on quantizing key tensors, we describe only this process. For value tensors, adopt a similar procedure with Algorithm in Liu et al. (2024e). Specifically, we maintain a residual buffer stores the most recent tokens' value tensors their original data type (e-g., FP16). When the residual buffer reaches size of R + 1, we quantize the oldest token's 's value tensor and remove it from the buffer. Value tensors are quantized per token, with each token's hidden dimension d divided into [d/G] groups of size G. For the sake of illustration, the hidden dimension d divisible by the number of elements quantized per iteration g the group size G; and that the residual buffer length R is divisible by G.
assume
Figure illustrates how our algorithm operates. Once the residual buffer reaches a prede- fined size, we apply our quantization algorithm to all stored tensors. We begin by quantizing a block elements, then update the remaining elements using (6),,), repeating this process until all elements are quantized. Compared to (3),,), Eq. (6) is more general formula- tion that supports block-wise quantization rather than quantizing one element at a time. For illustration, the figure assumes that the residual length equals the group size, though our method allows for setting the group size smaller than the residual length to enable finer-grained quantization.
In the pseudocode, we present our algorithm using multi-head attention. However, the method naturally extends grouped-query attention. For example, our experiments with uses grouped-query attention with 8 KV heads and 32
Llama-3. -ns----nstttttwwhhha-a- attention follow the same procedure. The only modification in Line for each KV head, we concatenate the query tensors from its associated query heads before applying SVD.
We assume batch size of 1 in the pseudocode. When the batch size exceeds 1, we share the same matrices first sample. This significantly reduces memory usage for large batch sizes. Empirically, we find that this does not degrade LLM performance compared to computing separate matrices for each sample. This aligns with our observation in Section 3.2 that query tensors tend to
A Piy across all samples in the batch, computing them from the
2If r is significantly smaller than the Woodbury identity can be applied to reduce this complexity to (rdP).
20

---

Published as a conference paper at COLM 2025
lie within small subspace, even when constructed from different sequences in the same dataset.
Algorithm 1 Main Algorithm for Key Tensor Quantization
Input: Residual length R, group size G, number elements to quantize per iteration g, hidden state dimension d, hyper-parameter A, subspace dimension
2: // Prefill Phase
3: Input: X R Rhmmp
4: Q = XW K XW
5: ,,,, V= (V)(())
D can be accelerated using block power method (Bentbib & Kanber, 2015)
6: Q Giagg([[ [[:
7: T d/g
total iterations required to run our algorithm
8: Compute AT = I+ Q and P = AQ+
9: Apply Lemma 3 to obtain A7 T-11" ,Ai
10: == |promp ,RR num_groou groupgrop--- = Pprmmp r\/G
11: Initialize Kquaat = and set Kfull = K promp r
12: // Quantize each group of keys (run in parallel)
13: for = 1 num group do
K2 [ii 1)G i:
P elements in the i-th group to be quantized
15: K quaat + QUAANT AT AT Pinv) Pin))
16:
Append KQuant K
17: end for
18: Decoding Phase
19: Input: E IR Rxd
20: k = xW and Append & to Kfull
21: num_group
R/G
22: if len(ftall))
R then for i to num, group do
23:
24:
K₂ (i
25:
quant
QUANT A ... Ar- Pinv)
26:
Append Kquant to Kquant end for
27:
28: Reset full =]
29:
end if
30:
function QUANT (K, A A Pinv)
31:
for f to T do
32:
Quantize K[: (t 1)g + 1: [g] per channel if < T then Update Kl: || using Lemma 2 with A, and Pinv end if end for
33:
34:
35:
36:
37:
Return: Quantized K
38:
end function
D More Experimental Results
D.1 Experimental Setup for Figure 2
We describe the experimental setup for the results presented Figure Unlike the ex- periments in Section 4, Figure 2 is based on semi-synthetic setup. Specifically, we first use the Llama-3. 1-8B-Instruct model to generate responses without KV cache quantization during decoding, ensuring that quantization errors do not alter the response trajectory. Additionally, we analyze query and key tensors before applying rotary position embeddings (RoPE) (Su et al., 2024) to isolate quantization effects from positional encoding. Figure (left)
21

---

Published as a conference paper at COLM 2025
Deviation from the subspace
0.30
different dataset
from subspace
0./
different dataset
Deviation from the subspace
different dataset
0.25
different sequence
different sequence
difterent sequence
prompt only
0.6
prompt only
0.4
prompt only
0.20
prompt response
0.5
prompt respanse
prompt respanse
0.15
0.4
0.10
0.3
0.2
0.2
0.05
0.1
0.00
0.0
0.0
75 Subspace Dimension
130
125
25
100
125
Subspace Dimension
Subspace Dimension
Figure 6: We reproduce the results from Figure 2 (left) for query tensors at different attention layers: the 1st layer (left), the 24th layer (middle), and the 32nd layer (right).
shows the deviation error for query tensors at the 12th attention layer. To further validate our findings, Figure extends these results across different layers.
D.2 Experimental Setup for Section 4
For SQuat, SQuatPre, we set the group size (i.e,, the number of elements quantized together) to 32 and the residual length (i.e., the number of most recent tokens whose KV cache is stored in full precision) to 32. The KV cache is quantized to bits. This configuration provides the best throughput and minimal memory usage (see Figure 4). For LM-Eval tasks and all LLMs, we set the subspace dimension to 5, the regularization coefficient to A = 0.001, and the number of elements to quantize per iteration to g = 64, resulting in just two iterations of our algorithm. For LongBench tasks, which involve longer inputs, we adjust the hyper- parametrts as follows. For Llama-2-7B-hf, we use
5, A 0.001, g = 32 for SQuat and r = 40, À = 0.001, g = 64 for SQuat. For Llama-3. 1-8--Instruct, set SQuatPre to 60, A 0.0005, 8 =32, and SQuat to 10, À 0.0005, g 64. For -attal--B-Instruct
3, we use r = =20, \ = 0.0005, = 64 for SQuatPre and r = 5, A 0.001, g 64 for SQuat. For our experiments using Reeseee-RI-Distll--ll.
8B the math-500 dataset (Figure 3), we ret = 40, À = 0.0005, and g = 64 for both SQuat and SQuatPre.
For KIVI and GEAR, we use the same configuration: a group size of 32, a residual length of 32, and 2-bit quantization for the KV cache. For KIVI, we use the official implementation from their GitHuub repository. For GEAR, we set rank = rankv = 2 and loop = 3, following the recommendations in their test script. Since their CUDA-based implementation does not support the Mistral model, we compare only on LLaMA models using their implementa- tion from GitHub repository. For ZipCache, we quantize important tokens to 4 bits and unimportant tokens to 2 bits, with streaming_gap = 32 to match the setup used for other baselines and our method. We set the unimportant_ratio to 0.6 for Llama-2-78 on LM-eval tasks, 0.7 on LongBench tasks, 0.5 for Llama-3. 1-88-Instruct on LM-eval tasks, and 0.6 LongBench tasks. These values ensure that their KV cache size is roughly aligned with that of other baselines, allowing for a fair comparison. As ZipCache is only implemented for LLaMA in their GitHub repository (as of the date of our paper submission), we include it in comparisons for LLaMA models only.
For LM-Eval, since MMLU_Pro Pro_Math and MMLU_Pro_Law all come from MMLU_Pro, we compute the weighted average in the follow way:
Average - GSM8K 10 (MMLU_Pro__M) Math + MMLU MMLU_Pro_Law) 1 IFEval GPQA + 5 BBH.
1
For LongBench, we follow the recommendation in Liu et al. (2024e) and set the maximum sequence length to 4,096 for Llama-2-78 and 8,192 for all other models. We report the KV size as the average percentage of the compressed cache relative to the FP16 cache at the end of the sequence. It includes all information needed to recover dequantize the KV cache back to its original data types. All our experiments are conducted on a single Nvidia H100 GPU (80GB).
22

---

Published as a conference paper at COLM 2025
À = 1 10
5
A = 1
A =
A =
1 10
r 5
71.72%
73.09%
72.71%
70.96%
4.47%
10
72.93%
71.57%
71.65%
70.81%
6.98%
r 20
73.62%
72.71%
71.27%
69.22%
5.61%
r 40
72.63%
72.48%
72.18%
66.95%
6.67%
= 60
73.24%
72.56%
71.87%
68.68%
16.83%
Table 3: Ablation study on the GSM8k dataset using the Llama-3.. 1--8-Innstruct model. We vary the regularization weight and the subspace dimension r in SQuat.
D.3 Ablation Study
We conduct ablation study two most important hyper-parameters in algorithm: the regularization weight in (2), and the subspace dimension of the query matrix Q Experiments are performed on the GSM8K dataset using the Llama-3. 1..1--B-inne 1-88-Instruct model.
Recall that the hyper-parameter original key vectors during (de)quantization and enforcing orthogonality of the residuals to the query subspace. When A 0, the algorithm simplifies to compression- based quantization method. The query subspace Q E Redd has top r important query directions. Results are summarized in Table 3. We find that performance degrades when > 10 3, This is because Q = diag(E[: v]: (Line in Algorithm 1) and the leading singular values Σ(: are typically large. Empirically, we observe that 0.001 and
[0, co) controls the trade-off between preserving the
consistently yield strong performance. Although this configuration does not achieve the best score GSM8K, we adopt it as the default for both SQuat and SQuatPre across all models and all tasks in LM-Eval.
23


Reviews:
[{"review_id": "0O0iRo6Jpr", "reviewer": "Reviewer_fNFy", "summary": "This paper introduces a tuning-free method SQuat for quantizing the KV cache in large language models (LLMs) by enforcing subspace-orthogonality between quantization errors and a task-relevant query subspace. Instead of minimizing raw quantization error, SQuat aims to preserve the inner products between queries and keys, which are critical to attention scores. It constructs this subspace using prompt query tensors via SVD and iteratively quantizes key tensors while constraining residuals to be orthogonal to it. This theoretically grounded approach improves inference efficiency without retraining or calibration data.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Unlike prior work that minimizes element-wise reconstruction error, SQuat introduces a theoretically grounded objective that minimizes the effect of quantization on attention outputs by ensuring key residuals are orthogonal to a task-relevant query subspace.\n- The method requires neither model fine-tuning nor additional calibration data, making it practical for deployment in existing LLM inference pipelines with minimal overhead.\n- SQuat is orthogonal to other cache management strategies (e.g., pruning, eviction) and compatible with state-of-the-art compression techniques, suggesting wide applicability in both research and production systems.", "reasons_to_reject": "- The query subspace is estimated solely from prompt tokens, assuming it generalizes well to future tokens. However, this assumption may not hold across diverse tasks or dialog settings where user responses shift semantics mid-generation. This could compromise the orthogonality constraint’s effectiveness in practice.\n- While the algorithm reduces the theoretical complexity to $\\mathcal{O}(d^3)$, it still introduces a non-trivial overhead compared to simpler methods like per-channel quantization. The practical cost of SVD, subspace projection, and iterative updates—especially for longer sequences and large-scale deployment—needs clearer profiling in the main paper.\n- The evaluation is focused on academic benchmarks and assumes full control of the inference stack. Real-world deployment scenarios, where key tensors may be quantized asynchronously or under hardware constraints (e.g., mobile inference), are not addressed, limiting the practical utility claims.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iSDQsU1J2G", "reviewer": "Reviewer_hftq", "summary": "This paper introduces SQuat, a KV cache quantization method that preserves task-relevant information by ensuring quantization errors remain orthogonal to the query subspace. Unlike traditional approaches that focus solely on minimizing numerical error, SQuat recognizes that preserving inner products between key and query tensors is critical for maintaining model performance. Experiments show memory reduction (2.17×-2.82×) and throughput improvement (2.45×-3.60×) while outperforming existing quantization methods across multiple LLMs and tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It introduces a novel conceptual framing for KV cache quantization that addresses fundamental limitations of current approaches.  \n- It provides rigorous mathematical derivation with proofs and an efficient implementation algorithm.  \n- It demonstrates consistent performance improvements across diverse models (Llama-2, Llama-3.1, Mistral, DeepSeek) and tasks (reasoning, long-context understanding).", "reasons_to_reject": "- Insufficient guidance on hyperparameter selection beyond ablation studies.  \n- Limited analysis of computational overhead during inference, particularly for initial tokens.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IUNb8u1XTC", "reviewer": "Reviewer_hLUL", "summary": "This paper proposes a method that considers potential future output errors during the quantization of the current key. The authors first demonstrate that these errors are upper-bounded by terms related to the product of queries and the quantization residuals of keys. Consequently, they propose regularizing the quantization process to encourage orthogonality between the quantization residuals and the subspaces spanned by queries. The proposed method is evaluated against several existing methods on a range of benchmarks. Overall, the motivation is compelling and supported by insightful analysis, suggesting strong potential for the proposed method. However, it should be noted that the quality of the experiments currently lags behind that of the other high-quality components of the work.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation is compelling and supported by insightful analysis that could inform future work.\n\n2. The proposed method is targeted and rigorous. Its derivation is easy to follow, and the solution is elegant. \n\n3. Empirical results demonstrate strong performance and introduce only negligible overhead.", "reasons_to_reject": "1. There are two new hyperparameters introduced. This paper lacks discussion on how to set them effectively. See Q1. \n\n2. Despite Llama 3.1's support for a 128k context length and the use of LongBench (a benchmark for long-context evaluation), the authors limited their experiments to a maximum sequence length of 8192 tokens. This is considerably shorter than the contexts explored in recent long-context modeling studies. Since KV quantization is primarily beneficial in long-context scenarios, evaluation on real long sequences (e.g., >64k tokens) is necessary to accurately assess its performance. \n\n3. The analysis lacks a detailed speed profile or computational cost breakdown for SQaut. Detailed and separated profiling of overheads in prefilling and generation is crucial for assessing practical performance and identifying bottlenecks. Refer to Q5 for a follow-up.", "questions_to_authors": "1. I find results in Table 3 puzzling. Intuitively, a larger $r$ defines a higher-rank subspace, which would presumably make the orthogonality constraints more difficult to satisfy. Conversely, a smaller $\\lambda$ diminishes the regularization effect. Therefore, the observation that reducing $\\lambda$ while increasing $r$ leads to better performance (in general) appears paradoxical. I would appreciate a more detailed analysis and discussion regarding the specific influence of $r$ on SQuat's behavior, especially in light of these seemingly conflicting effects.\n\n2. Why do SQuant and SQuant_pre use different values for the parameters $r$ and $\\lambda$ on the LongBench benchmark?\n\n3. What is the length of the input sequence used to generate Figure 2? Do the result reported in Sec 3.2, i.e., prompt only is a good enough approximation, hold for substantially longer sequences, for example, those with 100,000 tokens?\n\n4. What is the role of multiplying $\\diag(\\Sigma[:r])$ in Alg 1 line 6? Why is $V[:r]$ not enough to represent the subspace?\n\n5. If SVD is fast enough, can $\\hat{Q}$ be updated (periodically) using the residual buffer? If so, would this approach enhance performance, and what would be the consequent slowdown?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors introduce a new method for KV cache compression that relies on constructing a subspace spanned by query tensors to capture the most critical task-related information, which seems to yield improvmenets over sota methods.

The paper received positive scores (6, 6, 7) from three reviewers, all acknowledging the novelty and technical merit of the proposed approach. The Reviewers appreciated the method's ability to preserve task-relevant information via orthogonal constraints to a query subspace, distinguishing it from existing compression-based quantization techniques (putting a small asterisk here, discussed below).

Strengths:

- Introduces a theoretical framework beyond simple compression-based quantization methods.
- Demonstrates empirical results across multiple benchmarks and models.
- Requires no fine-tuning
- The authors addressed most of the reviewer concerns with additional experiments, and clarifications on computational breakdown

Weaknesses:

- Evaluation on very long sequences was limited (I suppose that's overall a harder eval, but it was capped at 30k tokens due to computational constraints).
- Hyperparameter selection of new tuning knobs adds extra overhead (this is a complain for any novel compression method to be honest)
- Practical deployment aspects was mentioned by a reviewer. I suppose this is always quite tricky and we have to rely on our belief that some of the findings transfer to a more practical setting.

expanding on the asterisk: 
- Many methods developed in the apst can be used for task specific quantization, eg Lexico by Kim et al is VERY similar to what the authors proposed. However, they didn't even compare in their benchmarks. with these methods that can be tuned to the task at hand.
Decision: Accept