Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Authors: Alina Shutova
Vladimir Malinovskii
Vage Egiazarian
Denis Kuznedelev
Denis Mazur
Surkov Nikita
Ivan Ermakov
Dan Alistarh
Venue: ICML 2025 poster (2025)
Keywords: Key-Value Cache Quantization
Compression
Primary area: deep_learning->large_language_models
TL;DR: Quantizing LLM KV cache better by predicting next layer entries from previous layer and quantizing the residue.

Abstract: Efficient real-world deployments of large language models (LLMs) rely on Key-Value (KV) caching for processing and generating long outputs, reducing the need for repetitive computation. For large contexts, Key-Value caches can take up tens of gigabytes of device memory, as they store vector representations for each token and layer. Recent work has shown that the cached vectors can be compressed through quantization, pruning or merging, but these techniques often compromise quality towards higher compression rates. In this work, we aim to improve Key \& Value compression by exploiting two observations: 1) the inherent dependencies between keys and values across different layers, and 2) the existence of high-compression methods for internal network states (e.g. attention Keys \& Values). We propose AQUA-KV, an adaptive quantization for Key-Value caches that relies on compact adapters to exploit existing dependencies between Keys and Values, and aims to "optimally" compress the information that cannot be predicted. AQUA-KV significantly improves compression rates, while maintaining high accuracy on state-of-the-art LLM families. On Llama 3.2 LLMs, we achieve near-lossless inference at 2-2.5 bits per value with under $1\%$ relative error in perplexity and LongBench scores. AQUA-KV is one-shot, simple, and efficient: it can be calibrated on a single GPU within 1-6 hours, even for 70B models.

Cache Me If You Must: Adaptive Key- Quantization for Large Language Models
Alina Shutova 12 Vladimir Malinovskii 21 Vage Egiazarian *3 Denis Kuznedelev 2
Denís Mazur 45
Nikita Surkov Ivan Ermakoy
Dan Alistarh
Abstract
Efficient real-world deployments of large lan- guage models (LLMs) rely Key-Value (KV) caching for processing and generating long out- puts, reducing the need for repetitive computation. For large contexts, Key-Value caches can take up tens of gigabytes of device memory, as they store vector representations for each token and layer. Recent work has shown that the cached vectors can be compressed through quantization, pruning merging, but these techniques often compromise quality towards higher compression rates. In this work, we aim to improve Key & Value compression by exploiting two observa- tions: 1) the inherent dependencies between keys and values across different layers, and 2) the ex- istence of high-compression methods for internal network states (e.g. attention Keys & Values). We propose AQUA-KV, adaptive quantiza- tion for Value caches that relies on compact adapters to exploit existing dependencies between Keys and Values, and aims to "optimally" com- press the information that cannot be predicted. AQUA-KV significantly improves compression rates, while maintaining high accuracy state- of-the-art LLM families. On Llama 3.2 LLMs, we achieve near-lossless inference at 2-2.5 bits per value with under 1% relative error in perplexity and LongBench scores. AQUA-KV is one-shot, simple, and efficient: can calibrated on a sin- gle GPU within 1-6 hours, even for 70B models.
$2.8
52.9
47.8
46.1
Average LongDench
37.2
20.5
70
Parameters (B)
Figure 1. Comparison of AQUA-KV to alternative Key- Key-Value Cache compression methods for Llama 3.x models in terms of average LongBench score on 14 english tasks (see Section 4).
attention- based Transformer models (Vaswani, 2017). To achieve faster inference and avoid wasteful recomputation of attention scores during autoregressive generation, KV caching is typically employed, where keys and values saved for later use. Unfortunately, K KV-caching comes with its own pitfalls: KV caches are large, especially when han- dling long sequences (Bai et al., 2023; Xiao et al., 2023). Thus, the memory footprint of a full-length Key-Value Value cache can reach tens of gigabytes of device memory, sometimes more than the model itself". This massive memory consump- tion increases the cost of deployment, and also slows down inference, the whole process become memory-boune for large caches (Hooper et al., 2024).
Previous work has proposed methods to compress KV caches using various methods such as quantization and prun- ing (Li al., 2024a), that can significantly reduce the mem- ory footprint of KV caches. Yet, as we increase the degree of compression, e.g. 2 bits per value, existing compression techniques begin to lose significant information, resulting in poor accuracy (Li et al., 2024c).
1. Introduction
Large Language Models (LLMs) are revolutionizing natural language processing, but come with major computational costs, in particular due to the input-quadratic complexity of
In this work, we aim to improve KV cache compression by taking advantage the inherent structure and dependencies in the cache tensors. Specifically, we analyze the Key-Value cache behavior for state-of the-trrtt LLMs and find several
'Equal contribution 'HSE University "Yandex 'ISTA "SberDevices 'MIPT "T- "T-Bank. Correspondence Dan Alistarh <dan.alistarh at
For the popular Llama 3.2 3B model (Dubey et al., 2024) with maximum context length of 2 takens 13x13) the 16-bit cache takes up 15 GB per sequence. For Ilama 3.1 70B and Qwen 72B, is 42.9 GB per sequence.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
strong inter-dependencces 1) between cached vectors from adjacent layers, but also 2) between the keys and values within one layer.
compression and decompression speeds maller, since we are dynamically adding new entries to the cache as well as decoding them at inference time. Another issue is posed by the inherent structure in caches, in particular, the existence of attention sinks (Xiao et al., 2023) and large outlier val- ues (Liu et al., 2024c; Hooper et al., 2024; Liu et al., 2024a), which may not be present in the weights. Next, we detail the main approaches for KV-cache quantization.
Starting from these observations, we formulate a practi- cal compression algorithm that explicitly leverages these inter-dependencies, by training compact linear predictors that capture mutual information between cache components. Further, to offiett prediction errors, we use data-free vector quantization to achieve superior ceapresssionacccuraee trade-off for the same bit-width. Our method requires only minimal calibration, is compatible with arbitrary quantiza- tion schemes and can be further combined with orthogonal compression techniques such as pruning.
K VV-Cache quantization approaches can be roughly catego- rized based on quantization granularity and error handling.
Quantization Granularity, Several approaches have been developed with varying quantization granularities. For in- stance, ZipCache (He et al., 2024) and WKVQuant (Yue et al., 2024) implement channel-separable token-wise quantization, while KVQuant (Hooper et al., 2024) and KIVI (Liu et al., 2024c) employ a hybrid approach, using per-channel quantization for key tensors while applying per- token quantization for value tensors. QJL (Zandich et al., 2024) introduces a specialized JL transform for key ten- sors combined with per-token quantization for value tensors. Methods like MiKV (Yang et al., 2024b), QAQ (Dong et al., 2024), and SK (Duanmu et al., 2024) employ variable bit widths to balance accuracy and memory reduction.
In summary, our contributions are as follows:
1. We analyze the structure of key-value caches in modern LLMs and highlight several sources of mutual informa- tion that can be leveraged for compression.
2. We propose AQUA-KV -a a novel compression frame- work that exploits inter- and intra-layer dependencies to improve quantization accuracy. AQUA-KV works in one shot, based on lightweight calibration procedure, and shows competitive aice-accrrcyy trade-offs. Additionally, it is compatible with arbitrary quantization techniques, and can be combined with additional compression, such as pruning.
Error Handling. Among the strategies used to address quantization errors, GEAR (Kang et al., 2024) compensates for errors using a low-rank matrix; to handle outliers, which can significantly impact model performance, IntactKV (Liu et al., 2024b) maintains full precision for outlier values. QuaRot (Ashkboos et al., 2024) transforms weight matrices using Hadamard orthogonal matrices to "smoothen" quanti- zation outliers without affecting model output. Palu (Chang et al., 2024) compresses KV cache through low-rank projec- tion, while 7DC (Zhang & Shen, 2024) aims to eliminate compression overhead through a novel zero-delay compres- sion scheme. Most methods maintain a window of recent historical KV cache in full precision to preserve accuracy.
3. We validate the effectiveness of AQUA-KV on modern LLM families in terms of both perplexity and zero-shot accuracy on long-range benchmarks, where AQUA-KV significantly improves accuracy across model types and bitwdiths, particularly for 2-b compression.
4. We test AQUA-KV compatibility with various quantiza- tion and pruning schemes, from simple uniform quantiza- tion, to modern data-free vector quantizers (Malinovskii et al., 2024b) and hybrid quantization & pruning regimes.
5. We develop a reference implementation for AQUA-KV calibration and inference, which is available online?.
Cross-Layer Merging. An alternative compression ap- proach, which has been relatively less investigated, has been to define layer groups, and keep single KV-Cache per layer group, reusing the one cache across all layers in the group. KVSharer (Yang et al., 2024c) observes that, sur- prisingly, sharing the caches that are most dissimilar, by the Euclidean distance, performs better than sharing similar ones. (This is identified using a calibration dataset.) Fur- ther, MiniCache (Liu al., 2024a) merges pairs of layers, and stores common interpolated directional vector and token-wise scalar scales. Additionally, outlier tokens are retained to increase accuracy. During inference, keys and values of the two layers are restored from the shared vector representation using saved scales.
2. Background and Related Work
2.1. KV-Cache Compression
So far, the main focus of work on LLM compression has been on the weight quantization, e.g. (Frantar et al., 2022: Lin et al., 2023; Tseng et al., 2024a: E Egiazarian et al., 2024). Recently, there has been a growing demand for KV-cache compression, especially in tasks requiring long contexts. The dynamic nature of K KV-caching poses unique challenges: while for weights it is acceptable to use "slow but accu- rate" compression such as codebook-basee methods (Tseng et al., 2024a; Egiazarian et al., 2024) given that weights are only decoded at inference time-for the KV-cache, both
KV Cache Pruning. Pruning methods aim to determine unnecessary parts of KV Caches and either evict them en-
Phttps: /tttps://gitie thub. www.b.coo//......ooo...o./oo.eoo htmm/goodeveningg 3/ aauukv
2

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
tirely, or offload to cheaper memory (e.g. CPU). Current research focuses token-level pruning. e.g. determining which tokens should be discarded or offloaded. They can be split into static methods and dynamic methods.
points on a lattice, which is specifically optimized for ac- curate quantization of normally-distribute vectors. This allows HIGGS to achieve fast data-free weight quantiza- rion. The technique requires several modifications to be applied to Key- caches, which we detail in Section 4.2. Finally, our approach is conceptually related to Residual Vector Quantization (RVQ) (Gray & Neuhoff, 1998), but using learned predictors instead of standard quantization.
Static methods use predefined position heuristics to deter- mine important tokens. An example is Fastgen (Ge et al., 2023), which employs knowledge of attention structure in attention heads, acquired during prefill, to identify one of the tailored attention structures for each head. This knowledge is then used during inference to efficiently evict unnecessary tokens. StreamingLLM (Xiao et al., 2023) notices that ini- tial tokens (attention sinks) and recent tokens consistently exhibit high importance. The method retains several tokens in the beginning and number of recent tokens, thus main- taining constant-size KV cache, facilitating deployment in memory scenario.
Linearity between adjacent layers. Prior work (Razzhi- gaev et al., 2024) has shown that there is almost linear relationship between activations in sequential layers in trans- former language models due to the presence of skip connec- tions. We leverage this property for the design of KV-cache predictors.
3. Method
Dynamic methods calculate importances dynamically, typ- ically using runtime information about attention distribu- tion. H2O (Zhang et al., 2023) calculates token importances dynamically during inference using accumulated attention scores. selects the least important tokens during each forward pass (if the cache is larger than desired) and thus maintains constant cache size. SnapKV (Li et al., 2024b) calculates token importances using attention features; un- like H2O, SnapKV thresholds token importances, allowing prompt compression during prefill.
The core idea of AQUA-KV is to leverage inter- dependencies between consecutive KV-caches to improve compression. For this, we train compact predictors that "guess" the value of Key & Value pair using other cache entries, then quantize the residual information that could not be predicted. This way, we only store the information that cannot be recovered from other sources.
In Section 3.1, we analyze the dependencies between var- ious KV cache components to determine the type of pre- dictors that achieves the best size-accuracy trade-off. In Section 3.2, we formulate practical one-shot algorithm that fits these predictors for use in KV cache compression. Finally, in Section 3.3 we describe a number of important implementation details for using AQUA-KV in practice.
For our work we choose to conduct experiments using H2O pruning as it well-established -ld-paay method. However, we emphasize that method is orthogonal to pruning and AQUA-KV can be combined with any cache eviction strategy, AQUA-KV integrates both Quantization and Cross-Layer Merging strategies. It can be regarded as more advanced Layer Merging approach via training a small supplementary model, referred to as predictor, capturing inter-layer dependencies of KV caches.
3.1. Analysis of Inter-Layer Dependencies
The efficacy of our approach depends on choosing which types of inter-depeneencis to exploit. To make this choice, we analyze the dependencies between cached vectors at different layers or tokens, and between (key & value) vectors within the same layer. Note that we do not expect the values in these vectors to be equal or even numerically close- indeed, simple examination shows that this is not the case. Instead, we look for consistent dependencies between these components that can be extracted with simple models.
Vector Quantization recently emerged as a popular option for LLM quantization (Egiazarian et al., 2024; Tseng et al., 2024b; van Baalen et al., 2024) since it allows to jointly quantize multiple individual model dimensions and can lead to state-of-the-ar cesuracc--s-compresse
(Tseng et al., 2024c; Malinovskii et al., 2024a). However, the extremely large computational cost of encoding makes such methods impractical in the context of KV-cache compression.
To measure dependencies between vectors at nearby layers, we adopt an approach similar to probing (Alain & Bengio, 2016): we train linear "probe" models whose goal is to predict the contents of a particular cache component, e.g. i- th layer keys or values, based inputs from various sources. As potential sources, we consider the previous layer keys and values, adjacent tokens, and different vector types (i.c. using the keys to predict values and vice versa).
We resolve this issue, and leverage the power of VQ in an efficient way, by adapting the HIGGS weight quantization technique (Malinovskii et al., 2024b) to KV-Cache compres- sion. HIGGS combines wriup-wise vector quantization with a Randomized Hadamard Transformation (KHT): the RHT projects the original values onto "rotated" space, where they will be Normally-distribuce In the quantization step, the rotated values are grouped and rounded to the nearest
We can then measure the relative prediction error for such probes, based on small amounts of calibration data. More
3

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Predict Keys
0.75
Block L
0.50
Cochey.
Keys Predicter
KoyaClache
0.25
Quantized Keys Residuals
0.00
L-
K(-) Vi-yRVi-1
Predict Values
0.75
0.50
Values Cache
Values Predlcttor
Values Cachey
0.25
Quanzizeed
Vallace
0.00
Resiluale
L-3
122
1-1
all
T-3
T:2
T-1
all
KL-J Vi-
Figure 2. Mean Explained Variance Ratios by linear probes from previous blocks (L), tokens (T) and role on Llama-3.2-3B.
Figure 3. An intuitive scheme of the AQUA-KV inference.
Only quantized residuals are saved for each black.
specifically, we measure the explained variance ratio to account for unequal scales of keys and values between lay- ers. Intuitively, if predictor captures 90% of the variance, it means that the subsequent quantization only needs to capture the remaining 10% of variance. For compression methods that are scale-independent 'Horváth et al., 2023) this would mean that the resulting quantization will also have roughly 10 times smaller error.
between several cache components. For attention keys, us- ing just one previous layer already achieves errors similar 2-bit quantization. For values, the dependency on previous layer is also strong: though less accurate than for keys, the previous layer values consistently explains more than half of the variance for the same token. We attribute this strong dependency to the fact that transformer architecture is resid- ual, and therefore, adjacent hidden states are only off by a single transformer layer. Since Key-Value representations are constructed linear projections of adjacent residual hidden states, they are also interdependent.
Note that not all predictors will be practical for Key-Value compression. For instance, if predictor uses subsequent block or future token KV vectors as inputs for the current ones, it would be difficult to inference the model with such KV cache as it goes against the order in which the blocks execute. Hence, we consider the following:
More distant layers are also predictive of the current layer, but the dependency quickly deteriorates with the distance. More importantly, there is almost no difference between using multiple past layers and just the previous layer, which allows to simplify the algorithm. See Appendices B & D for additional exploration and downstream results.
Previous blocks: same token vectors for for-1,-2,, -2 blocks;
Previous tokens: same layer, -1, -2, -3 previous tokens;
Different role: using keys to predict values and vice versa.
We also observe strong dependencies between keys and values within the same layer. This is also not unexpected, as these vectors are linear down- proje-roeections from the same input vector within the attention layer. The reason why there is additional information between matching keys/values is that in modern LLMs the key/value dimensions are both significantly smaller than the input vector dimension due to Grouped Query Attention (GQA, Ainslie et al. 2023).
We report these errors in Figure 2. For comparison, 1-bit and 2-bit quantizers usually explain 0.75 and 0.89 variance, respectively. In other words, they have 0.25 and 0.1 rela- tive quantization errors. Intuitively, if probe can predict keys/values with the same relative error as the 1- 2-bit quantizer, means that we can use less bit for quantization (e.g. 3-bit instead of 4-bit) after the residual with, on aver- age, the same accuracy. While this is not strict guarantee, we found that it holds well for real-world LLMs, can be seen in Sections 4.1 and 4.2.
In contrast, we found relatively little predictive power from past tokens. While there mutual information between adjacent tokens, using single previous token not enough to capture this dependency, and using many multiple tokens more expressive probe would make the resulting KV cache compression inefficient.
We train linear probes for 1.2m3-3.2-3B Key-Value Value cache on sample of RedPajama (Weber al., 2024) sequences, then evaluate relative error on hold-out sequences from the same source. For readability, the detailed experiment configuration is deferred for Appendix A.
Some of our observations above correlate with the results from Liu et al. (2024a), who were able to "share" some of the cache entries. However, such direct merging is drastic approach, that can lead to significant accuracy loss. From the perspective of Figure 2, this is because the information recovered by probe not enough by itself, never exceeding
The findings in Figure 2 demonstrate strong dependencies
Formally, scale-independent quantizer Q(.) satisfies YX, Vo>+, (Q(x)()(aX)) a.|]22 -Q()) X|122 Moder quantizers satisfy this due to the use of scales.

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
the equivalent of a 2-bit quantizer. In contrast, we employ more fine-grained approach that combines predictors with residual quantization to achieve more accurate compression.
account for this discrepancy, AQUA-KV trains predictors se- quentially, one transformer layer at time. Each subsequent set of predictors is trained using reconstructed cache entries as inputs, reflecting the way these predictors are used during inference. The first layer key-value cache is compressed as is, and each subsequent layer trains using previous layer key-value reconstructions as inputs. The full calibration procedure is described in Algorithm
Algorithm 1 AQUA-KV Calibration
Require: model, data, quantization method Q
I: X <- model. www.iinttuteeeeee
(data)
2: Kold; Vola f 0
3: predictors < 0
Here, Q(-) and Q-)(-) denote quantization and de- quantization operators. Our algorithm is agnostic to the choice of Q(-): simple uniform min-max quantization or any advanced method can be used (Frantar et al., 2022; Mali- novskii et al., 2024b). The notation arg min y (XX -Y fits a linear regressor to a given problem. By default, we use simple linear regression for all predictors, which allows for closed-form training and is easy to use during infer- ence. However, our algorithm can accommodate any other regressor type. In 4.1 we consider alternative algorithms: reduced-rank regression (Reinsel & Velu, 1998) and MLPs.
4: for = I,... mwdem. num layers do
5: block + model. trnee
6: (K, V) + block. w.ock.get.tteeniion..v/...
7:
X block
8: if Kold = @ and Vold then
9: Kou, Vold <- K, V
10: continue
11: end if
12: # Predict keys from past keys
13: frey + arg minyI1 [l.[
KI 12
14: Kresttue t +-Q-1 1-QQ-(Q2 (Q(K- Seay (kl)
Computational and memory overhead. In addition to the cache entries, AQUA-KV also needs to store the trained pre- dictors during inference. This requires additional computa- tion, and slightly increases the memory footprint. However, this overhead is small in practice: this is largely because modern LLMs use GQA (Ainslie et al., 2023) a popular at- tention variant that uses fewer key-value heads than queries. For AQUA-KY V, this means that the predictors contain sig- nificantly less parameters and require orders of magnitude less computation. For instance, For Llama 3.x 70B and Qwen 2.5 72B, inferencing AQUA-KV predictors for a to- ken requires at least 500x less floating point operations than running the base model for the same token (see Section 4.2).
15:
K Skey (Koua) + Kresiddue
16:
# Predict values from past values and current keys
17:
Jvalue + arg miny y [[[(Vout ou; Krec)) VI
18: V Vresidue + Q--1(QVV - -faaue ([Vold: )))
19: VICe + -value ([V ol: k)ee)) + V resiiuee
20: predictors [i] (Fkey, fvatue)
21: Kou, Vold f- Kree Viee
22: end for
23: return predictors
3.2. The AQUA-KV Algorithm
Based on the findings from the previous section we design cache compression algorithm that leverages the structure of Key-Value cache to improve compression.
Efficiency and limitations. Algorithm 1 is designed to work as a single-pass calibration procedure. It trains pre- dictors separately, one time. This allows our algorithm to run efficiently even on low-end hardware. For instance, calibrating the full set of KV predictors for Llama- 3. 100 model takes up 4 hours on a single GPU and takes up at most 16GB VRAM. AQUA-KV is designed as a sim- ple and lightweight algorithm that can be easily extended. As such, we deliberately forego more complex techniques such as global predictor fine-tuning dynamic bitwidth. We discuss these possible extensions in Section 5.
We choose the following predictor configuration: (1) we use the previous layer keys to predict the subsequent keys; and (2) we use both previous layer values and current layer keys to predict values. Note that we do not use the depen- dency between same layer K and V vectors in the Vz-KK direction: this is because we cannot predict in both direc- tions simultaneously during inference, and we found that the values are overall harder to predict (see Figure 2), so we opted to improve value predictor. We also purposefully leave out other possible input sources, such as more distant past layers. While using these layers can slightly improve reduce error, it would also make the predictors themselves larger and more compute- compute-intensive.
3.3. Implementation Details
Finally, we describe several practical details necessary for efficient implementation of our approach, whose experimen- tal validation is given in Section 4.1.
The way AQUA-KV trains those predictors is also different from Section 3.1. This is because, in practical KV cache compression, the predictors do not have access to ground- truth past KVs during inference. Instead, they can only use reconstructed (de-quantized) past key and value vectors. To
Backbone quantization. To validate the generality, we cou- ple AQUA-KV with three different quantization schemes: Quanto (HuggingFace,, 2024) (round-to-nearest with absmax normalization) and the more advanced vector quantization
5

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
schemes QuaRot (Ashkboos et al., 2024) and HIGGS (Mali- novskii et al., 2024b).
are originally stored without compression. This buffer has two positive effects: it improves accuracy on recent tokens and allows for efficient parallel processing. When the buffer is filled, its contents are quantized incrementally from the first layer to the last, in the same order as during calibration.
Relation between AQUA-KV and Attention Sinks. It well-known that modern LLMs tend to form attention "sinks" - tokens that have extremely high attention scores while being semantically unimportant (Xiao et al., 2023). Several prior works in KV compression propose special treatment for such attention sinks, such as keeping them in higher precision (Hooper et al., 2024) or introducing synthetic sinks (Xiao et al., 2023; Chen et al., 2025).
We define this procedure formally in Alg. 4. Due to the se- quential nature of our approach, we only need to materialize (de-quantize) a single layer at a time, which can be further improved with chunking. Note that the proposed scheme is seamlessly compatible with model parallelism, offloading, speculative decoding, merging multiple sequences and other popular LLM inference techniques. Furthermore, as we show in Section 4.4, our approach can be used in tandem with pruning techniques, such as H2O (Zhang et al., 2023).
When evaluating AQUA-KV, we found that attention sinks are indeed important. Compressing attention sinks poorly can affect the behavior of attention heads other to- kens and, hence, change the input/output distribution for the learned predictors. The AQUA-KV calibration algo- rithm computes input keys and values without accounting for quantization error (Alg. L5-7), as doing otherwise would significantly increase calibration time. Hence, we found that AQUA-KY benefits from keeping the first few tokens uncompressed, similarly to how they are treated in KVQuant (Hooper et al., 2024). For fair comparison, we keep the first 4 tokens uncompressed for both AQUA-KV and baselines without predictors, and explore this in more detail in Table 9 in suplementary materials.
4. Experiments
To test the real-world effectiveness of AQUA-KV, we apply it to modern LLMs in three setups: 1) in Section 4.1, we analyze the impact of individual components of our method and verify the design choices 2) Section 4.2 evaluates on a broader range of models and compression rates; 3) finally, Section 4.4 explores how our approach combines with other popular KV cache compression strategies.
Across all three sections, we use the same calibration and evaluation protocol. We use a sample from RedPajama (We- ber et al., 2024) dataset for calibration: namely, 256 se- quences of 8192 tokens sampled at random. We use 32 of those sequences as holdout for hyperparameter selection and the remaining 224 are used to train the predictors themselves. We use two popular evaluation metrics: WikiText-2 (Merity et al., 2016) perplexity and LongBench (Bai et al., 2023).
Positional embeddings. Most LLMs apply Rotary Po- sitional Embeddings (RoPE, Su al. 2021) to attention keys (but not values). This raises a natural dilemma about whether to apply predictors and quantizers before or after quantization. In our analysis, we found that linear predictors are beneficial in either case, but they offer better accuracy in pre-RoPE compression. We attribute this to the fact that the optimal predictor for post-RoPE compression needs to ootatio-eqqivvaiiant and the simple linear models we use not. As for the backbone quantization, we found that the uniform quantizer (Quanto) works slightly better post-RoPE, while HIGGS works equally well in both cases due to its use of the Hadamard transform. We provide experimental validation for these claims in Appendix D.
When evaluating perplexity, we adopt the same approach as in prior quantization works (Frantar et al., 2022: Lin et al., 2023; Egiazarian et al., 2024), with one exception: instead of processing sequences (of length 8192) in parallel, we en- code them auto-regressively and maintain the (compressed) Key-Value cache during inference. This results in the same perplexity for non-compressed KV cache, but allows us to properly account for the effect of recent token buffers and attention sinks during KV quantization, as described in Sec- tion 3.3. Here, we use base (non-instruct) models since they have better perplexity.
Per-token and per-channel compression. We use per-token quantization for both schemes. While some prior works suggest that keys are better quantized per- channel (Liu et al., 2024c; Hooper et al., 2024) due to dif- ferent outlier structure, we found that this is not necessary when quantizing predictor residuals. We use per-token com- pression as it is easier to implement, and we ablate this choice in Appendix D. For baselines, we always follow the quantization axes suggested in their respective papers.
In tur, LongBench v1 (Bai et al., 2023) contains long- context length evaluation benchmarks including QA tasks, summarization, and few-shot learning. We evaluated all the 14 English-language tasks without restricting the input length to 8192 tokens. This allows us to better explore the effectiveness of AQUA-KY on longer sequences. Since 14 individual tasks are often difficult analyze, we report the average score across all tasks and provide detailed per- task results in Appendices D. E & G. We use the official
Inference Algorithm. Finally, we explain how these design choices combine to the full AQUA-KV inference procedure. Both AQUA-KV and all our baselines maintain a small recent token buffer (up to r= 128 tokens for all setups) that
6

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
benchmark code and evaluate on Instruct models since many LongBench tasks were designed for such models (see Ap- pendix C for non-Instruct models and discussion).
fying their increase in size and inference time. In the other direction, using RRR for compression results in marginally worse perplexity. GPTQ offers favorable trade-off for use cases that need to further minimize size.
4.1. Detailed Evaluation & Ablation Analysis
First Layer Keys & Values. Since the first layer is not compressed by AQUA-KV predictors, we consider several strategies for it: keeping it as is, or quamtizing to 2-4 bits. Table shows that quantizing to 3 or 4 bits is nearly lossless, while 2-bit quantization leads to drops. As such, we use 4-bit quantization of the first layer our default configuration for Llama models.
First, we evaluate the effectiveness of the individual com- ponents of AQUA-KV in different combinations. To keep the number of experiments manageable, we have chosen the Llama 3.2 3B model and focused on 2-bit quantization. We explore additional models and compression targets in future sections. We report the evaluation results in Table and describe each sectiectinn below. Additional experiments and detailed LongBench scores in Appendix D.
Table I. Evaluation of Llama 3.2 3B with various Key- Value cache compression strategies. We report WikiText-2 perplexity for the base (non-Instruct) model and the average LongBench results for the Instruct model. Additional results in Appendix D.
Alternative quantizers. As discussed, AQUA-KV is com- patible with any "backbone" quantization scheme. We focus on three schemes described in Section 3.3: Quanto, QuaRot and HIGGS. For HIGGS, we use the quantization group size 1024 and use three grid configurations that have fast GPU support, at 2-, 3- and 4-bit precision (we use 2=2, {16, 64, 256) respectively) and one grid without fast GPU support (2-bit precision, d-4, n=256). For QuaRot we also use group size 1024, and for Quanto we use the default group size 64 and per-token compression (0-th axis). All methods use Round- To-Nearest (RTN) quantization. We evaluate each scheme with and without learned AQUA-KV predictors, and compare against two popular algorithms for KV cache compression: KIVI (Liu et al., 2024c) and KVQuant (Hooper et al., 2024). We also report additional Quanto configurations in Appendix D.
Quant. Bits
Wiki2 PPL.
LongBench Avg-t (instruct model)
Coufig
(base model)
Uncompressed
16
6.98
44.47
Quanto-2b-gs64
2.50
21.56
33.59
KIVI 222b-s118-r11 121-1188111 1228
2.25
9.33
39.63
KVQuant:22
2.33
9.43
20.56
QuaRot-2b-ee 2b-gs 002
2.03
44.68
32.80
HIGGS-2b-gs1002
2.02
7.47
43.25
KVSharer (1 pair)
15.43
7.45
36.81
KVSharer (4 pairs)
13:71
9.60
29.82
AQUA-KV (Quanto)
2.64
10.33
43.64
AQUA KV (QuaRal)
2.17
8.44
44.54
AQUA-KV KV (HIGGS)
2.16
7.03
44.26
AQUA-KV Predictor Architecture
Linear (162 MiB)
2.16
7.03
44.26
MLP (540 MiB)
2.16
7.03
44.61
RRR (68 MiR)
2.16
7.22
44.30
GPTQ (41 MiB)
2.05
7.03
44.19
Table shows that KVQuant, KIVI, QuaRot and Quanto have relatively poor results for bit quantization. Adding AQUA-KV to both Quanto and QuaRot provides a signif- icant improvement in both PPL and LongBench scores. While being calibration-free method, HIGGS outperforms all the -bove-mentioned methods in terms of PPL; HIGGS with AQUA-KV achieves the best results on both metrics.
AQUA-KV Ist Layer Keys & Values
Keep in BFF6
2.59
7.03
44.26
HIGGS 4 bit
2.16
7.03
44.26
HIGGS
2.13
7.03
44.28
HIGGS bit
2.09
7.05
44.41
HIGGS slow-grid 2 bit
2.09
7.01
44.43
Ablation Analysis
AQUA-KV KV (default)
2.16
7,03
44.26
who 16 bit Sink
2.16
7.15
44.13
Layer Sharing. As we discussed in Section 2, layer sharing is conceptually similar to AQUA-KV. To compare these two strategies, we evaluate against KVSharer (Yang et al., 2024c). We follow the original algorithm to share and 4 layer pairs chosen by the KVShrrer procedure. The results can be seen in a separate section of Table 1: while the technique can reduce model size, sharing multiple layer pairs causes major accuracy drops.
w/o V predictor wlo K prodictor
2.09
7.06
43.91
2.09
7.50
42.92
w/ pre- RoPF
2.16
7.05
44.13
Quantizer agnostic training
2.16
7.13
44.26
Ablation Analysis. Finally, we validate some of the AQUA- KV design choices described in Section 3.3. Namely, we examine the strategy of keeping 16-bit attention "sinks" by measuring the effect of quantizing them. We also measure the effect of AQUA-KV with only key or only value predic- tor, and validate the effectiveness of pre-RoPE predictors by comparing against Rost- We evaluate a simplified version of AQUA-KV calibration procedure that trains pre- dictors with non-quantized inputs: this way, the predictors could be trained once and can then used for arbitrary quanti- zation bitwidth. The results in Table demonstrate that each of the tested components is important for the effectiveness of our method, particularly the key predictors.
Predictor Architecture. Next, we compare several types of learned predictors: linear regression (our main proposal), Reduced-Rann Regression (Reinsel & Velu, 1998) with rank 256, a multilayer perceptron (MLP) with two layers, dou- bled hidden dimension and layer normalization. Further, we evaluate the quality loss from quantizing predictor weights to 4 bits with GPTQ (Frantar et al., 2022). Overall, MLP predictors are only marginally better than linear, not justi-
7

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Table 2. Evaluation of AQUA-KV (with HIGGS backbone) and baselines across five M.LSs for 2, 3 & 4 bit compression. The WikiText-2 Perplexity is evaluated on base -nontinstt models with sequence length 8192. The LongBench results are an average over 14 tasks evaluated with Instruct model with sequence length 2d (131K) tokens. The eache size corresponds to the total memory footprint for Llama 3.1 70B model with sequence length 2d (131K) tokens and batch size 1, including predictors. Additional details in Appendix E.
WikiText-2 PPT., (hase model)
LongBench Averaget (instruct)
Quant. bits
Cache size GiB, 70B
Method
Llama 3x
Qwen
2.5
Llama 3x
Qwen 2.5
3B
8R
70B
3B
7B
3B
8B
70B
3B
7B
Uncompressed
16
40
6.98
5.61
2.54
7.14
6.13
44.61
48.13
52.92
38.80
46.82
AQUA-KV
2,09
5.7
7.03
5.72
2.62
7.20
6.17
44,30
47.77
52.79
38.31
46.43
HIGGS
2.02
5.1
7.47
5.89
2.77
7.93
8.08
42.80
47.37
52.18
30.92
25.97
KIVI
2.25
5.6
9.34
7.37
3.06
9.05
7.02
99.64
46.28
52.45
28.66
32.78
KVQuant
2.33
5.6
9.43
6.64
3.28
20.56
37.17
46.
AQUA-KV KV
3.06
8.1
6.98
5.64
2.55
7.15
6.14
44.37
48.10
52.81
38.77
46.81
HIGGS
3.02
7.6
7.05
5.66
2.57
7.26
7.20
44.41
86
52.56
31.85
14.61
KIVI
3.05
7.7
7.87
6.04
2.87
7.63
6.37
41.40
46.98
52.87
30.37
32.63
KVQuant
3.33
8.3
7,26
5.84
2.75
41.40
46.42
50.74
AQUA-KV
4.02
10.9
6.98
5.61
2,54
7.15
6.14
44.48
48.10
52.95
38,92
46,77
HIGGS
10.6
7.01
2.55
7.16
6.88
44.41
$2.97
32.11
11,54
KIVI
4.25
10.6
7.03
5.64
2.61
7.17
6.14
43.11
47.57
52.88
31.50
33.40
KVQuant
4.33
10.8
7.04
5.65
2.58
43.62
47.77
$2.89
Table GSM8K-CoT (8-shot) accuracy (%) for Instruct models.
Table 4. IFEval accuracy (9) for Instruct models.
Method
Quant. Bits
I.aamn 3.x
8B
70B
3B
Qwen 2.5
7B
Method
Quant. Bits
3B
Llama 3xx
8B
70B
3B
Qwen 2.5
7B
Unsompressed
76.5
85.1
94.7| |
61.2 76.6
Uncomprossed
77.0
78.9
88.0
66.5
76.9
AQUA-KV
2.09
77.7 84.3
94.2 89.9 72.2
AQUA-KV
2.09
75.1
79.9
66.2
HIGGS
2.02
70.3
79.2 94,2 33..8
59.7
HIGGS
2.02
72.4
75.7
87.0
59.3
68.6
4.2. Large-Scale Evaluaton
Table 5, Sub- evaluation of Llama 3.2 3B on WikiText-2 (PPL, base models) and LongBench (average score, Instruct models) with the same setup and parameters as Table 2.
Next, we evaluate how AQUA-KV scales across different LLM sizes and compression bitwidths. We run our exper- iments using the popular Llama 3.x (Touvron et al., 2023; Dubey et al., 2024) and Qwen 2.5 (Yang et al., 2024a; Team, 2024) LLM families. For Llama 3.x models, we take the lat- est versions that have both Instruct and non-Instruct model variants: v for 8B and 70B and v3.2 for 3B. We need both variants for different evaluations (see Appendix C). We evaluate for 2, 3 & 4 bit KV quantization in WikiText-2 PPL & LongBench scores, and report the resulting KV-Cache footprint for a single full-length sequence.
Quant. Bits
Wiki2 PPI.
LongBench Avg.t
Method
Uncumpressee
6.98
44.61
AQUA-KV (d=8, n=256)
1.11
7.57
40.38
HIGGS (d=88 n=256)
1.02
16:34
20.99
AQUA-K -VV (d=4, n=64)
7.17
43.48
HIGGS (d==4, n==4)
1.52
8.57
38.92
4.3. Additional Benchmarks
The results in Table 2 summarize our findings: as before, AQUA-KV predictors can substantially improve over both the HIGGS quantizer and prior works on KV-Cache quanti- zation. The advantage from using AQUA-KV is particularly noticeable for extreme 2-bit compression, where AQUA- KV over 2-bit HIGGS quantizer is roughly equivalent to the 3-bit baseline quantizer, and sometimes outperforms it.
To better quantify the effectiveness of AQUA-KV cache quantization on different tasks, we evaluate it on several additional benchmarks on a subset of Instruct models. More specifically, we evaluate on three additional tasks: GSM8K (Cobbe et al., 2021) with 8-shot chain-of-thought (CoT) setup, MMLU-Pro (Wang et al., 2024) 5-shot chain- of-thought and IFEval (Zhou et al., 2023) zero-shot. We use the default evaluation configurations from LM Evaluation Harness (Gao et al., 2021) (see Appendix F) with the same AQUA-KV and HIGGS parameters as in Section 4.2.
In Table 5, we report evaluations with even lower bitwidths (<2 bits per value) for Llama 3.2 3B model. In these set- tings, AQUA-KV shows an even greater advantage over quantization without predictors. However, both methods have considerable loss perplexity and LongBench score. While the absolute improvements are impressive, most prac- tical use cases would benefit from using higher bitwidth with a smaller model or pruning the cache to fit into the memory budget. We report additional results in Appendix E.
Table 6. MMLU-Pro (5-shot) CoT accuracy (%), Instruct models.
Quant. Bits
Llama 3.x
Qwen
Method
3.2 3B
3.1 8B
3.1 708
2.5 3B
Uncompressed
16
34.47
44.34
63.29
43.7
AQUA-KV
2.09
32.79
43.25
62.21
42.11
HIGGS
29:25
38.40
60.43
23.97
8

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
We report the accuracy on each benchmark in Tables 3, 4, and 6, respectively. Overall, our results follow similar trend: AQUA-KV cache compression achieves significantly higher accuracy across all three benchmarks at the cost of a slight increase in memory footprint (to store predictor weights). In Appendix F, we also report accuracy scores on subsets of MMLU-Pro (e.g. physics, biology, economics).
sequence of up 32768 tokens with 2-bit HIGGS. In this setup, our AQUA-KV implementation has 188% overhead over 6 inference for sequences over 16384 tokens. This overhead includes the time to predict and dequantize thousands of past cache tokens on every step.
We also measure throughput by performing batched infer- ence with the maximum batch size that fits in GPU memory. In this setting, AQUA-KV allows for substantially higher throughput for short sequences since the quantized cache can fit larger batch size (up to 5x). Note, however, that AQUA-KV was primarily designed to reduce memory foot- print for longer sequences, and there are other quantiza- tion methods that target faster inference (Liu et al., 2024c; Hooper et al., 2024). We discuss inference in more detail and report speed benchmarks in Appendix H.
4.4. Compatibility with Pruning
As we discussed carlier. AQUA-KV is compatible with other KV cache compression techniques such as pruning. To test this in practice, we evaluate AQUA-KV with HIGGS quanti- zation in tandem with the popular H₂O (Zhang et al., 2023) token pruning method. We train our predictors normally as described in Section 3.2, without pruning. During inference, we first use the H2 O heavy hitter oracle to select which to- kens are to be preserved, and apply AQUA-KV compression those tokens. For this experiment, we always keep 20% of all tokens with the same protocol as in the original paper. We evaluate these mixed strategies in our main setup on subset of Llama 3.x models.
5. Discussion
We introduced a KV-Cache compression technique based on the idea of leveraging both inter- and intra-layer corre- lations in an efficient fashion. Empirical results suggest that AQUA-KV KV sets new state-of-the-ee compression-ye vs- accuracy trade-offs, while being compatible with different quantization and pruning techniques. Our approach bears several extensions: 1) predictors could be optimized (e.g. fine-tunedl to minimize a model-level objectives, similar to weight quantization techniques (Tssng et al., 2024b); 2) the bit-widths of different cache components could be ad- justed based on what fraction of them can be predicted; 3) it would be interesting integrate AQUA-KV with efficient LLM inference engines such vLLM. In general, it is in- teresting to consider the problem of efficient LLM inference with AQUA- which may require merging predictors with some of the base model computations to reduce overhead.
Table 7. Evaluation of AQUA-KV quantization in combination with H₂O token pruning. Instruct models. This config follows the same setup as Table 2, but every entry uses H₂O procedure to keep only 20% tokens, using hyperparameters from Zhang et al. 2023.
Quant. Bits
Memory Saved
LongBench Avg.1
Method
3B
8B
11,0 only
5.0x
38.82
41.42
H2O + HIGGS
2.02
39.6x
37.02
40.72
HaO AQUA-KV
2.09
3
38.43
41.11
H2O HIGGS
26.5x
38.27
41.37
H2O+ AQUA-KV
3.06
x
38.76
41.31
H₂O HIGGS
4.02
19.9%
38.74
41.32
H₂O+ AQUA-KV
4.02
19.9x
38.85
41.47
Last but not least, is curious why do LLMs learn pre- dictable key-value representations in the first place. If adja- cent keys and values can predict each other, it may hint at some redundancy within LLM attention heads. A promising direction for future work is to study the reasons why mod- ern LLMs learn inter-dependent representations, in hope of better understanding how LLMs use attention. If we can learn the root cause of this apparent redundancy in attention projections, could lead us to more efficient way of using LLMs in general, instead of relying on ad-hoc predictors.
The results, shown in full in Table 7 suggest that AQUA does not degrade pruning performance: our method combined with H2O shows little to no accuracy drop compared to using H2O in isolation. Furthermore, in this setup AQUA-KV with HIGGS quantization still outperforms HIGGS quantization without predictors, by similar margin. In Appendix G we evaluate AQUA-KV with H₂O for 50% token pruning. We also report individual task scores for LongBench.
Inference time.
The practical inference speed of AQUA- KV depends heavily on the backbone quantizer (e.g. HIGGS. Quanto, or others). Since AQUA-KV adds an extra prediction step, it can not be faster than the 6 baseline, but can be more memory-efficient. We benchmark Llama 3.2 3B and 3.1 70B models in 6 precision us- ing Transformers (Wolf et al., 2019) library for the LLM and custom CUDA kernels for HIGGS. We run the 3B model on a single A100 GPU and the 70B on 2x A100 in sequential mode, We measure token latency when generating a single
A Acknovledgements
Authors would like to thank Andrei Bocharnikov for imple- menting optimized CUDA kernels for AQUA-KV inference code and benchmarking inference speed (after the ICML submission deadline). Additionally, we would like to thank Vyacheslav Zhdanovskiy for helpful discussions and sug- gestions on inference code optimization.
9

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Impact Statement
M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L.,, Bavar- ian, M., Winter, C., Tillet, P., Such, F, P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E.,, Herbert-Vosse A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W. Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. 2021.
This paper presents work whose goal to advance the field of Machine Learning. Since we study a general problem of memory-efficient LLM inference, our work can contribute to a broad range of consequences stemming from LLM use and misuse. This also means that AQUA-KV does not introduce any principally new kinds of socictal benefit or harm, only making the existing LLM use cases more cost-efficient. The general societal impact of LLMs is an important area of research that cannot be easily summarized broader impact statement. As such, we do not highlight any specific impacts here and defer the reader to dedicated research on the broader impact of LLMs (Weidinger et al. 2021; 2022; Bender et al. 2021; Zhuo et al. 2023; Cui et al. 2024; Sheng et al. 2021; Durmus et al. 2023, among others).
Chen, M., Liu, Y., Wang. J., Bin, Y., Shao, W., and Luo, P. Prefixquant: Eliminating outliers by prefixed tokens for large language models quantization, 2025, URL https: //arxiv. org/ abs / 22000 05265.
References
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C.,, and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv: 141688 2021.
Ainslic, J., Lee- The-Thor,,.0 J., de Jong, M.. Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint a3X1v:2305.13245, 2023. URL https: //assv///xvvv.//// org/ http:/bs/223000 13245.
Cui, T. Wang, Y., Fu, C., Xiao, Y., Li, S., Deng, X., Liu, Y., Zhang, Q. Qiu, Z., Li, P., Tan, Z., Xiong, J., Kong, X., Wen, Z., Xu, K., and Li, Q. Risk taxonomy, mitigation, and assessment benchmarks of large language model systems. ArXiv, 0bs/2400.007775 2024. URL attps::/aapt ttttp:://ap..semaniii.. argg/CorpusTD:266333
Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 2016.
Ashkboos, S., Mohtashami, A., Croci, M. I.,, Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-fre bit inference rotated llms. arXiv preprint 2444::2404.00455 2024.
Dao-AlLab.
Fast hadamard
transform.
htttps://iithbb.coo ATLab/ fast sest--addnarr--rrasffooee 2025.02.17.
Bai, Y., Lv, X., Zhang. J., Lyu, H., Tang, J., Huang, Z., Du, Z.,, Liu, X., Zeng, A., Hou, L.,, et al. Longbench: A bilingual, multitask benchmark for long context under- standing. arXiv preprint ar arXiv:2308. 14508, 2023.
2023, Accessed:
Dong, S., Cheng, W., Qin, J., and Wang, W. Qaq: Quality- adaptive quantization for I1m kv cache. arXiv preprint arXhv:2403.0460 2024.
Bender, E. M., Gebru, T. McMillan-Major A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Account- ability, and Transparency, FAccT '21, pp. 610-623, New York. NY, USA, 2021. Association for Comput- ing Machinery. ISBN 978 9781450383097 doi: 10. 1442188.3445922. URL https: 1145/3442 .8.88883344455 3445922.
Duanmu, H., Yuan, Z., Li, X., Duan, J., Zhang, X., and Lin, D. Skvq: Sliding- winddwwwwidwwwwwooo key and value cache quantization for large language models. arXiv preprint mrXiv:2405.0622 0620002220000 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 200.221777733 2024.
Chang, C.-C., Lin, W.-C., Lin, C.-Y,, Chen, C.-Y., Hu, Y-F,, Wang, P.-S., Huang, N.-C., Ceze, L., and Wu, K.-C. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.211 401.21118. 2024.
Durmus, E., Nyugen, K., Liao, T. Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds Z., Hernandez, D., Joseph, N., Lovitt, L., McCan- dlish, S., Sikder, O., Tamkin, A., Thamkul, J., Ka- plan, J., Clark, J., and Ganguli, D. Towards mea- suring the representation of subjective global opin- ions in language models. ArXiv, abs/2306.16388,
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J.. Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov,
10

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
2023. URL https://api.
http://ppp..smmatttccchhhho... otr//Co-pusTD::25927555
recipe for near-lossless generative inference of Ilm. arXiv preprint arXiv:2403.05527, 2024.
Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E.,
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez. J., Zhang, H.,, and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 61 1-16, 2023.
Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. arXiv preprint awwiv::401....... 0tt::::200.0611111 18, 2024.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323, 2022.
Li, H., Li, Y., Tian, A., Tang, T., Xu, Z., Chen, X., Hu, N., Dong, W., Li, Q. and Chen, L. survey on large lan- guage model acceleration based on kv cache management. arXiv preprint arXiv:2412. 19442, 2024a.
Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L.,, Hsu, J., McDonell, K., Muennighoff, N., Phang. J., Reynolds, L.., Tang, E.,, Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot lan- guage model evaluation, September 2021. URL https: //doi. org/ / C 5281 zenodo. 5371628.
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Llm knows what you looking for before generation. arXiv preprint arXiv:2404. 14469, 2024b.
Li, Y., Jiang, H., Wu, Q.. Luo, X., Ahn, S., Zhang, C., Abdi, A. H., Li, D., Gao, J., Yang, Y., et al. Scbench: A kv cache-centric analysis of long-context methods. arXiv preprint arXiv:2412.10331 2024c.
Ge, S., Zhang. Y., Liu, I.., Zhang, M., Han, J., and Gao, Model tells you what discard: Adaptive kv cache compression for Ilms. arXiv preprint arXiv:2310.01801, 2023.
Lin, J., Tang. J., Tang, H., Yang. S., Dang, X., and Han, S. Awq: Activation aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00978 2023.
Gray, R. M. and Neuhoff, D. Vector quantization. IEEE Transactions on Information Theory, 44(6):2325-2383, 1998.
Guo, H., Brandon, W., Cholakov, R., Ragan-Kelley, J., Xing, E., and Kim, Y. Fast matrix multiplications for lookup table-quantized Ilms. Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 12419- 12433, 2024.
Liu, A., Liu, J., Pan, Z., He, Y., Haffari, G., and Zhuang, Minicache: Kv cache compression in depth dimension for large language models. arXiv preprint arXiv:2405.14366 2024a.
Liu, R., Bai. H., Lin, H., Li, Y.. Gao, H., Xu, Z., Hou, L., Yao, J., and Yuan, C. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint 2rXiv:2403.01244, 2024b.
He, Y., Zhang, L., Wu, W., Liu, J., Zhou, H., and Zhuang, B. Zipeache: Accurate and efficient kv cache quanti- zation with salient token identification. arXiv preprint arXiv:2405.1 14256, 2024.
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. Kivi: A tuming-free free asym- metric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750. 2024c.
Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length 11m in- ference with kv cache quantization. arXiv preprint arXIv:2401.1807, 2024.
Malinovskii, V., Mazur, D., Ilin, I., Kuznedelev, D., Burlachenko, K., Yi, K., Alistarh, D., and Richtarik, P. Pv-tuning: Beyond treme Ilm compression. arXiv preprint 4rX14:2405.14852. 2024a.
Horváth, S., Kovalev, D., Mishchenko, K., Richtárik, P., and Stich, S. Stochastic distributed learning with gradient quantization and double-variance reduction. Optimization Methods and Software, 38(1):91-106, 2023.
estimation for
Malinovskii, V., Panferov, A., Ilin, I., Guo, H., Richtárik, P., and Alistarh, D. Pushing the limits of large lan- guage model quantization via the linearity theorem. arXiv preprint arXiv:2411. 2024b.
HuggingFace. Optimum-quanto: A pytorch quantiza- tion backend for optimum. https: https://githuu com/ quanto, 2024. Accessed: huggingface wttpe//pppimmuuuuooooooomomm 2025-01-28.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609 1609.07434 2016.
Kang, H., Zhang, Q., Kundu, S., Jeong, G., Liu, Z., Krishna, T., and Zhao, T. Gear; efficient kv cache compression

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Paszke, A., Gross, S., Massa, F,, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, I.,, Bai, J., and Chintala, S. PyTorch: An imperative style, high-perfomae performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS). Neu- ral Information Processing Systems Foundation, 2019.
Vaswani, A. Attention is all you need. Advances in Neural
Information Processing Systems, 2017.
Wang. Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj. A., Hc, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark, 2024. URL https: //aa/aaxivvooo// org/ abs 200 01574.
Razzhigaev, A., Mikhalchuk, M., Goncharova, E., Gerasi- menko, N., Oseledets, I., Dimitrov, D., and Kuznetsov, A. Your transformer is secretly linear. arXiv preprint arXiv:2405. /2250, 2024.
Weber, M., Fu, D. Y., Anthony, Q. Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., Athiwaratkun, B., Chalamala, R., Chen, K., Ryabinin, M., Dao, T., Liang, P., Ré, C., Rish, I., and Zhang, C. Redpajama: an open dataset for training large language models. NeurIPS Datasets and Benchmarks Track, 2024.
Reinsel, G. C. and Velu, R. D. Multivariate Reduced- Rank Regression. Springer. New York, 1998. ISBN 978- 178-1-4757-2853-8. doi: 0..1077998-114757-25553 URL http::::///iikksprrrrss 1007/978-1-4755
Weidinger, L., Mellor, J. F. J., Rauh, M., Griffin, C., Ue- sato, J., Huang, P.-.., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S. M., Hawkins, W. T., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., Isaac, W. S., Legassic. S., Irving, G., and Gabriel, I. Ethical and social risks of harm from language models. ArXiv, abs/2112.04359, 2021. URL https: / api. semanticscholar. arg/CorpusTii 24495463999
wwt/book/1..c.
Sheng, E., Chang, K.-W., Natarajan, P. and Peng, N. Societal biases in language generation: Progress and challenges. ArXiv, abs/2105.04054, 2021. URL https: //api. swwanticscholateeee......c.c.aae o 234337004.
Su. J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.0988 098644 2021.
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-.., Mellor, J. F. J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., Biles, C., Brown, S. M., Kenton, Z., Hawkins, W. T., Stepleton, T., Birhane, A., Hendricks, L. A., Rimell, L.,, Isaac, W. S., Haas, J., Legassick, S., Irving, G., and Gabriel, I. Taxonomy of risks posed by language models. Proceedings of the 2022 ACM Con- ference on Fairness, Accountability, and Transparency, 2022. URL ht htwps::tt///..... http://aaaa.aa.....aaa///// semant. ierannicsccie cscholar. org htgg/CrrpuID::: 249872622
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https: / http//wwwnmmmmm////////////...mt github. blog/ hwtpg//wen.5///
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F. et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13977 2023.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint IrXiv0 2019.
Tseng. A., Chee, J., Sun, Q. Kuleshov, V., and De Sa, C. Quipft: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint 2rXiv:2402.04396 2024a.
Tseng, A., Chee, J., Sun, Q. Kuleshov, V., and Sa, C. D. Quip#: Even better Ilm quantization with hadamard inco- herence and lattice codebooks, 2024b.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ei- ficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
Tseng, A., Sun, Q. Hou, D., and Sa. C. D. QTIP: Quantization with trellises and incoherence processing. In The Tiihth Annual Conference on Neural In- formation Processing Systems, 2024c. URL https: //openreview.oo net.eeeweeeeeeeeeee.eeeee forum? 7
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu. D., Huang, F., Dong, G., Wei, H., Lin, H., Tang. J., Wang, J., Yang. J., Tu, J., Zhang, J., Ma, J.. Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang. K., Li, M., Xue, M., Ni, N., Zhang, P., Wang. P., Peng. R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z..,
van Baalen, M., Kuzmin, A., Nagel, M., Couperus, P., Bas- toul, C., Mahurin, E., Blankevoort, T., and Whatmough, P. Gptvq: The blessing of dimensionality for I1m quanti- zation. arXiv preprint arXiv:2402. 15319, 2024.
12

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Zhang, Z., and Fan, Z. Qwen2 technical report. arXiv preprint ar arXiv:2407. 10671, 2024a.
Yang, Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang, E., Kwon, S. J., and Lee, D. No token left behind: Reliable cache compression via importance-aware mixed precision quantization. arXiv preprint arXiv:2402. /8096, 2024b.
Yang. Y., Cao, Z., Chen, Q. Qin, L., Yang. D., Zhao, H., and Chen, Z. Kvsharer: Efficient inference via layer-wise dissimilar kv cache sharing. arXiv preprint arXiv:2410. 18517, 2024c.
Yue, Y. Yuan, Z., Duanmu, H., Zhou, S., Wu, J., and Nie, L. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint arXiv:2402. 10002..2006 12065, 2024.
Zandieh, A., Daliri, M., and Han, Qjt: 1-bit quantized jl transform for cache quantization with zero overhead. arXiv preprint orXiv:24066 2tl.:22200.34822, 2024.
Zhang. Z. and Shen, H. Zero-delay qkv compression for mit- igating kv cache and network bottlenecks in Ilm inference. arXiv preprint arXiv:2408.04100 2024.
Zhang, Z., Sheng. Y., Zhou, T., Chen, T., Zheng, L.,, Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 09.34661-37711 2023.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-folloiing evalu- ation for large language models, 2023. URL https: //arxiv. orgg abs abs/23111 221.1.11..000 07911.
Zhuo, T. Y., Huang, Y., Chen, C., and Xing, Z. Red team- ing chatgpt via jailbreaking: Bias, robustness, reliabil- ity and toxicity, 2023. URL https: wtw.iaaxvvo............ abs 2301 12867.
13

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
A. Additional Details for Section 3
Experiment configuration in Section 3.1. To measure the dependency between different layers, we first compute the Key-Value cache of Llama-3.2-3B (non-instruct) model on collection of 256 random sequenes of 8192 tokens sampled from RedPajama. We split these cache entries into 224 calibration sequences and 32 holdout sequences, and train linear probes (regressors) that learn predict keys values from one of the analyzed input sources: previous layers, past tokens, and different role vectors. The metric we report, explained variance ratio, is computed using per-channel variance, to account for biases in attention keys & values. We fit linear probes via close form solution with regularizer rate 10 10-3
Full inference algorithm from Section 3.3. To better formalize our approach, also provide detailed description of infereece with AQUA-KV compressed cache in Algorithms 2, 3 & 4 below.
Algorithm 2 encode
Require: layer.inden K, reconstructed keys and values of the previous layer Rpees V prev, predictors
1: if beder_indexxee = then
2: return K, V
3: end if
4: feey Svalue + preddcttorssaayer.ndee
5: pred fkey prov))
6: K2 t QQ(K- - K prea)
7: K (K)( K pred
8: Vpred + Svalue (LV previ KJ) {Concatenate prev and K}
9: Vg +- Q(V V yrea)
return kg, Vg
Algorithm 3 decode
Require: layer-index, K. Vg, reconstructed keys and values of the previous layer K Kpee. Vrev, predictors 1: if Layer index = then
2: return K,, Vy
3: end if
4: fay, Jvalue prredictors(laerrineee
5: K (Kt) + fkey (K prev))
6: # Value predictor expects both previous reconstructed V peev and current reconstructed K.
7: V (V,) fvatue (IV (IV p K)
8: return K, V
B. Extended Inter-Dependence Measurements
In addition to the abbreviated charts in Section 3.1, we also report extended inter-dependence probing results. Figure 4 explores additonal probe inputs and Figure 5 contains individual explained variance rations for each transformer block.
C. On LongBench Evaluation on non-Instruct models
As we discuss in Section 4, we only evaluate Instruct model variants on LongBench tasks.
Non-Instruct models treats the prompts with question as a plain that should be continued narratively. Namely, observe that, when mon-Instruct model is evaluated most LongBench tasks, it tends to behave follows:
1. The model encodes the task and produces an answer to that problem, whether correct or not;
2. Having produced the answer, the model "continues" the prompt by imagining new task;
3. The model produces the answer to the newly imagined problem;
Depending on the model and the allowed sequence length, steps 2-3 can repeat multiple times. Although generated text by the model may contain the correct answer, this can reduce the score drastically as LongBench scorer, on the most tasks, calculate the score between the whole generated sequence and ground truth answer.
14

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Algorithm 4 Inference with AQUA-KV Predictors
Require: model, input, key_cache, value.cache, predictors
1: K past: V past: K mev, V Vipprre f 0
2: X - poodel.input.ee.. htw.dee.innpttemeedddinn
(input)
3: for = 0,,., mode.,, mode.. nww.eaerrmmm...rr.. - do
4: # Recover previoully saved values
5: (K V pmm) +- decode(i, key.cache value.cachel]]. ( Rust, V pps, predictors)
6: # Run forward pass
7: block + model. model.trans www.:traassormmmo layers
9: block wtw.cckgge..ateenti......
8: (Kinp, Vinp) block.
p Ki IV past, mml) {Concatenate past and input keys/values}
10: # Compress new key-value entries
11: (K imp) encode(i, Kinp V inp K innpprev V i predictors)
key-cache + [key-cache[]]; K inp. {Concatenate key cache with new keys}
13: value-cachelp t mallueccachellj V {Concatenate value cache with new values} V
Kiip.prev. V Vipp.pev) + decode(i, K imp imp: Kinp.prev. V Vipp.prev predictors))sss
end for
16: return model compute
Predict Keys
0.50
0.25
0.00
Ku Prev. block keys
Van
Kt-1 K
[KV-
Prev. block values
Prov. token keys
Prev. token values
Mixed roles
Predict Values
0.75
0.50
0.00
K--2
Ki-2
Vi-+ VL-2
VL-1
Vas
Kf-1 kT-3
K
yr
vr-2
vr-3
Vil
K--1
Vi-1 KVL-
IKV, [VV
Figure Additional Mean Explained Variance Ratios by linear probes from previous blocks (L), tokens (T) and role Llama-3.2-3B.
Here an example of such generation for the original Llama 3.2 3B model:
Task name: NarrativeQA
PromptID:
Generation:
American
AnAnguestion: What is the name of the man who is the subject of the story?
Annnsnser:: Frank Saltram
An mnnnuesstio: What is the name of the man who is the subject of the story's story?
An Annnswer: George Gravener
In innquesstin: What is the name of the man who is the subject of the story's s story's s story? Annnswe:: Kent Mulville
15

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Explained Variance Ratio from Chosen Dependencies Across Transformer Blocks.
0.75
0.50
KL-1 2 K
0.25
VL-1-> > VL
0.00
20
25
Transformer block index (Llama-3.2-3B)
Figure 5. Explained Variance Ratios per Transformer Block for chosen sets of linear probes on Llama-3.22 3.2-3B.
Table 8, Evaluation of Llama 3.2 3B non non-Instruct model on the same 14 LongBench tasks as in Section 4, The model was evaluated as is, without an additional "chat template", using the official LongBench evaluation code (Bai al., 2023).
SamSum
WIKMMO
TREC
HorpotQA
MoluNews
Trivia
QMSuum
Psg Count
MFQA. on
Masique
Qasper
PoeRetr
NarrativeQA
GovReport
Quant. Bits
Wiki2 PPI.,
LongRench Avg (base model)
Config
(base model)
Uncompressedd
6.98
26.46
42244
11.25
22.64
23.79
35.03
7.28 12.25
11.45
30.16
HIGGS 2-bit
2,02
7.47
24.93
40.49
11.75
68.00
9.89 15.22 88.45
22.22
0.00 29.83
6.80
12.42
6.77
13,90
23.22
AQUA KV (HIGGS)
2.09
7.03
26.15
42.73
11.88
69.00
9.13
18.73
88.46
23.11
0.00
35.35
7.45
12.21
7.33
12.01
28.68
Quanto 2-bit gx64
2,50
21.56
15.54
11.63
AQUA-KV (Quanto gs64)
2.64
10.33
26.24
41.59
11.36 69.00 10.60 21.85
88.33 22.91
0.00
32.67
6.93
12.97
6.65
12.75
29.69
InAnQuestion: What is the name of the man who is the subject of the story's s story's story's story?
An An\nnnswer: Sir Gregory Coxon
An InInQuestion: What is the name of the man who is the subject of the story's story's s story
Ground truth answer: She is American.
To the best of out knowledge, the original LongBench study (Bai et al., 2023) evaluated on chat assistants, not base language models. However, several prior works on Key- quantization chose to evaluate non-Instruct models on LongBench. For instance, KIVI (Liu et al., 2024c) evaluates both Llama-2 Chat and non-Chat models on LongBench tasks Table 4. While non-chat models show inferior performance, we recognize that it may be interesting to see their LongBench evaluations for some niche cases. As such, we run several LongBench evaluations for AQUA-KV an baselines, using the same setup as in Section 4.1, 1 but for the non-Instruct model. The results of this evaluation are reported in Table
D. Detailed Evaluations for Section 4.1
We provide scores on each of the LongBench tasks in Table 9, as well as some additional experiments. The table is supplemented with results on QuaRot quantization method (Ashkboos et al., 2024) that combines Randomized Hadamard Transform (RHT) with standard non-vector quantization. We also report Quanto with larger group size (1024); ablation with post-RoPE quantization and additonal KVSharer setup with 2 shared pairs. The table also contains an ablation on the predictor architecture, including MLP predictors without Layer Nomalization, reduced rank regression (RRR with rank 256), quantized predictors and corner case with bias-only predictor. We also perform ablation analysis of keeping a longer sequence of unquantized tokens at the beginning of the sentence (4 and 64 tokens). Finally, we evaluate AQUA-KV with different predictor input configurations: removing one of the predictor components or using more distant past layers. Note that we also evaluated other AQUA-KV entries in Sections 4.1, 4.2 and 4.3 without quantizing the first layer and found no significant differences in perplexity and LongBench score, similarly to what we observe Table
16

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Table Evaluation of Llama 3.2 3B with various Key- Key-Value cache compression strategies. The left panel contains Wiki perplexity for the base -onnIIntruutt model and the average LongBench results for the Instruct model. The right panel reports detailed per-task LongBench scores for the Instruct model.
SanSum
2iIkMM
TREC
HotpotQA
MoliNows
TrivinQ4
OMSum
Psa Count
MFQ MPQA.o on
Musique
Qasper
PagRetr
NarrativeQA
Gov GovRsport
Quant.
WikiZ PPI.,
LongBench Avg-f (instruct model)
Conlig
Bits
(base model)
Uncompressed
16
6.98
44.47
42.57
40.65 71.5
53.07
26.17
88.78
24.50
3.53
$0.36
26.27
40.17
96.0
25. 16
33.89
Quantizers
KIVI-2h-gs125 r128
2.25
9.34
39.64
41.05
37.27
70.0
47.54
26.24
88.73
23.37
7.00
47.69
21.26
35.26
59.50
21.16
28.86
KVQuant-2--s18
2.33
9.43
20.56
26.75
17.38
50.50
23.58
21
56.96
18.28
2.11
22.93
8.25
1292
4.50
4.41
17.66
Quanto-2b-ga64
2,50
21.36
33.59
32.80
32.03
67.50
47.31
25 80
81.93
22.58
3.06
44.17
20.46
25.81
21.0
19:47
26.40
Quanto-2b-gs1222
2,03
5559.23
QuaRot 2b g-024
2,03
44.68
32,80
29.4
34.72
63.00
47.43
24.64
75.57
20.59
4.50
37.90
19.14
29.77
32.50
16.86
23.22
HIGGS-2h- 20CGG--h--g1100 1022
2.02
7.47
43.25
40.81
71.50
24.87
88
23.93
50.92
26.87
37.76
88.0
AQUA KV (Quanto g.64)
2.64
10.33
43.64
41.29
39.99
71.50
53.72
26.15
88.68
23.86
3.53
50.61
24.96
40.38
88
50
24.26 33.54
AQUA-KY (Quanto gs1024)
2.17
43.66
AQUA KV (QuaRoi g=1024)
2.17
8.44
44.54
42.31
39.28
72.00
53.21
25.77
88.48
24.03
4.50
52.
27.23
40.89
94.00
25.72 33.94
AQUA-KV (HIGGS 2h)
2.16
7.03
44.26
39.98
25.65
88.67
24
39.82
AQUA-KY (per-channel)
2,16
7,06
43.96
42.47
40.14
71.5
52.92
25.54
88.92
24.13
4,50
50.74
25.35
39.96
94.50
50
24.73
30.07
KVSharer
shured pair
15.43
7.45
36.81
23.07
2.50
2 shared pairs
14,86
8.04
40.89
40.64
28.74 71.50
36.05
25.70
87.59
2
4.14
48.60
18.03
39.48
81.67
31.03
31.64
shared pairs
13.71
9.60
29.82
35.06
24.64 59.0 37.23
23.12
78.16
20.96
3.06
37.5
14.28
31.96
12.58
16.59
23.3
Predictor Architecture (tntal predictor size)
MLP w/o LN (540 Miß)
2.16
7.03
44.33
42.42
40.02
71.50
53.47
25. 50
88.67
24.02
5.00
51.57
26.01
39.34
95.0
25.46
32.65
MLP (540 MiB)
2.16
7.03
44.61
42.60
39.60
71.0
53.77
25.29 88.41
24.17
4,50
50.12
26.45
40.96
96.0
30.31
31.29
AQUA-KV (162 MiB)
2.16
7.03
44.26
42.42 39.98
71.5
52.36
25.65
88.67
24.31
4.50
52.22
26.42
39.82
95.0
24.73
32,02
RRR (68 MiB)
2.16
7.22
44.30
42.52
41.07
71.00
52.90
25.65 88.78
24.73
4,50
49.70
26.75
40.39
96.0
24.69
31.54
GPTQ (41 MiR)
2.16
7.03
44.19
41.97 41.06 72.00
53.62
25.57
88.87
24.43
4.50
50.46
26.0
38.50
50
95.0
24.59
32
Only bias
2.02
7.24
42,83
38.96 39.66 69.50
518
24.14
88.23
24.07
3.50
50.20
25.96
38
36
91.5
26.25
27.48
First Layer Quantization
First layer 16b
2.59
7,03
44.26
42.42 39.98 71.5 52.36 25.65
88.67
24.31
4.50
52.22
26.42
39.82
95.0
24.73
32.02
Finst layer 4h
2.16
7.03
44.26
41.63 40.30 30
71.50
53.
25.71
88.32
24.07
4.50
50.67
26.24
40.75
95
50
25.10 32.21
First layor 3b
2.13
7.03
44.28
41.81 40.30 71.00
53.32
25.83
88.93
24.78
51.61
25.93
38
81
96.00
32.24
First layer 2h
2.09
7.05
44.41
42,74 41.03 72.00 53.45
25.57
88.70
24.47
4.50
49.70
26.67
39.47
96.00
25.11
32.38
Atlention Sink Quantization
All tokens quantized
2.16
7.15
44.13
25.46
31.54
Skip 4 tokens Skip 64 tokens
2,16
7.03
44.26
42.42 39.98 71.5
$2.36
25.65
88.67
24.31
4.50
52.22
26.42
39.82
95.0
24.73
32.02
2.17
44.26
42.01 40.30 72.00
88.67
24.73
Predictor Inputs
Kms + K. [Kne. Vau) + V.
2.16
7.03
44.26
42,42
39.98
71.5
$2.36
25
65
88.67
24.31
4,50
52.22
26.42
39.82
95.0
24.73
32.02
who predicter
2.09
43.91
25.53
wo K predictor
2.09
7.50
42.92
39.51
40.33
71
51.36
25
87.51
23.87
4.06
49.02
26.15
37.85
88.0
24.35
30.14
w/o Kix + V
2.16
7.04
44.23
42.42
40.26
71.50
52.89
25
97
24.64
4.00
50.52
26.63
40.46
94.50
25.30
31.38
w/o Vas V
2.16
7.04
44.28
42.79
39.71
71.50
52.75
25.86
88.68
24.42
4.00
50.94
26.92
39.73
95.0
26.07
31.54
Inputs from layers 7.
2.16
7.02
44.42
42.43
39.80
71.50
53.42
28
88
24.46
5.50
50.41
26.80
39.69
95 50
24.99
32.76
Inputs only from layor L-
2.16
7.04
44.01
42.25
39.66 71.0
53.12
25.38
88.77
24.47
5.50
50.51
26.15
38.82
31.97
Predictor Order
Before RoPE HIGGS
2.16
7.03
44.26
42.42 39.98 71.5 52.36 25.65
88.67
24.31
4,50
52.22
26.42
39.82
95.00
24.73
32.02
After RoPE HIGGS
2.16
7.05
44.13
41.27 40.06 71.50 53.05 25.72
88.77
24.64
4.50
50.
14
25
39.81
95.50
25.48
31.73
Before RoPE Quanto gs64
2.64
10.90
44.54
42.57 41.31 71.50 53.10 26.02
16
24.01
3.53
50.59
26.91
41
50
95
50
25
33.69
After Quanto gx64
2.64
10.33
43.64
71.50 53.72
Quantizer Errors Handling
Quamtizer specifffc
2.16
7.03
44.26
42.42 39.98 71.5 52.36 25.65 88.67 24.31
4.50
52.22
26.42
39.82
95.00 24.73 32.02
Quantizer agmostic
2.16
7.13
44.26
53.46
E. Detailed Evaluation of AQUA-KV and Baselines for Section 4.2
This appendix presents the evaluation results for AQUA-KV with the HIGGS backhone and baseline models across multiple architectures, including Llama 3.2 (3B, 8B, 70B) and Qwen2.5 (3B, 7B). The models are assessed under 2-bit, 3-bit, and 4-bit KV-cache compression settings.
We evaluate two key aspects of model performance:
17

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Table 10. Evaluation of AQUA-KV (with HIGGS backbone) and baselines Llama 3.2 3B for 2, 3 & bit compression. The WikiText-2 Perplexity is evaluated on base (non-instruct) version of the model with sequence length 8192. The LongBench results are an average over 14 tasks evaluated with Instruct model with sequence length 2 (131K) tokens. In addition to the overall average score across all tasks, the table also includes individual scores for each of the 14 tasks. The specific tasks in the original LongBench benchmark and the corresponding evaluation metrics can be found in the text.
SamSum
2WKKNNQ
TREC
HotporQ
MalaNews
Trivia
QMSam
Psg Count
MEQA en
Musique
Qusper
PstRetr
Narrative A
GovReporr
Quant. Bits
Wiki2 PPL.
LongBench Avg (instruct model)
Config
(base model)
Uncoompressed
16
6.98
44.61
42.50
40.32
70.50
52.77
25.79
88.78
24.38
5.00
51.13
26.21
40.74
97.0
97.00
24.93
34.54
AQUA-KV
2,09
7.03
44.30
41.76
40.66
72.50
52.72
25
88.78
24.43
4.50
49.07
26.21
39.36
97.
25.58
32.13
HHGGS
2.02
7.47
42.80
39.56
39.97
72.5
52.54
24.52
87.76
24.10
3.00
49.61
26.91
36.84
88
25.35
28.00
KTVI
2.25
9.34
39.64
41.05 37.27
70.00
47
54
26.24
88.73
23.37
7.00
47
69
21
35,26
59.50
21
28.86
KVQuant-2b-s18 %
2.33
9.43
18.28
26.75
17.38
50.50
23
28
64
56.96
18.28
2.11
22.93
8:25
12.92
4.50
4.41
17.66
AQUA-KV
3.06
6.98
44.37
43. 16.16.60
40.67
72.5
25.
77
24.71
4.00
49.25
26.55
96
24.43
33.52
HIGGS
3.02
7,05
44,41
42.74
40.67
73.00
52.54
25
58
88
80
70
4.50
48.
96
26.21
40.
96.50
24.61 32.87
KIVI
3.05
7.87
41,40
41.86
38.47
71.00
48.63
26.59
89.13
23.60
3.50
48.10
20.48
36.78
73.00
24.84 33.57
KVQuamt-3--s155
3,33
7.26
23.85
42.17
35.40
72.00
46.75
25.51
89.05
23.85
4.53
50.79
22.54
41
70.50
24.03
31.03
AQUA-KV
4.02
6.98
44,48
42.86
40.59
72.5
52.18
24.36
4.50
49.28
26.06
40.30
96.50
25.20 33.89
HIGGS
4.02
7,01
44.41
42.06
40.91
72.50
52, 40
25
63
88
22
24.
55
4.00
49.77
26
40.31
96.00
24,70 33.74
KIVI
4.25
7.03
43.11
42.82
38.41
71.00
48.91
26.64
89
28
23.61
6.50
50.53
21.51
40.62
86.00
24.08 33.65
KVQuant-Ib-s1%%
4.33
7.04
24.20
42.44
37.14
72.50
50.84
25. 48
24.20
2,00
53.12
23.06 38.99
94.50 24.52 33,05
Table 11. Evaluation of AQUA-KY (with HIGGS backbone) and baselines on Llama 3.1 8B for 2, 3 & bit compression. The WikiText-2 Perplexity is evaluated on base (non-instruct) version of the model with sequence length 8192. The LongBench results are average over 14 tasks evaluated with Instruct model with sequence length 217 (131K) tokens. addition to the overall average score across all tasks, the table also includes individual scores for each of the 14 tasks. The specific tasks in the original LongBench benchmark and the corresponding evaluation metrics can be found in the text.
SamSom
WRkMM
TREC
HorpatQA
MaluNews
TririaQ
OMMSum
Psg Count
MFQA. an
Musigue
Qusper
PsrReer
NarrativeQA
GovReport
Quant. Bits
Wiki2 PPI.,
LongBench Avg (instruct model)
Config
(base model)
Uncompressed
16
5.61
48.13
43.62
48.58
72.50
57.8
26.86
91
25
10.50
55
32,75
44.62
100.0
29.65
34.4
AQUA-KV
2.08
5.72
47.77
42.99
48. 16
73.50
$7
58
26
91.91
25.79
7.25
55.71
33.46
44.53
99
50.50
29.67
32,53
HIGGS
2.02
5,89
47.37
41.20
49.03
73.00
57.58
47
91.97
25.21
7.25
56.91
31.80
43.76
99.5
30.27
30.16
KIVI
2.25
7.37
46.28
43.41
43.33
71.50
55.12
26.79
91.14
24.57
5.67
52.56
30,9
41.02
99.50
29.10
33.3
KVQuant
2b.s1%
2.33
6.64
37.17
41.94
34.23
65.00
43.78
25.82
86.27
22.94
2.92
47.13
23.24
36.51
36.50
22.63
31.53
AQUA-KV
3.05
5.64
48.10
43.89
48.35
73.5
$7.43
26.85
91
25.46
7.43
56.64
33.96
45. 18
99.5
29.61
34.06
HIGGS
3.02
5.66
43.75
47.77
73.5
26
91
7.33
67.67
33.57
43.97
99.5
29.87
33.6
KTVI
3.05
6.04
46.98
43
43.86
72.50
54
46
26.93
91
76
25.21
6.78
54,
40
30.92
43.73
99.00
29.54
35.15
KVQuant-3b-s1$
3.33
5.84
46.42
44.56
42.73
72.50
53.91
26.54
92.01
24.96
5.42
53.42
28.03
44.31
99.50
28.64
33,40
AQUA-KV
4.02
5.61
48.10
44.14
48.81
73.5
57.2
11
91 63
45
7.43
56.17
33.69
44.6
99.5
29.68 34.45
HIGGS
4.02
5.62
43.6
73.5
57.23
7.6
55.72
34.29
44.43
99.5
30.28 34.21
KIVI
4,25
5.64
47.57
44.07
45.54
72.50
55.72
26.86
92
42
25.
58
8.34
54.89
31
44.42
99.50
29.46 35.22
KVQuant-b-s19
4.33
5.65
47.77
43.61
49.10
71.00
58.56
26.87
91.05
03
25
5.42
55
35.03
33
45.67
99.50
29.86 34.45
1. Language Modeling Quality- - Measured using WikiText-2 Perplexity, tested on the base (non-instruct) versions of the models with a sequence length of 8192.
2. Task-Specific Performance Measured on LongBench, a benchmark covering 14 diverse NLP tasks, with evaluation conducted on the instruct-tuned versions of the models using a sequence length of 2 (131K tokens) in total.
The Tables 10-14 summarize the average LongBench performance across all 14 tasks for each model, followed by the WikiText-2 perplexity scores. Additionally, detailed per-task results are provided for a deeper understanding of how different models and quantization settings affect specific NI.P capabilities.
To further analyze the impact of KV-cache compression various NLP applications, we report the performance on each of the 14 individual LongBench tasks in the following table 10. Each task has its own original name, evaluation metric and average length, reflecting its unique requirements.
18

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Table 12. Evaluation of AQUA-KV (with HIGGS backhone) and haselines on Llama 3.1 70B for 2, 3 & 4 hit compression. The WikiText-2 Perplexity is evaluated on base mnon-instruct) version of the model with sequence length 8192. The LongBench results are an average over 14 tasks evaluated with Instruct model with sequence length 2" (131K) tokens. addition to the overall average score across all tasks, the table also includes individual scores for each of the 14 tasks. The specific tasks in the original LongBench benchmark and the corresponding evaluation metrics can be found the text.
SamSum
WWBaMQ
TRE C
HatputQA A
MuluiVws
Trivia
OMS
Psgt Count
MFQA enn
Musiyue
Qusper
RsrRetr
Narrative
Gov RovRoport
Quant. Bits
Wiki2 PPL
LongBench Avg (instruct model)
Config
(base model)
Uncompreessed
16
2.54
52.92
44.004
68.63
76.50
65.71
26.72
94.21
23.93
18.00
$4.79
49.7
19.73
98.5
36.31
34.85
AQUA-KV
2.04
2.62
52.79
46.65
69:71 76.00
64.94
26.64
94.04
24.40
18.00
31
46.17
49.77
97
50
35.92
34,06
HIGGS
2.02
2.77
52.18
46.43
76.00
63.41 26.49
93.94
24.25 20.00
54.08
44.56
97.
35.04
32.11
KIVI
2.25
3,06
52.45
47.07
66.83 76.50
63 88
26.43
91.38
24.
50
20.00
53.91
46.66
49.
19
97
50
34,89
33,58
KVQuant-2b-s18
2.33
3,28
31.39
44.87
19.64
65.
51.52 52
26.09
89 42
23 44
11.50
51
38.50
43.59
97.50
30.92
31.85
AQUA-KV
3.03
2,55
52.81
46.74
68.65 63.633760 76.50
61
26
50
94.21
24.16
18.00
54.60
45, 89
49.73
98
36.17 34.59
HIGGS
3.02
2:57
52.56
46.26
75.50
64.76
26.
38
04
24,
18.50
54.74
45.
49
24
97
36.42 34.24
KIVI
3.05
2,87
$2.87
47.66
67.09
76.50
64.37
26.66
93.61
24.56
20.00
54.54
47.56
49.87
98.00
35.47 34.29
KVQuant-
3.33
2.75
35.02
45.83
59.45
76.00
58.21
26.32
89.42
23.9
17.50
93
48
35.8
34.18
AQUA-KV
4.02
2,54
$2.95
47.05
68.63
76.50
65.79
26.75
94.04
24.01
18.50
54.66
46
49
49
58.50
36.58 34.63
HIGGS
4.02
2.55
52.97
47.77
69:01
76.50
66.33
26.83
94.04
24.50
16.50
54.37
46.56
49.27
99.00
36.31 34.65
KIVI
4.25
2.61
52.88
47.24
67.27 76.50 64.37 26.75 94.04
24.47 20.00
54.01
47
85
50.19
97.
50
35.23 34.91
KVQuant-b-s118
4.33
2.58
35.42
47.66
68.
75.50 66.70 26.87 93.73 24.20
18.00
54.27
46.03
48.83 99.00
36.13 34.70
Table 13. Evaluation of AQUA-KV (with HIGGS backbone) and baselines on Qwen2.5 3B for 3 4 bit compression. The WikiText-2 Perplexity is evaluated on base (non- nonninntruuct) version of the model with sequence length 8192. The LongBench results are an average over 14 tasks evaluated with Instruct model with sequence length 2 (131K) tokens. In addition to the overall average score across all tasks, the table also includes individual scores for each of the 14 tasks. The specific tasks in the LongBench benchmark and the corresponding evaluation metrics can found in the text. The increased AQUA-KV bitwidth is due to not quantizing the 1st block for this model.
SamSium
2WikiMQ
TREC
HotpotQA
MoliNNww
TriviQ
QMsuu
Psg Count
MFQA.cm en
Musique
Qusper
PsgRetr
Narrative
GovReport
Quant. Bits
Wiki2 PPI.,
LongBench Avg f (instruct model)
Conlig
(base model)
Uncompressed
16
7.14
38,80
44.45
38.64
68.00
46.60
22.60
87.60
22.94
3.00
49.29
20.53
37.07
49.00
21.88 31.64
AQUA-KV
2.44
7,20
38.31
43.43
37 47
68 50
46.35
22.69 88
23.23
23
3.00
48
93
18.1
36.98
47.50
21.71
30.34
HIGGS
2,06
7,92
30.92
34.81
84.40
20.26
KIVI
2.25
9.05
28.66
42.07
12.48
69.00
18.60
23.37
87.05
24.09
5.05
29.96
10.56
11.10
31.27
10.31
26.31
AQUA-KV
3.42
7.15
38.77
45.19
38.64
68.00
46.81
22.90
87.67
23.35
2.50
48.43
20.39
37.34
47.50
22.79
31.29
HIGGS
3.06
7.28
31.85
42.90
30.92
53.00
41.33
21.64
79.05
22.53
2.50
42.94
15.98
29.94
20.50
19.78 22.94
KTVI
3.05
7.63
30.37
44.08
13.71
69:00
18.31
23.97
86.66
23.44
4.75
36.62
9.84
14.10
38.75
12.05
29.92
AQUA-KV
4.4
7.14
38.92
37.98
46.80
22.57
87.93
23.09
37.34
21.75 31.42
HIGGS
4.06
7.15
32.11
41.37
29.80
55.00
37.84
23.01
70.86
22.79
2.00
42.52
15.46
34.17
33.42
20.29 20.95
KIVI
4.25
7.17
31.50
45.01
15.42 69.50 21.40 24.59
87.53
23.76
3.50
38.99 12.06
16.13 42.50 8.65 31.9
Table 14, Evaluation of AQUA-KV (with HIGGS backbone) and baselines on Qwen2.5 7B for 2, 3 & 4 bit compression. The WikiText-2 Perplexity is evaluated on base (non-ing instruct) version of the model with sequence length 8192. The LongBench results are an average over 14 tasks evaluated with Instruct model with sequence length 2 (131K) tokens. In addition to the overall average score across all tasks, the table also includes individual scores for each the 14 tasks. The specific tasks in the LongBench benchmark and the corresponding evaluation metrics can be found in the text. The increased AQUA-KV bitwidth is due to not quantizing the 1st block for this model.
SamSum 2WikiMQ
TREC
HotpoiA
NeltiNws
Trisin
OMSum
MFQA.em en
Musique
Qusper
Pep
Narrative
Gov Repprrt
Config
Quant. Bits
Wiki2 PPL.
LongBench Avg (instruct model)
Psg
(base model)
Uncompressed
16
6.13
46.82
45.77
46.94 72.00
57.72
23,90
89
23.56
8.00
52.58
30.35 43.78
100.00 00.0029.44 49 31.93
AQUA-KV
2.48
6.17
46.43
45.99
45.71
72.00
56.88 23.84
89.18
23.44
8.50
52.15
29.63
42.77
99.50
29.21 31,15
HIGGS
2,03
8.08
25.97
26.98
17.98
52.50
31.67
12.04
55
45.46
13.14
7.25
27.19
14.55
22.22
55.79
14.72 12.05
KIVI
2.25
7.02
32.78
44.79
10.60
71.00
22.04
27.29
6.69
91.67
14.37
31.79
AQUA-KV
3.45
6.14
46.81
45.58
47.35
72.00
57.88
24.07
89.10
23.59
8.00
52.63
30.64
44.05
100.0028.40
32.03
HIGGS
3,03
7.2
14.61
14:02
13.38
45.00
15.27
6.07
29.29
10.00
5.54
18.29
4.65
14.81
15.90
6.10 6.19
KTVI
3.05
6.37
32.63
45.65
9.71
71.00
10.34
22.49
88.87
20.68
4.69
29.83
6.87
13.17
90.46
10.66
32,37
AQUA-KU KV
4.43
6.14
46.77
45.81
46.87
72.00
57.71
24.06
89.83
23.83
8.00
52.41
30.35
43.37
100.0028.65 31.86
HIGGS
4.03
6.88
11.54
11.80
7.62
37.75
10.13
4.39
22.54
10.52
7.20
10.50
4.83
5.85
16.26
5.02
KIVI
4.25
6.14
33.40
46.20
9.61
71.00
10.34
22.27
89.63
21.13
4.58
30.56
6.85
12.79
98.25
12.06 32.39
19

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Table 15. LongBench tasks mapping and evaluation metrics.
[Our task name]
Orig. task name
Eval metric Avg len
SamSum
SAMSum
Rouge-L
6,258
2WikiMQ
2WikiMullibepOA
4,887
TREC
TREC
Accuracy
5,177
HotporQA
HorpoiQA
F1
9,151
MultiNews
MultiNews
Rougo-L
2,113
TriviaQA
TriviQAA
F1
8,209
QMSum
QMSum
Rouge-L
10,614
PsgCount
PassageCount
Accuracy
11,141
MFQA_,n
MutiFielQAAenn
Fl
4,539
Musique
MUSiQue
Fl 11,214
Qusper
Qusper
Fl
3,619
PsgRetr
PassageRetriovalenn
Accuracy
9,289
NarrativeAA
NamrrrveQA
18,409
GovReport
GovReport
Rouge L
8,734
F. Additional Evaluations and Details for Section 4.3
In Section 4.3, we evaluate the following benchmark settings:
1. GSM8K (Cobbe et al., 2021) mathematical reasoning in 8-shot of-thugtt setup.
2. MMLU-Pro (Wang et al., 2024) general reasoning in 5-shot sfain-of-thoughle setup.
3. IFEval (Zhou et al., 2023) instruction following benchmark in zero-shot mode.
We use Language Model Evaluation Harness (Gao et al., 2021) version 0.4.7 with default parameters for each benchmark. For MMLUUPro, Table 6 in the main paper reports aggregated accuracy over different subsets. For completeness, we also report the individual per-task scores Table 16.
Table 16. Evaluation of AQUA-KV (with HIGGS backhone) and haselines on Llama 3.2 3B Instruct and Llama 3.1 8B Instruct with 2 bit compression. In addition to the overall average MMLU-Pro score, the table also includes individual scores each of the 14 tasks
Config
Quant. Bits
Overall score
Biology
Basiness
Chemisrry
CompSel
Ecoumiiss
Engincering
Health
History
Law
Math
Other
Philosophy
Physies Psychology
(instruct model)
Uncompreased
34.47
53,14
35.61 26.68
37.07 42.65 21.98
40.59
32.81
23,52
35.23
35.61
33.07
28.56 50.25
AQUA-KV
2.09
32.79
51.60
32.19
23.76 34.15 41.71 21,26
38.02
34.12
23.43
32,86
33.33
34,67
25.25
50.38
HIGGS
2.02
29.25
47.56
28.64
21.91
29.51 40.52 18.99
31.05
29.40
18.26
26.94
31.60
32.46
22.02
48.37
Llama 3.1 8B
Uncompreased
16
44.34
64.99
48.04
35,51 49.76 55.57 30.24 51.59
42,52
27.70
44.86
45.45
44.69
39.03
59.77
AQUA-KV
2.08
43.25
61.92
46.89
32.95
46.10 55.57 28.79 50.73 42.26
28.52
42.04
45.56
44.29
37.95 61.03
HIGGS
38.40
59.41
38.53 28.71 36.34 50.71 25.39 43.89 38.58 28.34
33.23
42.86
42.08
31.18 58.15
Additionally, we compare AQUA-KV and HIGGS compression the HumanEval (Chen et al., 2021) programming benchmark using the default LM Evaluaton Harness configuration. The results are summarized in Table 17.
While AQUA-KV similarly outperforms quantization without predictors, we caution the reader against drawing signif- icant conclusions from these HumanEval results. The absolute scores, even without quantization, do not match the publicly reported HumanEval scores for Llama 3.x. Upon further inspection, we found that this common issue and other researchers also report problems when reproducing HvaanEval. We uncertain whether this due Instruct
*F.g. see https: httpps://gt / thub. com www/meta 1 amma 1 ammma3/ saues/10000
20

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Table 17. HumanEval pass@ I results
Quant. Bits
Lluna 3.2
Qwen 2.5
Method
3B
3B
Uncumpressed
24.28
17.38
AQUA-KV
2.09
24.25
17.31
HIGGS
2.02
|
22.26
17.18
model prompting, answer extraction or other issues. The problem may potentially be resolved in the recently introduced humaneval 1 _instruct task in LM Evaluation Harness that was added after we ran our experiments, but we did not have the compute budget to test it further.
G. Additional Evaluations of AQUA-KV with H₂O
Here, we report more detailed evaluation results in setup where AQUA-KV is combined with H₂O Heavy Hitter Oracle. In addition to per-task LongBench scores, we also report several additional configurations for AQUA-KV backbone quantizer. The results for 3B and 8B models can be found in Table 20 and 19 respectively.
Table 18. Evaluation of Llama 3.2 Instruct with H2O pruning mixed with various Key- Value cache compression strategies. A 20% KV cache budget for HzO was used for all evaluations. The left panel contains the average I LongBench score for the model. The right panel reports detailed per-task LongBench accuracies and scores for the model.
SamSuni
2WikMMQ
TREC
HotpotQA
MoteNows
TriviaQA
QMSun
Pag Count
MFQ en
Masigue
Qusper
Ret
NarrativeQ.A
GovReport
Quant. Bits
LongBench Avg (instruct model)
Config
Uncompressed
16
44.61
42.5
40,32
70.5
52.77
25.70
88.7
24.38
51.13
26.21
40.74
97
24.93
34.54
H₂O
16
38.82
(22.92 39.87
68
43.91
22.64
88.88
22.14
7.5
48.52
16.38
31.78
65
18.39 27.54
H20+ + AQUA-KV
2.09
18.43
42.99
39.67
68
44.02
22.19
88.6
21.74
7.5
46.51
16.54
30.03
65
18.65
26.52
H20 HIGGS
2.02
37.02
40.04
37.93
67.5
44.11
22.15
87
21.19
6.5
46.5
16.05
29.07
56.5
18.07
25.24
H2O+ + AQUA- KV
3,06
38.76
43.2 39.73
68
43.99
22.34
89.04
21,93
7.5
48.02
16.51
31.5
64.5
18.88
27.45
H2 HIGGS
3.02
38.27
42.96
37.81
67.5
43.3
22.15
89.38
22.11
8.5
46.56
16.35
30.17
62.5
19.34 27.14
HgO+ AQUA-KV
4.02
38.85
43.16
39.96
68
43,9
22,86
89.04
22.17
7.5
48.44
16.38
31.88
64.5
18.48
27.61
H20 HIGGS
4.02
38.74
42.66
38.19
68
43.91
22.31
88.78
22.03
7
48.59
17.06
32.01
65.5
18.92
27.34
Table 19, Evaluation of Llama 3.1 8B Instruct with H2O pruning mixed with various ValuV cache compression strategies. A 20% KV cache budget for H₂O was used for all evaluations. The left panel contains the average LongBench score for the model. The right panel reports detailed per-task LongBench accuracies and Fl scores for the model.
SamSSm
2WiKMM0
TREC HotpotQA
NottiNows
TriviaQA
MMnum
Page Count
MFQA en
Musique
Qaojer
PouRetr
Report
Quant. Bits
LongBench Avg (instruct model)
Config
Uncompressee
(43.62
48.58
72.5
57.8
26,86
91.47
25.43
10.5
55.58
32.75 44.62
100
29.65
34.4
H₂O
16
41.42
|44.57
44.84
69.5
44.09
23.04
91.75
22.1
6.61
51.49
24.28
38.45
68
22.29
28.91
H2O + AQUA-KV
2.09
41.11
44.35
42.57
69
44.3
22.6
92.19
22.06
6:59
$1.87
23.92
37,57
68.5
21.9
28.16
H,O HIGGS
2.02
40.72
42.04
44
68.5
43.39
23.01
91.26
22.38
7.24
50.08
24.24
34.53
69
22.68
27.69
H2O AQUA-KV
3.06
41.31
44.63
44.69
69.5
44.32
23.06
92.28
22.12
6.51
50.68
23.98
37.45
68
22.
22.66
28.4
H2O+ HIGGS
3,02
41.37
44.39
45.14
69.5
44.88
22.86
91.84
22.5
6.95
50.72
24.24
37.27
68
22.57
28.31
H2O AQUA-KV
4.02
41.47
44,51
44.94
69.5
44.07
22.95
91.75
22.45
6.83
51.53
24.29
38.5
68
22.22
28.98
H₂0+ HIO+HIGGS HIGGS
4.02
41.32
44.47
44.08
69.5
43.97
23.03
91.93
22.11
6.83
50.52
24.34
38.67
68
22.21
28.8
H. Detailed Inference Benchmarks
In this section, we report detailed inference speed benchmarks using our AQUA-KY implementation. Our implementation consists of two CUDA kernels:
1. Quantization kernel: finds the nearest vector in a HIGGS lattice and returns its index. Since the original HIGGS (Mali-
21

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
Table 20. Evaluation of Llama 3.2 3B Instruct with H₂O pruning mixed with various Key-Value cache compression strategies. A 50% KV cache budget for HzO was used for all evaluations. The left panel contains the average LongBench score for the model. The right panel reports detailed per-task LongBench accuracies and scores for the model.
San
2WikMM
TREC
HotpotQ A
MoldNews
Trivin
QMSum
Par Cound
MFQA
Musique
Qasper
RstRetr
Narrative
Go RovReport
Quant. Bits
LongBench Avg (instruct model)
Config
Uncompressed
16
44.61
42.5
40.32
70.5
52.77
25.79
88.78 24.18
51.13
26.21
40.74
24.93
34.54
1g0
16
39,69
42.84
37.87
68.5
44.57
24.84 88.22
22.51
8.5
48.87
16.72
37.22
65
19.35 30.59
HgO+ + AQUA-KV
2.09
39.11
42,89
38.04
68.5
45.12
24.38
88.64
21,82
6
49.15
16.39
34.65
63.5
19.68
28.62
H₂O + HIGGS
2.02
37.42
39.72
36.61
68.5
44.01
24.15
87,52
21.77
5
47.9
16.01
32.32
54.5
18.85
27.07
H₂0+ + AQUA-KV KV
3,06
39.61
42.59 38.68
68.5
44.58
25.06
88.39
22.17
7.5
49.22
17.22
36.46
64.5
19.42
30.29
H₂O+ HIGGS
3.02
39.21
42.38
37.62
68
45.04
24.75
89.06
22.32
47.96
16.14
36.21
61
19.5
29.93
11.0 -VQUA-VV
4.02
39.72
42.65
37.8
68.5
44.58
25.07
89.08
22.35
8.5
49.06
17.06
36.83
64
20.07
30.52
H2O HIGGS
4.02
39.45
42.23
38.1
68,5
44.67
25
88.53
22.37
9
49.78
16.16
36.85
61
19.81
30,3
Table 21. Evaluation of Llama 3.1 8B Instruct with H₂O pruning mixed with various Key- Value eache compression strategics. A 50% KV cache budget for 0I, was used for all evaluations. The left panel contains the average LongBench score for the model. The right panel reports detailed per-task LongBench accuracies and Fl scores for the model.
SamSuum
2WRIMQ
TREC
HotpotQa
MoldNews
Triviaa
QMSuu
ParC
MFQA. en
Musique
Qasier
Rsr
NarrativeeAA
Gov RovRepprt
Quant. Bits
LongBench Avg (instruct model)
Config
Uncompressed
16
48.13
43.62
48.58
72.5
57.8
26.86
91.47
25.43
10.5
55.58
32,75
44.62
100
29.65
34,4
122
16
42.35
44.36
44.46
69.5
43.68
25.53 91.83
23.15
6.49
$2.93
24.8
43.63
68
22.78
31.79
HaO+ AQUA-KV
2.09
42,08
43.87
43.32
69
44.27
25.31
92.38
23.45
6.14
53.2
24,86
41.8
68.5
22.7
30.31
H₂O HIGGS
2.02
41.61
41.69
44.87
69.5
43.55
25.33
91.52
22.47
6.55
52.43
24.08
41.11
68
22,24
29.13
HaO AQUA-KV
3.05
42.37
44.1
44.65
69.5
44.12
25.38
92.62
23.47
6.5
53.57
24.73
43,3
68
22.13
31.14
+20 HIGGS
3.02
42,23
43.84
45.28
69.5
43.94
25.34
91.85
23.07
6.49
53.16
24.85
42.01
68
22:41
31.45
H2O + AQUA-KV
4.02
42.37
44.3
44.34
69.5
43.73
25.78
91.72
23.33
6.71
53.08
24.7
43.56
68
22.82
31.64
H2O+ HIGGS
4.02
42.17
44.38
43.65
69.5
43.54
25.81
91.93
22.84
6.76
53.25
24.54
42.53
68
22.13
31.58
novskii et al., 2024b) did not implement fast quantization, we developed this kernel specifically for our use case.
2. Dequantization kernel: selects the corresponding vector from the lattice, applies quantization scale and performs Hadamard transformation. The dequantized values are added to the predictor outputs in the same kernel.
We compare three inference modes: standard (bfloat16) inference, quantized AQUA-KV inference with CUDA kernels, and quantized inference with naive PyTorch (Paszke et al., 2019) implementation. The models we benchmark on are Llama 3.2 3B and Llama 3.1 70B to test our approach with different model sizes.
Inference latency. Our first set of experiments measures inference latency with batch size 1. We generate sequences of up to length 32768 and report the full forward pass latency (time per token). The results in Figure 6 show that our implementation introduces an average inference overhead of 60% for shorter sequences (fewer than 10k tokens), and an average overhead of 18% for longer sequences (10k to 32k tokens). This is expected since AQUA-KV performs all the computations required for 69 inference and additional quantization/ccquattaatiioo operations.
Throughput. Next, we compare infernce throughput in batched inference setting. For each method, we select the maximum batch size that fits on GPU (A100 for 3B, 2x A100 for 70B) at maximum sequence length and measure the number of tokens generated per second. In this setting, AQUA-KV compressed can achieve greater throughput for shorter sequences as shown in Figure 7. Note, however, that this improvement stems from the fact that our method can fit a larger batch size in the same GPU memory due to cache compression.
Overall, our CUDA implementation significantly outperforms the naive Python / PyTorch inference due to having significantly less DRAM 1/O operations (from fused dequantization). The latency overhead can be further reduced with additional optimizations: for instance, we can quantize the predictors themselves to 4-bit with almost no loss in accuracy (see Appendix D), which could allow us to speed up predictor computations. Additionally, it should be possible to fuse quantized cache reconstruction with the attention computation for further speedups.
22

---

Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models
0.6
blloat16
bíloat16
AQUA-KV PyTorch
AQUA-KV PyTurch
Time (seconds per token)
AQUA-KV CUDA
Time (seconds per token)
AQUA-KV CUDA
0.12
0.
0.08
03
0.06
0.1
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000
25000
30000
LLaMa 3.2 3B, Forward passes
LLaMa 3.1 70B, Forward passes
Figure Inference latency with batch size I for Llama 3.2 3B (left) and Llama 3.1 70B (right) A100 GPUs.
blloat16
blloat16
AQUA-KV PoTchh AQUA-KV CUDA
AQUA-KV PyTorch
AQUA-KV CUDA
Tokens per second
Tokens per second
400
300
200
1000
2000
3000
4000
5000
6000
7000
5000
5000
10090
15000
20000
LLaMa 3.2 3B, Forward passes
LLaMa 3.1 70B, Forward passes
Figure 7. Throughput of batched inference for Llama 3.2 3B (left) and Llama 3.1 70B (right) A100 GPUs.
23


Reviews:
[{"review_id": "ZZZ3FKlvvF", "reviewer": "Reviewer_AgDD", "summary": "The paper proposes a learned predictor based adaptive quantization for KV Cache compression. The idea is this -- transformer models are residual in nature, i.e. each subsequent layers add smaller and smaller deltas to the outputs -- this means that the intermediate representations are highly dependent. This turns out to be true for even KV Cache vectors. Paper leverages this understanding to train simple linear predictors for KV Cache and only quantize the residuals. since liner predictors explain significant variance the quantization on residuals are correspondingly accurate leading to further compression.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback and are glad that they appreciate our method's design and empirical results.\nBelow, we provide detailed answers to the questions posed in the review:\n\n> If you train your predictors post-ROPE for 8192 sequences, do you see any deterioration in inference on sequences beyond 8192 tokens?\n\nIn short, we found that AQUA-KV is not sensitive to the training sequence length: specifically, we did not see a significant impact on longer sequences present in our LongBench evaluation (with some tasks in excess of 100k tokens [1]). We attribute this to the fact that AQUA-KV only trains simple linear predictors.\n\n[1] https://github.com/THUDM/LongBench\n\nHowever, we agree that it is important to analyze the effect of training sequence length. To that end, we evaluate AQUA-KV for the Llama 3.2 3B model with **varying training sequence length** and measure the impact on perplexity.\n\nTable 1. WikiText-2 PPL for different training sequence lengths.\n\n|Eval \\ Train sequence length|128|1024|4096|8192|\n|-|-|-|-|-|\n|8192  |7.02|7.03|7.03|7.03|\n|128  |17.87|17.87|17.87|17.87|\n\nNote that in these evaluations, we control for the total number of tokens: every time we halve the sequence length, we also double the number of sequences in the calibration dataset. Otherwise, training with 64-token sequences would overfit because of insufficient training dataset size. For convenience, we report additional sequence lengths in https://anonymous.4open.science/r/rebuttal-pics-24A3/.\n\n> How do you think about combining multiple compression techniques together -- low-rank, quantization, predictor-based, pruning,etc.\n\nOur approach is indeed compatible with different cache compression techniques applied simultaneously. Specifically, **we combined AQUA-KV with H$_2$O pruning in Section 4.3 (detailed results provided in Appendix E)** and show that AQUA-KV can augment H$_2$O to further improve its size-to-accuracy trade-offs by combining three techniques together: pruning (H$_2$O), predictors (AQUA-KV) and quantization (HIGGS). We also explored low-rank *predictors* in Appendix C (Table 4), which can further reduce the memory footprint at the cost of degraded performance. In case you are interested in other specific combinations, we will consider them and add to the final version of the paper.\n\n\n\n\n\n> What in your opinion is the lower-bound (bits / token) on compression for reasonable performance. Like can we go below 2 bits?\n\nIn our work, we focused on ~2 bits per value because this setup can achieve favorable quality-to-size trade-offs for practitioners. However, it is indeed interesting to evaluate AQUA-KV in the extreme sub 2-bit setup. To that end, we evaluate AQUA-KV on top of a 1-bit HIGGS variant with d=8 group dimension and n=256 clusters, with the rest of hyperparameters matching our setup from Section 4.1. This results in circa 1 bit per stored value. We report our results for Llama 3.2 3B and 3.1 8B below.\n\nTable 2. WikiText-2 perplexity evaluation of AQUA-KV for 1 bit compression.\n| Method | Avg. Bits | Llama 3.2 3B | 3.1 8B |\n|--------------------|----------|--------------|--------|\n| - | 16 | 6.98 | 5.61 |\n| AQUA-KV ($d{=}8$, $n{=}256$) | 1.09 | **7.52** | **6.10** |\n| HIGGS ($d{=}8$, $n{=}256$) | 1.02 | 16.18 | 19.83 |\n\nTable 3. LongBench evaluation of AQUA-KV for 1 bit compression.\n\n| Method | Avg. Bits | Llama 3.2 3B (Instruct) | 3.1 8B (Instruct) |\n|--------------------|----------|--------------|--------|\n| - | 16 | 44.61 | 48.13 |\n| AQUA-KV ($d{=}8$, $n{=}256$) | 2.09 | **40.61** | **43.09** |\n| HIGGS ($d=8$, $n{=}256$) | 2.02 | 23.02 | 24.94 |\n\nTo summarize, AQUA-KV with 1-bit HIGGS quantization can achieve substantially better quality than raw 1-bit quantization, but both methods show higher quality deterioration. Still, this is an interesting setup and a potential frontier for future research. We will add these and additional sub 2-bit evaluations to the final version of the paper.", "claims_and_evidence": "Yes. The claims are well supported", "methods_and_evaluation_criteria": "Yes. The method is novel and well-suited application of micro-machine learning.", "theoretical_claims": "No theoretical claims made", "experimental_designs_or_analyses": "Experimental setup seems correct. Longbenchmark is a representative benchmark for KV Cache issue. The bit compressions used are reasonable. The baselines seem reasonable. ( Disclaimer: I am not very well versed with baselines in this field. For instance, are there any other adaptive methods that can be combined with quantization / pruning -- since no such method is discussed in the paper)", "relation_to_broader_scientific_literature": "The idea of using local learned components in compression is a new idea, in my understanding. Post-training compression is generally fixed algorithms. Learning is used in compression in two ways -- intermittent training of entire model for recovering compression loss (QAT or LTH) or from-scratch training of compressed models (SynFLOW like pruning OR ROAST). The idea of using compact ML models for reducing dimension / variance of data while natural is new in compression literature.", "essential_references_not_discussed": "I am not well versed with related literature", "other_strengths_and_weaknesses": "[Strengths]\n1. Good use of learned predictors\n2. Good gains in memory footprints at same accuracy\n3. The impact on efficiency in contained.\n4. Empirical evaluation including ablations is useful.\n\n[Weakness]\nNothing I can think of.", "other_comments_or_suggestions": "None.", "questions_for_authors": "[Questions out of curiosity , do not impact the evaluation of the paper]\n\n1. If you train your predictors post-ROPE for 8192 sequences, do you see any deterioration in inference on sequences beyond 8192 tokens?\n2. How do you think about combining multiple compression techniques together -- low-rank, quantization, predictor-based, pruning,etc. \n3. What in your opinion is the lower-bound (bits / token) on compression for reasonable performance. Like can we go below 2 bits?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RxDoWzgO8E", "reviewer": "Reviewer_8Gvk", "summary": "The paper presents AQUA-KV, an approach that leverages dependencies between keys and values across adjacent attention blocks. The method employs linear predictors trained to estimate KV caches for a given block based on previously generated keys and values. Subsequently, the residuals are quantized to low bit-widths to achieve efficient KV cache compression.\n\n## update after rebuttal\nThe paper presents a novel approach to reduce the KV cache quantization errors. After rebuttal, the authors clarify several parts in the paper and add more evaluations. Therefore, I recommend to accept this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and valuable suggestions. We agree that improving clarity and expanding evaluations would strengthen the paper, and we address these points below:\n\n> What does \"high-compression mechanisms for internal network states\" mean in the abstract? Is it an observation?\n\nWe meant to say that there exist methods that quantize KV states to low-bitwidth with small quality degradation. In the final version of the paper it could be rephrased to “In this work, we aim to improve Key & Value compression by exploiting two observations: … 2) the existence of high-compression methods for internal network states (e.g. attention Keys & Values).”\n\n> In L59-60 in the paper, the author mentioned \"vector quantization\". However, I can not find any details.\n\nBy “vector quantization” we referred to HIGGS, which is a vector quantization method, as described in L161. We meant that using HIGGS with predictors provides the best quality compared to other methods for KV quantization, as discussed further in Table 2.\nWe will clarify that by adding “and the more advanced **vector quantization scheme** HIGGS” in L272.\n\n> L196-197 needs to be elaborated.\n\nThese lines are indeed somewhat convoluted. We meant the following: \n1. we noticed that using 1- and 2-bit quantizers (e.g. HIGGS) can ‘explain’ ~0.75 and ~0.89 of the variance respectively. In other words, they have a ~0.25 and ~0.11 relative quantization error.\n2. If a probe can predict keys/values with the same error as a 1-bit quantizer, we found that we can use 1 less bit for quantization (e.g. 3-bit instead of 4-bit) after the residual with, on average, the same accuracy (e.g. see Table 2).\n\nWe will clarify this in the revised paper.\n\n> The experiments are only conducted on Wiki2PPL and some LongBench tasks. It lacks evaluations on more challenging tasks such as math and code tasks with CoT prompts.\n\nWe agree that AQUA-KV can benefit from additional evaluations on larger models. As requested, we have conducted evaluations on GSM8k (CoT) and IFEval and report them to in the tables below.\n\n**GSM8k accuracy (%) for Instruct models in the same setup as Section 4.2**\n\n|Method|Avg.Bits|Llama 3.2 3B|3.1 8B|3.1 70B|Qwen 2.5 3B| 7B|\n|-|-|-|-|-|-|-|\n|Uncompressed|16|76.5|85.1|94.7|61.2|76.6|\n|AQUA-KV|2.09|**77.7**|**84.3**|**94.2**|**59.9**|**72.2**|\n|HIGGS|2.02|70.3|79.2|**94.2**|35.8|59.7|\n\n**IFEval accuracy (%) for Instruct models in the same setup as Section 4.2**\n\n|Method|Avg.Bits|Llama 3.2 3B|3.1 8B|3.1 70B|Qwen 2.5 3B| 7B|\n|-|-|-|-|-|-|-|\n|Uncompressed|16|77.0|78.9|88.0|66.5|76.9|\n|AQUA-KV|2.09|**75.1**|**79.9**|**88.1**|**66.2**|66.9|\n|HIGGS|2.02|72.4|75.7|87.0|59.3|**68.6**|\n\nThe results generally align with the trends observed in LongBench evaluations, with AQUA-KV being substantially closer to the uncompressed baseline than raw HIGGS. We will add these evaluations to the final version of the paper and conduct additional experiments for code benchmarks (e.g. HumanEval).\n\nWe appreciate the reviewer’s insightful comments, which have helped us improve the paper’s clarity and experimental scope. The additional evaluations confirm AQUA-KV’s consistent performance across domains, as noted in our response. We hope these revisions address all raised concerns. If you add any follow-up suggestions in the next discussion phase, we will address them in the final version of the paper.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths: The idea is simple and effective, and it can quantize the stored KV cache residuals to 2-bit.\n\nWeaknesses: \n1. The paper is somehow difficult to follow. See Questions below.\n2. The experiments are only conducted on Wiki2PPL and some LongBench tasks. It lacks evaluations on more challenging tasks such as math and code tasks with CoT prompts.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. What does \"high-compression mechanisms for internal network states\" mean in the abstract? Is it an observation?\n2. In L59-60 in the paper, the author mentioned \"vector quantization\". However, I can not find any details.\n3. L196-197 needs to be elaborated.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ThhgDQKr4", "reviewer": "Reviewer_LU4T", "summary": "This work proposes AQUA-KV, a method of using inter-layer and intra-layer information to reduce the size of the KV cache with minimum overhead via a supplementary probe. AQUA-KV supplements a “backbone” quantization algorithm, where it functions to improve accuracy by using the information available from the previous layer (for both the keys and values) and within the same layer (for the values). By this method, only residual information unique to each token need be saved in the KV cache, allowing for better compression.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Overall, the review appreciates the efficacy and simplicity of AQUA-KV, but suggests additional evaluations on extra benchmarks and asks follow-up questions about implementation details. We do our best to address these below.\n\n# Additional evaluations\n\n> In Table 2 of the paper, there is only a small difference between the performance of HIGGS on Llama 3.1 70B compared to AQUA-KV with HIGGS. Much more rigorous evaluation is required to see if AQUA-KV is effective for large models as well as smaller models. \n\nFirst, we would like to note that the quantized 70B model already has very high quality, meaning that any absolute changes in score will be small.\nThus, we ask the reviewer to take into account the *relative* difference: for 2.x bits per value, the 70B model perplexity with 16-bit cache is 2.54. AQUA-KV @ 2-bit increases only PPL by +0.08, whereas raw HIGGS increases it by +0.23 (*almost 3x*). Likewise, the LongBench score drops by 0.13 for AQUA-KV and 0.74 for the nearest 2-bit baseline (>5x error increase). \n\nTo fully address the remark concerning larger-scale evaluations, we evaluate the 72B Qwen 2.5 model in the same setup as in Section 4.2:\n\n|Model|Method|Avg.Bits|Wiki2PPL (non-Instruct)|GSM8K (Instruct)|\n|-|-|-|-|-|\n|72B|-|16|3.49|95.8|\n|72B|AQUA-KV|2.09|**3.56**|**95.5**|\n|72B|HIGGS|2.02|3.66|93.7|\n\nWe also report additional benchmarks for Llama 3.1 70B below and in our response to Reviewer AgDD. To further explore the scalability, we will run the remaining evaluations for 70B+ models in the final version of the paper.\n\n> Language generation tasks by instruction-tuned models should be evaluated rigorously as these are closest to those used for actual LLM production. MMLU, GSM8K, HumanEval, and IFEval are frequently used.\n\nWe agree and evaluate GSM8K and IFEVAL across different models, including 70B. We prioritize 2.x bit evaluations due to time constraints and since this is where augmenting quantizers makes the most sense.\n\n\n**GSM8K accuracy (%) for Instruct models in the same setup as Section 4.2**\n\n|Method|Avg.Bits|Llama 3.2 3B|3.1 8B|3.1 70B|Qwen 2.5 3B| 7B|\n|-|-|-|-|-|-|-|\n|Uncompressed|16|76.5|85.1|94.7|61.2|76.6|\n|AQUA-KV|2.09|**77.7**|**84.3**|**94.2**|**59.9**|**72.2**|\n|HIGGS|2.02|70.3|79.2|**94.2**|35.8|59.7|\n\n**IFEval accuracy (%) for Instruct models in the same setup as Section 4.2**\n\n|Method|Avg.Bits|Llama 3.2 3B|3.1 8B|3.1 70B|Qwen 2.5 3B| 7B|\n|-|-|-|-|-|-|-|\n|Uncompressed|16|77.0|78.9|88.0|66.5|76.9|\n|AQUA-KV|2.09|**75.1**|**79.9**|**88.1**|**66.2**|66.9|\n|HIGGS|2.02|72.4|75.7|87.0|59.3|**68.6**|\n\nThe results show a similar trend to our LongBench evaluations, with AQUA-KV being substantially closer to the uncompressed baseline than raw HIGGS. \nWe will include these results in the final version of the paper and conduct additional experiments with the other two benchmarks (MMLU and HumanEval).\n\n> Although long sequence evaluation is also important, evaluations on shorter sequences should also be conducted.\n\nWe hope that this concern can be alleviated with the results we reported above, since those benchmarks have shorter sequences (e.g. GSM8K question plus answer takes up, on average, **198 tokens** for Llama-3.1/3.2 tokenizer). We also report perplexity with shorter sequence length (see the first table in our response to Reviewer AgDD). We do note, however, that KV-cache compression is most effective in the long-context regime. \n\n# Questions about overlapping AQUA-KV with model inference\n\n> Could the prediction for the keys be overlapped with the calculation of the previous FFN layer? \n\nThank you for this suggestion. It is indeed possible to overlap FFN/MLP computation with computing the next layer AQUA-KV cache. Furthermore, **since the value predictor is linear, we can overlap half of its computation (from previous values) as well,** then add the other half after the fact. \n\n> Could the benefit of applying the keys to reconstruct the values be investigated more thoroughly? This creates a dependency that prevents overlapping.\n\nWe have investigated this question via ablation analysis in **Table 4, section “predictor inputs”, on L809 (w/o $K_{rec}$ → V)**. To summarize, the key predictor does improve perplexity and LongBench scores, but only slightly (in the last digit). This component can indeed be removed in cases where one cares about better overlap. We will discuss this trade-off in Section 4.1.\n\n> I am willing to change my rating if these concerns are addressed.\n\nWe hope that the additional evaluations and discussions we provided can alleviate the reviewer’s concerns. If you add any follow-up suggestions in the next discussion phase, we will address them in the final version of the paper.", "claims_and_evidence": "The claim that AQUA-KV contributes more to accuracy than the baseline quantization is somewhat supported, especially for smaller models. However, there are too few evaluations of larger models (Llama 70B) to be certain that the proposed method adds value.", "methods_and_evaluation_criteria": "The limited number and types of evaluations conducted are the main weakness in this work. Although long sequence evaluation is also important, evaluations on shorter sequences should also be conducted. For example, using batched inference could also necessitate KV cache compression.\n\nMoreover, in Table 2 of the paper, there is only a small difference between the performance of HIGGS on Llama 3.1 70B compared to AQUA-KV with HIGGS. Much more rigorous evaluation is required to see if AQUA-KV is effective for large models as well as smaller models. Language generation tasks by instruction-tuned models should be evaluated rigorously as these are closest to those used for actual LLM production. MMLU, GSM8K, HumanEval, and IFEval are frequently used.\n\nI am willing to change my rating if these concerns are addressed.", "theoretical_claims": "There were no theoretical claims in this work.", "experimental_designs_or_analyses": "The experimental design of checking results for different backbone quantization algorithms and probes was sound. Also, the authors provided sufficient analysis of the effects of using multiple previous layers and previous tokens. The major issue was the paucity of evaluations.", "relation_to_broader_scientific_literature": "KV cache quantization is an emerging research topic with high practical value. With the increasing volume of LLM inference, reducing KV cache memory is a key consideration for LLM service providers. The authors propose a new method of improving outcomes from compressing the KV cache while retaining accuracy.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The method is simple to understand and has relatively little overhead, both during the training and inference stages. Although the authors do not provide an optimized implementation, there does not appear to be any fundamental barrier to integrating their solution to frameworks such as vLLM.\n\nAlso, by demonstrating the effectiveness of their method on the LongBench, the authors show that their method is competitive with other KV cache quantization algorithms on long-sequence evaluations, even matching the performance of the unquantized BF16 baselines.", "other_comments_or_suggestions": "There appears to be a missing word in paragraph 3 of Section 4. “We evaluate perplexity on base (non-instruct) models since they have better .”\n\nThe title displayed on the first page does not match the title displayed on the top of the subsequent pages.", "questions_for_authors": "Could the prediction for the keys be overlapped with the calculation of the previous FFN layer? This could reduce the overhead from the probe, although some KV cache quantization methods may not be compatible. \n\nCould the benefit of applying the keys to reconstruct the values be investigated more thoroughly? This creates a dependency that prevents overlapping.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1NLe7vNuqu", "reviewer": "Reviewer_MQGD", "summary": "The paper proposes AQUA-KV, a KV cache compression method for autoregressive LLMs that explots inter and intra layer dependencies for improving cache quantization accuracy. It can be combined with additional compression techniques such as pruning. To this effect, they train predictors for predicting the value of a Key & Value pair using other cache entries, and apply quantization on the residual information that could not be predicted. This method only needs to store the information that cannot be recovered from other sources. They use the previous layer keys to predict the subsequent keys, and use both the previous layer values and the current layer keys to predict values in their implementation. In their practical implementation they use linear regression for all predictors, and any quantization method can be used for quantizing the residual information. For evaluation they measure perplexity and end to end performance. They measure perplexity on WikiText-2 and end to end performance accuracy on LongBench tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and suggestions. We appreciate that you highlight the modularity of AQUA-KV and address your concerns below.\n\n> My main concern is that in Table 2, where the authors compare their method to other quantization approaches across various LLMs, their method shows a 0.3 improvement over HIGGS on the LongBench average score for Llama 3.X 8B and 70B in the 3-bit quantization setting but requires an additional 0.5 GB of cache memory. Similarly, for 2-bit quantization, it achieves a 0.4-0.6 improvement but demands 0.6 GB more memory on the LongBench average score for Llama 3.X 8B and 70B. This suggests that the experiment may not be fully controlled, making it unclear whether the observed gains in LongBench performance are significant or if any improvement would persist under a similar cache size constraint. Can the authors have a more controlled experiment, or explain why the existing experiment is controlled enough ?\n\nWe agree that our comparison can be improved by controlling strictly for the cache size. To address this, we conducted additional evaluations with raw HIGGS (our strongest baseline), where the quantizer was given more quantization clusters to increase the average bitwidth. To recall, a 2.x bit HIGGS splits the data into $d{=}2$-dimensional vectors and rounds each vector to one of $n{=}2^4{=}16$ clusters, with an additional 16-bit scale per $g{=}1024$ quantized values (the default configuration from [1]). This yields $\\log_2 n / d + 16/g \\approx 2.0156$ bits per parameter.\n\n[1] https://arxiv.org/abs/2411.17525\n\nTo offset AQUA-KV predictors, we evaluate HIGGS with a lattice of $n{=}18$ clusters instead of 16, with an average bitwidth of $\\approx 2.1$, resulting in a slightly larger overall cache size than for AQUA-KV.\n\n## Table: WikiText-2 Perplexity (non-Instruct models, setup from Section 4.2)\n\n| Method | Avg. Bits | Llama 3.2 3B | 3.1 8B |\n|--------------------|----------|--------------|--------|\n| - | 16 | 6.98 | 5.61 |\n| AQUA-KV | 2.09 |**7.03** | **5.72** |\n| HIGGS ($n=16$) | 2.02 | 7.47 | 5.89 |\n| HIGGS ($n=18$) | 2.10 | 7.40 | 5.85 |\n\n## Table: Average LongBench scores (Instruct models, setup from Section 4.2)\n\n| Method | Avg. Bits | Llama 3.2 3B | 3.1 8B |\n|--------------------|----------|--------------|--------|\n| - | 16 | 44.61 | 48.13 |\n| AQUA-KV | 2.09 | **44.30** | **47.77** |\n| HIGGS ($n=16$) | 2.02 | 42.80 | 47.37 |\n| HIGGS ($n=18$) | 2.10 | 43.31 | 47.16 |\n\nAs we can see, HIGGS with additional clusters does indeed perform better at the cost of a greater memory footprint, but AQUA-KV still outperforms it.\n\n\nPlease also note that the AQUA-KV memory overhead can be reduced by quantizing the predictor weights. We report this setup in Table 1 (see “GPTQ”, L359 for 4-bit quantization). We discuss this in L346-358 (right) and report more detailed results in Table 4 (Appendix).\n\nWe hope that these new results alleviate the reviewer’s concern and will add them to Section 4.1 and perform additional evaluations in Appendix.\n\n> The other question I have is whether the authors compared with KV cache compression techniques based on token selection and pruning, such as StreamingLLM, SnapKV since the authors have compared their method with H2O.\n\nIn our work, we chose the token pruning proposed in the H$_2$O paper as it was a middle ground between StreamingLLM [1] and more recent methods such as SnapKV [2]. In principle, AQUA-KV can also be combined with other token pruning strategies, including the two you proposed. We agree that exploring these combinations can further strengthen our paper, but we need additional time to incorporate them into our codebase and ensure that our experiment setup uses these pruning strategies properly. We thank the reviewer for the suggestion and will add these comparisons in Section 4.3 in the final version of the paper.\n\n\n[1] https://arxiv.org/abs/2309.17453 Xiao et al, 2023. Efficient Streaming Language Models with Attention Sinks\n\n\n[2] https://arxiv.org/abs/2404.14469 Li et al, 2024, SnapKV: LLM Knows What You are Looking for Before Generation", "claims_and_evidence": "I am mildly concerned about conclusions drawn from Table 2 based on the end to end performance of their method and other methods on LongBench. I highlight this concern and phrase it as a question to the authors below.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The paper does not have any theoretical claims.", "experimental_designs_or_analyses": "Yes I checked the evaluation of the proposed method with various other cache quantization schemes for 2 bit quantization, when evaluated on perplexity and end to end accuracy on LongBench. I also checked the evaluation of the proposed method with using HIGGS as the quantization mechanism as compared to other quantization methods for 5 different LLMs on the perplexity and end to end performance tasks.", "relation_to_broader_scientific_literature": "The main contribution of the paper is to the literature on KV cache compression and quantization techniques for memory efficient inference in LLMs.", "essential_references_not_discussed": "I am not aware of any important references not discussed.", "other_strengths_and_weaknesses": "I think the main strength of the proposed method is that it is modular and can be easily combined with any quantization method, and it can be applied on top of any token pruning method. This really increases the versatility of the procedure.", "other_comments_or_suggestions": "None.", "questions_for_authors": "My main concern is that in Table 2, where the authors compare their method to other quantization approaches across various LLMs, their method shows a 0.3 improvement over HIGGS on the LongBench average score for Llama 3.X 8B and 70B in the 3-bit quantization setting but requires an additional 0.5 GB of cache memory. Similarly, for 2-bit quantization, it achieves a 0.4-0.6 improvement but demands 0.6 GB more memory on the LongBench average score for Llama 3.X 8B and 70B. This suggests that the experiment may not be fully controlled, making it unclear whether the observed gains in LongBench performance are significant or if any improvement would persist under a similar cache size constraint. Can the authors have a more controlled experiment, or explain why the existing experiment is controlled enough ?\n\nThe other question I have is whether the authors compared with KV cache compression techniques based on token selection and pruning, such as StreamingLLM, SnapKV since the authors have compared their method with H2O.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents AQUA-KV, an effective method for adaptive KV cache quantization in LLMs. It introduces compact linear predictors to exploit inter- and intra-layer dependencies in the KV cache, enabling residual quantization that achieves higher compression rates with minimal loss in model quality. The idea is both simple and insightful, with strong empirical results across multiple model sizes and benchmarks. The authors also proactively addressed reviewers' concerns by providing additional evaluations on large-scale models and by comparing controlled memory setups, showing consistent gains over strong baselines. While some reviewers initially raised concerns about experimental scope and clarity, the rebuttal convincingly responds to them. Given the thorough empirical validation, I recommend acceptance.
Decision: Accept (poster)