KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
Authors: Xing Li
Zeyu XING
Yiming Li
Linping Qu
Hui-Ling Zhen
Yiwu Yao
Wulong Liu
Sinno Jialin Pan
Mingxuan Yuan
Venue: ICML 2025 poster (2025)
Keywords: Efficient LLM Inference
KV cache quantization
Mixed precision quantization
Attention patterns
Sensitivity analysis
Primary area: deep_learning->large_language_models
TL;DR: We propose a simple yet effective framework KVTuner to adaptively search for the optimal hardware-friendly layer-wise KV quantization precision pairs for coarse-grained KV cache with multi-objective optimization.

Abstract: KV cache quantization can improve Large Language Models (LLMs) inference throughput and latency in long contexts and large batch-size scenarios while preserving LLMs effectiveness.
However, current methods have three unsolved issues: overlooking layer-wise sensitivity to KV cache quantization, high overhead of online fine-grained decision-making, and low flexibility to different LLMs and constraints. Therefore, we theoretically analyze the inherent correlation of layer-wise transformer attention patterns to KV cache quantization errors and study why key cache is generally more important than value cache for quantization error reduction. We further propose a simple yet effective framework KVTuner to adaptively search for the optimal hardware-friendly layer-wise KV quantization precision pairs for coarse-grained KV cache with multi-objective optimization and directly utilize the offline searched configurations during online inference.
To reduce the computational cost of offline calibration, we utilize the intra-layer KV precision pair pruning and inter-layer clustering to reduce the search space. Experimental results show that we can achieve nearly lossless 3.25-bit mixed precision KV cache quantization for LLMs like Llama-3.1-8B-Instruct and 4.0-bit for sensitive models like Qwen2.5-7B-Instruct on mathematical reasoning tasks. The maximum inference throughput can be improved by 21.25\% compared with KIVI-KV8 quantization over various context lengths. Our code and searched configurations are available at https://github.com/cmd2001/KVTuner.

Sinno Mingxuan Yuan KVTuner: Sensitivity-Aware Layer-Wins Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
Xing L
Zeyu Xing
Yiming Li Linping Qu Hui-Ling Zhen Sinno Jialin Pan? Mingxuan Yuan'
Yiwu Yao
Wulong Liu
Abstract
LLMb with layerowise
Constrantts aujotives momary Rockares FX
Search spacee: lnyen-atie
eig
KV cache quantization improve Large Lan- guage Models (LLMs) inference throughput and latency in long contexts and large batch-size scenarios while preserving LLMs effectiveness. However, current methods have three unsolved issues: overlooking layer-wise sensitivity to KV cache quantization, high overhead of online fine- grained Necision-making and low flexibility to different LLMs and constraints. Therefore, we theoretically analyze the inherent correlation of wise transformer attention patterns to KV cache quantization errors and study why key cache is generally more important than value cache for quantization error reduction. We fur- ther propose simple yet effective framework KVTuner to adaptively search for the optimal hardware-friendd layer-wise KV quantization precision pairs for coarse-graines KV cache with multi multi-obbecctve optimization and directly utilize the offline searched configurations during on- line inference. To reduce the computational cost of offline calibration,
Lowserrativviy layera
Search g
Layer wio (Fi
Int kaysr Paroto officient KV codke cunatization
Layeroona
proming
woremaaannttttie thaliyy layare
proditoo paro
Intor Aayo--ayrrr clustering on
Altordonn ane vedar (ec (..)
V
Highh High-eaeeattvvty bnyars
Whilee mood P
Optinnal KV precision search nuti-doeettv optintzation
ICB
Fiall momzeaarraatt f.(P),
KVTuner
V2
Offivee efficlent search bused on layeniata asnaltvilinn quant.
Figure 1: The layer-wise KV cache quantization tuning framework KVTuner with two-stage search space pruning for efficient MOO search using the final memory and model accuracy.
144
utilize the intra-layer KV precision pair pruning and inter-layer clus- tering to reduce the search space. Experimen- tal results show that we can achieve nearly loss- less 3.t5-bit mixed precision KV cache quantiza- tion for L.MSs like Ilama-33 --S-strruettttt and 4.0-bit for sensitive models like Qwen2.5-7B- Instruct mathematical reasoning tasks. The maximum inference throughput can be improved by 21 11.55% compared with KIVI-KVS quantiza- tion over various context lengths. Our code and searched configurations are available at https
030
10
(a) Layer-O query head-2
(b) 21 query head-4
Figure 2: Token-level attention score of the 79-th query token to previous key tokens with the per-token-asym key cache quantization
Srve225-7B-Inntrruee GSM8K). Low- precision KV quantization (4-bit and 2-bit) causes signif- icant distribution shifts, resulting in errors of missing incorrect critical key identification.
1. Introduction
//github.com. hut.ccccccccmmmm200//////ee
Large language models (LLMs) and multi-modality large models can comprehend and generate text, audio, image, and video like humans, showing the strong capability of assisting and interacting with humans. LLM inference effi- ciency such as throughput and latency is critical to enhance user experience and reduce cost. To improve the inference efficiency of LLMs, previously processed KV tokens are cached to avoid redundant recomputation. However, the memory usage of the KV cache linearly grows with the
Equual contribution 'Huawei Noah's Ark Lab "The Chi- nese University of Hong Kong 'Huawei Computing Product @iixing2hhawei.ooms huawei.com>, Zeyu Xing <zeyuxing @ fink.cuhk.edu.h..
Line. Correspondence to: Xing Li <li.xing2
Jialin Pan <sinnopan www.h......uheee.... <yuan monan.minguuanee hum>
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
number of batch size and sequence length, the KV cache becomes the new bottleneck of LLM serving systems with large batching requests and long context. Valuable long con- text generation applications include multi-turn dialogues, long document understanding, and OpenAT ol-like level-2 reasoning. Commercial companies are releasing their sup- ports for long context generation and KV cache-based ser- vices like prompt eaching for better capability and efficiency (OpenAT] 2024| [DeppSeek] 2024). Efficient KV cache man- agement and compression can accelerate LLM inference and reduce hardware resource consumption, making it foundational technique for advancing both enterprise-scale LLM deployment and personalized Al agents.
cache quantization under constrained hardware resources: 1) Can we further almost losslessly compress KV cache with -rrinnly-rreenly and mixed precision quantization in plug-and-play way? 2) Are there any other inherent model properties such as attention patterns (Tang et al. 2025 Xiao ct al. (025) that can help better trade-off memory reduction and model accuracy? 3) There are normally multiple de- ployed LLMs in the industrial service systems and Artificial Intelligence (AI) agents. How to adaptively tune the KV cache quantization precision considering the accuracy re- quirement of requests and the LLM sensitivity to KV cache quantization?
To address these issues, we thoroughly study the sensitiv- ity of L.LM transformer layers to KV cache quantization and theoretically find out that error accumulation caused by KV cache quantization is strongly correlated with attention patterns in Section 4. and 4.4 According to our observation of the sensitivity of key and value cache in the same layer in Section 4.2 and [.3 and the layer-wise difference of transformer layers in Section [4.5] we propose to quantize coarse-grained key and value cache in the same layer with different precision and automatically search for the optimal layer-wise KV cache quantiza- tion precision pairs based on the inherent importance of intermediate layers in Section 5 During online serv- ing, the offline calibrated layer-wise KV cache quantization precision pairs are directly loaded without any additional overhead to improve inference throughput and latency. Our contributions are summarized as follows:
KV cache quantization is one of the most stable and eas- ily deployable KV cache compression methods to reduce the memory footprint and improve throughput (Yuan al.] 2024). INT8/FP8 KV cache with dynamic asymmetric token-wise (per-token-asym) or channel-wise (per-channel- asym) quantization can achieve lossless compression in most practical applications. However, lower-bit KV cache quantization easily leads to model accuracy degradation.
Intra-layer mixed precision KV quantization methods re- tain important KV tokens with high precision to reduce KV cache quantization errors and quantize other cache in the same layer with uniformly low precision such as 2-bit. KIVI (Liu et al. 2024e), IntactKV (Liu et al. (024c), and KVQuant (Hooper et al. 2024) statically keep prefix and initial KV cache blocks with high precision. They need specially designed operators for hardware like GPUs and require more careful KV cache management. Besides, the assumption that the static prefix and recent KV is important may not always hold as demonstrated in Figure 2 where low-precision quantization (4-bit and 2-bit) leads to dra- matic attention distribution shift in sensitive models like 1enn2.5-7B-Instruee Existing static and uniform KV preci- sion methods including KIVI b-bit cannot effectively handle these non-sparse retrieval heads. The only viable and effi- cient solution is to increase KV cache quantization precision of the whole model or some critical and sensitive layers.
We study the underlying mechanism of why key cache normally is more important than value cache. The LLM accuracy degradation with low-bit key cache quantization mainly caused by error accumulation and the layer-wise attention error distribution shift. We find out that the sensitivity of LLMs and intermediate layers to KV cache quantization is the model property and independent of input prompts.
We propose to automatically search for the hardware- friendly layer-wise KV cache precision pairs such as K8V4 and K4V2 with multi-objective optimization (MOO) under certain memory or accuracy constraints for efficient online inference. The intra-layer prun- ing and inter-layer clustering are used to significantly reduce the search space and the offline tuning cost.
In contrast, fine-gratn ane-ggained methods, such as QAQ (Dong et al. 2024). MiKV (Yang et al., 2024b), and ZipCache (He et al. 2024b), dynamically identify critical KV cache and update their precision on-the-fly to improve accuracy. However, they cannot be easily integrated with flash attention (Dao ee al. 2022) and vLL.M (K won et al., 2023), because of the intre- fine-grained KV cache precision difference and additional deployment efforts. In addition, the online com- putation and control flow logic for critical token identifica- tion introduce overhead and not fit into static graph- based inference acceleration methods.
We empirically demonstrate that our mixed -reci--pr KV tuning framework KVTuner can achieve almost lossless KV cache quantization with equivalent 4-bit even 3.25-bit precision in mathematical reasoning tasks for most LLMs with 21.25% inference throughput im- provement.
There are still several issues to improve the inference throughput and maximum supported context length with KV
2

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
Table 1: The error accumulation caused by low-bit KV cache quantization (KIVI-2) potentially leads to wrong responses of mathematical reasoning tasks. The input prompt is combined with 15-shot Chain-of-TTi Thoughts (CoTs) (Wei et al. (023) from GSM8K training dataset for Llama2-13B-chate 13B-chttt model generation. The official KIVI CUDA kernel and code are used to generate the output.
KV cache procision
Results with 15-th input peompt In dance class of 20 students, 20% enrolled in contemporary dance, 25% the remaiiinng ennolled in jazz dance, and the rest enroiled in hip-hop dance. What percentage of the entire suudents enrolied in hip-hop dance?
FP16
Let X the number students enrolied hip-hop dance. The number of students enrolled in contemporary dance is 20% of 20 4. The number
dents carolled in jazz dance is 25% of the remaining = 25% of (20- 0))23 25% of 16 = 4. Therefore, the number of students curolled in hip-hop dance is
30-4-4 220-4-1212-B The pereentage of the entire students cnrolled in hip-hop dance is 12/20 x 100% = 60% Atea 60%.
KIVI-4
Let X number students enrolled hip-hop dance.
number students enrolled in contemporary dance is 20% of
The number stu-
dents carolled in jazz dance 25% the remaining= 25% of (20- 4) 25% of 16 = 4. Therefore, the number of students curolled in hip-hop dance is 20 4- 4 20-1--111212, The pereentage of the entire students enrolled hip-hop dance 12/20 100% = 60%. ANeA 60%.
KIVI-2
Let X be the number of students who cnrolled in hip-hop dance. The number of stadents who enrolled in contemporary dance is 20% of 20 = 4. The number
of students (who enrolled in jazz dance is 25% of 10 4. Therefore, the total number of students (who enrolled in hip-hop dance is 20 10 4 28 The
percentuge of the entire students who enroiled hip-hop hop dance 28/20 228/20-144 14% n868 14.
2. Related Work
3. Background
KV cache management and compression methods include
3.1. Transformer and KV Cache
paged KV cache (Woon et al. 2023),
prefillng--dcodding disaggregation (Qin et al. 2024), quantization (Liu
In LLMs, there are multiple intermediate transformer layers stacked and executed to generate final output responses. For the I-th transformer layer, given i-th D-dimensional input hidden state æ E RR the I-th query, key, and value feedforward neural network layers generate q. = W aa ki = w/ and vf = =W',a, with the corresponding weight matrices W g' W and W respectively. Then the self-attention scores at are computed with the current query embedding and all key embeddings until the i-th step. Finally. the I-th self-attention layer generates the output state de which is forwarded to downstream sub-layers in the I-th transformer layer, with the softly weighted value embeddings V using the attention scores a/:
(PD)
2024e| c Hooper al.
2024 Zhang et al. 2024c
Yang ct al.
2024b He ct al.
2024b a Dong et al. 2024).
eviction Zhang et al.)
2023
Ge et al. 2024 Liu et al.
2023
Li et al. 2024a Adnan et al. 2224).
merging (Zhang 2024 Liu et al.
et al.
2024b Wang et al. 2024 Wan et al.
2024b), low-rank decomposition (Kang et al.
2024b Sun
jet al. 2024a), offloading (Sheng et al. 2023 Zhang et al.
2024a). prefetching (Lee et al., 2024b), and retrieval (Tang
et al. 2024). Among them, KV cache quantization is orthog- onal to most other KV cache management and compression methods, so it has been integrated with eviction, retrieval, and transferring (Tang et al. 2024 Liu et al. 2024d).
Model and activation quantization methods such as GPTQ
qg K
(Frantar et al., 2022). ) SmoothQuant (Xiao et al. 2023), AWQ (Lin et al. 2024a), SpinQuant (Liu et al. 2024f). and
a = softmax
of =v
(I)
VD
QServe (Lin et al. 2024b) are also used to reduce model memory usage and inference latency with low-bit computa- tion units. Model pruning and layer skipping reduce compu- tational cost by directly pruning unimportant layers or heads (Ma et al. 2023: Zeng et al.) 2023 Elhoushi et al. 2024).
where K' "'=oonnat(K", (K), -111kk) k)) and V "concctt (V', o of)
are the key and value embeddings generated in the prefilling and decoding stage in 1-th transformer layer until i-th step. They will still be re-used in subsequent generation steps for self atlf-attention computation. Therefore, we need to store them as KV cache in each layer independently to remove the additional computational cost of KV cache re-computation.
Speculative decoding is another promising direction for lossless LLM inference acceleration by reducing the LLM inference iteration times and KV cache memory movement cost in the memory- bounded decoding stage. LLMS verify multiple tokens speculated with smaller models (Li et al.
3.2. KV Cache Quantization
2024b), self-partial layers (Cai et al. 2024 Liu et al. 2024a Glocckle et al. 2024| Stern et al. 2018). or other training- free algorithms (Zhao et al.| 2024) in one forward step. In addition, Triforce (Sun et al.] 2024b) is proposed to inte-
Although storing KV cache can reduce the re-computation cost, the KV cache may become the new inference memory and latency bottleneck in the large batch size and long con- text scenario. KV cache quamtization can effectively address these problems. The and dequantization along the channel or token dimension to input X E S D are defined as
grate KV cache compression with hierarchical speculative decoding to improve long context generation efficiency.
B-bit quantization
X X--2
Q XX) = round
x
8 s
(2)
3

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
where the offset min(X) and the scale 8 2 ) We measure the relative KV cache and
Table 2: Word-perplexity of different KV cache quantization precision pairs with the huggingface transformers KIVI- HQQ implementation on the wikitext dataset and Im-eval- harness.
attention output errors and the absolute attention score er- as e mean [K'-K'| |
ey
IV
mean
V
Mede:
1394
K3V
KIN
K3V
KIV
KIVS
1C791
KVg.
ea mean((a' a') a')), and
ey mean |o' [''-o' o
where
Liare: Mestaite
555
11.82
13392
the attention score with dequantized key cache a!
softmax gik' VD
Owee23Mall
and the attention output with dequan-
Owes2 Ieame
648
6.87
1,99
681
6.0
1.30
16.27
38.2
Qwee2 Iearus
5.05
0.33
0.20
7.0
tized KV cache of a V.
4.2. Sensitivity to Quantization Mode and Precision
4. Observation
KV cache quantization errors strongly correlate with the quantization mode and precision as in Table 4 In terms of relative key error eg, the per-channel-asym quantization mode consistently outperforms the per-token-asym counter- part under the same precision for key cache, because key cache has strong channel-wis outliers (Liu al., 2024e Hooper al.. 2024), more detailed experiment results can Therefore, for specific KV cache,
4.1. Error Accumulation
Due to the sequential nature of LLMs along both the model layer and token sequence dimensions, the previous layer output with KV cache quantization errors is the input of the current layer and the previous step model output token with errors is the input of the input and subsequent transformer layers. Therefore, KV cache quantization leads to two- dimensional error accumulation. The error in the I-th layer and i-th token e depends on previous ~ -- layers and 1 ~ i - steps, as defined in
be found in Table 9 the quantization mode modification may lead to the shift of importance of key and value to attention output errors. As shown Table 4 the Pareto-optimal intra-layer KV cache quantization precision pairs significantly differ be- tween these two modes. Therefore, the KV cache preci- sion pairs need be adapted quantization modes. More detailed experimental settings and results available in Appendix B and D.I due to space limitations.
d = fe(e) ,ei- eli
(3)
The KV cache quantization error of a single token and layer may be ignorable. However, the error accumulation over the whole model and long context length noticeable and may lead to token flipping and generation error, which is similar to model quantization (Lee et al. 2024a). The error accu- mulation caused by low-precision KV cache quantization is a general problem in domain knowledge QA, AI Gener- ated Contents (AIGC), coding, and mathematical reasoning tasks, which may lead to critical factual errors and loss of instruction following ability.
4.3. Why Key Cache Is Generally More Important?
We discover the diverse model and transformer layer sensi- tivity to KV cache quantization mode and pairs, which is mainly caused by attention distribution shift as in Figure 2 In this section, we thus analyze the reason why key cache is normally more important than value cache from both the empirical and theoretical perspectives.
Accumulated emors and intermediate token flipping ren- der the entire mathematical and logical reasoning process ineffective, resulting in unnecessary computational over- head in long-come contect reasoning models like OpenAI ol. As demonstrated in Table D. KIVI-4 has exactly the same re- sponse with half-precision KV cache an example from the GSM8K 15-shot dataset, while the first three generated sentences with low-precision KIVI-2 are highly similar to original generation except for minor differences. Addition- ally, there is a small token flipping from to +, which leads to the arithmetic operation eтror in the fourth sen- tence with KIVI-2. The wrong 20 + + 4 = 28 instead of 20 20-4-4= = 12 finally leads to the arithmetic error 28/20 = =28/20=14> 14% and the completely wrong final answer 14.
(a) K8 ea 1-8x10 (b) K4 ea 2.5x10 (c) K2 ea 1.2x10
Figure 3: Layer-wise attention score error of per-token- asym KV cache quantization with simulated offline quanti- zation and dequantization (without eгror accumulation) of the 1-mma-..--BB--Inseei model and the first 20 prompts in the zero-shot GSM8K dataset.
Intermediate Attention Errors. Following the settings in Table we visualize the simulated layer-wise attention score errors of Llama-3.1 1.ma-3.1-8BB 3.1-8B- Instruct with the per-token-

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
Table 3: Layer-wise relative attention output error (eo) of per-token-asye asym KV Quant. method on Alama-3.1-88 Instruct on the first 20 prompts from the GSM8K dataset.
table exceptions: Qwen2.5- two LLMs are sensitive even int4 key cache quantization, indicating lower tolerance for precision reduction. Based on these findings, we conclude that the key cache plays a more critical role than the value cache during quantization. This characteristic can be leveraged to optimize memory usage while maintaining model effectiveness.
Baahh-7BJInnstruee These
KOVS
Rebatte Assentinn Ooaper Emmorte [
D414
0.00
030
0.83
0382
1092
asym KV cache quantization mode in Figure More results
of diverse L.MS and datasets are available in Appendix [F]
4.4. Correlation of KV Quantization Errors and Attention Patterns
Decreasing the key cache quantization precision from 8-bit to 4-bit and from 4-bit to 2-bit leads to 13.9x and x.6x average attention score error degradation in Figure 3 re- spectively. It may result in attention distribution shift in the
As shown in Figure heads with high KV cache quantiza- tion errors typically exhibit non-sparse attention pattems,. The sparsity patterns of the attention heads correlated with the head-wise and layer-wise sensitivity to KV cache quantization, Highly sparse streaming heads are generally more robust to KV cache quantization than retrieval heads. The proof of Lemma is available in Appendix A
token levels of specific sensitive heads as in Figure 2 and thus degrade the final accuracy. A similar phenomenon oc- curs in the final output token probability when implementing KV cache eviction (Adnan et al. 2024).
As shown in Table 3 the relative attention output errors of high-precision key cache quantization with the same overall memory usage e.g. K4V2 is significantly lower than the high-precision value quantization e.g. K2V4, which empirically validates that key cache is more important than value cache during KV cache quantization of intermediate transformer layers. More detailed experiment setting and results can be found in Figure 13 and 14
Lemma 1. Only attention heads with sparse and concen- trated putterns demonstrate consistent robustness to low- precision KV cache quantization.
The optimal strategy to mitigate attention shift and enhance accuracy is to increase key quantization precision, specif- ically reducing 9 in highly sensitive layers. This ap- proach is recommended when dynamic fine- mee-grained token or page-level KV cache quantization for better accuracy is not feasible, as such methods remain challenging to implement on existing hardware.
Final Generation Errors. We also study the final LLM generation performance with error accumulation enabled during decoding. Low-precision KV cache in all intermedi- ate layers are quantized with the same KV precision pairs such as K8V4 and K4V2. We utilize the KIVI implemen- tation with the HQQ backend in huggingface transformers v4.46.2 (Wolf et al. 2020), which supports popular LLMMs with different scales and proposes, and measure the word- perplexity with 2 As shown in Table 2 both KV8 and K8V4 quantization demonstrate similar perplexity levels across all models. Sim- ilarly, KV4 and K4V2 quantization demonstrate comparable patterns. These results suggest that we can achieve equiva- lent performance using either 6-bit (K8V4) 3-bit (K4V2) KV cache quantization while maintaining accuracy levels similar to those of KV8 or KV4 quantization, respectively. In contrast, K4V8 and K2V4 quantizations lead to substan- tial increases in perplexity scores, resulting in significant degradation of generation quality. A noticeable decline in generation quality occurs when reducing the precision of the key cache rather than the value cache. The 5-bit K8V2 precision pair achieves performance equal to or better than the higher 6-bit K4V8 precision pair while achieving an additional 12.5% reduction in memory usage. These LLMs demonstrate varying levels of sensitivity to KV cache quantization. Most models experience significant perplexity increases only with int2 key cache quantization, with two no-
8:25
calluation-harness (Gao al. 2024) in
1:00
(a) Layer-2 streaming head
(b) Layer- 13 retrieval head
Figure 4: Token-level attention distribution shift with the per-token-asym key cache quantization(LLlmaa_. Instruct, GSM8k)
4.5. Layer- Wise Sensitivity to KV Cache Quantization
According to the layer-wise attention score and relative out- put errors of different prompts and KV cache quantization precision pairs of Llama-3.1-8B -ns-trutt in Figure 3 and 13 transformer layers sensitive to KV cache quantization remain consistent across different input prompts. The ob- served shifts in layer-wise error distribution primarily stem from variations in key cache quantization precision. Both Qwen2.5-77 A.ve2..57B--. --n-tttntt and Mistral- Meatral-B-IInseeee IB-Instructt-O.. exhibit similar behavioral patterns in this respect. Further analysis results can be found in Appendix F We can thus conclude
5

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
that layer-wise sensitivity KV cache quantization is an inherent characteristic of LLMs.
fm (P) 2P(P) captures the average equivalent quanti- 21.
zation bits of all KV cache, [a(P) AALLM (Lan(KVinn Vhats) ALLM(KVV) V/)) measures the final LLM accuracy loss with the KV precision as P compared with LLM inference us- ing 16-bit half precision KV cache. For instance, we can limit the average KV cache quantization precision to 2.5-bit, while optimizing the equivalent quantization precision and inference accuracy.
KV cache quantization errors are accumulated over both the model layer and generation sequence dimensions, and the sensitive layer will further amplify errors and lead to dramatic model performance degradation. We can perform an offline search to identify the optimal coarse-grained KV cache quantization configuration, determining the most ef- fective precision pairs for each layer, particularly for sensi- tive layers, to achieve balance between memory reduction and generation efficiency without incurring any overhead during online inference.
5.2. Framework
To reduce the overhead of online fine-grained KV cache mixed-precision quantization tuning, we propose offline calibration of the optimal coarse-grained KV cache quan- tization precision pairs for each layer or head using multi- objective optimization algorithms (Akiba et al. 2019 Zhang & Li 2007). These pre-calibrated settings are then directly applied during online quantization. The efficiency of of- fline calibration is crucial for practical applications due to the large combinatorial search space of KV cache quanti- zation pairs across multiple transformer layers. Therefore, as demonstrated in Figure we propose the intra-layer and inter-layer search space pruning algorithms to accelerate the search process while preserving optimization opportunities. After the efficient preprocessing, the final LLM inference accuracy is utilized to search the Pareto optimal layer-wise KV precision pairs P capturing complex dependencies of the nonlinear error accumulation.
5. Method
KVTuner is an adaptive tuning framework for hardware- friendly pixed-precision KV cache quantization. It opti- mizes layer-wise KV precision pairs by considering their inherent sensitivity properties, aiming to achieve a better trade-off between inference efficiency and model accuracy.
Instead of making online decisions about fine- aine-graind token or page-level KV cache quantization precision for improved model accuracy, we conduct offline search to identify the Pareto-optimal quantization precision settings for coarse- grained KV cache in each transformer layer using multi- objective optimization algorithms. Here, we refer to the entire low-bit KV cache being quantized with a specific pre- cision pair, such as K8V4 or K4V2. This approach ensures that no additional overhead is introduced during dynamic quantization and online inference. Due to the flexibility introduced by layer-wise KV cache quantization precision tuning, V is able to accommodate more hardware and accuracy constraints of different deployed LLMs com- pared to uniform 8-bit or even lower precision quantization. Moreover, KVTunnr accelerates LLM inference and reduces memory footprint, while still maintaining lossless or slightly lossy final model generation.
5.3. Automatic Layer-Wise KV Cache Quantization Precision Pair Search
As analyzed in Section 4.2 and 4.3 the model-wise and layer-wise sensitivity to KV cache quantization mode and precision is the inherent model property and is independent of the input prompts. Therefore, we can search for the optimal layer-wise KV cache quantization precision pairs offline to eliminate the additional online decision-making overhead with high generalization. If the candidate layer- wise KV precision pairs are {2, 4, 8} {2, 8). then the number of possible combinations is 94, where the L is the number of transformer layers. For example, the Llama- 3. Instructt model with 32 layers has about 3.4 x 1030 which is intractable. Therefore, we design the following two-level search space pruning algorithm to reduce P from S to Sp where S, is the pruned candidate set in group and G is the number of clustered layer groups.
5.1. Problem Formulation
The offline layer-wise KV precision pair tuning problem can be formulated as a discrete combinatorial optimization task, considering hardware limitations and accuracy loss constraints. It can be solved using multi-objective optimiza- tion algorithms. We aim to minimize the quantized KV cache memory usage across all transformer layers while minimizing the final model accuracy loss, subject to the maximum M memory and A accuracy loss constraints:
INTRA-LAYER KV CACHE QUANTIZATION PRECISION PAIR PRUNING
min (Jm (P)) s.t. Jm (P) S M, < AA, (4) P
KV cache quantization errors in each layer accumulate across both the model layers and generation token dimen- sions. Therefore, we must control the layer-wise error by pruning KV cache quantization pairs to limit the final model
where the search space P E 5t is the KV cache precision pairs in L layers. The laser-wise search space S is defined
as the KV cache precision pair (P). P!) in the 1-th layer.
6

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
ervor. For all candidate KV cache quantization pairs in each layer, we prune those that are not part of the Pareto frontier, considering both the equivalent KV cache quantization pre- cision and the relative attention output errors. For example, the precision pairs KV8, K8V4, KV4, K4V2, and KV2 are in
Table 4: Intra-layer KV cache quantization precision pair pruning results of special transformer layers. The pruned Pareto efficient KV cache precision pairs in most layers are (KV8, K8V4, KV4, K4V2, KV2), so we omit them in the table. Value is always quantized with the per-token-asym mode. G1 of Destral7B-IInttruct-kk. is 2~44, 6, 7~10, 14, 18, 27, and 29. G2 of Qwen2.5-322 --t.e...533BnBnstrrreu Instruct is 5 ~ 10, 12, 14, 16, 18 ~ 21, 23, 26 ~ 28, and 32.
Pareto efficient for most layers in 3aama-3.1-8B-Ima Figure [3] except for the 0-th layer, where K4V8 results in smaller errors than K8V4.
INTER-LAYER CLUSTERING
Medd name
Kry quart, made perkokre-aay
KV8. KV4,
KV coche presiese pars
Lager ick
Although the above intra-layer prunning already significantly reduces the search space to Sp L such as 532 = 2.3 x 1022 in Llama-3.1-8B-Inntiee -8sttnttttt it is still too computationally costly for searching. Therefore, we further propose the inter-layer clustering algorithm based on relative attention output errors and the pruned candidate KV quantization pairs to Sy G such as 56 1 15625. The initial step involves partitioning layers based on distinct candidate sets of pruned KV cache quantization precision pairs. These candidate sets serve as indicators of how individual layers respond differently to specific KV cache quantization precision configurations. The subsequent step involves clustering layers that share the same candidate set, using quantization sensitivity as the clustering metric. This sensitivity is quantified with the relative attention output errors produced by the pruned precision pairs.
Liama 3.1-88 Instrnct
32
percharnd--ayn
KV8. KV4,
KV8. KV4,
3,
ps: Kilce doytE
KNA,
Misual 7B Instroct-00 30.3
32
prichaneel-ony
KVA,
pe. kikze dayti
Q 36
per-hhrndd-aay
KVA, K4V2, KV2
1.
K4V2,
33
KVG KZVA,
KV8 KAVS, KVA, KAVE, KV1
Nowe
Qwennt:5 fill-tmmmme
percharedl-ony
KVX KAVS, KVA, KIVA KVI
KVK, RAV8, KVA, KAVI, KV2
GLATS
Nove KVX, KAVS, KV4, K2V4 KV2
Qwen2.55-226 220 -matrec
64
per per-channcl cayes
KV8 RAVS, KV4, S4V2, XV2
W. 62
KVX, K3V4, KV4, K3V4, XV2
6.1. Pareto-Optimal KV Cache Precision Pair Search
KIVI. The mixed precision KIVI quantization mode can maintain high accuracy. As shown in Figure 5a KVTuner with KIVI effectively maintains Llama-3. 3.1 1-BB-BB-- 1--8--IInstttt Instruct per- formance while reducing the equivalent quantization preci- sion 3.06-bit. In addition, KVTuner also finds out four settings including lower-precision 4.91-bit in the Pareto frontier whose memory usage and accuracy are better than KV8. Most sampled settings are close to the Pareto fron- tier, indicating that Llama-3. 1-0ma-..--B-IInstruee is more robust to low-precision KV quantization. These demonstrate that KVTuner increases the flexibility of KV cache quantization and can achieve lower precision and even better precision than uniform KV precision.
CALIBRATION DATASET DESIGN
To effectively evaluate different quantization settings, we develop an approach that amplifies KV cache quantization error accumulation and distinguishes the performance of KV precision pairs during the calibration process. This approach utilizes dequantized KV cache for self-attention computation during the prefilling stage, enabling error ac- cumulation across model layers. Furthermore, we utilize long-context generation and challenging calibration datasets such as mathematical reasoning. In these tasks, minor errors propagating in decoding steps may result in intermediate generation token flipping and substantial mistakes in final answers as demonstrated by Table I
Per-token-asym. According to Figure 5b when using the per-token-asym quantization mode on the sensitive Qwen22.577B-Instrue model, the Pareto frontier identified by KVTuner consistently outperforms uniform precision quantization. Especially, KVTuner can achieve KV8 accu- racy with the equivalent 3.92-bit KV precision, while the uniform KV4 accuracy significantly degrades to around 0%. Therefore, even leveraging the simple and commonly used per-token-asym mode (Lin et al., 2024b Sheng et al.) 2023), KVTuner can reduce the memory footprint with the main- tained accuracy of models with high knowledge density.
6. Experimental Results
The detailed experimental settings are available in Section
The lntra-layer and inter-layer KV precision pairs prun- ing results of various LLMs are available in Appendix [.I] The proposed pruning algorithm can significantly reduce the search space to Sp G and speedup convergence of MOO search. The final model accuracy on mathematical reason- ing datasets and the throughput improvement validate the effectiveness of K Kuuner.
6.2. Mathematical and Scientific Reasoning Accuracy
Apart from the in-context few-shot GSM8K datasets, we also utilize them as the internal reasoning steps in multi- turn way to imitate OpenAl o1 like reasoning systems in Table KIVI-2 and KIVI-4 result in dramatic accuracy loss in Qwen2.5- 7B)- due to their high sensitivity
7

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
Table 6: Scientific reasoning accuracy comparison of dif- ferent KV cache precision settings with the per-token-asyee KV quantization mode on the GPQA Extended dataset.
Procuce
S-ker
OPON Baseded
10-dae
30
Awerge
Prection
5-due
CPQA Foredst
In-dve
20-h
BF16
0,3993
0L3114
0,265
5.995
55.5
143550
0,275
012705
KXB
3-222
112333
0.000
KVK
10308
107781
(a) Llama-3.1-8B-Innse with KIVI
(b) Qwenn.5-7B-Inssee with per-token-asym
KV2
0.996
02299
0.2003
0.2106
0.2344
KVRmaetS 45
03187
103277
0.318
53159
KV
UL3064
0.380M
012012
0.911
KV 59
03325
03335
0 0.309
0,3622
C5.7
03360
0.3857
0.3393
0.303
Qvee25.78- Ieeetrr
Figure 5: Pareto frontier on the first 200 GSM8K 4-shot prompts. Red points indicates the accuracy of 9 uniform layer-wise KV cache precision pairs including KV8, K8V4, K4V8, KV4, K4V2, and K2V4. For Qwen2.5-77-Instruut,, can easily see that K2V8, KV4, and other lower pre- cision pairs lose the capability of mathematical reasoning, obtaining around 0% accuracy. However, KVTuner still maintain nearly lossless overall 4-bit KV cache quantiza- tion.
ONN
K13
03327
0332
0.264
112711
0.266G
5.3000
K94
0U7751
0.859
KV2
01671
160905
0.100
KV2
12226
0.191
0.094
02001
K
0.383
00333
0.376
0.101
03333
0.000
10117
0.306
KV A3
0,249)
(L319)
0.265
0.291
KVitaer44
0.333
0,522
0.354
to low-precision KV quantization. KVTuner with KIVI can nearly losslessly quantizate KV cache to 3it2-bit, bit, and 5.96-bit of the three models, respectively, further re- ducing the memory footprint compared with KIVI-4 and KIVI-8. In addition, we find out interesting observation: KVTuner enables longer context and lower KV precision for better CoT and multi-turn mathematical reasoning ac- curacy than short-context and original BF16 precision KV. Most LLMs benefit from longer CoT and KVTuner enables nearly lossless lower-precision KV quantization. We ob- serve that KVTuner significantly reduces the performance gap between the per-token-asym and KIVI quantization modes.
Table 5: Mathematical reasoning accuracy comparison of different KV cache precision settings with the KIVI and per- token-asym quant. mode on the GSM8K few-shot CoT and CoT multiturn dataset. We highlight the average scores with significant accuracy degradation in red and those with moderate accuracy degradation in orange. Notably, for the Qwen2.5-3B -nnnstruct model using KIVI quantization mode, all configurations within the 4-bit to 6-bit equivalent preci- sion range exhibit lower accuracy on the calibration dataset compared to a configuration with an equivalent precision of bit As result, we choose this bit configuration as the highest-accuracy representative for cases where the equivalent precision is constrained to <6-bit.
We extend our evaluation to the GPQA dataset with few- shot CoTs, as detailed in Table 6 KVTuner successfully enables lower than 4-bit, such as 3.59-bit, KV cache quanti- zation with minimal performance degradation across various models. These results demonstrate the effectiveness of our method in maintaining high mathematical reasoning accu- racy while significantly reducing memory usage.
Quomi mathal
Protisio
Fousshir CoT
S-shet
16-boi
4shit
Fow-shat to multiumn
S-ibo
16-dbo
Avrrage
6.3. Long Context Generation Accuracy
Instruct
BF16
BF
0:7615
0,7741
0.58%
0815
0.39
08333
03838
KV8
0.7635
0,7110
0.7906
03340
038279
03029
We compare KVTuner on the sensitive 9wen2.5--B-Inntrree model with the baselines KIVI-8, KIVI-4, our proposed vari- ant KIVI-K8V4, and per-token-asym ones in the 20 Long-
Kv4
0.7240
0.746
0.735
01211
0.180
0.8007
0.766
Per-ullet-
XV2
0.124
0.199
0.020
KV Tunee
0.504
KVTane-C3.50
0.5210
03316
07460
0.766
KIVES
037733
0.756
03317
03317
KIVI-4
07718
Bench datasets (Bai et al. 2022).. The averaged scores are available in Table / KVTunnr pushes KV cache quantiza- tion for the nearly Iossless long context generation to 3.92- bit, outperforming the uniform KV precision. KVTuner
SIVE
2
0.073
06080
0.5929
0.6699
0.6543
0.6687
RVTine
07506
0.0055
03173
0,3755
KVTons-C325
0.7433
0.7566
0.7604
08362
0.8279
0.225
Qwen2.5-10
BF16
BF16
020000
0.490
0.70333
05679
0,003
0,690
01284
KVS
0.5959
0.653
01568
(0600)
0.333
KV4
0.1845
0.121
02312
02229
0.3616
02264
02205
with both KIVI and per-token-asym quantization methods achieve high accuracy and KV compression rates simultane- ously.
Pertooen-arym
KV2
0.099
00121
00100
000000
0.091
00129
0000
KC Gane 43.66
0,0015
0.064
00015
0.914
03133
01347
KVTune: C4.00
05116
06482
0.6622
05815
0.118
06422
06274
KTVI-
0.594
0A519
0164
01991
01355
063
KIVI-4
06156
06550
07766
05532
06414
06332
KIVI
KIVI-2
00546
00576
00615
0007
09%)
00856
KV
KVTuner C3.17
0.005
0.99%
05512
0,5891
0.405
06219
6.4. Throughput
Qwen2577
BF
0,5009
0.9287
0,218
077081
0.739
0,7544
07755
KVS
03998
03301
0.7584
0.354
02317
We measure the maximum throughput and the correspond-
KV4
0.006
00121
00121
0.00
0.05
0.0061
0003
Per toilee usy
KV2
00088
00099
0.0776
8008
0.0106
000105
80330
ing batch size under specific input prompt length with the implementation of the KIVI GPU kernel, which supports Llama series models. We follow the same settings and defi-
KV 0500
078855
03301
08205
01814
0.7455
0.666
07700
KVTanes C4.00
0.7877
0.1112
0.7726
0.929
0.731
0.2407
0.7599
KIVISS
0.423
03351
6K10
11
01306
017354
0.758
07751
RIVL-4
0.0735
0.117
11554
80687
0.0055
0.1463
0.1043
KIVI
RV Tiner 15
KIVL-2
00079
0.305
0/5017
01086
00030
0.055
01033
0.711
011326
nitions of KIVI. Throughput is defined as the the number of tokens generated per second (measured end-to-end, includ-
gVTune C3.92
0.5059
0.6644
0.658
0.558
0.6156
0.8035
0.6000
8

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
Table 7: Accuracy comparison between offline searched layer-wise KV cache precision using KVTuner in Table 5 and [ and uniform KV precision settings of the sensitive Qwen2.5-7B -B--ssuuutt on 20 LongBench long context gener- ation benchmarks.
26.3
KIVI
K64
BF16
KV8
K8V4
KV4
KVTuner-C5.96
KVTuner-C3.92
0.7956
0,7992
0.8001
0.7723
0,7956
0,7903
Per-token-asym
BF16
KV8
K8V4
KV4
CV5mmer-C55.
KVTuner-CA.0
0.7956
0.7971
0.7953
0.6343
0.8005
0.7960
Figure 6: Pareto frontier of Llama-3. 1-B--B...--B---- 1-88B-Inssttutt with the ver-token-asym KV quantization mode and without the proposed two-stage search space pruning on the first 200 GSM8K 4-shot prompts.
ing quantization/dequantiztiion overhead).
The layer-wise KV cache precision tuning in KVTuner are completely offline and no online overhead for precision se- lection is introduced. The model-level efficiency reflects the overall effects of layer-wise efficiency of all KV cache precision pairs. The memory movement cost from CPUs to GPUs and from GPU HBM to GPU cache linearly increases with the KV cache size in most case and attention is nor- mally memory bounded. We report the total model-level throughput comparison of 1aama-3.1-8--Isiiiii searched configuration in Table as below. Compared with KIVI-KV8, KVTuner-C3322 25 can improve decoding through- put by 16. 16.79%~~21.25% More efficient KV dequantization and attention kernels proposed in Qserve (Lin et al. 2024b) and TurboAAtention (Kang et al., (2244)) may further en- hance the throughout benefit of KVTunner than INT8 KV baselines.
with larger quantization errors. Reducing the quan- tization precision of the key for crucial group of layers can significantly degrade the performance. For instance, in Llama-3. Instuuet the layer group
[8 ~ 11, 14 ~ 17, 20, 30] is particularly sensitive to the reduction of the key precision, and if the pre- cision of the key is reduced from 4-bit to 2-bit, the performance would drop from 0.67 to 0.495.
using the
6.6. Ablation Studies
According to Figure 6 when using the per-token-asyne quantization mode on the 1--ma-3.--8B-Instie model, the search results deteriorate significantly if the proposed intra- layer and inter-layer search space pruning algorithms are not applied. In comparison with the counterpart with search space pruning as pre- pee-procsssing in Figure 9a] this highlights search space pruning is helpful for MOO search conver- gence and maintaining quantization performance.
Table 8: Throughput comparison between offline searched wise KV cache precision using KVTuner in Table S and uniform KV precision settings with KIVI of Llama-3. 1- 8B- Instruct.
B8
impunt en
KVKhhadline) 3836
K8V4
KV4
K4V2
KVTunesC4499
KVTuner433.25 4240
125
4193
4567
4652
311
100
1205
125 1304
12W maa
3
1024
349
397
632
645
600 waas
641 andes
7. Conclusion
6.5. Detailed Analysis
KVTuner enables efficient and adaptive layer-wise mixed- precision KV cache quantization via sensitivity-aware op- timization techniques. It systematically reduces KV cache quantization errors by prioritizing key cache precision while balancing memory efficiency and inference accuracy. Exper- imental results demonstrate that KVTunne achieves nearly lossless compression at 3.25-bit for Llama- -nnstruttt and 4-bit for sensitive Qwen2.5-7B-Insttee Instrruct.. KYTunnr also demonstrates that employing longer CoTs with lower and mixed precision KV cache quantization yields superior per- formance compared shorter CoTs utilizing higher pre- cision KV cache. This improvement is evident in both memory efficiency and accuracy, particularly in the con- text of mathematical reasoning tasks. KVTuner also greatly narrows the performance difference between the simple per- token-asym and accurate KIVI quantization modes, even when using overall similar low-precision settings.
By analyzing the detailed configurations in the Pareto fron- tier identified for Llama-3.1-88 I-Sttutst we observe that:
In most cases, all layer groups adopt a quantization configuration where the precision of the key is higher than the precision of the value. This supports our ear- lier observation from uniform quantization that the key plays a more critical role in quantization.
In other cases, in certain specialized layer groups, the value is set a higher precision than the key for certain specialized layer groups. This aligns with the patterns identified in Table 4 which highlight specific layer groups may require higher precision for values.
KVTueer tends to allocate higher precision to groups

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
Impact Statement
view.net, 2024. URL /
This paper thoroughly studies the layer-wise sensitivity of transformers to KV cache quantization methods, which is the inherent property of LLMs. Low-precision KV cache quantization may lead to significantly token-level atten- tion distribution shift in heads with non-sparse and non- concentrated attention patterns. The attention head related property may also be applied to LLM weight and activa- tion quantization and other KV cache compression fields. The proposed automatic KV cache precision pairs tuning algorithm makes inference acceleration of LLMs with low- precision KV cache possible, which can help reduce the deployment cost and carbon footprint. Low-precision KV cache quantization with ignorable LLM accuracy loss is an important direction to reduce the KV cache memory usage and cost in online inference, KV cache offloading (Sheng et al. 2023 Zhang et al. 2024a), storage (Jin et al. 2024). transferring (Liu al., 2024d), and more LLM inference related applications.
Cobbe, K., Kosaraju. V., Bavarian, M.. Chen, M., Jun, H., Kaiser, L.., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training ver- ifiers to solve math word problems. ArXiv preprint, 1lss2100.14166 2021. abs/2110.144
Contributors, L. Lmdeploy: A toolkit for compressing, deploying, and serving Ilm. htttps://github.com InternIM/ http://dmmpplyy 2023.
Dao, T. Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness.
Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Neural Infor- mation Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL httt:://paprrs.nibaast
Systems 35: Annual Conference
References
paper_files/pet haeer__Hlle/pppee/22///aaaa 67 77446722222220000027700000011663
Adnan, M., Arunkumar, A., Jain, G., Nair, P., Solovey- chik, I., and Kamath, P. Keyformer: Kv cache reduction through key tokens selection for efficient generative in- ference. Proceedings of Machine Learning and Systems, 6:1 12114 2024.
Abstract-Confferecce.nt
DeepSeek, 2024.
URL https://api
fbcs.deepsee..com/guiiooe kv_ccchee
Dong. S., Cheng, W., Qin, J., and Wang. W. Qaq: Quality adaptive quantization for Ilm kv cache. ArXiv preprint, abs/2403.04643, 2024. URL fittps://arxiv.ort abs abs/2403.04643.
Akiba, T. Sano, S., Yanase, T., Ohta, T., and Koyama, M. Optuna: A next- gext-generation hyperparameter optimization framework. In Teredesai, A., Kumar, V., Li, Y., Rosales, R., Terzi, E., and Karypis, G. (eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, KDD 2019, Anchor- age, AK, AK, USA, August 4-8, 2019, pp. 2623-2631. ACM, 2019. doi: 10. 00.11445/329250033337000 URL https:
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783, 2024. URL httpss:::: /ttp:///rxvv/////////// aba/2407.21783
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z. Du, Z., Liu, X., Zeng, A., Hou, L.,, Dong, Y., Tang, J., and Li, LongBench: A bilingual, mul- titask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pp. 3119-3137, Bangkok, Thailand, Au- gust 2024. Association for Computational Linguistics. URL https: 1ong.172
Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., et al. Layer skip: Enabling early exit inference and self-speculative decoding. ArXiv preprint, abs/2404.167100 2024. URL hhttps://arxiv.orr abs Abs/2404.16710
Ester, M., Kriegel, H.PP., Sander, J., Xu, X., et al. A density- based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pp. 226-231, 1996.
doi: 10. 7/Aclannhholoyy.org/2222..
Cai, T., Li, Y., Geng, 7., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple LLM inference accelera- tion framework with multiple decoding heads. In Forty- first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenRe-
Frantar, E., Ashkboos, S., Hoefler, T. and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. ArXiv preprint, Abs/2210.17323. 2022. URL https: hupps://azze http:///xxxvv//////// abs/2210.17323
10

---

KVTuner: Sensitivity-Aware Layer- Wiser Mixed-Precision KV Cache Quantization
Gao, L.,, Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L.,, Hsu, J., Le Noac"h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J. M., and Zhang, C. (eds.), Advances in Neural Infor- mation Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15,
J., Reynolds, L., Schoelkopf, H.. Skowron, A., Sutawika, I.,, Tang, F.,, Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evalua-
2024, 2024. URL http: http:://ppprr..ipps.co
tion, 2024, URL 12608602
paper paper__fleees/paeee/22//////
72550008855343331004466222222993 Abstract--Confeeemc..h
Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive KV cache compression for llms. In The Twelfth In- ternational Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Openke- view.net, 2024. URL forum? d=rrrrdDDuurrr
Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T. Liu, J., Lv, C., Zhang, Y., Lei, J., Fu, Y., Sun, M., and He, J. C-eval: A multi-level multi multi-discipline chinese evaluation suite for foundation models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M.. and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neu- ral Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL lattp://papers.nipscom
Gloeckle, F., Idrissi, B. Y.. Rozière, B., Lopez-Paz, D., and Synnaeve, G. Better & faster large language models via multi-token prediction. In Forty-first In- ternational Conference on Machine Leurning, ICML. 2024, Vienna, Austria, July 21-27, 2024. OpenRe- view.net, 2024. URL https:://ppereeeeit..t Barummid-pEWAae 1102
paper /aee_f/lels//pper/22///asss 66ceB84bee9666d22225555562222223 Abstract-Datassets_ _ww_aannd___eaaeeeeeee rks.ttmkkshhttm
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. ArXiv preprint, abs/2310.06822 2023. URL htttps://arxiiite abs/2310.06825
He, Y., Chen, F., Liu, J., Shao, W., Zhou, H., Zhang, K., and Zhuang, B. Zipvl: Efficient large Eision-language models with dynamic token sparsification and kv cache compression. ArXiv preprint, abs/2410.08584 2024a. URL httph:tt/://// www.aaiwwvov..a.a. wttpxii.orr/abs/2210000 abtp://..///22./......
Jin, C., Zhang. Z., Jiang, X., Liu, F., Liu, X., Liu, X., and Jin, X. Ragcache: Efficient knowledge caching for retrieval-augmenned generation. ArXiv preprint, abs/2404.12457, 2024. URL httttt://arxiiv.rrr abs/2404.12557
He, Y., Zhang, L.. Wu, W., Liu, J., Zhou, H., and Zhuang, B. Zipcache: Accurate and efficient KV cache quantization with salient token identification. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomezak, M., and Zhang, C. (eds.), Advances in Neural Infor- mation Processing Systems 38: Annual Conference Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024b. URL htttp://paprrs.iiiii
Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. Trivi- aQA: A large scale distantly supervised challenge dataset for reading comprehension. In Barzilay, R. and Kan, M.-Y. (eds.), Proceedings of the 55th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, 2017. Association for Computational Linguis- tics. doi: 10.18653/v1/P17-1144) URL https:// aclantholoog..or/P11-111
paper (e5773116ae0811574443666633 Abstract-Conferrenee
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M.. Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021. URL httpss://penneeveee.eettt Forum?id=d7ke jm 3Gm)
Kang, H., Bharadwaj, S., Hensman, J., Krishna, T., Ruhle, V., and Rajmohan, S. Turboattention: Efficient attention approximation for high throughputs 1lms. arXiv preprint arXhv:4412.08585, 2024a.
Kang. H., Zhang, Q. Kundu, S., Jeong, G., Liu, Z., Kr- ishna, T., and Zhao, T. Gear: An efficient ky cache compression recipefor noar-lsssless generative inference of 11m. ArXiv preprint, abx/2403.05521 2024b. URL
Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kyquant: Towards 10 million context length LLM inference with KV cache quantization. In Globersons, A.,

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 61 6166626 2023.
compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024a.
Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measur- ing how models mimic human falsehoods. In Mure- san, S., Nakov, P., and Villavicencio, A. (eds.), Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18663//1//0222..00
Lai, G., Xie, Q., Liu, H., Yang, Y., and Hovy, E. RACE: Large-scale ReAding comprehension dataset from aminations. In Palmer, M., Hwa, R., and Riedel, S. (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, Copenhagen, Denmark. 2017. Association for Compu- tational Linguisties. doi: 10.18665//1/DD17--0333 URL
long. 229. URL 2022.ac1 long.229
Lin, Y. Tang. H., Yang, S., Zhang, Z., Xiao, G., Gan, C., and Han, S. Qserve: W4a8kv4 quantization and sys- tem co-design for efficient Ilm serving. ArXiv preprint, abs/2405.04533 2024b. httpssttt:///axiiie abs 0Ds/2405.04532)
https: /aclaaaaaa http:///...annooooooooorr///-/1.
Lee, J., Park, S., Hong, S., Kim, M., Chang, D., and Choi, J. Improving conversational abilities of quantized large language models via direct preference alignment. In Ku, L., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 11346 11364. Association for Computational Li Linguistics, 2024a. URL https: //6oi.02g/10.116633///22 -2tp:////2222....11111o1115512 10ng.6112
Liu, A., Feng. B., Xue, B., Wang. B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., al. Deepseek-v3 technical report. ArXiv preprint, abs/2412.11933 2024a. URL
Liu, A., Liu, J., Pan, Z., He, Y., Haffari, R., and Zhuang, B. Minicache: KV cache compression in depth dimension for large language models. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomezak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neu- ral Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024b. URL htttp::/papers.nipsscom
doi: 10. 10011666111222444111-006....
Lee, W., Lee, J., Seo, J., and Sim, J. [InfiniGen]:: Effi cient generative inference of large language models with dynamic {KV} cache management. In 18th USENIX Symposium Operating Systems Design and Implemen- tation (OSDI 24). pp. 172-172,20 2024b.
Sateer_fllee/pape////0 Edo 04070777000000338003333333333333 www.racct-Cooffeence...
Li, Y., Huang, Y. Yang, B., Venkitesh, B., Locatelli, A., Ye, H. Cai, T., Lewis. P., and Chen, D. Snapkv: LLM knows what you are looking for before generation. In Glober- sons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomezak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Con- ference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver. BC, Canada, December 10 15, 2024, 2024a. URL. http::::///pppeeeip..coo
Liu, R., Bai, H., Lin, H., Li, Y., Gao, H., Xu, 7.,, Hou, L., Yao, J., and Yuan, C. Intactkv: Improving large language model quantization by keeping pivot tokens intact. In Ku, L., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11- Association for Computational
paper fi wttp://eeeeeaapeee//////////////
16. 2024, pp. Linguistics. 2024c. doi: 10.11866//1//2022FFIIDDN ACL.460. /ttp:t/pp...//.oog11111166////// 2024 www.ddiinnnnnccc..5.00 ac1.460
23ad412222203e017222322222 9bbe- Abstract-Confezeee
Li, Y., Wei, F,, Zhang, C., and Zhang, H. EAGLE: speculative sampling requires rethinking feature uncer- tainty. In Forty-first International Conference on Ma- chine Learning, ICML 2024, Vienna. Austria, July 21- 27, 2024. OpenReview.net, 2024b. URL https:// spenreview.net/forze
Liu, Y., Li, H., Cheng, Y., Ray, S., Huang, Y., Zhang, Q. Du, K., Yao, J., Lu, S., Ananthanarayanan, G., et al. Cachegen: Ky cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, pp. 38 38-56, 2024d.
2 -d= NNN 7exyb4
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device llm
Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., Kyrillidis, A., and Shrivastava, A. Seissorhands: Exploiting the persistence of importance hypothesis
12

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
for LLM KV cache compression test time. In Oh, A., Naumann, TT, Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Infor- mation Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16.
and Scarlett, J. (eds.), International Conference on Ma- chine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 31094-31116. PMLR, 2023, BuRLhttps:://prceeeee ng..mm sheng23a.html
2023, 2023. URL httpp://papers.nipgp
paper files/ Flless/papee/2223/aaaee
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper,
a452a 4454a1666663eeee5fccc3334333 Abstract Ctn--
J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. ArXiv preprint, 0tbs/1999.000033 2019, URL
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braver- man, V., Chen, B., and Hu, X. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenRe- view.net, 2024c. URL https: http:///pnneeeeeeeee./. /t Eorum2dd-105722Rqg3
SoftAge-AI, 2024.
URL https://
http//aatttttt.aaaaaaaaaee Sargg//gface.o/ccaaattttt/eoeo.e multi-turn, datasst
huggingface
AI/
Stern, M., Shazeer, N., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models. In Bengio, S., Wallach, H. M., Larochelle, H., Grau- man, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 10107-10116, 2018. URL #ttos:/pppoeed.nss.nerii...
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant-IIm quantization with learned rotations. ArXiv preprint, abs/2405 16406, 2024f. URL hntps:
Vaaxiv..rrg/bbs/24555.2
Ma, X., Fang, G., and Wang, X. LIm-pruner: On the structural pruning of large language models. In Oh, A. Naumann, T. Globerson, A., Saenko, K., Hardt, M. and Levine, S. (eds.), Advances in Neural Infor- mation Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA. USA, December 10 16. 2023, 2023.
paper 2016 hashh/ 14127b91920 (112779118te652222cccc0f11..211 Abstract
Sun, H., Chang, L.-W., Bao, W., Zheng, S., Zheng, N., Liu, X., Dong, H., Chi, Y., and Chen, B. Shadowkv: Kv cache in shadows for high-throughput long-context I1m inference. ArXiv preprint, abs/2410.21465, 2024a. URL abs/2210.212gg0
URL hattp::/paperr.nips..t
paper fi wttp://leeeeaaeee/222////////a//
https://arxivi:
Abstract-Confezennn
Sun, H., Chen, Z., Yang. X., Tian, Y., and Chen, B. Tri- force: Lossless acceleration of long sequence generation with hierarchical speculative decoding. ArXiv preprint, abs/2404.11912, 2024b. /ttpss/sss///rxxvv... abs abs/2404.11912
OpenAL, 2024. 2024. URL/ api gp1-prromtt-ccchine
Qin, R. Li, Z. He, W. Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: A kveache-centric disag-
gregated architecture for Ilm serving. ArXiv preprint, abs/2407.00077, 2024. httttss://axiv..rr abs abs/2407.00077
Tang, H., Lin, Y., Lin, J., Han, Q., Ke, D.,, Hong, S., Yao, Y., and Wang, G. Razorattention: Efficient KV cache compression through retrieval heads. In The Thir- teenth International Conference on Learning Representa- tions. ICLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net, 2025. URL http:::tpp//.ccm. http:///ppeeeeeeeieeeeeee Forum?id=zkee forum?idd=ck2201004
Rein, D., Hou, B. L.,, Stickland, A. C., Petty, J., Pang. R. Y. Dirani, J. Michael, J., and Bowman, S. R. Gpqa: A graduate-level progle-proof q&a benchmark. ArXiv preprint, abs/231 2023. URL https:
/
haaxiv..rg//bss/311111
Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. QUEST: query-aware sparsity for efficient long-context LLM inference. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. Open 2024. URL htttst/:: /// p oatp:///fffoomm.oaao/////tx
Sheng, Y., Zheng, L.,, Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. Flexgen: High-throughput generative inference of large language models with a single GPU. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S.,
13

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed- KV Cache Quantization
Wan, Z., Wu, Z., Liu, C., Huang, J., Zhu, Z., Jin, P. Wang, L,, and Yuan, I.. LOOK-M: look-once optimization in KV cache for efficient multimodal long-conters inference. In Al-Onaizan, Y., Bansal, M., and Chen, Y. (eds.), Findings of the Associa- tion for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pp. 4065 4078. Association for Computational Linguis- tics, 2024. URL
inference with retrieval and streaming heads. In The Thir- teenth International Conference on Learning Representa- tions, ICLR 2025, Singapore, April 24-28. 2025. OpenRe- view.net, 2025. URL httLhhtpss//// httpas/////pnneeeeeeee.. forum?id=cFup 7xUm)
Yang. A., Yang. B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. ArXiv preprint, abs/2412.15115, 2024a. abs attp://///aa//211251/5 22t.//11222111115.
URL BtlLhttps://// a
emm 1p.235
Yang. J. Y., Kim, B., Bae, J., Kwon, B., Park, G., Yang. E., Kwon, S.J., and Lee, D. No token left behind: Reliable kv cache compression via importance mixed precision quantization. ArXiv preprint, aws//202..1009909 2024b. htttt::/:/aat http:///axxvv.oo/aa/a////sa
Wang, Z., Jin, B., Yu, Z., and Zhang. M. Model tells you where merge: Adaptive kv cache merging for Ilms long-contens tasks. ArXiv preprint, abs/2407.08455422 2024. URL https: http://wwwii.oora//////////
Wei, J., Wang, X., Schuurmans, D., Bosma, M.. Ichter, B., Xia, F., Chi, E., Le, Q.. and Zhou, D. Chain-of- thought prompting elicits reasoning in large language /
Yuan, J., Liu, H., Zhong, S., Chuang. Y., Li, S., Wang, G., Le, D., Jin, H., Chaudhary, V., Xu, Z., Liu, Z., and Hu, X. B. KV cache compression, but what must we give in return? A comprehensive benchmark of long context capable approaches. In Al-Onaizan, Y., Bansal, M., and Chen, Y. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pp. 4623-4648. Association for Computational Linguis- tics, 2024. URL hhtpp:: https//aaaaaatoooo...or. indiiinn emm p.266.
models, 2023. URL
2201.11900
Wolf, T., Debut, L., Sanh, V., Chaumond, J.. Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtow- icz, M., Davison, J. Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transform- ers: State-of-the-a natural language processing. In Liu, Q. and Schlangen, D. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations, pp. 38 45, Online, 2020. Association for Computational Linguistics. doi: 100118633/1122222eemmddeeeeme 6. URLhtttps hww.://hwwooooggoog/222..oo
Zeng, D., Du, N., Wang, T., Xu, Y., Lei, T. Chen, Z., and Cui, C. Learning to skip for language modeling. ArXiv preprint, abs/2311.15436, 2023. https:
(/arriv.org/abs//11.1555
Zhang, H., Ji, X., Chen, Y., Fu, F., Miao, X., Nie, X., Chen, W., a.d Cui, B. Pyeache: Product quantization- based kvcache for long context Ilm inference. ArXiv preprint, 2024a.
6emoo..66
Xiao, Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Ma- chine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 38087-38099. PMLR, 2023. URL httphhttt////occeeeeeee www.anl....s............... w.w.wl/ppp.eee.....m. /VT2e///222/// / xiac23c.htm
abs 0bs/2247.112220
Zhang, Q. and Li, H. Moea/d: A multiobjectivve evolutionary algorithm based on decomposition. IEFE Transactions on evolutionary computation, 1 11(6):712-73)) 2007
Zhang. Y., Du, Y., Luo, G., Zhong, Y., Zhang, Z., Liu, S., and Ji, R. Cam: Cache merging for memory-efficien 1lms inference. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b. URL https! //openreviee.net//ooommm LoTmmmppe11dTTmmmmm51010
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representutions, ICLR 2024, Vienna, Austria, May 7- 11. 2024. OpenReview.net, 2024. URL https:// openrevieew.eet/fooum id= http://ooruuniddd////SSS7/7
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song. Z., Tian, Y., Ré, C., Barrett, C. W., Wang, Z., and Chen, B. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural
Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y., and Han, S. Duoattention: Efficient long-context L.M
14

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
Information Processing Systems 36: Annual Conference
Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16,
2023, 2023. URL http:///apprss.nps..co
paper_ earilee/pappa popee__
6cceffaaaa15772577773ccccc222223 Abstraact-onffeeece..tt
Zhang, Z., Liu, S., Chen, R., Kailkhura, B., Chen, B., and Wang, A. Q-hitter: A better token oracle for efficient Ilm inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems, 6:381-394, 2024c.
Zhao, Y., Xie, Z., Liang, C., Zhuang, C., and Gu, J. Looka- head: An inference acceleration framework for large language model with lossless generation accuracy. In Baeza-Ya Bacza-Yates, R. and Bonchi, (eds.), Proceedings of the 30th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024, pp. 6344 6355. ACM, 2024. doi: 10. 10.1145/6637558.3661166 16144 URL https:://oi.oo. 10.1145/3637528367722
Zheng, L.,, Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao. S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C. W., and Sheng, Y. Sglang: Efficient execution of structured language model programs. In Globersons, A., Mackey, I.., Belgrave, D., Fan, A., Paquet, U., Tomczak, M., and Zhang, C. (eds.), Advances Neural Infor- mation Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15,
2024, 2024. URL http: http::/papprs.nipp.cco
paper files/ /llles//aper/222/haah
7f 5ddecc
Abarrract-Conffeence...l
15

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
A. Proof of Lemma 1
Lemma T claims that only attention heads with sparse and concentrated patterns demonstrate consistent robustness to low-precision KV cache quantization. Its proof is below.
Proof. Given the query token q E RIID and key cache K TR the attention score without errors is dg expigk.)
Ej= exp(ak,)) The key asymmetric uniform quantization error AK E R N o²) follows normal distribu-
tion, where a = ma=(K))min(() Therefore, low precision quantization leads to exponential larger quantization errors. 28-1
Then, the i-th attention score with key errors is
à exp(a(K; + AK,))
explgK,)exp(q Σラ= E jeppqa
exp(qK,)) explaaк, J
(5)
AK,))
ΣS exp(gk eXplaa..)
If the key quantization error vector A A with low quantization precision B is noticeable, the inner product of query and error vector q qåK; j is also not ignorable. There are two cases where à; equals to a, for all tokens. In other words, the attention distribution before and after key quantization are identical.
Case 1) exp(gAK,) c 1, where each key token quantization errors have the same inner product result with the query token qåK; = q which normally does not happen.
expięk,)
Case 2) There is a dominating key token i. j # i, exp(qk,) > e and exp(ak.)
~ 0, then
ex)(qAK,))
exp(gAK.)) exp(qAK:))
à ΣE= exp(q exp(ęK.)
1.
(6)
Other dominated key token thus has the attention score å = 0. The exactly identical attention distribution with dominating key token may be special case, but it indicates that attention heads with small amount of dominated key tokens, which have highly attention scores and result in sparse and concentrated attention patterns, are consistently robust to low-precision KV cache quantization.
B. Effects of KV Cache Quantization Mode and Precision
In this section, analyze the effects of KV cache quantization mode and precision. We collect the full precision query tensor in the decoding phase and KV cache in both prefilling and decoding stages of the Llama-3,1-8B 1-8B-Inntruct model when processing the first 20 mathematical GSMSK zero-shot prompts without KV cache quantization. After that, we quantize KV cache along the channel or token dimension with uniform precision to compute errors of KV cache and attention score and output vectors of each self-attention layer as defined in Section 3.2 caused by KV cache quantization without any error accumulation. Finally, we average the simulated errors over different prompts and all layers in Table to study the inherent sensitivity of KV cache to quantization mode and precision.
The non-accumulated relative attention output errors e. of INTS KV cache quantization with the per-token-asym per- channel-asym are lower than 3%. Minor single-token errors may cause slight shifts in intermediate attention patterns and final output distributions, yet these shifts are typically insufficient to alter the generated output tokens. However, when implementing extremely low-precision 2-bit KV2 cache quantization, the relative key quantization error e, increases to 40.1% or 77.5 77.5%, which may lead to substantial attention distribution shift for non-sparse retrieval heads as demonstrated in Figure e, increases dramatically to 81.4% with the per-channel-asym mode even 96.2% 26%2% with the per-token-asym mode, The noticeable errors may thus lead noticeable token flipping and generation errors as in Table [
The relative key error e of the INT8 per-token-asym key quantization is 0.012280, which is 2.5x larger than the per-channel- asym counterpart 0.004869. Dynamically asymmetric quantization along the channel dimension leads to significantly smaller error of both key cache and attention score compared with token dimension quantization. indicating that key cache is strongly sensitive to quantization dimensions. The phenomenon can be explained with the strong channel-win -wie outliers of key cache (Liu et al. 2024e; Hooper et al. 2024]. While value cache can not benefit from switching the quantization dimension, as the relative value errors the channel or token dimensions over different precision are quite close.
16

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
Table 9: Key and value cache quantization relative error analysis of different precision and quantization methods. We collect BF16 KV cache of 20 prompts from the GSM8K zero-shot dataset with 1aae-3..-8--I--s-- and then perform offline quantization to compute the mean error between BF16 and dequantized KV cache.
KV cache precision
KV quant mode per-channel-asym
Relative ej
Relative e,
6a
Relative 6o
0.004869
0.007754
0,000013
0.027686
KV8
per-token-asym 0.012280
0.007865
0.000018
0.014589
per-channel-asym
0.080991
0.125457
0.000172
0.158429
KV4
per-token-asym
0.196476
0.126894
0.000251
0.206909
per-channel-asym 0,401151
0.604678
0.000868
0.814023
KV2
per-token-asym
0.774668
0.607898
0.001 166
0.961792
As shown in Figure 7 there are clear layer-wise diversities of KV quantization errors eg and eo with different quantization modes including per-token-asym and per-channel-asym and different precision like INT8, INT4, and INT2. In addition, changing the quantization dimension or mode can result in the significant distribution shift of layer-wise key quantization error. For example, the most sensitive layer with the per-token-asym quantization mode is layer-29, while it changes to layer-11 and layer-13 13 with the per-token-asym mode. Statically retaining the first or last several layers with more sparse budgets (Tang et al., 2024) may not general well in KV cache quantization. Therefore, we need an automatic KV cache quantization tuning framework to adaptively adopt to these layer-wise differences and configuration modifications.
C. Experimental Settings
KVTuner is an automatic KV cache quantization precision tuning framework and can be applied to any quantization mode. We choose two representative and efficient KV cache quantization algorithms KIVI (Liu al. 2024e) and per-token-asym with uniform KV8, KV4, or KV2 precision pairs in all layers as baselines. Specifically, for the KIVI quantization method, we set the residual length to 32 and the group size to 32. KVTuner is currently implemented based huggingface transformers, but it can be applied to inference frameworks such as vLL.M (Kwon et al. 2023), Megatron (Shoeybi et al.) (019)), LMMeploy Conttibbuoor 2023), and SGLang (Zheng et al., 2024). To ensure compatibility, we integrate KV cache quantization methods including KIVI, per-token-asym, and KVTuner within the In-evaluationn-haress (Gao et al.
2024). allowing for
seamless adaptation and reproducibility of KVTuner.
We select three popular and recently released LLMs series Llama3.1 (Dueey al. 2024p.
Mistral-vO.3 mang al., and Qwen2.5- In-Inttt represent the most studied model size. To cover more LLMs application scenarios with different scales, Swen2.5-3B-Imm Instuctt and its quantized version Qwen2.5-3B B...5-3----sstuct--Wee are selected for personal devices with limited GPU memory, while Qwen2.5- 14B, 32B) )- Instuct with larger model scale and better performance are also tested. We also test
2023), and Qwen2.5 (Yang et al., (024a). Among them, Llama-3.1 1--8B- Instrucc,
for mathematical reasoning tasks.
1) General tasks: CEvan et al. 2023), MMLU (Hendryeks et al., 2021). TriviaQA (Joshi al., 2017). RACE (Lai et al.) 2017). and TruthfulQA (Lin et al., 2022). 2) Math, science, and logic tasks: GSM8K (0-shot, 4-shot, 8-shot, 16-shot) (Cobbe et al.
We cover 5 general AIGC and 2 mathematical reasoning tasks available in Im-evaluation-harree
202I. GSM8K multi-round with Im-evaluation-hhares (Gao et al. 2024). GPQA (Rein et al. 2023).
For the final layer-wise KV cache quantization precision pair searching with multi nulti-objecttve optimization, we use the open-sourced and widely used Optuna framework (Akiba et al., 2019) and MOEA/D (Zhang & Li] (007) algorithm. In which case, we treat the LLM inference accuracy under different layer-wise KV precision pairs and input prompts as block-box. The intra-layer and inter-layer search space pruning only takes several minutes but significantly improves sampling efficiency of the downstream Optuna.
We first preprocess the available quantization precision options for each layer group and store them in array. The indices of this array are then treated as integer parameters, which are optimized by Optuna through multi-objective optimization. The first objective is to maximize the accuracy on the first 200 samples of the GSM8K dataset, while the second objective is to minimize the equivalent quantization precision or memory usage of KV cache. For each combination of model and quantization mode, we set soft constraint on the equivalent precision at 4-bit and 6-bit for optuna, conducting 200 search iterations for each setting. The total time cost of offline KV cache precision pair tuning with Optuna mainly depends on the
17

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
0.5
124
M
0.31
828
0.75
0.12
0022
034
846
0334
832
L180
(a) K8 per-token-asye
(b) K4 per-token-asyn
(c) K2 per-token-asye asym
01390
0.0300
0.040
8:07
8.6
00325
(d) K8 per-channel-asyyee
(e) K4 per-channel-asym
(f) K2 per-channel-asym
032
8.04
324
8.00
0.33
0.227
0.3
V8 per-token-aa asym
(h) V4
(i) V2 per-token -nmm
1.2
0.11
0.35
0.22
0.3
0027
(i) V8 per-channel-asym
(k) per-channel-asym
() V2 per-channel-asym
Figure 7: Layer-wise relative key errors e and value errors e, of -.mma-..18B-IInsii with the per-token-asym and per-channel-asym quantization modes, KV cache precision 8, 4, and 2-bit, and the same settings in Table 9
18

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
hardware and operator implementation efficiency.
D. Search Space Pruning and Multi-objective Optimization results
D.I. Intra-Layer and Inter-Layer Search Space Pruning Results
D.I.I. INTRA-LAYER PARETO OPTIMAL KV CACHI PRECISION PAIR PRUNING
The intra-layer KV cache quantization precision pair pruning based on Pareto frontier are available in Table The calibration dataset is the first 20 prompts from the zeroshot GSM8K dataset. The Pareto optimal KV cache precision pairs in most layers are the key-first set (KV8, K8V4, KV4, K4V2, KV2], indicating that the observation that key cache is more important than value cache holds.
When both key and value cache are quantized along the token dimension, only the first layer in Llama-3. Mistral-7B-Inn 1-nnr--tt--ttt prefers other KV precision pairs and all layers in Qwen2.5- 14B, 32B )- I-Instruct select the key-first set. In contrast, K8V2 outperforms KV4 in four important layers of Qwen2.5- 7B IInstruct indicating that uniform 4-bit key quantization may lead to model accuracy degradation as in Table 13
1-1-aa--.---B-mmm 188--nntttutt and
When utilizing the KIVI-like key per-chamncl-asyne and value per-token-asyne quantization mode, more layers show diverse preference of Pareto optimal KV cache quantization precision pairs. these layers, K4V8 and K2V4 outperform K8V4 and K4V2, which means that lower precision key is better than lower precision value in terms of attention errors. It indicates that per-channel key quantization can effectively reduce quantization errors.
D.1.2. INTER- LAYER CLUSTERING BASED ON ATTENTION ERRORS
After the intra-layer KV cache quantization precision pair pruning, we apply the inter-layer clustering among the layers with the same precision pair set. The clustering algorithm is DBSCAN (Ester et al. 1996) with the hyper-parameter epsilon=0.05 and bain_samples_t As demonstrated in Table I0 we successfully reduce the exponential component of search space size from the number of transformer layers T. e.g. 28--64 to the number of clustered layer groups G e-g. 4--8. Utilizing the two-level search space pruning, the total number of combinations of candidate KV cache precision pairs is significantly reduced from 9dh to 5 or 6C, In 1 1-8B-IIntruuu 9r = =93 3.4 10 while 5 = 5 15625.
In the layer-wise relative attention output errors with per-token-asym KV cache quantization of Llama-3.1 1-----Instuu in Figure [3 the highly sensitive layers include layer-(0, 1, 2. 3, 4, 23, 24, 25, 27, 28, 29), while the insensitive layers include layer- (8, 9, 10, 11, 13, 14, 15, 20, 30). Layers in these two classes are correctly clustered into different groups. Similar per-token-asye asym and KIVI-like quantization modes in Figure and [8] respectively. Therefore, we can conclude that the proposed multi-objective Pareto frontier based intra-layer pruning and inter-layer clustering algorithms successfully reduce the search space by considering the inherent layer-wise sensitivities.
phenomenon can also be observed in 1esn22.5-7B-Intrruue
D.2. Searched layer-wise KV precision pairs
The final searched layer-wise mixed precision KV cache quantization precision pairs of different LLMs and KV quantization modes are available in Table II] Some clustered layer groups in Table [occhoose the same KV cache quantization pairs under the given memory consumption and/or accuracy degradation constraints. The number of utilized KV cache quantization precision pairs is reduced from 6 ~ 8 to 2 ~ 5 in the tested Llama-3. -ns-Iust Qwen2.5-3B Instruct, and Qwen2.5-7B Instruct models. In addition, the significantly diverse layer-wise KV precision pair distribution in Table [T indicates that there are not clear hcuristic rules based on layer depths to identify layer importance and sensitivity to KV cache quantization. Therefore, we need to measure the model accuracies considering their complicated nonlinear dependencies to layer- wise KV cache precision pairs and utilize accuracies to distinguish the whole model level KV cache precision pair combinations.
D.3. Pareto Frontier with the GSM8K Calibration Dataset
We use the open-sourced package optuna (Akiba et al., 2019) with the MOEA/D algorithm (Zhang & Li, 2007) for the final search with the model memory usage and inference accuracy of the first 200 4-shot GSMSK prompts. The multi-objective search of the Pareto optimal layer-wise KV cache quantization precision pairs of Llama-3. 1--.3..------e Instruct Mistral-7B-Innttuen v0.3, Qwen2.5- 5--3--BIInnttue and Qwen2.5-7B I--ntruuttt with the KIVI and per-token-ass -sym quantization modes are available
19

---

KVTuner: Sensitivity-Aware Layer- Sayer-Wise Mixed-Precision KV Cache Quantization
Table 10: Inter-layer clustering results by clustering among the layers with the same pruned intra-layer KV cache quantization precision pairs. For example, layers 14 and 20 demonstrate higher sensitivity than layers 3, 13, and 27 as visualized in Figure They are clustered into different group, validating the effectiveness of our intra-layer pruning and inter-layer clustering.
Model name
Key quant. mode
G
Grouped layer ids
[0]. [1~4, 13, 18, 25, 27, 31,, 15, 6, 12, 21, 26, 28), 18~11, 14-17, 20, 30),
per-token-asym
221, (23, 24, 291
Lluma-3.1-80 1-mma-3.-8---nsttuuu
32
[0]. 7, 29, 31), (4, 25,
14 18 22. (1, (5,,,, (7-22, 24-30)
per-channel-asym
30), 17)
per-token-asym
(0, 31), (2~4), [6, 27, 29), 17. 8, 10, 18), 19, 14), (3, 21~26, 28, 30).
Mistral-7B- ---nsnrct..
32
per-channel-asym
(11~13, 15, 17, 20), [16]
[0), (1,366 8, 12, 13, 15. 20). (2, 14, 23, 35). 11, 16. 25, 32), (10, 19.
per-token-asym
24, 33), (37, (17,30, 31, 34). (21, 22), (18, 27, 29)
Qwen2.5-3B -wenn.5-3B-Inttruct
36
(0, 13, (2,4). (34,35), (3, 11, 23), (5.
25, 32, 33), (8, 16, 18, 21, 22, 17, 19, 20, 29, 31), (12, 28)
per-channel-ssym
24, 27, 30), (9, 10, 14,
[0]. (1.2,,,,,, (6. (6) (7. (7.
11, 13), (8, 24). (9. 12, 16~ 18, 21, 22.
per-token-asym
26). (14,200, (3.
Benn2.5-7B-Instruee
28
(0, (1,3), (4.5, 12, (7.9, 10, 13, 14, 16, 18~21, 27), 18, 26). (II.
par-channcl-asym
15, 17), [6)
[0~2, 6, 12, 13. 41), (3-5, 8). (7. 10. 15 19. 13, 14. 31, 38, 39).
per-token-asym
[16~188 20, 27, 28, 30, 37,
40. 42--44 46, 47). (22, 26, 45)
Bren2.5-14B-Instmee
48
(0. (L. 3. 4). (5, 6, 8. 9, 12). (7. 10, 13, 15-211, 23, 24, 26~33, 35-38,
per-chaneelasym
44--77, (11, 25. 41. 42). (14, 39, 40), (22, 34)
[0,2, II, 15, 54, 571, [1. 7-
14, 17-32, 34-053, 55, 56, 58-63),
per-token-asym
(3,4). (6. 16)
(wen2.5-32B-Insstte
64
[0~-4). [11]. (5~100 12, 14. 16, 18-23, 26-28. 32). (13, 15, 17, 22, 24, 25.
per-channelasym
29 (63)
in Figure g andd? g
In order to validate the effectiveness of the proposed two-stage intra-layer and inter-layer search space pruning, we disable the pre-processing process and directly use the original full S search space in Figure 10
For each combination of model and quantization mode, we set soft constraint the equivalent precision at 4-bit and 6-bit for optuna, conducting 200 search iterations for each setting. The search results are then merged for visualization. In cases where the two-stage intra-layer and inter-layer search space pruning is not applied, we set the maximum equivalent quantization precision to 6-bit and similarly perform 200 search iterations. Specifically, for the KIVI quantization method, we set the residual length to 32 and the group size to 32.
Note that for Qwen-2.5-7B with the KIVI quantization mode, the result from 200 search iterations appeared abnormal. Therefore, extended the search 500 iterations obtain the final result.
E. Correlation of Model- and Layer-wise KV Cache Quantization Sensitivity with Attention Patterns
According to the layer-wise attention score errors of Llama-3. 1--tma--.1-8B-In-see --ntruntttt in Figure 3 and Qwen2.5-7B-Instruct in Figure 16 we can observe the clear layer-wise difference in the same LLM. In this section, we try to explain the reason of the difference from the attention pattern perspective as in Figure and 12] In which, we visualize block level attention scores of the first 4 heads with block size 4 in the prefilling and decoding stages, and horizontal and vertical axes represent the key and query dimensions respectively. Yellow, green, and purple points indicate high, medium, and low attention scores, respectively. We find out that the more complex and dynamic attention patterns usually lead to larger attention score errors and sensitivity to KV cache quantization of intermediate transformer layers and the whole LLMs.
and
Take Llama-3.1
an example, layer 12 and 13 are in the group with high attention score errors, while layer 0 and 31 are in the medium error group and layer 2 and 23 are in the low error group. Analyzing the attention patterns of these layer in the below Figure IT we can conclude that heads in the layer 12 and 13 have dynamic and non-sparse attention patterns, which are called as retrieval heads (Tang et al.) 2025 Xiao et al., (025). In contrast, heads in layer 0, 2, 23 and 31 have more static attention patterns like attention sink and recent window, which are called as streaming heads (Xiao et al.
2025 2024).
20

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
L-ama-3.1-8B-Inntte
(b) Mistral-
20
(c) Swen2.553B-Insttuue
(d) Qwen2.5-7B-Instruct
Figure 8: Pareto frontier of different models with the KIVI quantization mode on the first 200 data slices of the 4-shot GSM8K dataset.
Llama-3.1-8B-Instree
(b)
(c) Qwen2.5-30 3-BBB 3---5------sss Instruct
(d) Qwen2.5-78 7B-IIntrucctt
Figure 9: Pareto frontier of different models with the per-token-asym quantization mode on the first 200 data slices of the 4-shot GSMSK dataset.
21

---

KVTuner: Sensitivity-Aware Layer- Jayer-Wise Mixed-Precision KV Cache Quantization
Table 11: Detailed searched layer-wise KV cache quantization precision pairs of different LLMs and KV cache quantization modes by KVTuner.
Model name
Quant, mode
Equivalent precision
Quant. precision
Layer ids
K4V8
0
3.59
KV4
8-1-
21, 28, 14-17, 20.
per-token-asym
1-4,7,
19,
22-25, 29,31
K8V4
20,
23-25,
29-31
5.44
KV4
0.5.6. 12.
19. 26, 28
K3V4
13,17
1-3, 29,
LJama-33 3.B.BB It-Instruc
3.25
K4V2
8-12,
14-16,
18-24, 26,
KV2
25,
KIVI
K3V4
4,6, 8-12,
14-16,
18-20, 22, 23
29,
4,90
K4V2
2,,
K2V4
0
KV2
13,
K8V4
17. 29.
KSV2
4,00
KV4
11,
19,
28,
22,
24
per-token-asym
K4V2
8.9, 12,
13-16,
20,
23,35
KVS
0
KSV4
1-3-6,
8-
13,
17-20,
29-12
K4V2
2, 14, 21-23, 28, 32,
K4V8
n-1
Qwen2.5-3B-Instrur
K4V2
3,5-11, 13-27, 29-33
3.17
34-35
K2V4
2,4
KY2
12, 28
KIVI
KV8
[-]
3, 11. 23,
3.44
K4V2
8-10, 14-22,
26, 27, 29-31
K2V4
34-35
KV2
2,4, 12, U
KVS
K8V2
13,
4,00
KV4
6,9.9-12, 14-23,
per-loken asym
K4V2
1, 24,
K8V4
8,9, 12, 14, 16 20, 22,24.
KV4
2, 10, 19,
Qwen2.5-7P 5.B Instruct
KV8
2, 11,
3.92
KV4
4,5, 22-226
22-26
1.3, 1.3.7. 10,
14, 18-21,
KV8
7.9, 10, 13, 14,
21,
KIVI
K8V4
4.5. 12, 22, 23, 24
5.96
K4V2
15, 17
K2V4
1,3
KV2
6,8, 26
Compared with Llama-3.1-3 1----..1-----ttuuec which has the high ratio of heads with static attention patterns, Qwwen..5-B-Innttrues consists of many heads with mixture dynamic retrieval heads and other static patterns. It may explain why Qwen2.5- Instruct more unstable to KV cache quantization as Table Layer 5, 12, 21, and 27 have similar attention patterns, but the relative strength of retrieval and streaming heads leads to the difference of layer-wise sensitivity to KV cache quantization.
However, the sensitivity to KV cache quantization is the inherent model property which can be learned offline. Therefore, it is necessary to apply layer-wise mixed precision KV cache quantization and maintain high precision of key cache than value cache with multi-objective optimization KV precision pair tuning as proposed in this work. KVTuner thus makes equivalent 4-bit and even lower KV cache quantization nearly lossless the sensitive models like
F. More KV Cache Quantization Results on General and Mathematical Reasoning Datasets
The experimental results of the selected 5 LLMSs on the general and mathematical reasoning datasets with uniform KV cache quantization precision pairs are available in Table 12 and 13 To simulate the Openai ol like long CoT reasoning process, the fho-shot CoT' the GSM8K dataset are treated as multi-turn conversation, which enabled with the flags
22

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
Indeepertine
pemtoe
(a) 1aimma-..-8B-Inaiiii
(b)
Figure 10: Pareto frontier of different models with the per-token-asym quantization model on the first 200 data slices of the shohot GSM8K dataset without intra-layer and inter-layer search space pruning.
fewshot_an _as___________ multiturn and papll_chat__empleee in Im-evaluation--aanes. In which cases, questions are provided as user content and answers are provided as assistant responses instead of directly using the given standard answers. Table [4] summarizes the results of 8 LLMs including Qwen2.5-(144 32B) Instruct under the fewshot_ _astoot_________ us_multiturn setting.
There are limited long output mathematical reasoning datasets in Im-evaluation-haarees (Gao et al. 2024) and the evaluation of the long context generation is an open question. Therefore, we enable KV cache quantization in both the prefilling and decoding stages amplify the effects to final generation results caused by KV cache quantization error accumulation, which makes distinguishing different quantization methods easier. For the KIVI quantization mode, we utilize the HQQ quantizer from HuggingFace's implementation, with both the residual length and group size 32.
According to Table 12 13 and 14 most LLMs, including Llama-3.1-88 1-B..--B-Immmme Instruct, Mistral-7BB Instrucc--t.... (wwe2..5-114B.
-Instuut, are robust to low-bit KV cache quantization. Although eтror accumulation caused by KV cache quantization starts from the prefilling stage, the high KV cache quantization precision pair KV8 with KIVI or per-token-asym quantization mode are still generally lossless, except Qwen2.5- 1-t.5----h--77 Maathh7B-Instruct The uniform KV cache quantization precision pairs KV4 or even K4V2 with the KIVI quantization mode can achieve nearly lossless 4x or even 5.3x KV cache compression, respectively. KV4 with the simple per-token-asym mode also results in negligible accuracy loss in Llama-3. --sttutttttc and -
shown in Table 12 KIVI does outperform the per-token-asym quantization mode in the low-precision settings such as KV4, K4V2, and KV2, especially in
and (wwen.5--7BIInstrul as
demonstrated in Figure 13
As shown in Figure 14 the larger ( Instrruct models are robust than the smaller Qwen2.5- 7B, models. In addition, comparing Qwen2.5-3B- Math-7B (1n-7BB)--nstuul IInstruct and the weight quantized -aennn.5----Insruutt----
Instruct-AWQ and Siee2.5-33-IInsrucee we can conclude that model weight quantization with AWQ does not affect the model-level sensitivity to KV cache quantization. The increasing GSM8K accuracy with the longer CoTs under the half precision BF16 KV cache setting indicates that most Qwen2.5 models benefit from longer CoTs. We also obverse that 16-shot CoTss with K4V2 KV cache precision outperforms the 4-shot CoT' with BF16 KV cache precision on the larger Qwen2.5-(1 (wen2.5-(14B,332) 43BBB Instruct models. It indicates that longer CoT with lower and mixed precision KV cache outperforms uniform precision counterparts as in Section 6.2 In other words, mixed precision key cache quantization with higher precision key can achieve both memory usage and inference accuracy improvement than equal precision key and value cache quantization.
23

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
(a) Layer-0 with recent attention patterns (medium attention score errors)
(b) Layer-2 with attention sinks (low attention score errors)
(c) Layer- 12 with retrieval heads (high attention score errors)
(d) Layer- 1.3 with retrieval heads (high attention score errors)
(e) Layer-23 23 with attention sink (low attention errors)
Hust
(f) Layer-31 with mixture of retrieval and streaming heads (medium attention score errors)
Figure 11: Selected layer-wise attention patterns of 1aamm--.1--8BInsstie model and the first prompt in the 0-shot GSM8K dataset. Many layers and heads of Llama-3. I-St-In have simple and streaming attention patterns which highly concentrated and sparse attention scores. As result, the attention score errors in these layers are medium or low. In contrast, layers with retrieval or mixed attention patterns, whose attention scores are non-sparse, normally show high attention score errors. We also observe that the attention patterns of query heads in the same group and sharing the same key cache are highly similar, which may indicate that we can apply attention head group-wise KV cache management for better accuracy.
24

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
(a) Layer-0 with mixture of recent window, re-access, and retricval heads (high attention score errors)
(b) Layer-1 with mixture of recent window and re-access patterns (medium attention score errors)
(c) Layer-5 with mixture of retrieval and streaming heads (low attention score errors)
(d) Layer- 12 with mixture of retrieval and streaming heads (medium attention score errors)
(c) Layer-21 with mixture of retrieval heads and attention sinks (medium attention score errors)
(f) Layer-27 with mixture of retrieval heads and attention sinks (high atten- tion score errors)
Figure 12: Selected layer- attention patterns of Qwen2.5-7B-Inntrrus model and the first prompt in the 0-shot GSM8K
dataset. Most layers and heads of Dwwen.55-7B-Instreee have complex attention patterns, such as retrieval, and mixture of retrieval and recent or attention sink patterns. These non-sparse and non-concentrated attention patterns result in the high sensitivity of Qwwen2.5-7B-Instrret to KV cache compression including low-precision quantization and even model weight and activation quantization.
25

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
Table 12: Final generation accuracy comparison of different KV cache quantization modes and precisions and Llama-3.1 1-
SB-Instruct and Eestral-BB-Instrucss Instruct-vO.3 on the AIGC and mathematical datasets. KV cache quantization enabled during both prefilling and decoding stages amplify the effects of emor accumulation.
GSM8K
Quant. method
Precision
CEVAL
MMLU
TriviaQA
RACE
TruthfulQA
Averuge
0-shot
4-shot
8-shot
16-shot
Llama-3.1-80
--Sttrutt
BF16
BF16
0.5386
0.6802
0.5161
0.4469
0.6267
0.2866
0.7635
0.7741
0.7854
0.6020
KV8
0.5416
0.6798
0,5162
0,4469
0.6304
0.2752
0.7597
0.7657
0.7809
0,5996
K8V4
0.5394
0.6792
0.5138
0.4498
0.6450
0.2858
0.7695
0.7794
0.7923
0.6060
K8V2
0.4807
0.6381
0.4989
0.4383
0.6499
0.2358
0.7074
0.7036
0.7195
0.5636
K4V8
0.5327
0.6694
0,5144
0.4488
0,5851
0.2623
0.7566
0.7566
0.7710
0.5885
KIVI
KV4
0.5245
0.6689
0.5135
0.4498
0,6132
0.2782
0.746
0.7589
0.7680
0,5912
K4V2
0.4703
0.6236
0.5016
0.4450
0.5373
0.2464
0.6694
0.6694
0.6854
0.5387
K2V4
0.3247
0.4628
0.4761
0.3675
0.4639
0.0978
0.1122
0.1054
0.0842
0.2772
KV2
0.2771
0.3600
0,4584
0.3301
0,3182
0.0508
0.0432
0,0318
0.0250
0.2105
KV8
0.5342
0.6800
0.5175
0.4459
0.6206
0.2805
0.7657
0.7809
0.7801
0.6006
K8V4
0.5386
0.6776
0.4709
0.4450
0.6169
0.3154
0.7733
0.7688
0.7847
0.5990
K8V2
0.4792
0.6183
0,4984
0.4239
0.5887
0.1501
0.6391
0.6262
0.6550
0,5199
K4V8
0.5163
0.6579
0.5123
0.4411
0.6781
0.2517
0.7180
0.7293
0.7240
0.5810
Per-token-asye asym
0.5141
0.6570
0.4849
0.4325
0.6340
0.2782
0.7240
0.7202
0.7157
0.5734
K4V2
0.4413
0.5910
0.4779
0.4306
0.5447
0.1289
0.5709
0.5633
0.5519
0.4778
K2V4
0,2400
0.2350
0.0249
0,2593
0.3268
0.0212
0,0159
0.0296
0,0212
0.1304
KV2
0.2444
0.2338
0.0052
0.2478
0.2277
0.0227
0.0174
0.0197
0.0273
0.1162
Mistral-
BF16
BF16
0.3923
0.5911
0.6081
0.4057
0.4296
0.0766
0.3389
0.3753
0.3601
0.3975
KV8
0.3945
0.5901
0.6072
0,4115
0.4259
0.0735
0.3412
0.3639
0.3624
0.3967
K8V4
0.3945
0.5909
0.6068
0.4067
0.4394
0.0781
0.3457
0.3723
0.3669
0.4001
K8V2
0.3819
0.5776
0.6042
0.4086
0.4370
0.0675
0.3404
0.3518
0.3609
0.3922
K4V8
0.3990
0.5875
0.6069
0,4048
0,4308
0.0697
0.3442
0.3563
0.3738
0.3970
KIVI
KV4
0.3945
0.5886
0.6074
0.4105
0,4455
0.0751
0.3434
0.3662
0.3586
0.3989
K4V2
0.3752
0.5753
0.6035
0.4000
0.4223
0.0705
0.3434
0.3397
0.3616
0.3879
K2V4
0.3128
0,4926
0.5982
0.3847
0.3917
0.0637
0.0978
0.0910
0.0773
0.2789
KV2
0.2905
0.4571
0.5920
0.3885
0.4688
0.0478
0.0766
0.0644
0.0516
0.2708
KV8
0.3900
0.5892
0.6071
0.4067
0,4284
0.072
0.3419
0.3745
0.3571
0.3963
K8V4
0.3967
0.5897
0.6040
0.4057
0.4357
0.0751
0.3533
0.3715
0.3707
0.4003
K8V2
0,3692
0.5760
0.5797
0.4029
0,3929
0.0675
0.3328
0,3381
0.3548
0.3793
K4V8
0,3871
0.5862
0.6070
0.4077
0.4259
0.0629
0.3450
0.3578
0.3692
0,3943
Per-token-asym
KV4
0.3871
0.5865
0.5994
0.4048
0.4321
0.072
0.3450
0.3556
0.3685
0.3946
K4V2
0.3618
0.5672
0.5774
0.4086
0.3623
0.0599
0.3048
0.3389
0.3571
0.3709
K2V4
0.2786
0.4360
0.4688
0.3914
0.3268
0.0303
0.0334
0.0281
0.0211
0.2238
KV2
0.2741
0.3926
0.4045
0.4019
0.2999
0.0281
0.0265
0.0167
0.0220
0.2074
26

---

KVTuner: Sensitivity-Aware Layer- Layer-Wise Mixed-Precision KV Cache Quantization
Table 13: Final generation accuracy comparison of different KV cache quantization modes and precisions and Qwen2.5
LLMs on the AIGC and mathematical datasets. KV cache quantization is enabled during both prefilling and decoding stages to amplify the effects of error accumulation.
GSMSK
Quant. method
Precision
CEVAL
MMLU
TriviaQA
RACE
TruthfulQA
Average
0-shot
4-shot
8-shot
16-shot
Swwen25--B--sstrutt--WW
BF16
BF16
0.7125
0.6382
0.2299
0.3904
0.4700
014866
0.5815
0.6353
0.6861
0.5367
KV8
0.7073
0.6389
0.2283
0.3885
0.4761
0,4882
0.5762
0.6361
0,6816
0.5357
K8V4
0,7080
0.6388
0.2321
0.3895
0.4871
0,4852
0.5625
0.6315
0.6823
0.5352
K8V2
0.6872
0.6204
0.2225
0.3856
0.4847
0.4928
0.5368
0.6058
0.6641
0.5222
K4V8
0.7125
0.6275
0.2326
0.3923
0.4761
0,4814
0.5580
0.6096
0,6550
0.5272
KIVI
KV4
0,7013
0.6249
0.2322
0.4048
0.4627
0.4761
0,5474
0,6240
0,6368
0.5234
K4v2
0.6709
0.6038
0.2216
0.3885
0,4700
0.4519
0.5284
0.5732
0.6171
0.5028
K2V4
0.3566
0.3626
0.1986
0.2995
0.4186
0.0197
0.0099
0.0099
0.0091
0.18
KV2
0,3507
0.3203
0,1983
0.2727
0,4308
0.0136
0.0144
0.0144
0.01 36
0, 1818
KV8
0,7043
0.6379
0,2248
0.3866
0.4798
0,4913
0,5823
0.6331
0,6740
0.5349
K8V4
0.6969
0,6364
0.2402
0.3837
0,4676
0,4784
0,5671
0.6209
0.6717
0.5292
K8V2
0.4926
0.4979
0.0100
0.3732
0,4749
0.3616
0.3798
0,4200
0.4640
0.3860
Per-token-asym
K4V8
0.2489
0.2306
0.0000
0.2258
0,1591
0.0008
0
0
0.0008
0.0962
KV4
0,2377
0.2325
0,0000
0.2220
0.1469
0
0
0.0015
0.0015
0.0936
K4V2
0,2600
0,2323
0.0000
0.2258
0,0979
0.0038
0.0911
K2V4
0.2318
0.2335
0.0001
0.2201
0.1677
0.0023
0.0083
0.0045
0.0099
0.0976
KV2
0,2489
0.2372
0.0000
0.2249
0.1310
0.0023
0.0053
0.0106
0.0061
0.0963
Qwen2.5-7B-Insttee
BF166
BF16
0.7949
0.7178
0.3239
0.4612
0,5104
0,7233
0.8059
0.8287
0.8218
0.6653
KV8
0.7949
0.7174
0.3235
0.4603
0,5092
0.721
0,7915
0,8249
0,8302
0.6637
K8V4
0,7979
0.7174
0.3222
0.4651
0,5104
0,7119
0.7915
0,8180
0.8226
0,6619
K8V2
0,7734
0.7035
0.3165
0.4459
0.4994
0,6581
0,7832
0,8059
0.8105
0,6440
KIVI
K4V8
0.5780
0.5024
0.2757
0.3311
0.3660
0.0136
0.0076
0.0038
0.003
0.2312
KV4
0.5802
0.5028
0.2761
0.3206
0.3758
0.0182
0.0068
0.0038
0.003
0.2319
K4V2
0.5245
0.4704
0.2754
0.3167
0.3745
0.0152
0.0099
0.0053
0.0038
0.2217
K2V4
0,2719
0.2645
0.2742
0.2507
0,2399
0.0053
0.0015
0.0008
0.0008
0. 141455
KV2
0.2756
0,2568
0.2741
0.2632
0,2338
0.0099
0.0038
0.0023
0.1466
KV8
0.7883
0,7119
0.3192
0.4593
0,4957
0.7149
0.8044
0.8052
0.8203
0.6577
K8V4
0.7920
0.7117
0.2978
0.4545
0,5018
0.7111
0.7847
0.8044
0.8067
0.6516
K8V2
0.7169
0.6757
0.1127
0.4488
0.4957
0,577
0.7233
0.7453
0.7513
0.5830
Per-token-asym
K4V8
0,2192
0.2305
0.0000
0.2220
0,0318
0
0.0782
KV4
0,2400
0.2327
0.0000
0.2115
0.0171
0.0008
0.0015
0.0782
K4V2
0,2400
0.2301
0.0001
0.2172
0.0245
0.0023
0.0008
0
0
0.0794
K2V4
0,2273
0.2347
0.0001
0.2077
0,0575
0,0061
0.0068
0.0015
0.0015
0.0826
KV2
0,2489
0.2376
0.0000
0.2230
0.1346
0.0045
0.003
0.0076
0.0015
0.0956
Oween..-MMath-7B-inne
BF16
BF16
0.4881
0.5383
0.0074
0.3464
0.4015
0.4109
0.8863
0.8870
0.8840
0.5389
KV8
0.4844
0,5379
0.0072
0.3397
0,3966
0.4041
0.8878
0.8878
0.8772
0.5359
K8V4
0.4874
0.5361
0.0071
0.3445
0.4002
0.4102
0.8886
0.8870
0.8840
0.5383
K8V2
0.4606
0.5291
0.0071
0.3426
0.4162
0.4139
0.8779
0.8802
0.8696
0.5330
KIVI
K4V8
0.4428
0,5061
0.0073
0.2660
0.4100
0.0834
0.1501
0.2024
0.1259
0.2438
KV4
0.4368
0.5070
0.0074
0.2718
0.4284
0,0879
0.1516
0.1895
0.1236
0.2449
K4V2
0.4294
0.4862
0.0069
0.2699
0.4100
0.0819
0.1145
0.1433
0.1024
0.2272
K2V4
0.2712
0.2780
0.0059
0.2230
0.3941
0.0152
0.0061
0.0023
0.0008
1330
KV2
0,2741
0.2757
0.0057
0.2220
0,3501
0.0167
0.0023
0,003
0
0,1777
KV8
0.3975
0,5905
0.6064
0.4038
0.4308
0.0728
0,3457
0,3685
0.3571
0.3970
K8V4
0,3878
0,5891
0.6035
0.4010
0.4443
0,0735
0,3450
0.3616
0.3632
0.3966
KXV2
0.3522
0.5590
0.5452
0.3971
0,3452
0.0462
0,3116
0,3397
0.3359
0.3591
K4V8
0,3804
0.5822
0.6016
0.4010
0,3831
0.0667
0,3252
0,351
0,3381
0,3810
Per-token-asym
KV4
0.3767
0.5803
0.5967
0.4038
0,3953
0.0622
0.3093
0.3146
0,3404
0,3755
K4V2
0.3470
0.5463
0.5372
0.3943
0.4211
0.0462
0.2631
0.2752
0.2911
0.3468
K2V4
0.2429
0.2401
0.0262
0.2900
0,2693
0.0121
0.0038
0.0045
0.0083
0.1219
KV2
0.2363
0.2351
0.0110
0.2766
0.1787
0.0121
0.0061
0.0091
0.0091
0.1082
27

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
8338
8.2
0,299
022
0.330
00125
0:2
8:00
00875
Liyer
Leveril
(a) KV8 Gat 0.014
(h) K8V4 6at 0.100
(c) K8V2 6at 0.401
0.80
0:25
0.25
0.28
0.2
023
0.13
02
0.18
0:3
0.53
16
25
23
25
10
15
23
23
20
(d) K4V8 c.t 0.168
(e) KV4 C. 0.207
(1) K4V2 eo: 0.453
2.38
7.00
1.75
1.59
3.2
2.08
0.75
0.75
0.42
0.25
Lewert
Leverit
(g) K2V8 ea: 0.882
(h) K2V4 co: 0.892
(i) KV2 eo: 0.962
Figure 13: Laser-wise relative attention output error e, of per-token-asym KV cache quantization with simulated offline quantization and dequantization (without error accumulation) the Llama-3.1-8B-Instte in the 0-shot GSM8K dataset. When the key quantization precision decreases to 2-bit, the layer- wise relative attention output error distribution significantly shifts. Especially, the errors of layer-3 and layer-1 significantly larger than other layers.
model and the first 20 prompts
F. Layer-wise Attention Score and Relative Output Error
In this section, visualize more layer-wise attention errors with KV cache quantization covering different LLMs, datasets, and KV cache quantization mode and precision. We select the first 20 prompts from the mathematical reasoning dataset GSM8K (Cobbe et al. 2021) and the AIGC multi-turn conversation dataset multiturn-softae (SoftAge-Al 2024). Tested and
-nnt-7-----nttut-O...... The layer-wise sensitivity to KV cache quantization of different LLMs are consistent to different prompts and datasets. Key cache quantization generally leads to the layer- wiser attention output error distribution shift. When the layer-wise attention error distribution significantly changes, the final model accuracy also dramatically degrades. For example, the perplexity and final generation accuracy of Qwen2.5-7B-unl Instruct dramatically degrades when the key quantization precision decreases 4-bit and 2-bit with the KIVI per-token-asyne quantization mode demonstrated in Table 2 and 6 The attention distribution of it also significantly shifts as shown in Figure 16 and 18
LLMs include Llama-3, 3.1.8BB--BBB Instructt Bwen2.5--B-Innsture
The most
As visualized in Figure 12 most layers of Qwen2.5-D S----B-I-nntte have a high ratio of non-sparse retrieval heads, which are sensitive to low-precision key cache guantization as analyzed in Section 4.4 As result, 4-bit or 2-bit key quantization leads to noticeable errors of attention score and critical KV identification in these layers with medium attention errors such layer-1, 12, and 21.
28

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
0.000
80006
2203115
05022
860306
SC00U5
Ligeric
Lager
(a) K8 eu: 5.0 X 10
(b) K4 e.: 6.7 X 10
(c) K2 e.: 3.26 X 10
8.00
0.0230
1.2
0.3
8.34
023
8.28
00225
8.00
025
Liyerr
(d) KV8 cot 0.017
(e) K8V4 fat 0.110
K8V2 <o: 0.418
0:10
0.65
0.25
0:00
0.22
8:38
022
0.19
10
30
23
(g) K4V8 e.: 0.199
(h) KV4 0.240
(i) K4V2 e.:. 0.484
2.58
2.5
2.2
3.7
1.5
5.25
0.5
(i) K2V8 eat 1.092
(k) K2V4 ear 1,103
()) K2V2 ozr 1.148
Figure 14: Layer-wise attention score errors e, and relative attention output error co of per-token-asym KV cache quantization with simulated offline quantization and dequantization (without error accumulation) the Llama-3.1-88 Instruct model and the first 20 prompts the AIGC multiturn softage dataset. When the key quantization precision decreases to 2-bit, the layer-wise relative attention output error distribution significantly shifts. Especially, the errors of layer-3, layer-1, and layer-27 27 are significantly larger than other layers.
29

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
0.007
0.030
66023
22022
03022
Mapar
Lager
(a) K8 eu: 4.0 x 10
(b) K4 ea: 6.7 X 10
(c) K2 e.: 3.26 X 10
0.723
2:34
0.0
224
0.48
023
0.2
1.00
0.22
025
(d) KV8 fat 0.008
(e) K8V4 Fat 0.110
K8V2 <o: 0.418
0122
0128
8:355
0.18
8.50
6.55
8:5
0.14
0280
0335
0.48
8399
0.15
10
2.5
20
25
20
25
(g) K4V8 e.: 0.138
(h) KV4 0.187
(i) K4V2 e.: 0.484
Layer/d
(i) K2V8 eo: 1.092
(k) K2V4 ear 1,103
(1) K2V2 cz 1.148
Figure 15: Layer-wise attention score emrrr 6, and relative attention output error c, of key per-channel-asym and value per-token-asym quantization with simulated offline quantization and dequantization (without error accumulation) the (lama-3.1-8B-Inttree model and the first 20 prompts in the AIGC multiturn softage dataset. When the key quantization precision decreases to 2-bit, the layer-wise relative attention output error distribution significantly shifts. Especially, the errors of layer-2 and 27 significantly larger than other layers.
30

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
84275
6.275
0.0330
1.625
6G225
0.320
14430
00933
24
K8 ca: 1.74 10"
(b) K4 Ca: 1.54 x 10
(c) K2 ca 4.68 x 10
0.23
8.28
025
0.2
0:2
Leyerid
Layerid
(d) KV8 eu: 0.033
(c) K8V4 eat 0.117
(f) K8V2 ext 0.446
36-
3.0
0.2
Laveill
(g) K4V8 ca: 0.292
(h) KV4 cot 0.324
(i) K4V2 cat 0.557
16
28-
20
25
10
15
22
15
(i) K2V8 e.: 0.948
(k) K2V4 C.:: 0.958
(I) KV2 Got 1.038
Figure 16: Layer-wise attention score e, and relative attention output eπrоr of per-token-asym KV cache quantization with simulated offline quantization and dequantization (without error accumulation) of the Qwee2.5--B-Instruee model and the first 20 prompts in the 0-shot GSM8K dataset. When the key quantization precision decreases to 4-bit or 2-bit, the layer-wise relative attention output error distribution significantly shifts. It also explains the performance degradation of the wikitext and other datasets. Especially, the errors of layer-3 and 13 are significantly larger than other layers. Note that in the 8-bit key cache quantization precision, only the first layer-0 and last layer-27 show significantly high errors, while in the 4-bit and 2-bit key cache quantization precision, the attention output errors of layer-3, 7, 10, 13, and 23 become noticeable compared with the first and last layers. Although these layers have relative simpler attention patterns as demonstrated in Figure 12] the low- low-precision b-t and 2-bit key cache quantization results in significantly token-level attention distribution shift.
Qwen2.5-7B-Instruu
31

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
011910
EX0L50
0.0020
10012
660000
220102-
60977
000310
0.0000
020052
8.0000
Leyeria
Leser 8
(a) K8 ca: 1.8 x 10
(b) K4 ea: 1.68 x 10
(c) K2 ea: 5.00 x 10
2.2
228
0.1
0.3
0.45
0.48
0.2
Layerkl
(d) KV8 co: 0.031
(e) K8V4 ea: 0.110
(f) K8V2 co: 0.427
68-
20
25
10
15
22
10
15
(g) K4V8 e.: 0.280
(h) KV4 0.310
(i) K4V2 c.:: 0.531
112
0.-
(i) K2V8 eo: 0.901
(k) K2V4 cg: 0.909
(l) K2V2 cg: 0.961
Figure 17: Lase attention score ea and relative attention output error e, of aer-token-asym KV cache quantization with simulated offline quantization and dequantization (without error accumulation) of the Qwwn22.-77-Inntruct model and the first 20 prompts in the AIGC multiturn softage dataset. The layer-wise attention error shift is similar to Figure [16] indicating that the laser-wise sensitivity to KV cache quantization is independent of the input prompts and even domains.
32

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
11003
50020
848200
8004
52012
Lapar 8
14ya 8
Layer/2
(a) K8 eu: 8.0 x 10
(b) K4 eu: 8.3 x 10-5
(c) K2 e.: 3.92 x 10
0.38
0.2
132
0.5
0.79
3.4
8.58
0.34
0.32
2.2
0:4
2.0
234
Lxwertl
Lay
(d) KV8 6ac 0.017
(e) K8V4 fot 0.101
(f) K8V2 <o: 0.424
0.291
0.80
0281
0335
8.25
0.48
0.122
0350
035
20
25
16
25
(g) K4V8 e.: 0.131
(h) KV4 0.174
(i) K4V2 e.:. 0.454
Layerid
(i) K2V8 eo: 0.679
(k) K2V4 ea: 0.696
(l) K2V2 ea: 0.838
Figure 18: Layer-wise attention score ea and relative attention output error e, of key per-channel-asym and value per-token-asye quantization with simulated offline quantization and dequantization (without error accumulation) the -wen2.5-7B-Instruct model and the first 20 prompts the AIGC multiturn softage dataset. Key quantization along the channel dimension significantly affects the distribution of critical layers for 4-bit and 2-bit precision compared with those in Figure 17 The averaged attention output errors e, under the same KV precision pairs also dramatically reduced.
33

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
0.210
0.0000
2002
000215
66038
000020
86875
0c000
008006
00025
0.200
2201
0.000
Laperiaa
Liper
Lager
(a) K8: 5.90 X 10
(b) K4: 8.51 X 10
(c) K2: 4.04 X 10
0.60
036
0.34
022
8.48
0.22
020
0.5
0.03
0.08
(d) KV8 fot 0.013
(e) K8V4 fot 0.102
(f) K8V2 co: 0.411
0.75
0.50
8.28
8.48
0.23
0:5
0.83
0.77
0.99
0.5
25
20
25
20
(g) K4V8 e.; 0.149
(h) KV4 0.191
(i) K4V2 e.:: 0.453
2.0
25
Layerid
(i) K2V8 eo: 0.823
(k) K2V4 egt 0.837
(I) KV2 eat 0.939
Figure 19: Layer-wise attention score 6, and relative attention output error e, of per-token-asym KV cache quantization with simulated offline quantization and dequantization (without error accumulation) of the Mistral-BB-Instrucct and the first prompts in the 0-shot GSM8K dataset. When the key quantization precision decreases 2-bit, the layer-wise relative attention output error distribution significantly shifts. Especially, the emors of layer- 2, 3, and 4 are significantly larger than other layers.
model
34

---

KVTuner: Sensitivity-Aware Layer- Wise Mixed-Precision KV Cache Quantization
Table 14: KIVI-HQQ KV cache quantization results of different precision and L.M models on the GSM8K few-shot Co'ss as multiturn conversation dataset.
GSM8K
GSM8K
Precision
Average
Average
4-shot
8-shot
16-shot
4-shot
8-shot
16-shot
3aama-3-8--Intruuee
Qwen2.5-7B-Inttee
BFI6
0.7794
0,8006
0.7847
0.7882
0.6998
0.7377
0.7506
0.7294
KV8
0.7801
0.8006
0.7824
0.7877
0.6801
0.7369
0.7460
0.7210
K8V4
0.7688
0.7862
0.7809
0.7786
0.6793
0.724
0.7468
0.7167
K8V2
0.7566
0.7763
0.7642
0.7657
0.6801
0.7491
0.7437
0.7243
K4V8
0.7445
0.7695
0.7498
0.7546
0.0076
0.0038
0.0053
0.0056
KV4
0.7422
0.7688
0.7384
0.7498
0.0038
0.0053
0.0023
0.0038
K4V2
0.7346
0.7437
0.7430
0.7404
0.0061
0.0023
0.0038
0.0041
K2V4
0.0152
0.0167
0.0159
0.0159
0.0045
0.0045
0.0023
0.0038
KV2
0.0159
0.0144
0.0152
0.0152
0.0023
0.0015
0.003
0.0023
Bittral-B--Intructt-e
Sween25-Mathh-7B-nnstree
BF16
0.5019
0.4890
0.4973
0.4961
0.8901
0.8666
0.8658
0.8742
KV8
0.5042
0.4890
0.4966
0.4966
0.8901
0.8658
0.8666
0.8742
K8V4
0.5064
0.4890
0.4913
0.4956
0.8931
0.8688
0.8628
0.8749
K8V2
0.4837
0.4663
0.4632
0,4711
0.8719
0.8741
0.8491
0.8650
K4V8
0.4754
0.4701
0.4534
0.4663
0.0500
0.0576
0.0697
0.0591
KV4
0.4875
0.4754
0.4822
0.4817
0.0455
0.0516
0.0796
0.0589
K4V2
0.4428
0.4503
0.4579
0.4503
0.0425
0.0516
0.0607
0.0516
K2V4
0.0258
0.0288
0.0250
0.0265
0.0023
0
0.0008
KV2
0.0190
0.0220
0.0208
0.0206
0.0023
0.0008
0.0015
0.0015
Owen2.5--B-Instrrun
Qwen2.5-14
14B..--1BBIIsttruc
BF16
0.5732
0.5997
0.6459
0.6063
0.7536
0.7862
0.8180
0.7859
KV8
0.583
0.6035
0.6353
0.6073
0.7491
0.7877
0.8158
0.7842
K8V4
0.5603
0.5967
0.6513
0.6028
0.7551
0.7953
0.8264
0.7923
K8V2
0.5133
0.5481
0.5997
0.5537
0.743
0.7733
0.8029
0.7731
K4V8
0.5118
0.5057
0.5049
0.5075
0.7430
0.7779
0.7998
0.7736
KV4
0,5080
0.4845
0.4837
0.4921
0.7339
0.7908
0.8112
0.7786
K4V2
0.4587
0.4124
0.4170
0.4294
0.7475
0.7733
0.7953
0.7720
K2V4
0.0083
0.0061
0.0136
0.0093
0.0220
0.0144
0.0174
0.0179
KV2
0.0061
0.0076
0.0076
0.0071
0.0288
0.0152
0.0167
0.0202
Swen2.5-3B-Instruut--ae
Swen2.5-32B-Instruu
BF16
0.5656
0.6209
0.6399
0.6088
0.7619
0.7809
0.7961
0.7796
KV8
0.5686
0.6149
0.6550
0.6128
0.7650
0.7877
0.8021
0.7849
K8V4
0.5747
0.608
0.6406
0.6078
0.7726
0.7801
0.7998
0.7842
K8V2
0.5466
0.5694
0.6149
0.5770
0.7384
0.7703
0.7877
0.7655
K4V8
0.4845
0.4564
0.4443
0.4617
0.7597
0.7794
0.8135
0.7842
KV4
0.4845
0.4807
0.4352
0.4668
0.7680
0.7718
0.8097
0.7832
K4V2
0.4177
0.3730
0,3518
0.3808
0.7559
0.7733
0.7801
0.7698
K2V4
0.0114
0.0091
0.0053
0.0086
0.0379
0.0281
0.0311
0.0324
KV2
0.0167
0.0114
0.0129
0.0137
0.0258
0.0136
0.0311
0.0235
35


Reviews:
[{"review_id": "UAaPgyclLO", "reviewer": "Reviewer_fdwh", "summary": "The authors propose KVTuner, a sensitivity-aware layer-wise mixed-precision KV cache quantization framework for LLM inference. KVTuner addresses key challenges in KV cache quantization, including layer-wise sensitivity to quantization errors, high overhead of fine-grained online adjustments, and inflexibility across different LLM architectures. Instead of applying uniform quantization across all layers, KVTuner performs an offline search for optimal layer-wise key and value precision pairs (e.g., K8V4, K4V2) using multi-objective optimization (MOO). This search considers both memory constraints and model accuracy. The precomputed precision pairs are then applied directly during inference, reducing computational overhead while maintaining nearly lossless accuracy.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the thoughtful feedback and constructive critiques. Below, we address each concern and outline planned revisions to strengthen the paper:\n\n---\n# 1. Computational cost\n**The profiling and layer-wise KV cache precision tuning are completely offline and no online overhead for precision selection is introduced.** Only online quantization for the newly generated key and value token using the calibrated precision similar to uniform KV precision is applied. \n\nWe only perform 200 rounds of offline multi-objective optimization search with limited data, which is efficient and **one-time cost**. This is a huge advantage compared with other mixed-precision approaches that incorporate token-importance ranking for KV selection.\n**The two-stage search space pruning only takes seconds.** The MOO search with pruned search spaces is the most time consumping part, which performs 200 rounds of the target LLM on the selected 200 prompts. The optuna framework is also quite efficient. **The simple huggingface transformers may takes hours** in the above settings with single batch inference in Nvidia RTX 4090. The offline tuning time cost is acceptable compared with the saved cost during large-scale online service and model pretraining cost. In addition, better hardwares, torch graph compiling, and multi-batch inference during tuning may reduce the tuning cost to less than one hour.\n\n---\n# 2. Latency experiment\nThe layer-wise FLOP cost difference is mainly caused by efficiency difference of the KV cache precision pairs.  The model-level efficiency reflects the overall effects of layer-wise efficiency of all KV cache precision pairs. The memory movement cost from CPU sto GPUs linearly increases with the KV cache size in most case and attention is normally memory bounded. We also report the total model-level throughput comparison of Llama-3.1-8B-Instruct using the searched config in Table 5 as below.  The hardwares are Nvidia RTX 4090 24G. **Compared with KIVI-KV8, the throuhgput of KVTuner-C3.25 can be improved by 16.79%~21.25%.**\n|BS, inputLen|KV8(baseline)|K8V4|KV4|K4V2|KVTuner-C4.92|KVTuner-C3.25|\n|-|-|-|-|-|-|-|\n|64,128|3836|4193|4567|4697|4240(10.53%)|4652(**21.25%**)|\n|16,512|1102|1205|1275|1304|1239(12.41%)|1296(**17.55%**)|\n|8,1024|549|597|632|645|600(9.22%)|641(**16.79%**)|\n\n---\n# 3. Long context effectivenss\nWe compare KVTuner with the basedlines KIVI-8, KIVI-4, our proposed variant KIVI-K8V4, and per-token-asym ones in the 20 Longbench datasets and the averaged scores are as below.  **KVTuner pushes KV cache quantization for the nearly lossless long context generation to 3.25-bit, outperforming the uniform KV precision.**\n### Qwen2.5-7B-Instruct KIVI\n |BF16|KIVI-8|KIVI-K8V4|KIVI-4|KVTuner-C4.92|KVTuner-C3.25|\n|-|-|-|-|-|-|\n|0.7956|0.7992|0.8001|0.7723|0.7956|0.7903|\n\n### Qwen2.5-7B-Instruct per-token-asym\n\n|BF16|KV8|K8V4|KV4|KVTuner-C5.0|KVTuner-C4.0|\n|-|-|-|-|-|-|\n|0.7956|0.7971|0.7953|0.6343|0.8005|0.7960|\n\n---\n# 3. Global optimum\nDue to the complex and nonlinear dependency of error accumulation, there are no theoretically global optimization solutions for the NP-hard problem. The MOO formulation balances Pareto-optimal solutions rather than seeking global optimality. The two-stage search space pruning aims to help MOO converge and reduce searching cost, comparing Figure 5 and 6.\n\n---\n# 4. Scaling over batch size\nWhen the input and output sequence lengths are fixed at 512 and 128, respectively, the following table presents the throughput (in Token/s) of KVTuner.\n\n|BS|4|8|16|32|64|80|\n|-|-|-|-|-|-|-|\n|Tokens/s|494|938|1776|2889|4655|4964|\n\n---\n# 5. Integration with eviction\nWe agree that integrating with eviction methods is promising and leave it in our future work. KVTuner is fully compatible with KV cache eviction methods including StreamingLLM, H2O, and SnapKV, because quantization and eviction are two orthogonal approaches.\n\n---\n# 6. MHA\nWe mainly test on LLMs with grouped query attention (GQA), because most recent and powerful models are GQA based and GQA is a variant of MHA. The layer-wise sensitivity to KV cache quantization is the inherent property of LLM with multi-layer transformers. In addition, key and value in attention layers have different sensitivity to quantization. There are the only two assumptions of KVTuner. We also analyze MLA models such as Deepseek-v2-lite-chat, in which we also observed similar layer-wise patterns.\n\n---\n# Conclusion\nKVTuner’s practical impact lies in its deployability: it requires no inference-time overhead and achieves near-lossless accuracy, making it a compelling solution for production LLM systems and most LLM acceleration hardwares.\nIn addition, we study the underlying mechasim of the higher importance of key cache. We also theoretically analyze that the layer-wise sensivity of attention heads to KV cache quantization strongly correlates with attention patterns, which is novel and may provide more insights to model design and compression.", "claims_and_evidence": "The paper claims that KVTuner significantly improves LLM inference efficiency while maintaining accuracy close to full-precision KV caching. As shown in Table 7, KVTuner-C6 achieves a 38.3% throughput improvement compared to KV8, and KVTuner-C3 achieves an even higher 76.4% improvement. However, the selection method may introduce additional computational complexity. In lines 275-295, the authors discuss how KVTuner avoids online decision-making overhead.", "methods_and_evaluation_criteria": "The methodology is well-structured and based on a layer-wise sensitivity analysis of KV cache quantization. The evaluation uses standard mathematical reasoning benchmarks such as GSM8K and GPQA, which are appropriate for testing the impact of quantization errors. However, as noted in lines 220-250, additional profiling of the computational overhead per layer and the impact on inference latency in real-world applications (e.g., batched inference on vLLM) would strengthen the claims. A comparison of layer-wise FLOP costs before and after applying KVTuner’s selection would provide a clearer picture of its computational efficiency.", "theoretical_claims": "The paper correctly identifies that key cache quantization errors accumulate across both model layers and generation steps, leading to significant degradation in long-context inference. The discussion in lines 330-350 formalizes the optimization problem for selecting layer-wise precision pairs, but it does not analyze whether the proposed selection strategy guarantees global optimality. Additionally, while KVTuner reduces memory usage, it does not completely eliminate online computational overhead.", "experimental_designs_or_analyses": "The experiments comprehensively evaluate KVTuner across different models (Llama-3.1-8B, Qwen2.5-7B, Mistral-7B) and various KV precision configurations. The results in Table 5 show that KVTuner maintains accuracy while achieving significant memory savings. However, a few aspects could be further explored.", "relation_to_broader_scientific_literature": "The paper is well-situated in the literature on KV cache quantization and memory-efficient LLM inference. It correctly cites works on uniform KV quantization (KV8, KV4) and hybrid eviction strategies. However, as discussed in lines 275-295, it does not sufficiently compare with recent approaches that integrate quantization with eviction (e.g., SnapKV. A direct comparison would strengthen the positioning of KVTuner as a practical alternative to existing methods.", "essential_references_not_discussed": "The paper does not discuss alternative mixed-precision approaches that incorporate token-importance ranking for KV selection.", "other_strengths_and_weaknesses": "The paper makes an important contribution to memory-efficient LLM inference with strong empirical results. However, there are areas for improvement:\n\n1) The additional computational cost of  computation is not fully analyzed.\n\n2) The practical impact on multi-head attention efficiency is unclear.\n\n3) The method’s effectiveness in extremely long-context settings (e.g., 100K+ tokens) is not evaluated.", "other_comments_or_suggestions": "Including a runtime profiling analysis of KVTuner’s selection method would strengthen claims about efficiency.", "questions_for_authors": "What is the additional FLOP overhead per generation step compared to traditional KV quantization methods?\n\nHow does KVTuner scale with batch size increases? \n\nCan KVTuner be integrated with KV cache eviction methods like SnapKV for improved memory efficiency? Have you considered hybrid approaches?\n\nHow does KVTuner handle extreme long-context inference (e.g., 100K+ tokens)? Does performance degrade due to accumulated quantization errors?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z8b9BDCM31", "reviewer": "Reviewer_FF8e", "summary": "This paper proposed an innovative quantization technique for KV caches, which can reduce the inference throughput with a negligible quality drop in the output.\n\nThis paper's key insight is that the key cache is more important than the value cache in terms of reducing the quantization error. Its key contribution is an adaptive framework called KVTuner that can tune the KV quantization configurations offline and use them online for different objectives.\n\nThe experiments show that the solution can produce similar quality compared with the state-of-the-art solutions. And its inference efficiency is higher than a default KV8 quantization.\n\n## Update after rebuttal\nThank the authors for the clarifications. I will maintain the decision of \"weak accept\".", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and constructive feedback. We sincerely appreciate your acknowledgment of the innovativeness and feasibility of the proposed methodology and theoretical analysis of attention patterns to KV cache quantization, as well as your recognition of the comprehensive implementation of the experimental investigations.  Below, we address your concerns and suggestions point by point:\n\n---\n# 1. Efficiency evaluation clarifications\nWe acknowledge the lack of clarity in our efficiency evaluation setup. Here are key clarifications:\n* Baseline Implementation: In Table 7, the static SOTA baseline KIVI and our solutions are both based on the official KIVI code with flashattention enabled in their kernels during decoding. KV8 and KV4 are KIVI-8 and KIVI-4, respectively. We will correct them in the revised version. We slightly modify the KIVI CUDA kernels to support INT8 and mixed precision of key and value cache. KIVI, per-token-asym, and our KVTuner is a online static method and compatible with FlashAttention. In addition, It uses lightweight post-training calibration and offline bit-width selection. This design choice ensures compatibility with existing inference frameworks. However, other online mixed precision KV cache quantization methods with attention scores based online token importance estimation is normally not compatible with flashattention.\n\n* New GQA efficiency results with KVTuner: We modify the KIVI code to support GQA models including Llama-3.1-8B-Instruct. We thus report the total model-level throughput comparison of Llama-3.1-8B-Instruct using the searched config in Table 5.  The hardwares are Nvidia RTX 4090 24G. **Compared with KIVI-KV8, the throuhgput of KVTuner-C3.25 can be improved by 16.79%~21.25%.**\n|BS, inputLen|KV8(baseline)|K8V4|KV4|K4V2|KVTuner-C4.92|KVTuner-C3.25|\n|-|-|-|-|-|-|-|\n|64,128|3836|4193|4567|4697|4240(10.53%)|4652(21.25%)|\n|16,512|1102|1205|1275|1304|1239(12.41%)|1296(17.55%)|\n|8,1024|549|597|632|645|600(9.22%)|641(16.79%)|\n\n* Throughput definition:  We follow the same settings and definitions of KIVI. Throughput is defined as the the number of tokens generated per second (measured end-to-end, including quantization/dequantization overhead). For example, if the batch size is 128 and one generation step takes 50ms, the throughput is 128 * 1000 / 50 = 2560 tokens/s.\n\n---\n# 2. Discussion of conflicting literature\nWe appreciate the opportunity to clarify our position relative to prior works (Atom and KIVI):\nAtom claims that KV cache is more amerable to quantization than activation matrices in Section 4.4 and utilizes INT4 precision for both key and value. **The importance difference of key and value is not clearly discussed in Atom. KIVI implicitly implies that key is more important than value by applying more complex per-channel per-group quantization to key and simple per-token quantization to value.**\n\nThe conclusion that key cache is normally more important than key cache is validated with extensive empirical studies, which include final perplexity on the **wikitext** dataset in Table 2, layer-wise attention errors on the **GSM8K** dataset in Table 3 (Page 4), final model accuracy on the **general CEVAL, MMLU, TriviaQA, RACE, and TruthfulQA** datasets with both **per-token-asym and KIVI** quantization modes in Table 11 (Page 25), layer-wise attention score and output errors with the key per-channel-asym and value per-token-asym quantization on **AIGC** multiturn softage dataset of **Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct** in Figure 10 (Page 18) and Figure 13 (Page 21).\n\n---\n# 3. OOM with 8K sequence\nWe use 4 batch size with 8K tokens (~4GB). The OOM error for the 8K FP16 setup may arise from an unoptimized memory allocation strategy in our prototype implementation (e.g., redundant intermediate tensors were not freed). The default efficiency testing in the KIVI repo does not enable FlashAttention during prefilling, which may also result in the OOM issue with 8K long sequence.\n\n---\n# Conclusion\nThe proposed layer-wise KV cache precision pair tuning naturally suits the layer-wise sensitivity to KV cache quantization, making KVTuner a practical solution to reduce the memory usage and improve inference efficiency of LLMs with various sensitivities. KVTuner successfully pushes the nearly lossless KV cache quantization in complex mathematical and sciencitific tasks to 3.25-bit for Llama-3.1-8B-Instruct and 4-bit for sensitive Qwen2.5-7B-Instruct. KVTuner also greatly narrows the performance difference between the simple per-token-asym and accurate KIVI quantization modes, even when using overall similar low-precision settings.\n\nMany KV cache quantization approaches are proposed recently, but the correlation to LLM attention patterns is not well studied. We theoretically prove that the sparse streaming heads are robust to KV cache quantization than sensitive retrieval heads, which is the cause of layer-wise sensitivity to KV cache quantization.", "claims_and_evidence": "The key insight of the paper (the key cache is more important than the value cache in terms of reducing the quantization error) is backed up with good experimental results (20 samples from the standard dataset on an up-to-date llama 8b model).", "methods_and_evaluation_criteria": "The evaluation section contains 2 parts: quality evaluation and efficiency evaluation.\n\nThe quality evaluation is comprehensive and the metrics are suitable for the use case.\n\nHowever, the efficiency evaluation needs to be improved on the following points:\n\n1. The setup of the benchmark is not clear. The paper does not mention how the baseline and their solution are implemented. Is there any new CUDA kernel in the paper's solution? Is the baseline using state-of-the-art attention frameworks like FlashAttention?\n2. The definition of \"throughput\" is not clear. Does it refer to the speed of token generation (i.e., tokens per second), or number of finished requests per second, or something else?", "theoretical_claims": "The theoretical claims and the algorithm design make sense and there are no obvious problems.", "experimental_designs_or_analyses": "The experimental design of quality evaluation is comprehensive and good.\n\nHowever, the design of the efficiency evaluation is not clear (because of the problems mentioned above)", "relation_to_broader_scientific_literature": "There are plenty of works focusing on KV cache compressions. Some of them suggest the key cache is less important than the value cache [1], and some of them suggest we should have different quantization methods for the key cache and value cache respectively [2].\n\nThe key claim in this paper seems to conflict with those prior works. Please discuss the difference and the potential reason.\n\n\nReferences\n----\n[1] Zhao, Yilong, et al. \"Atom: Low-bit quantization for efficient and accurate llm serving, 2024.\" URL https://arxiv. org/abs/2310.19102.\n[2] Liu, Zirui, et al. \"Kivi: A tuning-free asymmetric 2bit quantization for kv cache.\" arXiv preprint arXiv:2402.02750 (2024).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "In Table 7, the 8K FP16 setup has an OOM error. I'm wondering why it could happen. 8K context length only corresponds to ~1GB of KV cache when using llama-7B models. If the GPU is 48GB, how could it be OOM? Could it be because there is something wrong with the experiment?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ah85Aeb9Ls", "reviewer": "Reviewer_uJyY", "summary": "This paper proposes a multi-objective optimization-based algorithm to search for the optimal layer-wise mixed precision KV cache quantization configuration. The authors observe that key caches generally require more bits for quantization than value caches, and thus propose allocating more bits to the key cache. Additionally, they find that different layers exhibit varying sensitivity to KV cache quantization. To address this, they apply mixed precision across different layers and employ a search algorithm to identify the optimal configuration. To reduce the search space, they incorporate pruning and clustering strategies. Experimental results across various LLMs and tasks demonstrate the accuracy and latency benefits of their approach.\n\n## update after rebuttal\n\nThe authors addressed my concerns regarding the baseline comparison and provided additional experimental results to support the effectiveness of their method. Therefore, I will raise my rating to a weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thorough feedback. Below, we address the concerns raised and outline revisions to improve the clarity and rigor of our work.\n\n---\n# 1. Key Cache Importance \n* The reviewer correctly observes that in certain layers (e.g. Layer 0,1,2,31 of Llama-3.1-8B-Instruct), the proposed intra-layer KV cache precision pair pruning algorithm selects K4V8 rather than K8V4. We will clarify in the revised manuscript that Claim 1 (\"key cache is generally more important\") reflects an overall trend across layers. KVTuner assigns more bits to value cache than key cache in specific layers.\nThose layers where higher value bitwidths outperform higher key bitwidths normally have more streaming head patterns than retrieval heads, where key is more robust to low-bit quantization. Therefore, more memory bitwidths should be assigned to value cache, which is more sensitive in this case.\n\n* **This phenomenon verifies our observation and theoretical analysis about the strong correlation of KV cache quantization errors and attention patterns in Section 4.4.** In addition, the proposed KVTuner is adaptive to the inherent model structure patterns and the proposed layer-wise KV cache quantization precision tuning makes sense. \n\n---\n# 2. Comparison with KIVI\nWe appreciate the reviewer’s attention to fairness in comparisons. **KVTuner offers more flexible accuracy and efficiency tradeoff, which is not available in the static, uniform and non mixed-precisionbaselines KIVI and per-token-asym.**\n\n* The accuracy of KVTuner-C3.44 only decreases by 0.52% but the memory usage is reduced by 14% compared with KIVI-4 in Qwen2.5-3B-Instruct. We also push the frontier of nearly lossless KV cache compression (only 0.04% accuracy loss than the BP16 baseline) to 3.44-bit in this model.\n* In Table 11 (Page 24~25), we compare more uniform precision including K8V4 (C6), K8V2 (C5), and K4V2 (C3). Llama-3.1-8B-Instruct and Mistral-7B-v0.3 are more robust to low bit key quantization than Qwen2.5-7B-Instruct. However, Qwen2.5-7B-Instruct outperforms others in terms of accuracy.  \n* In Table 5 and 6 (Page 8), the baselines INT4 KIVI and per-token-asym quantization lead to significant (67% and 26%) accuracy degradation in Qwen2.5-7B-Instruct. However, **KVTuner successfully reduces the accuracy loss to 16% with lower 3.92-bit memory usage and 0.18% with similar 4-bit memory usage, respectively. It indicates that the accuracy improvement of KVTuner is noticeable and KVTuner offers more robustness and flexibility to sensitive but powerful models.** \n* Figure 5 also visualizes the accuracy and equivalent KV cache bitwidth of different layer-wise KV cache precision pairs during KVTuner offline searching. The red circles are uniform precision across all layers. From Figure 5, we can easily observe more Pareto-optimal layer-wise KV cache precision pairs than uniform ones. **Especially, the accuracy of the uniform KV4, K4V2, and K2V4 is around 0% in Qwen2.5-7B-Instruct, while the searched equivalent 4-bit and 3-bit configs of KVTuner achieve 80% and 50% accuracy, respectively. It is a huge improvement in terms of accuracy with similar memory usage.**\n* Latency Baseline Clarification\nIn Table 7, \"KV8\" refers to 8-bit KIVI quantization. We thus report the total model-level throughput comparison of Llama-3.1-8B-Instruct using the searched config in Table 5.  The hardwares are Nvidia RTX 4090 24G. Compared with KIVI-KV8, the throuhgput of KVTuner-C3.25 can be improved by 16.79%~21.25%.\n|BS, inputLen|KV8(baseline)|K8V4|KV4|K4V2|KVTuner-C4.92|KVTuner-C3.25|\n|-|-|-|-|-|-|-|\n|64,128|3836|4193|4567|4697|4240(10.53%)|4652(**21.25%**)|\n|8,1024|549|597|632|645|600(9.22%)|641(**16.79%**)|\n---\n# 3. Long context evaluation\nWe compare KVTuner with the basedlines in the 20 Longbench datasets and the averaged scores are as below. The conclusion is that KVTuner push the nearly lossless long context generation to 3.25-bit.\n\n### Qwen2.5-7B-Instruct KIVI\n |BF16|KIVI8|KIVI-K8V4|KIVI4|KVTuner-C4.92|KVTuner-C3.25|\n|-|-|-|-|-|-|\n|0.7956|0.7992|0.8001|0.7723|0.7956|0.7903|\n\n### Qwen2.5-7B-Instruct per-token-asym\n\n|BF16|KV8|K8V4|KV4|KVTuner-C5.0|KVTuner-C4.0|\n|-|-|-|-|-|-|\n|0.7956|0.7971|0.7953|0.6343|0.8005|0.7960|\n\n---\n# 4. Per-channel vs. per-token quantization\nKIVI-HQQ also supports key per-channel quantization by tuning the axis_key config. KIVI requires new operators and careful management as discussed in Line 58~69. In contrast, per-token-asym can be easily implemented and is supported in common inference frameworks such as LMDeploy.\n\n---\n# 5. Layer-wise sensitivity analysis\nWe also study the layer-wise sensitivity to KV cache quantization of Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct with both KV per-token-asym and KIVI like quantization modes on the non-math AIGC multiturn softage dataset in Figure 9 (Page 17), 10 (Page 18), 12 (Page 20), and 13 (Page 21). The sensitive layers are consistent with those in math tasks e.g. Figure 8.", "claims_and_evidence": "The authors make three claims:\n\n1. The key cache is more important than the value cache.\n\n2. Different layers exhibit varying sensitivity to KV cache quantization.\n\n3. Their proposed method outperforms baseline methods that use static quantization bits for all KV caches, such as KIVI.\n\nHowever, claims 1 and 3 are problematic.\n\nFor claim 1, Table 4 shows that in some layers, the value cache is actually more important than the key cache. The authors should explain why this occurs and clarify whether claim 1 holds universally or if it depends on specific conditions.\n\nFor claim 3, it is not entirely clear that their method outperforms KIVI. In Table 5, KVTuner-C4.90 performs slightly worse than KIVI-4 for Llama-3.1-8B-Instruct, and KVTuner-C3.44 is worse than KIVI-4 for Qwen2.5-3B-Instruct. Moreover, the authors do not include a baseline of KIVI-3, making it difficult to provide a fair comparison between KIVI and the proposed method. For instance, although KVTuner-C3.25 outperforms KIVI-2 for Llama-3.1-8B-Instruct, KVTuner-C3.25 uses a higher bitwidth than KIVI-2, making the comparison unfair. This issue also applies to Table 6.\n\nRegarding latency, the baseline is unclear. Is KV8 referring to KIVI-8 or just standard 8-bit quantization? KIVI-n is used in Tables 5 and 6 but not in Table 7, which adds to the confusion. If KV8 is not KIVI-8, why not compare with KIVI-8? Furthermore, Llama2-7B is used for the latency comparison, but it is not included in the accuracy comparison, leaving the accuracy difference between KV8 and KVTuner-C6 unknown. Without this information, the latency comparison lacks context. It is difficult to assess whether the proposed method provides a better latency-accuracy trade-off than the baselines.", "methods_and_evaluation_criteria": "The proposed method is reasonable, given the varying importance of key/value caches and the different layer-wise sensitivity to KV cache. However, the evaluation lacks clarity, as the accuracy benefits are subtle, and the latency comparison, as previously mentioned, is problematic. Additionally, since KV cache quantization is particularly important for long-context generation, the authors should evaluate their method on long-context benchmarks, such as LongBench, to more effectively validate its performance.", "theoretical_claims": "All proofs are correct.", "experimental_designs_or_analyses": "The accuracy experiments have the issue of lacking appropriate baselines with roughly the same quantization bitwidth as their proposed method.\n\nThe latency experiment in Section 6.3 is problematic, as it does not provide the accuracy results for Llama2-7B, making it difficult to assess the accuracy-latency trade-off.\n\nIn Section 6.5, it is unclear which points in Figure 18 correspond to the unified precision configuration.", "relation_to_broader_scientific_literature": "The attempt to apply a multi-objective optimization algorithm to KV cache quantization in this work has the potential to broaden the application of MOO.", "essential_references_not_discussed": "All essential references are discussed.", "other_strengths_and_weaknesses": "1. Most experiments in this paper mention the use of per-token KV cache quantization, such as in Figures 2, 5, and 6. However, as demonstrated in KIVI, per-token key quantization performs significantly worse than per-channel quantization. Why not consistently apply per-channel quantization to the key?\n\n2. Table 2 compares word-perplexity for the KIVI-HQQ implementation. However, in the HQQ implementation, both the key and value are quantized per-token. Since using per-token quantization for the key naturally leads to greater loss compared to per-channel quantization, it is unreliable to draw conclusions about whether the key is more important than the value when per-channel quantization is not applied to the key and per-token quantization is used for the value.\n\n3. Section 4.5 aims to conclude that layer-wise sensitivity to KV cache quantization is an inherent characteristic of LLMs. However, the analysis is based only on prompts for math problems, and it is unclear whether this finding applies to general prompts, such as non-math tasks like retrieval and summarization.\n\n4. Although the search space is greatly reduced, it is still considerably large (15,625, as mentioned in Line 319). The search cost may still be high, yet there is no explanation provided regarding the search cost, such as the total search time.", "other_comments_or_suggestions": "Table 4 typo: Pateto -> Pareto", "questions_for_authors": "1. What is the search cost of the proposed method?\n\n2. The accuracy comparison is unreliable, as it lacks appropriate baselines with competitive quantization bitwidths for a fair comparison. Additionally, the accuracy improvement appears to be subtle.\n\n3. The latency comparison is unreliable, as it lacks the accuracy results for the Llama2-7B model, making it impossible to assess the latency-accuracy trade-off.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies KV cache efficiency, a critical topic for today's LLM research. It provides an adaptive framework that prioritizes the key cache over the value cache for quantization, which brings additional performance improvements. 

While the reviews are mixed, the rebuttal somewhat alleviated some of the concerns with one reviewer raising their rating. I agree with most of the reviewers that this result can benefit the community due to the importance of the topic and the promising experimental results. Thus, I recommend weak acceptance. If the paper is accepted, I strongly advise the authors to address all concerns raised by the reviewers.
Decision: Accept (poster)