ICML 2025

KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache
Quantization for Efficient and Nearly Lossless LLM Inference
Xing Li * 1 Zeyu Xing * 2 Yiming Li 1 Linping Qu 1 Hui-Ling Zhen 1 Yiwu Yao 3 Wulong Liu 1
Sinno Jialin Pan 2 Mingxuan Yuan 1
Abstract
KV cache quantization can improve Large Lan-
guage Models (LLMs) inference throughput and
latency in long contexts and large batch-size
scenarios while preserving LLMs effectiveness.
However, current methods have three unsolved
issues: overlooking layer-wise sensitivity to KV
cache quantization, high overhead of online fine-
grained decision-making, and low flexibility to
different LLMs and constraints. Therefore, we
theoretically analyze the inherent correlation of
layer-wise transformer attention patterns to KV
cache quantization errors and study why key
cache is generally more important than value
cache for quantization error reduction. We fur-
ther propose a simple yet effective framework
KVTuner to adaptively search for the optimal
hardware-friendly layer-wise KV quantization
precision pairs for coarse-grained KV cache with
multi-objective optimization and directly utilize
the offline searched configurations during on-
line inference. To reduce the computational cost
of offline calibration, we utilize the intra-layer
KV precision pair pruning and inter-layer clus-
tering to reduce the search space. Experimen-
tal results show that we can achieve nearly loss-
less 3.25-bit mixed precision KV cache quantiza-
tion for LLMs like Llama-3.1-8B-Instruct and
4.0-bit for sensitive models like Qwen2.5-7B-
Instruct on mathematical reasoning tasks. The
maximum inference throughput can be improved
by 21.25% compared with KIVI-KV8 quantiza-
tion over various context lengths. Our code and
searched configurations are available at https:
//github.com/cmd2001/KVTuner.
*Equal contribution
1Huawei Noah’s Ark Lab 2The Chi-
nese University of Hong Kong 3Huawei Computing Product
Line.
Correspondence to: Xing Li <li.xing2@huawei.c...