ICLR 2025

Published as a conference paper at ICLR 2025
PALU: KV-CACHE COMPRESSION WITH LOW-RANK
PROJECTION
Chi-Chih Chang1,3∗
Wei-Cheng Lin1∗
Chien-Yu Lin2∗
Chong-Yan Chen1
Yu-Fang Hu1
Pei-Shuo Wang1
Ning-Chi Huang1
Luis Ceze2
Mohamed S. Abdelfattah3
Kai-Chiang Wu1
1National Yang Ming Chiao Tung University
2University of Washington
3Cornell University
ABSTRACT
Post-training KV-Cache compression methods typically either sample a subset of
effectual tokens or quantize the data into lower numerical bit width. However,
these methods cannot exploit redundancy in the hidden dimension of the KV
tensors. This paper presents a hidden dimension compression approach called
Palu, a KV-Cache compression framework that utilizes low-rank projection to
reduce inference-time LLM memory usage. Palu decomposes the linear layers
into low-rank matrices, caches compressed intermediate states, and reconstructs
the full keys and values on the fly. To improve accuracy, compression rate, and
efficiency, Palu further encompasses (1) a medium-grained low-rank decompo-
sition scheme, (2) an efficient rank search algorithm, (3) low-rank-aware quan-
tization compatibility enhancements, and (4) optimized GPU kernels with oper-
ators fusion. Extensive experiments with popular LLMs show that Palu com-
presses KV-Cache by 50%, while maintaining strong accuracy and delivering up to
1.89× speedup on the RoPE-based attention module. When combined with quan-
tization, Palu’s inherent quantization-friendly design yields small to negligible
extra accuracy degradation, while saving additional memory than quantization-
only methods and achieving up to 2.91× speedup for the RoPE-based attention.
Moreover, it maintains comparable or even better accuracy (up to 1.19 lower
perplexity) compared to quantization-only methods. These results demonstrate
Palu’s superior capability to effectively address the efficiency and memory chal-
lenges of LLM inference posed by KV-Cache. Our code is publicly available at:
https://github.c...