Beyond Homogeneous Attention: Memory-Efficient LLMs via
  Fourier-Approximated KV Cache
Authors: name: Xiaoran Liu; name: Siyang He; name: Qiqi Wang; name: Ruixiao Li; name: Yuerong Song; name: Zhigeng Liu; name: Linlin Li; name: Qun Liu; name: Zengfeng Huang; name: Qipeng Guo; name: Ziwei He; name: Xipeng Qiu
Published: 2025-06-13 15:35:54

Abstract: Large Language Models struggle with memory demands from the growing Key-Value
(KV) cache as context lengths increase. Existing compression methods homogenize
head dimensions or rely on attention-guided token pruning, often sacrificing
accuracy or introducing computational overhead. We propose FourierAttention, a
training-free framework that exploits the heterogeneous roles of transformer
head dimensions: lower dimensions prioritize local context, while upper ones
capture long-range dependencies. By projecting the long-context-insensitive
dimensions onto orthogonal Fourier bases, FourierAttention approximates their
temporal evolution with fixed-length spectral coefficients. Evaluations on
LLaMA models show that FourierAttention achieves the best long-context accuracy
on LongBench and Needle-In-A-Haystack (NIAH). Besides, a custom Triton kernel,
FlashFourierAttention, is designed to optimize memory via streamlined
read-write operations, enabling efficient deployment without performance
compromise.