Beyond Homogeneous Attention: Memory-Efficient LLMs via
  Fourier-Approximated KV Cache
Authors: name: Xiaoran Liu, name: Siyang He, name: Qiqi Wang, name: Ruixiao Li, name: Yuerong Song, name: Zhigeng Liu, name: Linlin Li, name: Qun Liu, name: Zengfeng Huang, name: Qipeng Guo, name: Ziwei He, name: Xipeng Qiu

Large Language Models struggle with memory demands from the growing Key-Value
(KV) cache as context lengths increase. Existing compression methods homogenize
head dimensions or rely on attention-guided token pruning, often sacrificing
accuracy or introducing computational overhead. We propose FourierAttention, a
training-free framework that exploits the heterogeneous roles of transformer
head dimensions: lower dimensions prioritize local context, while upper ones
capture long-range dependencies. By projecting the long-context-insensitive
dimensions onto orthogonal Fourier bases, FourierAttention approximates their
temporal evolution with fixed-length spectral coefficients. Evaluations on
LLaMA models show that FourierAttention achieves the best long-context accuracy
on LongBench and Needle-In-A-Haystack (NIAH). Besides, a custom Triton kernel,
FlashFourierAttention, is designed to optimize memory via streamlined
read-write operations, enabling efficient deployment without performance
compromise.