Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache

## Abstract

Abstract Large Language Models struggle with memory demands from the growing Key-Value (KV) cache as context lengths increase. Existing compression methods homogenize head dimensions or rely on attention-guided token pruning, often sacrificing accuracy or introducing computational overhead. We propose FourierAttention , a training-free framework that exploits the heterogeneous roles of transformer head dimensions: lower dimensions prioritize local context, while upper ones capture long-range dependencies. By projecting the long-context-insensitive dimensions onto orthogonal Fourier bases, FourierAttention approximates their temporal evolution with fixed-length spectral coefficients. Evaluations on LLaMA models show FourierAttention achieves the best long-context accuracy on LongBench and Needle-In-A-Haystack (NIAH). Besides, a custom Triton kernel, FlashFourierAttention, is designed to optimize memory via streamlined read-write operations, enabling efficient deployment without performance compromise.

## 1 Introduction

Figure: Figure 1: Overview of FourierAttention.
Refer to caption: x1.png

Large Language Models (LLMs) have transformed natural language processing with breakthroughs in text generation, comprehension, and reasoning (OpenAI, 2023; Sun et al., 2024; OpenAI, 2024; Guo et al., 2025). However, their autoregressive decoding relies heavily on a memory-intensive Key-Value (KV) cache, leading to significant memory allocation as context lengths scale (Vaswani et al., 2017; Fu, 2024; Liu et al., 2025). This overhead limits LLM deployment in resource-constrained environments. While approaches like quantization and sparse attention have been explored to reduce memory needs, they often compromise accuracy or add complexity (Liu et al., 2024c; Hooper et al., 2024; Yuan et al., 2025). Developing memory-efficient methods that preserve performance remains critical for broader LLM applicability.

Existing training-free KV cache compression methods, like token eviction strategies (Xiao et al., 2024; Zhang et al., 2023; Li et al., 2024b), prune sequence subsets but overlook the heterogeneous roles of head dimensions, leaving dimension-aware allocation largely unexplored. Similarly, quantization methods (Liu et al., 2024c; Hooper et al., 2024; Duanmu et al., 2024) reduce memory by fixed bit-widths, and hidden dimension compression (Chang et al., 2024; Saxena et al., 2024) methods apply uniform ratios, both neglect their distinct contribution across dimensions (Liu et al., 2024b; Peng et al., 2024). These approaches treat head dimensions as homogeneous, static units rather than dynamically allocating resources based on their importance.

Another critical limitation of existing methods lies in their reliance on attention-guided strategies (Zhang et al., 2023; Li et al., 2024b). While these approaches enable selective token pruning with minimal accuracy degradation, they impose prohibitive memory and latency overheads due to attention score recalculation. We address this challenge by adapting the HiPPO framework (Gu et al., 2020), a mathematically grounded approach for long-sequence modeling. HiPPO approximates infinite-length sequences as compact finite states by projecting inputs onto finite-order orthogonal basis functions, such as polynomial bases or Fourier bases (Gu et al., 2020; He et al., 2023). This retains global critical and contextually vital patterns while filtering out redundant signals. By leveraging HiPPO’s theoretical foundations, we can bypass attention recomputation entirely, achieving both memory efficiency and computational efficiency.

Building on these insights, we introduce FourierAttention, a training-free KV cache compression framework using a translated Fourier transform. Departing from prior methods that uniformly process all head dimensions, FourierAttention identifies localized, context-insensitive dimensions in KV states and approximates their temporal evolution via a fixed set of orthogonal Fourier basis functions. By retaining only the dominant Fourier coefficients ($k\ll L$, where $L$ is the sequence length), our method projects sequences into a compact spectral representation. Unlike polynomial bases that are widely used in HiPPO, which require recurrent state updates, FourierAttention exploits the shift-invariance and temporal parallelism of Fourier transforms, allowing for efficient computation in a single pass. During decoding, a customized Triton kernel FlashFourierAttention is used to decompose KV cache states during attention calculation, minimizing memory overhead via streamlined read-write operations.

Our contributions can be summarized as follows:

- •
We reveal a bifurcation in Transformer head dimensions: lower dimensions prioritize local context, while upper ones capture long-range dependencies. This inspires us to compress long-context-insensitive dimensions without sacrificing contextual awareness.
- •
We introduce FourierAttention, which optimizes KV cache by projecting its temporal evolution onto a fixed set of orthogonal Fourier bases. This method efficiently eliminates redundant components while preserving contextual fidelity, achieving a balance between memory and computational efficiency.
- •
We evaluate FourierAttention’s performance on the LLaMA Series using LongBench and NIAH. Our FourierAttention achieves the best long-context performance on average while maintaining lower memory consumption.

## 2 Related Work

KV cache optimization is a crucial technique for enhancing efficiency in attention-based LLMs (Fu, 2024; Liu et al., 2025). As context length increases, the KV cache in LLMs grows linearly, creating substantial memory overhead that becomes a bottleneck for long-context applications. Beyond architectural modifications during pretraining (Ainslie et al., 2023; Liu et al., 2024a), existing training-free optimization approaches mainly involve token eviction or compression. The former discards tokens based on positional or attention patterns, including StreamingLLM (Xiao et al., 2024), H2O (Zhang et al., 2023), SnapKV (Li et al., 2024b), and PyramidKV (Cai et al., 2024), while the latter compresses KV cache through quantization or low-rank projection, such as KIVI (Liu et al., 2024c), KVQuant (Hooper et al., 2024), and Palu (Chang et al., 2024). However, these methods lack fine-grained consideration of different head dimensions in KV cache, applying uniform optimization across all dimensions. In contrast, our FourierAttention compresses most dimensions to a fixed length while preserving long-context-sensitive dimensions, effectively reducing KV cache size while maintaining the original long-context capabilities.

Figure: (a) LLaMA3.1-8B
Refer to caption: x2.png

## 3 Methodology

Due to considerations of different head dimensions of KV cache, we propose FourierAttention. In FourierAttention, most dimensions of the KV cache are compressed to a fixed length through translated Fourier transform, as shown in Figure [1](https://arxiv.org/html/2506.11886v1#S1.F1).

### 3.1 Head Dimension Specialization

We analyze the heterogeneous sensitivity of transformer head dimensions to varying context lengths. By visualizing attention scores across 128 dimensions in LLaMA architecture (Figure.[2](https://arxiv.org/html/2506.11886v1#S2.F2)), we identify a bifurcation in attention patterns: the first 70 dimensions (0–69) exhibit sharp focus on short-range context, with score distributions concentrated on recent tokens, while the latter 58 dimensions (70–127) maintain a persistent bias toward initial "sink tokens"—positional embeddings that serve as static reference points. This divergence suggests distinct contextual roles encoded within head dimensions, where specialized subsets prioritize local versus global signal retention.

To further validate this hypothesis, we evaluate the model on a Needle-In-A-Haystack retrieval task across sequences of up to 32,000 tokens. As shown in Figure 1(a), the baseline model achieves perfect retrieval accuracy (100.0). Introducing Gaussian noise to the first 70 dimensions, performance remains robust (97.02), confirming their limited role in long-range dependency resolution. Conversely, perturbing the latter 58 dimensions catastrophically reduces accuracy to 53.20 on average, with failures consistent across all tested depths and context lengths (Figure 1(b) mirrors this trend). This stark contrast empirically demonstrates that upper dimensions in transformer are indispensable for retaining long-range information, while lower dimensions specialize in local context encoding. These findings provide critical insights for optimizing memory-efficient architectures, as strategically prioritizing dimensions specialized in long-range retention enhances contextual awareness within memory limits. For more details on dimension selection, please refer to Section [3.4](https://arxiv.org/html/2506.11886v1#S3.SS4).

### 3.2 Preliminary: HiPPO Framework

Inspired by HiPPO (Gu et al., 2020), we compress these less context-sensitive dimensions into fixed-length states to reduce KV cache storage. Under the HiPPO framework, an infinitely long sequence, $f_{1\cdots{L}}$, can be approximated by finite-length states, $\bm{c}\in\mathbb{R}^{k}$, as the combining coefficients of finite-order basis functions. HiPPO designs different state update equations for various basis functions under different measure functions, such as LegT based on Legendre Polynomial in a translated fixed window size. Among these methods, FourierT measure based on Translated Fourier Transform is most suitable for token-wise parallelism in transformers, because it can be expressed in matrix form and performed independently in different order states. Therefore, we adopt FourierT to compress cache, $\bm{K},\bm{V}\in\mathbb{R}^{L\times{d}}$, which also achieves better downstream performance in Section [5.1](https://arxiv.org/html/2506.11886v1#S5.SS1).

### 3.3 Online Compression via HiPPO-FourierT

We set the translated window length in FourierT to the maximum context length, ensuring effective compression within valid input-output ranges. In the prefilling phase, we preserve all dimensions of the initial $L_{\text{init}}$ and the local $L_{\text{local}}$ tokens,

$$ $\begin{gathered}\bm{K}^{i},\bm{K}^{l}=\bm{K}[:L_{\text{init}}],\bm{K}[-L_{ \text{local}}:]\\ \bm{V}^{i},\bm{V}^{l}=\bm{V}[:L_{\text{init}}],\bm{V}[-L_{\text{local}}:]\end{gathered}$ (1) $$

and distinguish the dimension indices $\mathcal{D}^{ku}$, $\mathcal{D}^{kc},\mathcal{D}^{vu}$, $\mathcal{D}^{vc}$ in KV cache for uncompressing and compressing, to enable training-free integration.

$$ $\displaystyle\bm{K}^{mn}=\bm{K}[L_{\text{init}}:-L_{\text{local}},\mathcal{D}^ {kc}],$ (2) $\displaystyle\bm{V}^{mn}=\bm{V}[L_{\text{init}}:-L_{\text{local}},\mathcal{D}^ {vc}],$ $\displaystyle\bm{K}^{mu}=\bm{K}[L_{\text{init}}:-L_{\text{local}},\mathcal{D}^ {ku}],$ $\displaystyle\bm{V}^{mu}=\bm{V}[L_{\text{init}}:-L_{\text{local}},\mathcal{D}^ {vu}].$ $$

We preserve $\bm{K}^{mu},\bm{V}^{mu}$, compress $\bm{K}^{mn},\bm{V}^{mn}$ to fix-sized $\bm{K}^{mc}\in\mathbb{R}^{2k\times{|\mathcal{D}^{kc}|}},\bm{V}^{mc}\in\mathbb{
R}^{2k\times{|\mathcal{D}^{vc}|}}$ and use the original KV for forward propagation.

$$ $\begin{gathered}\bm{K}^{mc}=\bm{\mathcal{F}}\bm{K}^{mn},\ \ \bm{V}^{mc}=\bm{ \mathcal{F}}\bm{V}^{mn},\[2.15277pt] \bm{O}=\texttt{flash_attention}(\bm{Q},\bm{K},\bm{V}).\end{gathered}$ (3) $$

When it comes to the compression matrix in FourierT, originally, $\bm{\mathcal{F}}\in\mathbb{C}^{k\times{L}}$, where $\bm{\mathcal{F}}_{nt}=e^{i\frac{2\pi{nt}}{T}}$. However, since caches in mainstream LLMs are real-valued, we convert complex numbers to corresponding 2D vectors, transforming $k$-order complex states into $2k$-order real states. Therefore, the real compression matrix in FourierT is $\bm{\mathcal{F}}\in\mathbb{R}^{2k\times{L}}$ as shown in Equation [4](https://arxiv.org/html/2506.11886v1#S3.E4).

$$ $\bm{\mathcal{F}}={\begin{bmatrix}1&1&\cdots&1\\ 0&0&\cdots&0\\ 1&\cos{\frac{2\pi}{T}}&\cdots&\cos{\frac{2\pi(L-1)}{T}}\\ 0&\sin{\frac{2\pi}{T}}&\cdots&\sin{\frac{2\pi(L-1)}{T}}\\ \vdots&\vdots&\ddots&\vdots\\ 1&\cos{\frac{2\pi{(k-1)}}{T}}&\cdots&\cos{\frac{2\pi{(k-1)(L-1)}}{T}}\\ 0&\sin{\frac{2\pi{(k-1)}}{T}}&\cdots&\sin{\frac{2\pi{(k-1)(L-1)}}{T}}\\ \end{bmatrix}}.$ (4) $$

In the decoding phase, FourierAttention compresses tokens out of the local range individually,

$$ $\begin{gathered}\bm{K}^{mc}\leftarrow\bm{K}^{mc}+\bm{f}_{t+1}\bm{K}^{l}[0, \mathcal{D}^{kc}]\\ \bm{V}^{mc}\leftarrow\bm{V}^{mc}+\bm{f}_{t+1}\bm{V}^{l}[0,\mathcal{D}^{vc}]\\ \bm{f}_{t+1}=\begin{bmatrix}0&1&\cdots&\cos{\frac{2\pi(k-1)t}{T}}&\sin{\frac{2 \pi(k-1)t}{T}}\end{bmatrix}^{\top}\end{gathered}$ (5) $$

and reconstruct intermediate cache $\tilde{\bm{K}}^{m},\tilde{\bm{V}}^{m}$ via inverse Fourier transform in attention computation with the current query vector $\bm{q}_{t+1}$.

$$ $\begin{gathered}\tilde{\bm{K}}^{m}[\mathcal{D}^{ku}]\leftarrow\bm{K}^{mu},\ \tilde{\bm{K}}^{m}[\mathcal{D}^{kc}]\leftarrow\frac{1}{k}\bm{\mathcal{F}}^{T} \bm{K}^{mc}\\ \tilde{\bm{K}}=\texttt{cat}(\bm{K}^{i},\tilde{\bm{K}}^{m},\bm{K}^{l})\\ \tilde{\bm{V}}^{m}[\mathcal{D}^{vu}]\leftarrow\bm{V}^{mu},\ \tilde{\bm{V}}^{m} [\mathcal{D}^{vc}]\leftarrow\frac{1}{k}\bm{\mathcal{F}}^{T}\bm{K}^{mc}\\ \tilde{\bm{V}}=\texttt{cat}(\bm{V}^{i},\tilde{\bm{V}}^{m},\bm{V}^{l})\\ \bm{o}_{t+1}=\texttt{flash_attention}(\bm{q}_{t+1},\tilde{\bm{K}},\tilde{\bm{ V}}).\end{gathered}$ (6) $$

To eliminate intermediate read-write cost in decompression, we try to implement a custom kernel, FlashFourierAttention, using Triton (Tillet et al., 2019), integrating the decompression into standard FlashAttention2 (Dao, 2024) and FlashDecoding (Dao et al., 2023). FlashFourierAttention loads compressed intermediate states once and decompresses at corresponding sequence positions during iterative KV cache loading. FlashFourierAttention is still in progress to achieve better computational efficiency compared with standard attention.

### 3.4 Fine-Grained Compression Schema

In FourierAttention, a crucial point lies in how to select the dimension to be compressed. To address this, we directly compress and decompress all KV caches, prioritizing dimensions with smaller mean-squared error in reconstruction to a fixed length.
Based on further observations of the KV cache, we adopt a fine-grained compression schema, where more dimensions of the V cache and lower-layer caches are compressed to a fixed length. We analyze the standard deviation of KV cache dimensions along the temporal direction across different layers and find that for both LLaMA3.1-8B and LLaMA3.2-3B, as shown in Figure [3](https://arxiv.org/html/2506.11886v1#S3.F3). The standard deviation of the K cache is consistently higher than that of the V cache, and the standard deviation in upper layers exceeds that of lower layers. Consequently, we compress more dimensions of the smoother V cache and lower-layer caches to a fixed length, while retaining more K cache and upper-layer caches to extend with sequence length. Thus, FourierAttention exhibits an asymmetric, inverted-pyramid compression pattern.

Interestingly, this differs from most KV cache compression approaches. Works like Cai et al. (2024) and Xing et al. (2024) suggest preserving more KV caches in lower layers, as attention becomes sparser in upper layers. However, in FourierAttention, the optimization criterion is whether the dimension can be well reconstructed. Since caches in upper layers exhibit more oscillatory features due to more deterministic predictions, we retain more dimensions to maintain output stability.

Figure: (a) K cahce in LLaMA3.1-8B
Refer to caption: extracted/6539870/figures/std_chart-llama3_1_8b-k_rope_rmprefix.png

## 4 Experiment

**Table 1: Results of LLaMA Series (Dubey et al., 2024; Meta, 2024b) on LongBench (Bai et al., 2023). Our FourierAttention achieves a superiority over StreamingLLM (Xiao et al., 2024), Palu (Chang et al., 2024), and KIVI (Liu et al., 2024c) and shows closest performance with LLMs with SnapKV (Li et al., 2024b).**
|  | Single-Doc QA | Multi-Doc QA | Summarization | Few-shot Learning |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | NQA | Qsp | MulF | HQA | 2WQA | MSQ | QRpt | QSum | MulN | TREC | TrQA | SSum |
| LLaMA3.1-8B | 13.22 | 20.24 | 32.81 | 11.97 | 13.60 | 8.72 | 29.70 | 25.09 | 0.90 | 73.50 | 90.97 | 47.26 |
| + StreamingLLM | 7.87 | 13.86 | 15.59 | 7.79 | 10.08 | 4.49 | 19.93 | 21.52 | 9.88 | 61.50 | 84.66 | 43.48 |
| + SnapKV | 12.67 | 19.84 | 32.48 | 12.00 | 13.76 | 8.59 | 29.19 | 24.90 | 12.56 | 73.00 | 90.97 | 46.53 |
| + Palu | 4.50 | 18.03 | 21.58 | 9.47 | 11.32 | 5.23 | 17.25 | 6.90 | 8.98 | 68.50 | 83.38 | 32.29 |
| + KIVI | 12.73 | 20.94 | 32.79 | 11.51 | 13.93 | 8.77 | 30.33 | 25.18 | 2.41 | 73.50 | 10.83 | 45.98 |
| + Ours | 14.94 | 18.47 | 30.47 | 13.46 | 14.29 | 9.64 | 22.34 | 22.69 | 5.00 | 72.50 | 89.18 | 43.80 |
| LLaMA3.2-3B | 10.27 | 21.69 | 35.52 | 9.58 | 12.78 | 6.75 | 30.15 | 23.77 | 28.24 | 70.00 | 87.24 | 38.19 |
| + StreamingLLM | 9.14 | 17.59 | 21.57 | 7.06 | 9.78 | 3.99 | 19.02 | 21.32 | 23.23 | 53.00 | 84.33 | 39.76 |
| + SnapKV | 8.92 | 21.04 | 34.97 | 9.50 | 12.77 | 6.62 | 29.45 | 23.36 | 27.77 | 69.50 | 86.39 | 38.33 |
| + Palu | 1.98 | 19.17 | 20.37 | 5.84 | 10.28 | 2.65 | 13.37 | 4.10 | 13.97 | 57.00 | 47.39 | 21.48 |
| + KIVI | 10.21 | 22.17 | 35.08 | 9.68 | 12.29 | 6.94 | 30.84 | 23.52 | 23.21 | 70.00 | 64.09 | 43.46 |
| + Ours | 8.91 | 21.27 | 31.20 | 11.99 | 17.77 | 8.26 | 23.41 | 22.16 | 23.76 | 69.00 | 86.18 | 37.12 |

**Table 2: Continuous table of Table [1](https://arxiv.org/html/2506.11886v1#S4.T1).**
|  | Synthetic | Code | Avg. |  |  |
| --- | --- | --- | --- | --- | --- |
|  | PsgC | PsgR | LCC | Re-P |  |
| LLaMA3.1-8B | 0.75 | 26.75 | 72.00 | 69.27 | 39.49 |
| + StreamingLLM | 1.25 | 6.18 | 58.36 | 56.35 | 31.52 |
| + SnapKV | 0.75 | 26.75 | 59.97 | 59.74 | 37.07 |
| + Palu | 0.60 | 14.58 | 56.68 | 54.57 | 30.68 |
| + KIVI | 0.75 | 30.00 | 29.43 | 19.86 | 23.18 |
| + Ours | 2.25 | 16.07 | 67.32 | 63.02 | 36.98 |
| LLaMA3.2-3B | 0.00 | 7.00 | 70.01 | 66.38 | 38.04 |
| + StreamingLLM | 1.37 | 6.47 | 55.47 | 53.53 | 31.19 |
| + SnapKV | 0.00 | 6.75 | 58.20 | 56.36 | 34.83 |
| + Palu | 1.45 | 3.08 | 55.10 | 49.54 | 25.53 |
| + KIVI | 0.00 | 6.00 | 40.01 | 37.87 | 28.98 |
| + Ours | 1.62 | 8.32 | 67.29 | 59.91 | 36.33 |

### 4.1 Setup

We conduct experiments on LLaMA3.1-8B (Dubey et al., 2024) and LLaMA3.2-3B (Meta, 2024a). For all models, we set the length of initial tokens $L_{\text{init}}$ to 4, the length of local tokens $L_{\text{local}}$ to 1024, and the number of states $k=512$. We evaluate the reconstruction loss using the prompt portion of the 32k Needle-In-A-Haystack benchmark in OpenCompass (Contributors, 2023). As mentioned earlier, we employ an asymmetric inverted pyramid compression strategy: for the first 4 layers, we compress 90% of K dimensions and 95% of V dimensions; for the last 8 layers, 50% of K and 70% of V; and for the remaining layers, 80% of both K and V. Overall, 76% KV caches are compressed to a fixed length. All experiments are performed on an NVIDIA H100 GPU with FP16 precision and accelerated with FlashAttention2 (Dao, 2024).

### 4.2 Long-Context Evaluation

We evaluate our method against other KV cache optimization approaches with two long-context benchmarks in OpenCompass (Contributors, 2023), LongBench (Bai et al., 2023) and Needle-In-A-Haystack (NIAH) (Kamradt, 2023; Li et al., 2024a), with a truncation context length of 32K. We compare with StreamingLLM (Xiao et al., 2024), SnapKV (Li et al., 2024b), Palu (Chang et al., 2024), and KIVI (Liu et al., 2024c), covering both token eviction and feature compression. For fair comparison, we retain 4 initial tokens and 1024 local tokens in StreamingLLM, additionally keep 1024 recalled middle tokens, matching our compressed dimension count, in SnapKV, compress KV feature dimensions to 70% in Palu, and apply 4-bit quantization, 75% compression, in KIVI.

For LongBench as shown in Tables [1](https://arxiv.org/html/2506.11886v1#S4.T1) and [2](https://arxiv.org/html/2506.11886v1#S4.T2), our FourierAttention achieves performance closest to the original model on LLaMA3.2-3B and is slightly inferior to SnapKV on LLaMA3.1-8B. For the NIAH task as shown in Figure [4](https://arxiv.org/html/2506.11886v1#S5.F4) and [5](https://arxiv.org/html/2506.11886v1#S5.F5), we similarly achieve performance closest to the original pretrained models at 32k context length. While SnapKV is theoretically suitable for retrieval tasks like NIAH, it still exhibits recall errors. Though Palu and KIVI maintain stable attention approximation under moderate compression, 30-50%, they show significant performance degradation at  75% compression due to insufficient granular analysis of KV cache features. In contrast, our FourierAttention optimizes compression by identifying and preserving KV dimensions insensitive to compression, thereby maximally retaining the long-context capabilities and demonstrating superiority across both models and benchmarks.

In addition to comparisons in downstream performance, we will also conduct efficiency experiments. Since our custom kernel FlashFourierAttention is still in progress, we will report this in detail in the next version of the paper.

## 5 Discussion

### 5.1 Choice of Basis Functions

**Table 3: Validation of basis function and compression schema in LLaMA3.2-3B (Meta, 2024a)**
|  | MK1 | MK2 | MK3 | MV | Avg. |
| --- | --- | --- | --- | --- | --- |
| LLaMA3.2-3B | 99.00 | 100.00 | 99.00 | 100.00 | 99.50 |
| + FourierT | 99.00 | 99.00 | 99.00 | 96.00 | 98.25 |
| + LegT | 89.00 | 93.00 | 50.00 | 93.75 | 81.44 |
| + uniform | 100.00 | 99.00 | 98.00 | 90.50 | 96.88 |
| + KV inv. | 98.00 | 100.00 | 98.00 | 93.00 | 97.25 |
| + layer inv. | 99.00 | 98.00 | 93.00 | 86.00 | 94.00 |

Although FourierAttention employs HiPPO-FourierT for compression, Gu et al. (2020) proposes and claims polynomial basis functions like LegT with superior performance. While maintaining identical sliding window sizes, we compare LegT and FourierT in reconstructing KV caches from LLMs. As illustrated in Figure [6](https://arxiv.org/html/2506.11886v1#S5.F6), we evaluate their reconstruction effects on 4 randomly selected KV cache dimensions from layer 0 of LLaMA3.2. Under equivalent state dimensions(^1^11FourierT uses lower-order basis functions since FourierAttention’s state size is twice the number of states), FourierT consistently achieves lower reconstruction loss than LegT.

We further evaluate FourierT and LegT compression on LLaMA3.2-3B using more discriminative NIAH variants, Multi-Key NIAH (MK) and Multi-Value NIAH (MV) (Hsieh et al., 2024) in 4k context length. For fair comparison, we employ the same method to identify dimensions suitable for LegT compression and apply an identical compression schema. Results in Table [3](https://arxiv.org/html/2506.11886v1#S5.T3) show FourierT still performs better, demonstrating that FourierT offers better parallelizability for compression efficiency and performance in downstream evaluation.

Figure: (a) Pretrained
Refer to caption: x4.png

Figure: (a) Pretrained
Refer to caption: x10.png

Figure: (a) Cache reconstruction by HiPPO-LegT
Refer to caption: x16.png

Figure: (a) Layer 0 in LLaMA3.1-8B
Refer to caption: x18.png

### 5.2 Ablation on Compression Schema

As mentioned in Section [3.4](https://arxiv.org/html/2506.11886v1#S3.SS4), we propose a more fine-grained compression scheme based on additional observations of the KV cache. As shown in the Table [3](https://arxiv.org/html/2506.11886v1#S5.T3), we compare three approaches: uniform compression across all layers and between KV (uniform), inverted KV compression schema by K-priority over V (KV inv.), and inverted layer-wise compression schema by upper-layer priority over lower-layer (layer inv.). Results demonstrate that our original V-priority and lower-layer-priority compression schema achieves superior performance on discriminative NIAH variants. This further illustrates that frequency-based sequence-wise KV cache compression exhibits fundamentally different optimization characteristics compared to conventional KV token eviction methods (Cai et al., 2024; Xing et al., 2024).

### 5.3 Compressed Dimension Distribution

Finally, we analyze the compressed dimensions selected by our FourierAttention. We count the number of each dimension selected for compression, averaged across attention heads in different layers, grouped every 16 dimensions. Results in Figure [7](https://arxiv.org/html/2506.11886v1#S5.F7) show that in both LLaMA3.1-8B and LLaMA3.2-3B, starting from layer 2, lower dimensions are more frequently compressed while the upper dimensions tend to preserve complete temporal information in our FourierAttention. This phenomenon is more evident in upper layers, where fewer dimensions are chosen to be compressed. As illustrated in Figure [2](https://arxiv.org/html/2506.11886v1#S2.F2), these uncompressed upper dimensions primarily contribute to forming attention sinks and capturing long-context semantic relationships, thus requiring complete retention, whereas other dimensions can be stored with limited length.

## 6 Conclusion

We propose FourierAttention, a novel KV cache optimization approach that compresses long-context-insensitive dimensions without sacrificing contextual awareness based on an interesting phenomenon in transformer head dimensions, that lower dimensions capture local features, while upper ones capture long-context dependencies. Inspired by HiPPO, we optimize the long-context-insensitive KV cache through a translated Fourier transform into fixed-length states in the prefilling phase and reconstruct the KV cache in the decoding phase. FourierAttention shows the best performance on the LLaMA Series in LongBench and NIAH on average. We are trying to improve the efficiency of FourierAttention through a customized Triton-based kernel, FlashFourierAttention, eliminating intermediate read-write operations and effectively reducing memory overhead.

## Limitations

We will continue optimizing our customized Triton kernel FlashFourierAttention, which injects KV cache decomposition in FlashAttention and FlashDecoding (Dao, 2024), minimizing memory overhead via streamlined read-write operations. Moreover, our performance still shows gaps compared to the pre-trained model. These aspects will be thoroughly investigated in future work.

## Acknowledgments

We thank Jian Yuan from Shanghai Jiao Tong University for assisting with experimental verification.

## References

- Ainslie et al. (2023)
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023.
Gqa: Training generalized multi-query transformer models from multi-head checkpoints.
*arXiv preprint arXiv:2305.13245*.
- Bai et al. (2023)
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2023.
Longbench: A bilingual, multitask benchmark for long context understanding.
*arXiv preprint arXiv:2308.14508*.
- Cai et al. (2024)
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and 1 others. 2024.
Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.
*arXiv preprint arXiv:2406.02069*.
- Chang et al. (2024)
Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. 2024.
Palu: Compressing kv-cache with low-rank projection.
*arXiv preprint arXiv:2407.21118*.
- Contributors (2023)
OpenCompass Contributors. 2023.
Opencompass: A universal evaluation platform for foundation models.
[https://github.com/open-compass/opencompass](https://github.com/open-compass/opencompass).
- Dao (2024)
Tri Dao. 2024.
Flashattention-2: Faster attention with better parallelism and work partitioning.
In *The Twelfth International Conference on Learning Representations*.
- Dao et al. (2023)
Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. 2023.
[Flash-decoding for long-context inference.](https://crfm.stanford.edu/2023/10/12/flashdecoding.html)
- Duanmu et al. (2024)
Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. 2024.
Skvq: Sliding-window key and value cache quantization for large language models.
*arXiv preprint arXiv:2405.06219*.
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*.
- Fu (2024)
Yao Fu. 2024.
Challenges in deploying long-context transformers: A theoretical peak performance analysis.
*arXiv preprint arXiv:2405.08944*.
- Gu et al. (2020)
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. 2020.
Hippo: Recurrent memory with optimal polynomial projections.
*Advances in Neural Information Processing Systems*, 33:1474–1487.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*.
- He et al. (2023)
Ziwei He, Meng Yang, Minwei Feng, Jingcheng Yin, Xinbing Wang, Jingwen Leng, and Zhouhan Lin. 2023.
Fourier transformer: Fast long range modeling by removing sequence redundancy with fft operator.
*arXiv preprint arXiv:2305.15099*.
- Hooper et al. (2024)
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024.
Kvquant: Towards 10 million context length llm inference with kv cache quantization.
*arXiv preprint arXiv:2401.18079*.
- Hsieh et al. (2024)
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024.
Ruler: What’s the real context size of your long-context language models?
*arXiv preprint arXiv:2404.06654*.
- Kamradt (2023)
Greg Kamradt. 2023.
Needle in a haystack - pressure testing llms.
[https://github.com/gkamradt/LLMTest_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack).
- Li et al. (2024a)
Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. 2024a.
Needlebench: Can llms do retrieval and reasoning in 1 million context window?
*arXiv preprint arXiv:2407.11963*.
- Li et al. (2024b)
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024b.
Snapkv: Llm knows what you are looking for before generation.
*arXiv preprint arXiv:2404.14469*.
- Liu et al. (2024a)
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, and 1 others. 2024a.
Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.
*arXiv preprint arXiv:2405.04434*.
- Liu et al. (2025)
Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, and 1 others. 2025.
Thus spake long-context large language model.
*arXiv preprint arXiv:2502.17129*.
- Liu et al. (2024b)
Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. 2024b.
Scaling laws of rope-based extrapolation.
In *The Twelfth International Conference on Learning Representations*.
- Liu et al. (2024c)
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024c.
Kivi: A tuning-free asymmetric 2bit quantization for kv cache.
*arXiv preprint arXiv:2402.02750*.
- Meta (2024a)
AI Meta. 2024a.
Introducing meta llama 3: The most capable openly available llm to date.
*Meta AI.*
- Meta (2024b)
AI Meta. 2024b.
Llama 3.2: Revolutionizing edge ai and vision with open, customizable models.
*Meta AI.*
- OpenAI (2023)
OpenAI. 2023.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*.
- OpenAI (2024)
OpenAI. 2024.
[O1: Openai’s first model](https://openai.com/o1/).
Accessed: 2024-12-25.
- Peng et al. (2024)
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024.
Yarn: Efficient context window extension of large language models.
In *The Twelfth International Conference on Learning Representations*.
- Saxena et al. (2024)
Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. 2024.
Eigen attention: Attention in low-rank space for kv cache compression.
*arXiv preprint arXiv:2408.05646*.
- Sun et al. (2024)
Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Xiangyang Liu, Hang Yan, Yunfan Shao, Qiong Tang, Shiduo Zhang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, and 5 others. 2024.
[Moss: An open conversational large language model](https://doi.org/10.1007/s11633-024-1502-8).
*Machine Intelligence Research*.
- Tillet et al. (2019)
Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019.
Triton: an intermediate language and compiler for tiled neural network computations.
In *Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages*, pages 10–19.
- Vaswani et al. (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.
[Attention is all you need](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf).
In *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc.
- Xiao et al. (2024)
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024.
Efficient streaming language models with attention sinks.
In *The Twelfth International Conference on Learning Representations*.
- Xing et al. (2024)
Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and 1 others. 2024.
Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction.
*arXiv preprint arXiv:2410.17247*.
- Yuan et al. (2025)
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025.
Native sparse attention: Hardware-aligned and natively trainable sparse attention.
*arXiv preprint arXiv:2502.11089*.
- Zhang et al. (2023)
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, and 1 others. 2023.
H2o: Heavy-hitter oracle for efficient generative inference of large language models.
*Advances in Neural Information Processing Systems*, 36:34661–34710.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: - Could you frame the contribution wrt KV Cache compression, Quantization and Low Rank reduction.
- Could you provide exact numerical results for latency and memory consumption rather than just plots?
- Why aren't other architectures included in the evaluation ? Could you include them ? 
- Could you provide results in Table1 for different compression ratios ?
soundness: 2
strengths: - The empirical finding that different head dimensions serve distinct roles (local vs. global context) is interesting and well-validated through ablation studies
- Leveraging the HiPPO framework provides mathematical rigor, and the adaptation from complex to real-valued representations is sensible for practical implementation.
- The custom Triton kernel shows practical engineering effort and is very nice.
summary: This paper proposes a training-free KV cache compression method that exploits heterogeneous roles across transformer head dimensions. The authors observe that lower dimensions capture local context while upper dimensions capture long-range dependencies, and compress the former using Fourier basis functions from the HiPPO framework. A custom Triton kernel (FlashFourierAttention) is implemented for efficient deployment. The observation and the custom kernel are nice, but the evaluation is not comprehensive enough. Both the choice of the baselines (compression vs quantization) and the range of performed experiments and models selected are not enough. Also, the paper would benefit from a better framing of the method with respect to the literature about KV Cache compression.
weaknesses: - (General) The title and abstract claim this is "KV cache compression," but the actual mechanism is better described as lossy approximation or dimensionality reduction. For this reason, while comparing to Palu (dimensionality reduction) makes sense, comparing to SnapKV and PyramidKV is kind of strange as these perform KV Cache eviction in a different setting. The authors should (a) discuss the differences between KV Cache compression, quantization, reduction in a clear way and (b) include quantization in the baselines.
- (Method) The observation about dimension heterogeneity is a bit over-claimed and not entirely new, as prior work has noted similar patterns. This is cited by the authors themselves.
- In Table 1. The comparison shows that the methods have been tested with different compression ratios. Why not use the same compression ratio for fair comparison? 
- In Table 1 and in general across the paper, for a fair and comprehensive evaluation one should consider a range of compression rations for all methods instead of a specific one.
- The choice of L-init=4, L-local=1024, N=512 appears arbitrary with no ablation study or principled justification. How sensitive is performance to these choices?
- The method is only evaluated on Llama models. This strongly hinders the generalization of the method to different architectures.
- I am not sure I understood how is the dimension selection actually performed? The authors say "we directly compress and decompress all KV caches, prioritizing dimensions with smaller mean-squared error" but doesn't specify the algorithm. Is this done once offline or adaptively?

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: - What happens on other families on models?
- Are results statistically significant?
- How sensitive are the results to errors in identifying the two types of latent dimensions?
soundness: 2
strengths: - Interesting analysis about how latent dimensions can be characterised into long-context-sensitive/insensitive (also from a mechinterp point of view)
- Strong results compared with competitive baselines like SnapKV, PyramidKV, and Palu at comparable budgets on LongBench and NIAH
- Interesting custom FlashFourierAttention kernel
summary: This paper proposes FourierAttention, a training-free KV cache compression method. The idea is that KV cache tokens have long-context-insensitive and long-context sensitive channels -- the former can be projected onto translated Fourier bases, while the latter can be ;eft uncompressed. Authors also propose a Triton kernel for on-the-fly reconstruction of the tokens. Experiments on long-context benchmarks with LLaMA-3.* models show improvements over baseline KV cache compression methods like SnapKV and PyramidKV. One concern I have is that results in Tab. 1 often show very marginal improvements, and results are often worse than those of SnapKV and PyramidKV -- are those statistically significant? Also, authors only experiment with Llama-3.1-8B and Llama-3.2-3B -- what happens on models not in the Llama3 family (e.g. any of the recent Qwens)? On the other hand, the baselines are very competitive (according to e.g. KVPress, https://github.com/NVIDIA/kvpress) so any improvements on those is more than welcome.

Tiny typo: line 189 -- "We denode"
weaknesses: - Absolute gains are *very* modest -- are they statistically significant?
- I was not able to find quantitative results on latency, please let me know if I missed those
- Experiments only on Llama3-based backbones

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: See above
soundness: 3
strengths: + This paper discovers that the different dimensions of Q and K in attention computation play different roles. Initially, this finding seemed counterintuitive to me, as I typically assumed that different dimensions were homogeneous—this was because I had overlooked the effect of ROPE. The paper innovatively leverages this insight by applying different compression strategies to different dimensional ranges, achieving better compression efficiency.

+ The paper also implements the proposed method's corresponding FlashFourierAttention operator, which is an important contribution to the open-source community.

+ The experiments provide thorough comparisons in terms of performance and latency, effectively demonstrating the superiority of the proposed method.
summary: This paper proposes FourierAttention, an online compression method for the KV cache. FourierAttention uses Hippo-FourierT to compress the KV cache, significantly reducing the memory overhead of the KV cache while maintaining better performance compared to baseline methods.
weaknesses: + The paper only conducts experiments on LLaMA. It would be better to include comparisons with other open-source models as well.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: * Regarding the main finding — why can the locally focused and globally focused dimensions be separated by contiguous index ranges? Wouldn’t one expect them to be more interleaved or randomly distributed across dimensions?
soundness: 2
strengths: * **Novel Observation.** This work presents an interesting finding that within a single attention head, some dimensions focus on local context while others capture global dependencies. This insight provides a new perspective on KV-cache compression, distinct from prior token-wise, head-wise, or precision-wise (quantization-based) methods.

* **Solid System Implementation.** The paper adapts kernels from FlashAttention and FlashDecoding to build FlashFourierAttention, effectively integrating Fourier-based compression into the attention computation and avoiding extra memory movement during inference.
summary: This work is based on an interesting observation that, within attention heads, some dimensions focus on local context while others capture long-range dependencies. To exploit this heterogeneity, the paper introduces FourierAttention, which compresses the locally focused dimensions of the KV cache into fixed-length states using the HiPPO framework combined with a translated Fourier transform, achieving efficient and training-free KV-cache compression.
weaknesses: * **Limited Latency Evaluation.** The experiments only report prefill latency, while the majority of KV-cache read and update operations occur during decoding. Including TPOT (Time per Output Token) or throughput comparisons would provide a more complete performance evaluation.
* **Limited Model Generalization.** FourierAttention is built upon the finding of heterogeneous dimension roles within attention heads, yet the evaluation is limited to LLaMA-3.1-8B and LLaMA-3.2-3B. Models with different head sizes (e.g., LLaMA-3.2-1B with only 64 dimensions per head) may not exhibit the same behavior, raising questions about general applicability.
* **Lack of Per-Head Analysis.**
  Prior works such as MInference [1], RazorAttention [2], DuoAttention [3], and HeadKV [4] have shown that sparsity and attention patterns vary across heads. This paper provides limited analysis on whether the observed dimension heterogeneity holds consistently across all attention heads, leaving this aspect unclear.



[1] Jiang, Huiqiang, et al. "Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention." *Advances in Neural Information Processing Systems* 37 (2024): 52481-52515.

[2] Tang, Hanlin, et al. "Razorattention: Efficient kv cache compression through retrieval heads." *arXiv preprint arXiv:2407.15891* (2024).

[3] Xiao, Guangxuan, et al. "Duoattention: Efficient long-context llm inference with retrieval and streaming heads." *arXiv preprint arXiv:2410.10819* (2024).

[4] Fu, Yu, et al. "Not all heads matter: A head-level kv cache compression method with integrated retrieval and reasoning." *arXiv preprint arXiv:2410.19258* (2024).


Decision: Reject