FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension
Authors: Jushi Kai
Yixuan Wang
Boyi Zeng
Haoli Bai
Bo Jiang
Ziwei He
Zhouhan Lin
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
KV Compression
Context Extension
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper introduces FreqKV, an efficient context extension method that iteratively compresses key-value states in the frequency domain.

Abstract: Existing key-value (KV) cache compression methods for large language models (LLMs) often rely on token eviction, which risks losing critical local information in both long prefilling and decoding scenarios. When extrapolating beyond the pretrained context length, their performance degrades sharply on long-context benchmarks. Motivated by the observation in the frequency domain that the context information is concentrated in the low-frequency components, we propose FreqKV, a parameter-free and architecture-agnostic approach. It iteratively compresses the increasing KV cache in the frequency domain, allowing models to process lengthy contexts efficiently. With minimal training at 8K length, FreqKV extends the context window of LLaMA-2-7B up to 256K tokens while maintaining stable perplexity. Extensive experiments on both prefilling and decoding stages demonstrate that FreqKV enables robust context window extension and consistently outperforms existing KV cache compression methods, highlighting its effectiveness for both understanding and generation in long contexts.

Published as a conference paper at ICLR 2026
COMPRESSION IN FREQUENCY DOMAIN FOR CONTEXT WINDOW EXTENSION
FREOKV:
Jushi Kai', Yixuan Wang', Boyi Zeng'. Haoli Bai,, Bo Jiang', Ziwei He Zhouhan Lin'*
'LUMIA Lab, Shanghai Jiao Tong University, "Huawci Noah's Ark Lah "Shanghai Jiao Tong University, *Shanghai Innovation Institute
json.kai @ @jtu.edu..cn ze @ sii.edu.cn, lin.zhouhan @ gmail.com
ABSTRACT
Existing key-value (KV) cache compression methods for large language models (LLMs) often rely on token eviction, which risks losing critical local informa- tion in both long prefilling and decoding scenarios. When extrapolating beyond the pretrained context length, their performance degrades sharply on long-context benchmarks. Motivated by the observation in the frequency domain that the con- text information concentrated in the low-frequency components, we propose FreqKV, parameter-free free and architecture-agnen aggnostic approach. It iteratively com- presses the increasing KV cache in the frequency domain, allowing models to process lengthy contexts efficiently. With minimal training at 8K length, FreqKV extends the context window of LLaMA-2-7B up to 256K tokens while maintaining stable perplexity. Extensive experiments across prefilling and decoding demon- strate that FreqKV enables robust context window extension and consistently out- performs existing KV cache compression methods on LLaMA-2 and LLaMA- 3, highlighting its effectiveness for both understanding and generation in long contexts. Our code is available at https: /t////gttt///// com/ LUMIA LUMIA-Group/ FreqKV.
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable performance in natural language understanding and generation. However, their inference capabilities are fundamentally constrained by pre-defined context window size established during pre-training. Consequently. LLMS strug- gle to maintain their performance when processing sequences that exceed the preset context size, significantly limiting their applicability in long-context scenarios.
To alleviate this limitation, several position encoding methods, such ALiBi (Press et al., 2022), PI (Chen et al., 2023), and LongRoPE (Ding et al., 2024) have been proposed to extend the context window. Relying on full self-attentoo,, the quadratic computational cost renders them prohibitively expensive at scale. LongLoRA (Chen et al., 2024) trains LLMs using shifted sparse attention. De- spite training efficiency, their sparse attention fails to be applied during inference, and they still require the original attention on the full sequence.
A popular strategy improve efficiency is compress the key-value (KV) cache during inference. Methods such as SnapKV (Li et al., 2024), PyramidKV (Cai. et al., 2024b), and FastKV (Jo et al., 2025) evict tokens deemed less important according attention scores. However, these compression strategies cannot be extrapolated beyond the context window. The information associated with the evicted tokens will be lost, leading to performance degradation when decoding future tokens.
Another line of work (Zhang et al., 2024; Wan et al., 2025) attempts to merge tokens rather than dropping them, preserving more information. Nevertheless, their performance remains suboptimal without fine-tuning. Concurrently, LoCoCo (Cai et al., 2024) and Activation Beacon (Zhang et al., 2025) introduce additional modules to compress KV states and incorporate the fine-tuned compress- ing pattern into the decoding procedure of LLMs. However, the increasing memory requirement for additional parameters is inevitable.
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
In this work, we observe that the KV states of LLMs exhibit strong energy concentration in the low- frequency components of the frequency domain, suggesting that high-frequency parts are largely re- dundant. Building this observation, method that iteratively compresses key-value states in the frequency domain. Unlike eviction meth- ods, FV preserves the information of all tokens by transforming KV states into the frequency domain and discarding high-frequence components that carry limited information. By compressing iteratively the cache grows, VreqKV retains recent tokens with minimal loss while progressively compressing earlier ones. This parameter-fre method requires no architectural modifications and can be applied to both the prefilling stage (long-context understanding) and the decoding stage (long- context language modeling and generation).
introduce FreqKV, an efficient context window extension
Extensive experiments show that FreqKV enables robust context window extension. With minimal fine-tuning at 8K length, V maintains low perplexity as the evaluation context length scales to 256K. achieves comparable performance to methods that employ full cache or additional com- pressors in long context language modeling. Experimental results across LongBench, RULER, and Needle-in-aaHHaatttc indicate that FreqKV surpasses recently studied KV compression methods in the prefilling stage. Furthermore, FreqKV V also delivers substantial improvements in long-gen generation capability on LongGenBench, validating its effectiveness in decoding scenarios. These results es- tablish frequency-domain compression viable approach for extending the context window.
2 RELATED WORK
KV Compression for LLMs. To alleviate memory and computation costs, growing body of research explores compressing the KV cache when processing long contexts. One widely stud- ied approach is selective token eviction, which be traced back to attention sinks (Xiao et al., 2024a;b), suggesting that preserving a small set of initial tokens helps stabilize model performance. Furthermore, MInference (Jiang al., 2024) propose accelerate prefilling with unique patterns in long-context attention matrices. More recent methods, SnapKV (Li et al., 2024), InfiniPot (Kim al., 2024), PyramidKV (Cai. et al., 2024b), GemFilter (Shi et al., 2024), and FastKV (Jo al., 2025), use attention scores to measure the importance of tokens and discard less important ones. However, LLMs suffer from the permanent loss of the information associated with evicted tokens during decoding. To address this limitation, some researchers introduce cache merging techniques to approximate the original full attention of the existing contexts, such as CaM (Zhang et al., 2024), K KVMerger ( (Wang et al., 2024), and D20 (Wan et al., 2025). Nevertheless, these inference methods often sacrifice performance for efficiency and cannot be extrapolated beyond the context window.
Efficient Context Extension for LLMs. Recent advancements in context extension for LLMs have focused on efficiently scaling models handle longer input sequences without significantly increasing computational costs. LongLoRA (Chen et al., 2024) employs shifted sparse attention dur- ing the parameter-efffcient fine-tuning. However, this sparse attention mechanism is not applicable during inference, necessitating return to the original full attention post-training. Other techniques, such as LoCoCo (Cai et al., 2024), integrate convolutional operations into LLMs for compressing long contexts. They fine-tune the compression modules together with LLMs. Landmark attention (Mohtashami & Jaggi, 2023) uses landmark tokens to retrieve previous input blocks. Similarly, Ac- tivation Beacon (Zhang et al., 2025) introduces special token to represent the previous context for compression. However, they need a copy of multi-head attention, which amounts to approximately 2B for 7B models. In contrast, our proposed method achieves context extension without introducing any additional parameters.
Learning in the Frequency Domain. Learning in the frequency domain is well-established technique to compress images and accelerate CNNs (Gueguen et al., 2018). has been observed that CNNs are more sensitive to low-frequency channels than high-frequency channels (Xu et al., 2020). Moreover, Frequency Principle (Xu et al., 2019; Luo et al., 2019) theoretically shows that low frequencies converge faster than high frequencies for deep neural networks. These works have inspired efforts to process natural language. FNet (Lee- -eehThopp et al., 2022) improves the efficiency of Transformer encoder architectures by replacing the self-attention layers with the Fourier transform to serve the purpose of mixing tokens. Additionally, Fourier Transformer (He et al., 2023) eliminates redundancies in the context through frequency domain processing within encoder architectures.
2

---

Published as a conference paper at ICLR 2026
31
0.00
0.20
0.01
2.8
0.0
(a) The average power spectrums of key states.
laye:
layer
layer
loyer 16
layer 31
8.0013
8.310
8.330
375
2.5
03004
8:39
0.275
0300
200
683
830
1330
253
630
500
1002
353
630
900
1003
200
800
230
85
1530
The average power spectrums of value states.
Figure 1: The average power spectrums of key states and value states in different layers of LLaMA- 2-7b. 1000 documents are sampled from CNN/Daily Mail (Hermann et al., 2015). We transform key states and value states to the frequency domain, and average power spectrums over these samples and hidden dimensions.
However, because of the autoregressive nature, it remains unclear how to leverage frequency com- ponents for decoder-only Transformer, which is the main architecture of generative LLMs. To the best of our knowledge, FreqKV is the first work that explores compressing key-value states in the frequency domain for decoder-only LLMs.
FREOKV
In this section, we present power spectrum analysis of KV states for LLMs. Building on this obser- vation, V compresses KV states in the frequency domain and extends the context window via iterative compression.
3.1 PRELIMINARIES
Energy Concentration in the Frequency Domain. We use DCT (Discrete Cosine Transform) to transform key states and value states from the time domain, which the sequence dimension, to the frequency domain. The formulations of DCT and the inverse DCT (IDCT) can be referred to in Appendix
The average power spectrums in different decoder layers of LLaMA-2-7B calculated and pre- sented in Figure 1. The figure reveals that the energy of key states and value states increasingly concentrated in the low-frequency components. The distributions of their power spectrums are sim- ilar since they are projected from the same hidden states. While the initial embeddings of natural languages in the first layer exhibit strong low-frequency bias, subsequent layers gradually shift energy toward low frequencies along the decoding procedure. This observation suggests that the low-frequency components capture most of the essential information, whereas high-frequency parts carry limited signal and can be discarded with minimal loss, Further analysis of frequency compo- nents is provided Appendix
Self-Attention with KV Cache. For the incoming token vN, the prefilled N tokens X = [zo..... -1] utilized as the cache during decoding. Denote the cached KV states for the previous tokens X - as Ko-N-1 and Vo For simplicity. indices corresponding to layers and heads have been omitted.

---

Published as a conference paper at ICLR 2026
When calculating attention scores, the incoming token N attends to all cached KV states well as to itself:
(IN|KEN-1 + kNl'
[IN
A(N) Softmax
VV:N-1 + UN),
vd
where di is the hidden dimension. QN: KN and uN are the query, key and value states of x respec- tively. means the concatenation of the KV cache and KV states of æx.
3.2 EFFECT OF FREQUENCY COMPONENTS
To empirically investigate what infor- mation different frequency bands en-
code, we have conducted targeted per-
5.9
turbation experiment. We sample 100 CNN/DailyMail documents and intro- duce controlled perturbations by ran- domly replacing 1% words with their synonyms. We then transform the KV states of LLaMA-2 into the frequency domain and compare the cosine similar- ity between perturbed and original in- puts for both fow-frequency (lowest 50%) and figh-frequency (highest 50%) com- ponents. The results across different lay- ers are presented in Figure 2:
High components cfvalue itate
Casine Similarity
2.0
0.4
shows that the output states are robust to the perturbation with low- frequency components retained while high-frequeen
Figure 2: Cosine Similarity between perturbed and orig- inal inputs for different frequency components of KV states.
more sensitive, supporting the interpretation that low fre- quencies encode global information, while high frequencies capture local details.
We further evaluate this distinction on the summarization track the downstream benchmark Long- Bench, comparing the performance of Ilama-2-chat-2 Ab with low and high frequencies retained in Table 1. Summarization requires capturing global semantic structure and long-range coherence. Accordingly, retaining low-frequency components yields substantially better performance.
Table 1: Performance on summarization tasks with high/low-frequene components retained.
Components Retained GovReport
QMSum
MultiNews
high-frequency 14.21
16.54
15.55
low-frequency 25.51
21.81
26.9
Together, these results strengthen motivation: low-frequency components primarily encode global semantic contexts and long-range dependencies, whereas high-frequency components en- code local details, which explains why preserving low-frequency information is both more robust and more beneficial for long-context understanding.
3.3 KV COMPRESSION THE FREQUENCY DOMAIN
To reduce redundancy in the key-value (KV) cache, we compress KV states in the frequency domain as shown Figure 3a. Specifically, we conduct DCT along the sequence dimension transfer the KV cache to the frequency domain:
2 - DCT((K.N-)), Z = DCT((V.N-1);
(2)
where z - and ZV_ - the spectral representations of the KV cache.
4

---

Published as a conference paper at ICLR 2026
Frequency Compression
Input Tokens
What is
the
hurram life
expect ancy
loday
DCT
Context Window
Frequency
Filter
Frequency Compression
Frequency Compression
sink tokens
IDCT & Rescale
incoming tokens
compressed tokens
(h)
Figure 3: The overview of our FreqKV. (a) The illustration of the frequency- domaan compression. (b) The KV cache will be compressed in iterative manner to extend the context window. Sink kens remain uncompressed throughout the process. The tokens after sink tokens will be compressed in the frequency domain and subsequent tokens will continue get into the cache. When the cache is filled again, the compressed tokens and incoming tokens will be compressed together.
As illustrated in Figure 1, since the low-frequenny components exhibit higher magnitudes and carry more information, will retain them and remove the high-frequene parts for compression. Given a retaining ratio the cache size is reduced to L = by filtering out N L high- frgg-frre components.
Then we conduct IDCT along the frequency dimension to convert the compressed components back the time dimension. should be noted that the signals normalized by the square root of the component number when applying DCT and IDCT (Appendix A). Therefore, the compressed
signals should be rescaled with the factor of V N to restore the original amplitude:
O::-1 K 0:0-1 = V IIDTT
D.N -
OON- VOL = - IDCT
-- 1)).
L 1]), (3)
0:N-1 K Ko 11 and 0.L-1 are the compressed KV cache of size L in the time domain. The superscript "O:N - 1" indicates the original token range, while the subscript "O:L- - means the retaining size.
0:N-1
The incoming token NN will attend to the compressed KV cache as follows:
O.N-1 gN K 0.L-1 + kNI"
O:N-1 (V
À(N, Softmax
0:L-1 @UN].
(4)
Vd
3.4 CONTEXT EXTENSION VIA ITERATIVE COMPRESSION
Extending the context window of LLMss is fundamentally constrained by memory and computation cost. To address this, FreqKV employs an iterative compression strategy in the frequency domain that constrains the effective cache size while enabling processing of arbitrarily long sequences. The overall pipeline is illustrated in Figure 3b.
For tokens within the context window, the standard attention will be conducted. Once the cache reaches the maximum size, we will compress it in the frequency domain. Specifically, cached KV states converted to the frequency domain via DCT, where low- www-frequency components tained before applying IDCT to transform them back to the time domain. The incoming tokens are appended the compressed cache, and the process repeats when the cache is filled again. As result,

---

Published as a conference paper at ICLR 2026
earlier tokens experience more iterations of compression the context window expands, whereas less compression will be performed on the more recent tokens a property well aligned with the autoregressive nature of LLMs. The compression is triggered only once every N L tokens, with the complexity of O NNlog.N) for each compression. Therefore, the computational overhead of the compression could be negligible.
Recent work has found the phenomenon of attention sinks that LLMs tend to assign high attention scores initial tokens (Xiao al., 2024b; Han et al., 2024). Therefore, we maintain these initial tokens uncompressed in the cache and only compress tokens that come after them.
During training and prefilling, VV will process the sentence chunk-wise, interleaving attention computation with the compression operation. After each compression. N - S incoming tokens form chunk that fills the cache, while S sink tokens remain uncompressed. and L compressed "tokens" represent earlier history. Consequently, the newly incoming token will attend to S sink tokens, compressed tokens, M N previous incoming tokens, and itself:
S.N qm gm.sKos-1 @ K D-z-1 @ KNAMI")))
S:N-1
A(S, N, L,, M) = Softmax
[Vois @ VoL @ Vw.dd, (5)
where K Ko-.-1 and VoL.11
S:N-1
S:N-1
0:I-1 denote the compressed KV cache of size L for tokens Xs:N-
[ss,.... ,.S...,INNII IN.
should be noted that key states in LLMs are often equipped with position embeddings like RoPE (Su et al., 2024). In FreqKV, key states are compressed and cached before applying RoPE. The cached key states are then encoded with RoPE when conducting self-attention. They are allocated with position indices within the cache rather than the original sequence, which results in the context window extension without requiring position extrapolation or interpolation. Further discussion of RoPE integration is provided in Appendix C.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
We employ VV on LLaMA-2-7B/13B (Touvron et al., 2023) and LLaMA-3-8B (AI@Meta, 2024). Unless otherwise specified, the retaining ratio y is set to 0.5, and kept uncompressed. For LLaMA-2, the preset context window size is 4096, which also defines the maximum KV cache size N. Accordingly, during each compression step, the N = 4092 states since the 5-th state in the cache will be compressed into L = 7 (N - S) 2046 states. As for LLaMA-3, the native context window size is 8192. is equipped with GQA (Grouped-Quee) Attention). which means it has lower proportion of parameters for attention modules than LLaMA- 2 (Multi-Heal Attention, MHA). Minimal training is introduced to adapt models to this frequency- domain compression method, following LongLoRA (Chen et al., 2024). Details of training settings are summarized in Appendix D.
sink tokens are
We assess KV on long context language modeling and generation tasks for decoding perfor- mance, and long context understanding tasks for prefilling performance. For long context language modeling, we fine-tune LLaMA-2-bass. on the pre- pre-training dataset RedPajama (Computer, 2023), extending the context window from 4K to 32K. Perplexity (PPL) evaluation is conducted on PG-19 (Rae et al., 2019). For long context generation and understanding, the instruction following dataset LongAlpaca (Chen et al., 2024) is used for the supervised fine-tuning (SFT) of LLaMM-22chhal and LLaMA-3-insttnn
The context window is extended from 4K to 8K, and from 8K to 16K respec- tively. Models are evaluated on LongGenBench (Liu et al., 2024), LongBench (Bai et al., 2024), RULER (Hsieh et al., 2024), and Needle-in-a-Hie Hayasaas (gkamradt, 2023).
4.2 LONG CONTEXT LANGUAGE MODELING
We use FreqKV to train LLaMA-2-7B and L-M-----3BB RedPajama. Perplexity is measured on the test set of the book corpus dataset PG-19 with the evaluation sliding window of 256. We compare our method with other baselines, including full fine-tuning (Full FT), LongLoRA, and

---

Published as a conference paper at ICLR 2026
Table Perplexity evaluation on the test set of PG-19. The superscript "** means that we reproduce LoCoCo following their official code for evaluation. The results of full fine-tuning and LongLoRA are reported from LongLoRA (Chen al., 2024). Full FT and LoCoCo encounter the OOM (Out- of-Memory) issue with the training length of 16K.
Training Length
Inference Cache
Evaluation Context Length
Size
Method
2048
4096
8192
16384
32768
Full FT
Full
7.55
7.21
6.98
8192
LoCoCo* LongLoRA
Compressed
8.15
8.08
7.27
Full
7.70
7.35
7.14
FreqKV
Compressed
7.45
7.12
7.04
7.02
7.02
7B
Full FT
Full
OOM during Training
LoCoCo
Compressed
OOM during Training
16384
LongLoRA
Full
7.65
7.28
7.02
6.86
FV
Compressed
7.46
7.13
7.03
6.99
6.98
LongLoRA
Full
8.29
7.83
7.54
7.35
7.22
32768
FreqKV
Compressed
7.47
7.14
7.04
7.00
6.98
Full FT
Full
6.95
6.60
6.43
13B
8192
LongLoRA
Full
7.03
6.73
6.58
FreqKV
Compressed
6.82
6.52
6.44
6.42
6.41
LoCoCo (Cai et al., 2024). While Full FT and LongLoRA leverage full KV cache during inference, LoCoCo and V use compressed cache.
PPL scores on different evaluation context lengths are reported in Table 2. More Evaluation results Proof-pile (Azerbayev al., 2022) are presented in Appendix E. While both LongLoRA and LoCoCo sacrifice performance within the original context length (2K and 4K), FreqKV matches or even surpasses Full FT. With the training length of 8K, FreqKV performs comparably LongLoRA trained at 32K and consistently outperforms LoCoCo in terms of extended context length as well as the shorter lengths. Additional ablation studies on retained frequency components, retaining ratio, and sink size are reported in Appendix F.
4.3 LONG CONTEXT UNDERSTANDING
To further assess the performance on long-prefilling scenarios, we apply FreqKV to extend the context window of LLaMA-2-chat-77 from 4K 8K and LLaMA-3-instruct-B8 from 8K 16K. Evaluation is conducted on three long context understanding benchmarks: LongBench, RULER, and Needle-in-a-Haasttck
LongBench. Scores the 5 categories of LongBench reported in Table 3. FreqKV sistently improves the performance of LLaMA-2-chat and LLaMA-3-instruct across a majority of these tasks. We compare method against eviction- strategies GemFilter (Shi et al., 2024), SnapKV (Li et al., 2024), PyramidKV V (Cai. et al., 2024b), LaCache (Shi et al., 2025), and FastKV (Jo et al., 2025), merging-bel based strategies CaM (Zhang et al., 2024), KVMerger (Wang al., 2024), and D20 (Wan et al., 2025), and a channel compression method ThinK (Xu et al., 2025) that im- proves SnapK Fr achieves SOTA (state-of-the-arr) tasks and obtains the highest average score. In particular, it shows significant gains on question an- swering and summarization tasks such as HotpotQA, 2WikiMQA. and QMSum. Moreover. even with an extremely low retaining ratio (y = 1%), where other methods degrade sharply, FreqKV remains stable and effective, highlighting its robustness under aggressive compression.
most of the long context understanding
Results RULER presented in Table 4. FreqKV consistently outperforms baselines across different tasks. Notably, baseline methods collapse entirely at 16K evaluation length, which exceeds the original context window of I.LaMA-3-instnue (8K). This failure arises because they
RULER.

---

Published as a conference paper at ICLR 2026
Table 3: Scores of different KV compression methods on LongBench. The superscript **" means that results of CaM and K KVMerger are reported from KVMerger (Wang et al., 2024). We reproduce the other baselines following their official codes for evaluation. The second-bes scores are also underlined in the table.
Sangle-Documen
QA
Maulii-Dccumnt QA
Summarization
Few-shol Learning
Code
Refaining
Ratio
Method
NirQA
Qasper
MF-en
HeepoiQA
2WikiMQA
Musique
GovRepor
QMSmi
MulniNews
TREC
TrivisQA
SAMSam
LCC
RB-P
LaMMA-2-hhtt
c-hal---
100%
Fall Cache
18.7
19.2
36.8
25.4
32.8
9.4
27.3
20.8
25.8
61.5
77.8
40.7
52.4
43.8
35.17
CaM
17.08
20.00
33.98
30.69
24.46
24.66
63.5
82.17
KVMerger*
18:50
20.04
36.89
32.99
25.31
26.29
64.0
83.62
LaCache
16.55
31.55
12.62
26.27
7.72
65.0
47.97
D20
20.32
19.16
33.09
30.62
26.76
9.65
23.72
20.54
25.42
63.0
84.23
41.22
58.61
52.77
36.37
GemFilter
15.22
17.76
32.92
27.29
26.76
8.66
16.17
57.5
31.96
50%
F
18.25
22.46
35.83
28.84
31 14
8.49
24.4
20.96
25.8
60.0
82.67
04
57
51.52
36.33
SnapKV
17.91
22.9
35.3
26.03
28.21
8.77
24.99
20.94
25.92
64.0
83,36
41.0
60.83
55.14
36.81
Think
15.57
21.76
37.98
30.21
31.55
8.06
26.24
20.98
26.
65.0
80.08
40.97
60.2
54.26
37.07
PyramidKV
17.36
23.58
34,89
26.28
28.39
9.17
24.95
20.9
26.04
64.0
83.36
40.92
60.68
55.17
36.84
FregKY
20.41
21.05
31.2
34.54
34.78
14.47
25.51
21.81
26.9
56.0
84.09
41.53
58.99
58.65
37,85
LaCache
4.37
12.72
6.34
14.77
14.91
2.23
11.18
7.39
9:94
16.75
15.13
6.42
7.56
7.09
9.77
D20
15.53
12.61
17.64
29.0
21.71
8.49
16.56
16.05
16.63
52.5
81.03
20.2
31.88
33.47
26.66
FastKV
10.55
13.49
13.53
13,27
6.8
4.11
11.94
14.45
11.92
27.0
$2.05
17.22
32.1
29.38
18.41
GemFilter
3.22
3.99
2.09
0.78
11.07
13.14
10.9
12.5
21
11,50
1%
SnapKV
11.49
12.47
13.61
12.79
7.05
3.48
11.96
14.61
12.16
28.0
52.11
17.03
32 48
28.39
18.40
Think
14.5
14.71
4.94
11.97
11.63
18.0
PyramidKV 18.17
23.8
35.97
28.42
31.7
8.76
15.85
16.64
16.2
64.5
83.65
32.41
60 38
54.89
35 35.10
FV
19.44
20.2
31 68
38.89
32.0
12.52
15.96
17.39
15.91
60.5
83.98
33,07
62.47
56.57
35.54
(LaMA-3-instrueet
100%
Full Cache
22.52
31.83
41.04
44.14
36.68
24.05
28.87
23.25
26.46
75.5
90.23
42.09
56.7
$1.84
42.51
LaCache
22.95
28.91
40.99
46.34
36.26
23.02
16.92
16.02
13.59
74.0
91.11
32.97
55 09
49.31
39.25
D20
26.61
28.19
38.85
44.95
36.57
22.9
27.66
23.34
25.62
74.0
90.31
41.79
57.57
54.54
42.36
Geml Gemliller
23.6
29.26
43.71
42.18
36.35
24.48
17.75
16.46
15.7
73.0
90.79
33.3
59
38.54
38.45
50%
SuapKV
FastKV
23.88
22.99
29.67
31.89
40.75
41.11
44.66
43,92
34,02
36.42
23.07
38
16.86
17.14
16.45
16.45
15.61
15.53
74.5
74.0
90.31
90.23
42.02
33.72
59.21
59.03
53.97
53.3
39.63
40.02
Think
21.98
30.97
41.9
45.1
34.18 18
22.51
27.73
23.38
26.4
74.5
90.16
41.5
62.05
58.37
42.91
PyramidKV
31.93
23.86
28.77
26.62
74.5
42.53
FregKV
23.14
41.6
43.62
46.48
40.54
20.2
29.99
23.67
28.58
69.0
88.95
41.47
60.33
51.11
43.48
LaCache
14.27
5.87
33.07
18.58
15.01
7.76
10.89
40.0
77.92
40.7
27.31
D20
24.95
12.77
20.19
36.61
27.2
15.43
22.57
15.18
15.12
56.0
89.34
24.75
32.79
36, 18
30.65
Gemfilter
15.35
12.28
20.51
19.37
21.79
1%
SnapKV
FistKV
19.57
18.29
12.61
14.25
31.87
31.28
42,59
38.03
26.82
29:44
20.03
20.71
13.96
14.29
14:57
15.32
13.08
12.86
61.0
60.5
87.79
88.52
29.95
29.55
56.74
89
53.36
$4.18
34.25
34.83
ThinK
18.02
10.38
31.46
39.1
26.08
17.46
17.31
21.48
17.97
39.5
88.18
37.27
54.69
19
33,86
PyramidKV
22.79
31.91
41.24
44.2
36.67
22.53
17.29
16.52
15.57
74,5
90,23
33.3
58.95
53.12
39.92
FregKY
22,86
39.7
36.44
44.29
39.59
22.57
29.1
22.92
28.35
65.0
89.0
41.9
62.17
55.42
42,81
compress the long prompt after the first decoding step. Out-of-bound position embeddings are used when calculating attention, resulting in a performance breakdown. In contrast, FreqKV remains robust and demonstrates its effectiveness for context extension. FreqKV first conducts self-attention within the context window, and then compresses the KV states for out-of wfindoww tokens to get in. Compressing key states before applying ROPE, FreqKV extends the context window without requiring position extrapolation or interpolation.
Figure 4 illustrates evaluation results on Needle-in---Haasstae, Freq performs well in extending the context window of LLaMA-3-instruct In contrast, other KV com- pression methods fail beyond the original window boundary. This breakdown is consistent with our observations in RULER.
4.4 LONG CONTEXT GENERATION
We use LonghenBench to evaluate the long-context generation performance of FreqK V. This bench- mark constructs extended generation tasks by concatenating multiple chain-of-thouee questions and answers, followed by several real questions in GSM8K and MMLU for long-context responses. The model is then required to generate answers with a maximum output length of 4096 tokens.
8

---

Published as a conference paper at ICLR 2026
Table Scores of different KV compression methods RULER. D2O encounters the OOM issue with the evaluation length of 16K.
Evaluation Length
Method
CWE
FWE
QA
VT
NIAH
Avg.
LLaMA-3
96.40
82.33
62.50
95.00
97.81
86.81
+D20
93.20
81.00
62.00
88.80
47.53
74.51
+GemFilter
96.40
82.33
62.50
95.00
97.81
86.81
+FastK
86.10
77.67
63.50
93.00
87.03
81.46
8K
+SnapKV
96.00
80.33
62.50
93.60
92.28
84.94
Think
95.7
78.67
62.00
96.8
94.91
85.62
PyramidKV
96.40
82.33
62.50
95.00
97.81
86.81
+FregKV
91.80
84.67
64.50
100.00
97.44
87.68
+D20
OOM
+GemFilter
0.00
0.00
0.50
0.00
0.00
0.10
+FastKV
0.00
0.00
0.00
0.00
0.00
0.00
16K
+SnapKV
0,00
0.00
0.50
0.00
0.00
0.10
+Think
0.00
0.00
0.00
0.00
0.00
0.00
PyramidKV
0.00
0.00
0.50
0.00
0.00
0.10
+FregKV
45.10
87.33
34.00
44.60
25.47
47.30
(a) GemFilter.
(b) FastK
(c) SnapK V.
11.
(d) ThinK.
(e) PyramidKV.
(l) FreqKV.
Figure 4: The Beedle-in-a-Haysteed results of different KV compression methods on LLaMA-3- instruct-8B, with the x-axis representing the document length ranging from 1K to 16K tokens, and the y-axis showing the position of the "needle" within the document.
The accuracy scores (%) are provided in Table 5. Fre- qKV demonstrates substantial improvements in long- generation capability and consistently outperforms ex- isting KV compression methods across both GSMSK and MMLU. FreqKV preserves essential contextual information through iterative frequency- com- pression, enabling robust long-conter decoding.
Table 5: Performance on LongGenBench.
Method
GSM8K
MMLU
LLaMA-3
42.8
23.8
+D20
8.0
24.8
GemmFittr
48.5
25.4
FastKV
48.5
25.4
ANALYSIS
SnapKV
49.6
25.8
ThinK
45.5
25.7
PyramidKV
49.6
26.2
5.1 EFFICIENT EXTRAPOLATION
FreqKY
54.2
31.2
Furthermore, we evaluate the performance FreqKV longer lengths the validation of PG-19

---

Published as a conference paper at ICLR 2026
Table 6: Extrapolation performance of FreqKV on PG-19 up to 256K context length. We measure latencies (seconds) and PPL scores at different lengths. "Compression Count" denotes the number of compressions during decoding. "Compression Latency" stands for the latency introduced by all the frequency-domain compressions during decoding.
Evaluation Length
16K
32K
64K
128K
256K
Compression Count
7
15
31
63
127
Decoding Lateney
214.96
445.69
907.49
1832.77
3692.03
7422.66
Compression Latency 0.62 (0.29%)
1.09
(0.24%)
2.18
(0.24%)
4.00
(0.22%)
8.31
(0.23%)
16.39 (0.22%)
PPL
7.76
7.74
7.73
7.73
7.73
7.73
Table 7: The training time and memory requirements of Freq and Full FT.
Method
Training Length
Time (hours)
Memory (GB)
Full FT
8K
28.88
44.04
8K
17.07
24.58
FreqKV
16K
38.57
34.00
32K
89.35
45.37
shown in Table With the training length limited to 8K tokens, it effectively extrapolates the context window of LLaMA-2-7B to 256K, which is 64 times larger than the original size. This is enabled by its iterative compression strategy, which applies fewer compressions recent tokens while progressively compressing earlier ones. Conse- quently, FreqKV maintains low perplexity as the evaluation context length scales from 8K to 256K, far exceeding the training length.
In addition, latencies (seconds) of decoding and rrequency-domal compression at different lengths are also reported in Table 6. It shows that the decoding time of FreqKV increases approximately linearly, with a negligible time spent on compression. The overhead of the compression accounts for less than 0.3% and does not grow with the context length since the number of compressions increases linearly with the context length. Further analysis for the compression overhead is provided Appendix G. In summary, FreqK V demonstrates strong extrapolation ability ultra-long contexts while preserving both efficiency and stability.
5.2 TRAINING COST
The training time and memory requirements of FreqKV and Full FT on LLaMA-2-7B are reported in Table 7. 8 NVIDIA RTX6000 Ada-48GB GPUs are used. For comparison, we set the batch size per device to and gradient accumulation steps to 8. Statistics show that FreqKV significantly reduces both training time (by 40.89%) and memory requirements (by 44.19%), demonstrating its computational efficiency.
CONCLUSION
In this paper, we introduce FregKV, a frequency-domain KV cache compression framework that exploits the energy concentration of KV states to retain low-frequency components while discard- ing redundant high-frequency ones for LLMs. By iteratively compressing the KV cache, FreqKV achieves efficient long-context extension while preserving decoding quality. Without introducing additional parameters architectural modifications, FreqKV exhibits remarkable efficiency and effectiveness on both prefilling and decoding stages. Extensive experiments across language mod- eling, generation, and understanding benchmarks demonstrate that FreqKV not only consistently surpasses existing KV compression methods but also offers a viable and efficient approach for con- text window extension.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENTS
Our work pertains to key-value compression and context extension of large language models. this work, only publicly available data and artifacts. There ethical issues in paper, including its motivation and experiments.
REPRODUCIBILITY STATEMENTS
We have provided detailed implementations of our method throughout the paper. Our method is comprehensively elaborated in Section 3. Detailed settings of our experiments and analyses given in Section 4, 5, and Appendix F. Our code is included the Supplementary Material for reproducibility.
THE USE OF LARGE LANGUAGE MODELS
We used ChatGPT solely language editing tool to polish grammar, improve clarity, and refine the academic style of the manuscript. All research ideas, methods, experiments, and analyses were independently developed and conducted by the authors.
ACKNOWLEDGEMENT
This work is sponsored by the National Natural Science Foundation of China (NSFC) grant (No. 62576211) and the National Key Research and Development Program of China (No. 20232DD0211022..
REFERENCES
AetMeta. Llama 3 model card. 2024. URL http:////ggiaubb.c///ttat--aaaa
CARD md.
Zhangir Azerbayev, Edward Ayers, and Bartosz Piotrowski. Proof-pile, 2022. URL https://
huggingface
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, mul- titask benchmark for long context understanding, 2024. URL https:///aaaae org/abs/ 2308. 14508.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URL https: arxiv org/ abs 2004.001511
Ruisi Cai, Yuandong Tian, Zhangyang Wang, and Beidi Chen. Lococo: Dropping in con- volutions for long context compression. In Forty- first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27. 2024. OpenReview.net, 2024. URL https: //openrevi ew. fortff d=NU1 MgMyh09
Zefan Cai., Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, 2024b. URL hittps://arke wwtprxiivi.rrr.ooooooooooo wwwg/aaaaa abs 2406. 02069.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 155995 2023.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models, 2024. URL https: /arxivv. org/abs/220 9309. 12307.
Together Computer. Redpajama: an open dataset for training large language models, 2023. URL hittps://githhe www..////eeeeooooeoooeeeoee /Reee jama- Daam--aaaa.

---

Published as a conference paper at ICLR 2026
Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning, 2023, URL https: pttppp://ariivv org/ abs///ss///// 2200/23372220 08691.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan
Yang. and Mao Yang. Longrope: extending Ilm context window beyond 2 million tokens. In Proceedings of the 41st International Conference on Machine Learning, pp. 11091-11104, 2024.
gkamradt. Limtest needle in haystack pressure testing llss. https: /ttp//gttttttttttt/ com/ BataraaattLLMMettt_eeeeee InAaalstttaaaaaaakckkk 2023.
Lionel Gueguen, Alex Sergeev, Ben Kadlec, Rosanne Liu, and Jason Yosinski. Faster neural net- works straight from jpeg. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 31, Cur- ran Associates, Inc., 2018. URL https: http:///poccceco.oe//oom/ neurips cct pappr_fiieeee/////ppppppppe Petpppee666..tttc-ape........
Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm- infinite: Zero-shot extreme length generalization for large language models, 2024. URL https:
/208 16137.
Ziwei Hc, Meng Yang. Minwei Feng, Jingcheng Yin. Xinbing Wang, Jingwen Leng, and Zhouhan
Lin. Fourier transformer: Fast long range modeling by removing sequence redundancy with fft op-
erator. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 8954 8966. Association for Computational Linguistics, 2023. doi: 10. 116653/1/20222 htwp:/22.i.ninng-aal..33. URL http:// doi. org/ 10. 1 v1 /2023. findings acl ac1.570.
Karl Moritz Hermann, Tomás Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend, 2015.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024. URL https: /awxiv..vvvv..////////aa.///vvvv org/ 2404 06654.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https: www.aiiw.rgg....ee.ooooo.ooooeee /aabbs/2100 09685.6666.6.....
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yev Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context Ilms via dynamic sparse attention, 2024. URL https: / /atxxvvivvv/v////////ee///aaavl. org/oaggaoo httg////////e 2005/2400 2407 02490.
Dongwon Jo, Jiwon Song. Yulhwa Kim, and Jae-Joon Kim. Fastkv: Ky cache compression for fast long-context processing processs with token-selective propagation, 2025. URL htt.ttts::///t http:ps://axxx..rrrre abs/2502. 01068.
Minsoo Kim, Kyuhong Shim, Jungwook Choi, and Simyung Chang. Infinipot: Infinite context http://awwwww..aaaa//aa
processing on memory-constrained llms, 2024. URL. 01518.
James Lee- Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms, 2022. URL https: http://wwxvv...g.....///// www.oor///bb.//o...///........ea /aabs/211. 2200 03824.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947-22970, 2024.
Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. Longgenbench: Long-context gener- ation benchmark. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16. 2024, pp. 865-883. Association for Computational Linguistics, 2024. doi: 10. 18653/V SEEIINGG-EEMLPPA... URL https: /doi. org/10 11t.11..11111111111 find inndings-emn] 48.
12

---

Published as a conference paper at ICLR 2026
Tao Luo, Zheng Ma, Zhi-Qin John Xu, and Yaoyu Zhang. Theory of the frequency principle for general deep neural networks, 2019. URL https: /arxivi///xivv////aa//aa org/ abs/ 09235.
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context
length for transformers, 2023. URL https: : /ttp//axivvv.///////eee///eeella www.iii...g/abaa.... ats//30s/33055 0tt.//255.......0 16300.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases
enables input length extrapolation. In ICLR, 2022.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling, 2019. URL https://azxiv.ot /abs/
Dachuan Shi, Yonggan Fu, Xiangehi Yuan, Zhongzhi Yu, Haoran You, Sixu Li, Xin Dong, Jan Kautz, Pavlo Molchanov, Yingyan, and Lin. Lacache: Ladder-shaped kv caching for efficient long-context modeling of large language models, 2025. URL https://azxii: org/ /bs/ 2507.1420
Zhenmei Shi, Yifei Ming, Xuan-Phi Nguyen, Yingyu Liang, and Shafiq Joty. Discovering the gems in early layers: Accelerating long-context Ilms with 1000x input token reduction, 2024. URL
https: /ttp:///aavvv/ee www. 2409.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut
Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojmic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: / arxiv. org/ abs 2307 09288.
Zhongwei Wan, Xinjian Wu, Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang. Siqi Luo, Jing Xiong, Longyue Wang, and Mi Zhang. D2o: Dynamic discriminative operations for efficient long-context inference of large language models, 2025. URL https:: //tp://aaiivvv///i///ai org/ / 2406. 13035.
Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adap- tive kv cache merging for llms long-context tasks, 2024. URL https: /ttp:///axvvvooooooo abs/ / 2407 08454.
Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-fres free long-context extrapolation for Ilms with an efficient context memory, 2024a. URL
wtt.oor/g/aaaaa/egeeeae 04617.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL htttps://openreve/
net http:/ofmmmmdddcc7s1111...
Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren. Learning in the frequency domain, 2020. URL https:
wtw.////////////////////// 12416.
Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. Think: Thinner key cache by query-driven pruning, 2025. URL https:/ ptttss///arivve org 22122220021211
13

---

Published as a conference paper at ICLR 2026
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. CoRR, abs/1901.065222. 0000001..623.... 2019. URL http: //arxiv. www.vorgg/oaoo abs/ 1901. 06523.
Peitian Zhang. Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Long context compression with activation beacon. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL
httts:://ppeneviies net o-
Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji. CaM: Cache merging for memory-efficiett LLMs inference. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the sl International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 58840-58850, PMLR, 21-27 Jul 2024. URL nttps://pzoceeding mlr press/ressseeeeceee /22 zhang2 html.
A DISCRETE COSINE TRANSFORM
The Discrete Cosine Transform (DCT) transforms signal from the spatial domain (time or position) into the frequency domain. Several variants of DCT exist, with DCT-II being the most common. For a real-value discrete signal Xo:N-1 = IN] of length N, it defined as:
xt(2n +
N
if O,
21
In
(6)
2N
otherwise
n=0
where 0, 1,... .N- 1. ar is the normalization factor. The time-domain signal Xo:N-1 can be recovered by applying the inverse DCT (IDCT) on the frequency components Zo.N-I:
N
Tt( + 1)
Fy 0g
COS
2N
2
The frequency components are expressed as combination of the original signals. The values can be computed using the Fast Fourier Transform (FFT) with a complexity of (NlogN). The amplitudes of frequency components are utilized in the power spectrum analysis to represent the energy or magnitude of components. The components of higher energy in the frequency domain indicate they more informative (He et al., 2023).
For time signals (e.g.. texts) and space signals (e.g., images), their energy is most concentrated in low-frequency DCT components. This phenomenon occurs due the mathematical properties of DCT and the inherent smoothness of real-world signals. The DCT expresses a signal as a sum of cosine functions oscillating at different frequencies. In the case of text embeddings, neighboring tokens exhibit strong temporal correlations, meaning adjacent values in the semantic space tend to be similar. low-frequency cosine components are slowly varying and capture these smooth variations well, while high-frequency components mainly represent abrupt changes or noise.
B HEAD-WISE ANALYSIS OF POWER SPECTRUMS
We explore the power spectrum distribution of key states and value states in different attention heads of LLaMA-2-7B, as shown in Figure 5. Although the values of power spectrums vary in different heads, their distribution exhibits similar patterns. They have consistent tendency to aggregate energy in the fow-frequency components along the decoding procedure. could be promising to study specific differences and associations in different heads or other modules.
C COMPRESSING KEY STATES WITH ROPE
We further investigate to compress key states after applying RoPE. Power spectrums key states with RoPE provided in Figure 6. Components of certain frequencies filtered out by RoPE.
14

---

Published as a conference paper at ICLR 2026
byyr
Ityer
layer
liger
12.0
66
280
1000
3.15
0.18
400
80
090
1000
200
403
030
80
200
280
690
86
1008
230
44
960
636
1206
236
480
006
090
10.0
6.12
6.22
0.100
3.5
heal
2.0
0.00
1230
120
(a) The average power spectrums of key states.
Iayer
loper
liger
layer
Jaye
0.00071
0.09000
hoal
000001
0.900
100.2
2.0
1100000
1230
12.0
(b) The average power spectrums of value states.
Figure 5: The average power spectrums key states and value states in different heads of Llama-2- 7b.
layer
layer
layer
layer
layer
120
3.0
150
3.20
123
330
30
30
50
Figure 6: The average power spectrums of key states with RoPE.
We train LLaMA-2-7B on RedPajama with the variant method that key states are compressed after applying ROPE. The training length is 8K. Models are evaluated on the test set PG-19 and Proof- pile. The results in Table 8 demonstrate that the variant suffers from performance degradation, which is probably due to the usage of position embeddings out the original window. In contrast, FreqKV compresses and caches key states before applying RoPE. Therefore, they allocated with position indices within the original window rather than the lengthy sequence, which enables context window extension with need of using out-of-bound position embeddings.
15

---

Published as a conference paper at ICLR 2026
Table Perplexity evaluation on the sets of PG-19 and Proof-pile. "Variant" stands for Fr with key states compressed after applying RoPE.
Evaluation Context Length
Dataset
Method
2048
4096
8192
Full FT
7.55
7.21
6.98
PG-19
FreqKV
7.45
7.12
7.04
Variant
7.53
7.19
7.13
Full FT
3.14
2.85
2.66
Proof-pile
FreqKV
3.15
2.88
2.79
Variant 3.16
2.88
2.80
Table 9: Perplexity evaluation on the test set of Proof-pile. The superscript *** means that we reproduce LoCoCo following their official code for evaluation. The results of full fine-tuning and LongLoRA are reported from LongLoRA (Chen et al., 2024). Full FT and LoCoCo encounter the OOM (Out-of-Memory) issue with the training length of 16K.
Training Length
Inference Cache
Evaluation Context Length
Size
Method
2048
4096
8192
16384
32768
Full FT
Full
3.14
2.85
2.66
8192
LoCoCo LongLoRA
Compressed
3.40
3.20
2.88
Full
3.20
2.91
2.72
FreqKV
Compressed
3.15
2.88
2.79
2.76
2.75
7B
Full FT
Full
OOM during Training
LoCoCo
Compressed
OOM during Training
16384
LongLoRA
Full
3.17
2.87
2.66
2.51
FreqKV
Compressed
3.15
2.88
2.78
2.74
2.73
LongLoRA
Full
3.35
3.01
2.78
2.61
2.50
32768
FreqKV
Compressed
3.16
2.89
2.78
2.74
2.72
Full FT
Full
2.96
2.69
2.53
13B
8192
LongLoRA
Full
3.04
2.77
2.60
FreqKV
Compressed
2.99
2.74
2.65
2.63
2.63
D TRAINING SETTINGS
For long context language modeling, we fine-tune LLaMA-2-base on the RedPajama (Computer, 2023) pre-training dataset for 1000 steps. For long context understanding, we SFT LLaMA-2-chatt and I.LaMA-3-instmu on the instruction following dataset LongAlpaca (Chen et al., 2024) for epochs, which consists of 6.28K long-context QA samples.
The total batch size (GPU number Batch_size.ppepe device _eevic 64. The learning rate increases linearly from le-6 to 2e-5 with 20 warm-up steps and remains constant in the following steps. The rank used in the LoRA (Hu et al., 2021) fine-tuning is set to 8. Following LongLoRA (Chen al., 2024), the embedding and normalization layers are learnable during training. Moreover, we equip our method with FlashAttemtion-2 (Dao, 2023) for further acceleration and memory saving.
coaudiintt.ccuuulattinnn
16

---

Published as a conference paper at ICLR 2026
ym ym0.5 y=
-1000
1750
FreoKV
1500
Decoding Time (5)
1250
8.0
Decading
1000
7.9
750
500
250
7.7
2K
4K
Evaluation Context Length
8K
16K
32K
64K
1K
4K
aK
12K
16K
20K
24K
28K
32K
Sequence Length
Figure 7:
Perplexity (Line) and decoding la-
Figure 8: Decoding time with the full cache and
tency (Bar) with different retaining ratios.
FreqKV V the increasing sequence length.
E EVALUATION ON PROOF-PILE
In Table 9, present the evaluation results the Arxiv math dataset Proof-pile. On Proof-pile, FreqKV / trails LongLoRA at longer context lengths. This gap arises because Proof-pile, unlike natu- ral language corpora, contains dense symbolic and mathematical expressions where minor variations (e.g., single symbol or index) drastically alter semantics. Such localized variations are more diffi- cult to preserve under compression, making it particularly challenging for models using compressed cache to predict the next token.
LongLoRA retains the full KV cache during inference, avoiding any information loss and thus per- forming better on formula-heavy tasks. Nonetheless. FreqKV remains competitive and notably out- method, LoCoCo. Additionally, the performance of FreqK V
performs another compression- does not degrade further as the context length increases, highlighting its robustness.
F ABLATION STUDY
F.1 RETAINING HIGH/LOW-PREQUENC COMPONENTS
We further compare variants of V that retain only high-frequency vs. only low-frequency components using LLaMA-2-base-7B on the validation set of PG-19. Results (PPL) are provided in Table 10.
It shows that low-frequency components adapt better than the variant that retains high-frequency parts, which is consistent with our analysis in summarization tasks (Table 1). While the variant deteriorates when exceeding the original context window, low-frequeney components, which carry more information, benefit the model, and the perplexity remains stable. This result aligns well with Frequency Principle (Xu al., 2019; Luo al., 2019) that low frequencies converge faster than high frequencies for Deep Neural Networks.
Table 10: Perplexity evaluation on PG-19 with high/low-frequencc components retained.
Components
Evaluation Context Length
Retained
2048
4096
8192
16384
gigh-frequency
8.29
7.92
8.42
8.70
-reffrreeency
8.20
7.84
7.74
7.73
F.2 RETAINING RATIO
We train LLaMA-2-7B RedPajama (Computer, 2023) with different retaining ratios. The training length is 8K. Models are evaluated on the validation set PG- 19 with the evaluation length ranging from 2K 64K.
17

---

Published as a conference paper at ICLR 2026
The performance of FreqKV with different retaining ratios is presented in Figure 7. The decoding time (seconds) is averaged over five runs. FreqKV performs better with more frequency compo- nents retained. However, larger retaining ratios lead to more frequent compression steps and denser attention computation, hence higher latency.
We also evaluate the performance of FreqKV when modifying its retaining ratio during evaluation on the validation set of PG-19. The PPL scores at a length of 8K are presented in Table 11. These results suggest that VreqKV robust mismatched training/evaluation retaining ratios, especially when using larger inference ratios than training (which retains more information).
Table 11: Perplexity evaluation on PG-19 with mismatched training/evaluation retaining ratios.
Retaining Ratio during Training
Retaining Ratio during Evaluation
0.25
0.5
0.75
0.25
7.84
7.81
7.82
0.5
7.89
7.76
7.77
0.75
7.97
7.79
7.75
F.3 SINK SIZE
The effect of sink token count has been explored in StreamingLLM (Xiao et al., 2024b). We also ablate the effect of the sink token number S for FreqKV on LLaMA-2-7B, Results given in Table 12. It shows that a threshold of four sink tokens appears to be enough, with subsequent additions contributing marginal effects.
Table 12: Perplexity evaluation on PG-19 with different numbers of sink tokens.
Evaluation Context Length
Sink Size
2048
4096
8192
0
8.23
7.87
7.84
2
8.21
7.85
7.78
4
8.20
7.84
7.76
8
8.20
7.83
7.75
F.4
EFFECT OF BREQUENCY-DDMANE COMPRESSION
(a) Dropping.
(b) FregKV.
Figure 9: Comparison of FreqKV V and its variant Dropping on LLaMA-2-chat-77
To demonstrate the effectiveness of our compression operation, a variant method, Dropping, is also implemented and evaluated Needle-in-a--Haystacc for comparison, as in Figure 9, Instead of com-
18

---

rescaled as presented Published as a conference paper at ICLR 2026
2.3
with rescaling without rescaling
2.2
1.9
1.8
800
Training Step
Figure 10: Curves of training loss for V with and without rescaling.
pressing in the frequency domain, Dropping retains the most recent tokens like Streaming (Xiao al., 2024b). Although trained under identical conditions, Dropping suffers from irreversible loss of carly-context information. FreqKV, by contrast, preserves essential signals and achieves superior performance.
F.5 EFFECT OF RESCALING
In Equation 3, V - V works rescaling factor for the compressed signals to restore the original amplitude when conducting IDCT. In DCT, an N-length time-domain signal is normalized by VN; during IDCT, the retained L frequency components are correspondingly normalized by v L. When iterative compression is applied across multiple steps, this mismatch in normalization scales can cause the reconstructed signals to be progressively amplified. In our pilot study, the training loss significantly higher the early stages when the compressed signals
in Figure 10. Therefore, we introduce the rescaling factor / N restore the original amplitude in the time domain.
G COMPRESSION OVERHEAD
We conduct further studies the compression overhead of Fveq FreqKV performs compressions on-the-fly during decoding. We compare the decoding time required by LLaMA-2-7B with the full cache and our FVeqKV when the sequence length increases. As shown in Figure 8, the decoding time starts to diverge at the length of 4K. While the full cache utilization leads to a quadratic growth in decoding time, the decoding time of Freq V increases approximately linearly, with a negligible time spent on compression, showcasing its efficiency.
As introduced in Section 3.4, chunk- wise attention is performed for the prefilling tokens. The size of the attention matrix in each chunk is (N - L- S) N except for the last few tokens. Therefore, the computational cost of self-attention grows approximately linearly with the input length, like sliding window attention (Beltagy et al., 2020).
We use torchprofile' to count the number of Floating Point Operations (FLOPs) with input sequences of different lengths for LLaMA-2-7B. We calculate FLOPs for the model with the original attention, which leverages full KV states, and with FreqKV with the retaining ratio of 0.5. To quantify the compression overhead, instead of compressing in the frequency domain. shares the same sink size and retaining size as VV The difference in FLOPs between the methods shows the overhead of compression.
have measured FLOPs of Dropping, which retains the most recent tokens
The statistics are given in Table 13. The compression times of FreqKV with different context lengths also reported in the table. It shows that the computation overhead of compression process grows less than 0.5% even with a length of 32K, which could be negligible. FreqKV reduces more FLOPs the input length grows from 4K 32K compared to Full KV. This is because the compres-
19

---

Published as a conference paper at ICLR 2026
sion is performed every N - L S tokens with the complexity of ((VlogN), which is negligible compared to the quadratic self-attention.
Table 13: FLOPs (TFLOPs) with input sequences of different lengths. Experiments are conducted single NVIDIA RTX6000 Ada-48GB GPU, where full KV cache of 12K tokens with float will cause an OOM (Out-of-Memory) issue. The difference between V and Dropping shows the computation overhead of compression.
Models
4K
8K
12K
16K
32K
Full Cache
62.93
143.46
OOM
OOM
OOM
Dropping
62.93
125.86
188. 79
251.72
503.43
FreqKV
62.93
125.90
188.85
251.81
503.63
Compression Overhead
0 (0%)
0.039 (0.031%) 1%)
0.064 (0.034%)
0.090 (0.036%)
0.193 (0.038%)
Compression Count
20


Reviews:
[{"review_id": "sNPz3ioNcI", "reviewer": "Reviewer_DurU", "summary": "authors propos FreqKV, a parameter-free, architecture-agnostic KV-cache compression method for LLMs. It applies a DCT along the sequence axis to KV tensors, keeping low-frequency components, then IDCTs back, with an iterative schedule that re-compresses older tokens as the cache grows.", "questions": "see. weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Clear idea with strong intuition that low-frequency energy concentration in KV states\n\n2. Comprehensive experiments on multiple benchmarks illustrated the effectiveness of the proposed method.\n\n3. The ablation study provides more detailed information on frequency choice.", "weaknesses": "1. While the low-frequency concentration is a great motivation, the paper doesn't explore why this happens or what information is stored in which frequency bands. For instance, is the low-frequency \"global context\" and the high-frequency \"local token-specific details\"? Authors may provide a deeper analysis here to provide valuable insights\n\n2. Attention heads, layers may carry different spectral content. A per-head adaptive $\\gamma$ or power-based cutoff might outperform fixed ratios. Do the authors have ablation or adaptive strategies here?\n\n3. Authors may need to compare the most recent baseline [1]\n\nReferences:\n\n[1] LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models, ICML'25", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RaQJTYibgx", "reviewer": "Reviewer_35T2", "summary": "The paper proposed FreqKV, a novel KV cache compression method. FreqKV performs KV compression in the frequency domain using the DCT, retaining only the low-frequency components. After the fine-tuning, LLMs (Llama2 and Llama3) show comparable performance to the uncompressed one, and are superior to existing KV compression methods. The paper also reports generation speed improvements.", "questions": "* The related work section would benefit from a broader discussion of recent streaming-based long-context KV management methods, such as InfLLM, InfiniPot, and Minference.\n* The rescaling formulation (Eq. 5) is somewhat questionable. Instead of scaling by the number of retained coefficients, it might be more reasonable to adopt an (spectral) energy-preserving normalization. \n* (minor) The abstract could be updated to refer to Llama 3 rather than Llama 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea of applying frequency-domain compression to the KV cache is both intuitive and novel. The similarity to JPEG compression makes the concept easy to understand, and the empirical results demonstrate that this approach is competitive with or superior to existing baselines.\n* The paper includes extensive comparisons with a wide range of prior methods on multiple datasets.", "weaknesses": "* The paper lacks a detailed analysis or intuitive explanation of why low-frequency components dominate in the KV cache. Moreover, the large magnitude of low-frequency components does not necessarily imply that they are semantically important, yet the paper seems to make this assumption. While the empirical evidence supports the method’s motivation and effectiveness, a more analytical approach would strengthen the argument.\n* Although the authors claim speed improvements in both the prefill and decoding stages, the experiments seem to focus on decoding latency. The paper should provide either empirical results or additional clarification regarding the prefill stage speed-ups.\n* In Table 3, competing methods fail completely (near zero point) at the 16K context length. This might not solely reflect a generalization failure but could be caused by an implementation difference: FreqKV applies pre-RoPE compression (on-the-fly RoPE just before inference), whereas others employ post-RoPE compression. It would be essential to clarify this difference.\n* FreqKV requires a training phase for parameter learning; however, it remains unclear how it performs without such fine-tuning. Furthermore, comparisons in Table 3 should also include other training-based methods (e.g., LoCoCo).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "76XHV2FKpR", "reviewer": "Reviewer_phR3", "summary": "The authors propose a form of cache compression based on converting the tokens in the cache into the frequency domain for compressed storage. This results in lower storage requirements, lower attention complexity in decoding, and a kind of natural context extension since positional embeddings since the compression projects the cache into a fixed cardinality which will not expand to use up all of the positional embeddings.", "questions": "For a clearer understanding of how the DCT and IDCT matrices transform the inputs, can you add the dimensions for both the DCT and IDCT matrices?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- KV cache compression is an important topic due to the efficiency gains and power consumption concerns of modern transformers.\n\n- In addition to compression the cache, the method also seems to cause an efficiency gain in the decoding attention operation.", "weaknesses": "- Llama 3 is used as a baseline model. This is important because I believe the only reason some baselines show poor performance is because they have exceeded the number of training positional embeddings. However, Llama 3 is already 1.5 years old at this point. There are already 3 releases of Llama3 which go up to 3.3 and have 131K native positional embeddings. Can FreqKV be applied to these models and show the same good performance past 131K?\n\n- There is no comparison of latency with baselines such as SnapKV.\n\n---\n\nOverall, it would be more convincing if the authors could provide a case on what happens with extremely long contexts. Due to the compressive nature of the cache, it may only be able to hold information up to a certain point before the compression becomes noise. However, if the same trend witnessed at 8K-->16K can be witnessed on a 131K-->232K model, I would find this very compelling. This could be done with the exact same experimental setup and swapping in the Llama 3.x models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)