FASA: FREQUENCY-AWARE SPARSE ATTENTION
Authors: Yifei Wang
Yueqi Wang
Zhenrui Yue
Huimin Zeng
Yong Wang
Ismini Lourentzou
Zhengzhong Tu
Xiangxiang Chu
Julian McAuley
Venue: ICLR 2026 Poster (2026)
Keywords: Functional sparsity of FC; KV cache
Primary area: foundation or frontier models, including LLMs
TL;DR: We discover that RoPE is intrinsically sparse at the "frequency chunk" level and leverage this to build a zero-cost, query-aware KV cache pruner that rivals full-attention performance.

Abstract: The deployment of Large Language Models (LLMs) faces a critical bottleneck when handling lengthy inputs: the prohibitive memory footprint of the Key Value (KV) cache. To address this bottleneck, the token pruning paradigm leverages attention sparsity to selectively retain a small, critical subset of tokens. However, existing approaches fall short, with static methods risking irreversible information loss and dynamic strategies employing heuristics that insufficiently capture the query-dependent nature of token importance.
We propose FASA, a novel framework that achieves query-aware token eviction by dynamically predicting token importance.
FASA stems from a novel insight into RoPE: the discovery of functional sparsity at the frequency-chunk (FC) level. Our key finding is that a small, identifiable subset of "dominant" FCs consistently exhibits high contextual agreement with the full attention head. This provides a robust and computationally free proxy for identifying salient tokens.
Building on this insight, FASA first identifies a critical set of tokens using dominant FCs, and then performs focused attention computation solely on this pruned subset.
Across a spectrum of long-context tasks, from sequence modeling to complex CoT reasoning, FASA consistently outperforms all token-eviction baselines and achieves near-oracle accuracy, demonstrating remarkable robustness even under constraint budgets. Notably, on LongBench-V1, FASA reaches nearly 100\% of full-KV performance when only keeping 256 tokens, and  achieves 2.56$\times$ speedup using just 18.9\% of the cache on AIME24.

Published as a conference paper at ICLR 2026
FASA: FREQUENCY-AWARY SPARSE ATTENTION
Yifei Wang' Yueqi Wang Zhenrui Yue Huimin Zeng'
Yong Wang
Ismini Lourentzou
Zhengzhong Tu Xiangxiang Chu
Julian McAuley²
1 AMAP, Alibaba Group 2 UCCS 3 SUIC
a Texas A&M University
ABSTRACT
The deployment of Large Language Models (LLMs) faces critical bottleneck when handling lengthy inputs: the prohibitive memory footprint of the Key Value (KV) cache. To address this bottleneck, the token pruning paradigm leverages attention sparsity to selectively retain a small, critical subset of tokens. However, existing approaches fall short, with static methods risking irreversible information loss and dynamic strategies employing heuristies that insufficiently capture the query-dependent nature of token importance. We propose FASA, a novel frame- work that achieves query-aware token eviction by dynamically predicting token importance. FASA stems from a novel insight into RoPE: the discovery of func- tional sparsity the frequency-chung identifiable subset of "dominant" FCs consistently exhibits high contextual agree- ment with the full attention head. This provides robust and computationally free proxy for identifying salient tokens. Building this insight, FASA first identifies a critical set of tokens using dominant FCs, and then performs focused attention computation solely this pruned subset. Across a spectrum of long-contex, tasks, from sequence modeling to complex CoT reasoning, FASA consistently outper- forms all token-eviction baselines and achieves near-oracle accuracy, demonstrating remarkable robustness even under constraint budgets. Notably, on LongBench- V1, FASA reaches nearly 100% of full-KV performance when only keeping 256 tokens, and achieves 2.56x speedup using just 18.9% of the cache on AIME24
(FC) level. Our key finding that a small,
INTRODUCTION
Despite recent advances in Large Language Models (Dao et al. 2022 Ainslie et al. 2023 Li
et al. 2024a Wang al., (025c) in long-context processing, requirements such repository-leve code analysis (Chen et 2021 Shi et al. 2025) Wang et al.) 2026 Chen et al., 2023b) and
document summarization (Goyal Durrett 2020 pose both memory and computational challenges, especially the linear growth of the KV cache. As the sequences grow, each token generation requires accessing the entire KV cache, leading to increased memory 1/O latency. This memory-bound process underutilizes perforreffrmecce GPUs, ultimately limiting the overall throughput. To optimize KV cache management, previous studies have proposed mainly five directions: token eviction (Akhauri et al.
2025 Fang et al. 2025), low-rank compression (Chang et al. 2025 Singhania et al., 2024
Zhang et al. 2025). quantization (Hooper et al.) 2025b Liu et al. (024d)) KV merging (Wang et al.
2025d. Wan et al.) 2025 Liu et al. 2024b),, and budget allocation (Cai et al. 2025b).
An intuitive and widely explored approach is token eviction (LI et al. 2025 Liu et al. 2023). The rationale is that only small subset of tokens contributes significantly to outputs, enabling the selective removal of trivial ones. Existing token eviction methods can be classified into three types: (1) Static strategies remove tokens with fixed rules (Xiao et al. 20244), therefore risking irreversible information loss; (2) Adaptive strategies either permanently evict less critical tokens (Zhang et al.
2023 Li al. 2024) preserve the full cache while retrieving subset of entries [Tang et al. 2024
Ge et al. 2024). Yet such heuristic rankings provide an imperfect proxy for the truly dynamic nature
of token importance; (3) Learning-based strategies (Akhauri et al. 2025 Yang et al. 2025 Chen et al. (222aa) rely on a trained token predictor, suffering from poor generalization on different datasets. Can a token predictor achieve query-awareness without resorting to costly training? ?
PProject lead and corresponding author.
Cooe available at httt:://tthaab.oo/AAAAAMMMM CAA.AAAAAMAAPIIAAAAAACCCCCAAO

---

Published as a conference paper at ICLR 2026
In response this question, we introduce FASA (Frequency-Awar Sparse Attention). a training- free, high-granularity, query-aware predictor designed to evaluate token significance during the decoding phase, in training-fite frre manner. The design of FASA is rooted in an intriguing observation that differential frequencies within RoPE (Su et al. 2023) induce functional sparsity among frequency chunks (FCs). Only a sparse subset of FCs, termed dominant FCs, contribute significantly to contextual awareness, while others construct robust positional patterns. We empirically verify that these dominant FCs are sparse, universal, and task-agnostic in Section 3.3 thereby providing a robust
foundation for accurately predicting token importance.
Building upon this insight, FASA employs two-stage framework for efficient inference. The first stage, Token Importance Prediction, harnesses dominant FCs to dynamically estimate attention scores, obtaining critical tokens. the second stage, Focused Attention Computation then performs precise and focused token generation on this reduced set. The overhead of FASA is minimal because the identification of dominant FCs a one-time and task-invariant process. Ultimately, FASA achieves high efficiency by fetching only small fraction of the KV cache, which significantly reduces the data transferred between memory and the processor and thereby lowers memory bandwidth consumption. The overview of FASA is in Figure 2 Grounded the same principles above, we introduce two variants of FASA: FASA-M and FASA-C. While they differ in implementation strategies, both achieve equivalent downstream task performance while offering different efficiency profiles, specializing in memory and computation, respectively. Crucially, despite FASA leverages a low-rank subspace, its primary objective the dynamic prediction of token importance, not mere dimensionality reduction. This design makes FASA orthogonal to and compatible with most other KV cache compression methods. For example, it can be seamlessly integrated with layer-wise budget allocation schemes like PyramidKV (Cai et al. 2025b).
We evaluated FASA across range of LLMs with varying KV cache budgets, concentrating on three core tasks: Long-contex, benchmark, long-sequence modeling, and long chain-of- of--thugtttt (LongCoT) reasoning. Our method achieves performance comparable to that of full KV cache, with reduction of less than 0.7%, while consistently surpassing all baseline methods across these tasks. FASA-M provides an 8x compression of the KV cache, substantially optimizing memory usage, and FASA- C delivers 2.6x speedups, enhancing computational efficiency, with 25% of FCs selected. Our contributions are summarized as follows:
We are the first uncover an intriguing finding: functional sparsity at FC-level induced by RoPE. Leveraging the functional sparsity of FCs, we introduce FASA, a training-free framework for dynamically predicting token importance.
We present two variants of FASA: FASA-M, optimized for settings with memory constraints, and FASA-C, designed for scenarios with computational constraints.
Extensive experiments across three paradigm tasks demonstrate that FASA consistently achieves near-oracle accuracy in both long-context and long-generation tasks.
2 RELATED WORKS
Token Eviction. A central theme in recent KV cache optimization (Hooper et al. 2025a Wang let al. 2025a) is the exploitation of inherent, query-dependent attention sparsity (Liu et al. 2024c 2025a Behnam et al. 2025). Stream (Xiao et al. 2024) employs a rigid heuristic, preserving only initial and recent tokens, which invariably discards potentially crucial information from intermediate positions. SnapKV (Li et al. 2024) improves on this by introducing a one-time, prefill-stage filtering based on empirically estimated attention scores. However, the static nature of this estimation cannot adapt to the evolving relevance of tokens as generation progresses. Quest (Tang et al.) (024)) offers a more dynamic solution by organizing the KV cache into pages and selectively fetching them. Despite its dynamism, its efficacy hampered by a coarse, page-level granularity, which incurs significant overhead by forcing the retrieval of entire pages even when only few tokens are needed.
Low-rank Compression. Another prominent paradigm for KV cache compression low-rank approximation (Zhang et al., 2025 Dong et al. 2024), predicated on the observation that the cache's 's information content is concentrated in a low cww-dimesiiona subspace (Sun et al., 2025 sax 2024] Behnam et al. 2025). For instance, SparQ (Ribar et al. 2024) employs a heuristic that selects key dimensions based on high query-vector magnitudes, a strategy that proves suboptimal due to its head-agnostic nature and its simplistic reliance on magnitude a proxy for importance. Similarly,
2

---

Published as a conference paper at ICLR 2026
LoKi (Singhania et al. 2024) leverages Principal Component Analysis (PCA) to project key states into a compact subspace for efficient computation, but at the cost of significant memory overhead from storing the requisite projection matrices. In contrast, our proposed FASA cireumvents these limitations by operating in in-place on the KV cache, thereby incurring no auxiliary memory overhead.
3 OBSERVATION
3.1 PRELIMINARY: ROTARY POSITIONAL ENCODINGS (RoPE)
RoPE embeds relative position information into the self-attention computation. Specifically, for query vector qt, and a key vector k,, at positions 4 and 12, the attention score is formulated as At,,t = (q, Re, (k,, R,,)² R,,) T qr, ,. Due to the orthogonality, the product of RA, and R elegantly simplifies to single rotation matrix parameterized solely by the relative offiet At = t1 22
RoPE. From frequency-domain perspective, the RoPE mechanism can be interpreted through the concept of "frequency chunks" (FCs). This framework posits that any d-dimensional vector Ed (e.g.. query and key) is partitioned into d/2 orthogonal 2D subspaces. We denote the i-th such subspace, or FC, v associated with unique base angular frequency, calculated as 02 = B- 2(1-1// for i {1..... d/2), where B is predefined frequency base. This design establishes a direct mapping from a chunk's dimensional indices (2i, 2i + 1) to its rotational frequency. Lower dimension indices (i) result in higher frequencies which implies that the corresponding FCs rotate very quickly physically. For a token absolute position m, its i-th FC is rotated by angle m0f; through specific 2 x 2 rotation matrix Rm, The global rotation matrix R& is block- where each diagonal block is a x 2
A Frequency- Chrnkk Perspective
(V21, U24+1)² Each FC is
rotation matrix Rsc,0, and defined as RA
RAr,,2...... RAn. +
4/2 Rac..
d/2
d/2
(cos(m).)
-ii((m0)))
①
(V2,, ,am,.)
(1)
m
(mn)))))) cos(me;)
k=1
3.2 MOTIVATION AND HYPOTHESIS
Different Roles of FCs. The varying rotational velocities across FCs inherently lead to functional heterogeneity. This principle is substantiated by two key observations from prior literature. First, a distinet division of labor exists within RoPE (Barbero et al.] 2025 Wei et al. 2025), where high-frequency FCs (in low dimensions) primarily responsible for constructing robust positional patterns, and in contrast, low-frequency counterparts specialize in carrying the semantic information and model long-range dependencies. Second, this functional specialization is structurally reflected by ioduucddde concentration of high magnitude values within specific query and key dimensions (Sun et al.) 2024), reinforcing the non-uniform functional importance of FCs. This functional heterogeneity suggests that FCs can be grouped into two distinct categories:
Position vs. Semantics:
1. Contextual FCs: A small, critical subset responsible for dynamic, context-specife attention.
These FCs identify which tokens are semantically relevant to the current query.
Structural FCs: The remaining majority primarily injects inherent, positional attention patterns, mainly recency bias (Peysakhovich & Lerer 2023) and attention sinks (Xiao et al., 2024).
The model's contextual awareness overwhelmingly driven by the Contextual FCs. A few contextual FCs could replicate the contextual selection behavior of full attention head. If their index set is denoted Isom C. .d/....d//),. the full attention dot product can be effectively
Hypothesis:
approximated by summing only over Ialoms namely At, = qr, RAcK!, Eieton q RAL.o. k!
3.3 QUANTIFYING FUNCTIONAL SPARSITY
Quantifying hypothesis of FC-level functional sparsity requires a metric assess the "dominance" of individual FCs. Therefore, we propose the Contextual Agreement (CA) metric, which measures the alignment between the attention pattern from single FC and that of the full attention head.
Formal Setup. For a query qe E Rdd and key matrix Kitt E Rdxtt in an attention head (1, h), we
define score vectors: the standard full-head scores an, and the single-FC scores a The
(1)
3

---

Published as a conference paper at ICLR 2026
Heta-LLame-3.1163 11-88 Layer 25
Mistral-7B Layer 10
Index
Frequency www.oee
Prequency www.
Figure 1: Functional sparsity of FCs revealed by Contextual Agreement (CA) heatmaps. Each heatmap shows CA per FC (æ-axii)) across all heads (y-axis). A few "dominant" FCs (bright vertical bands) consistently capture contextual information across attention heads. Results on Qasper (K = 256); see Appendix
latter are computed using only the 2D components of the i-th FC. These are expressed as:
011 (q: K.te) = [q: R (ko) ..... qi Ro (k,)
(2)
(8) aihh (4, k1t)) [q. R kg ! q i k. 10
Ro.e,
Metric Definition. The CA score, CA
L.h.i quantifies the agreement between the full-head on and single-FC (1) scores by measuring the normalized intersection of their top-K token index sets: (q, = [TopK-I(au,, K1:)), KC) n TopK-I(a,,, (q:, K))/,,
(4) where the operator TopK-I(ax, K.) retrieves the top-IC values of vector a. To assess FC's importance robustly, we compute its mean CA score, by averaging across several samples from specific dataset. Figure reveeals the distinct functional contribution of each FC across all heads.
Sparse and Universal Zdom- (1) Sparsity: A small subset of FCs (dominant FCs) exhibits dispropor- tionately high agreement with full attention pat- terns, while the vast majority of other FCs have negligible CA scores (typically 0.15). In Table 9. dominant FCs account for less than 1% of all FCs, while non-dominant FCs with low CA scores comprise approximately 90% or more; (2) Univer- sality: The functional sparsity is widely observed across model architectures and scales (Appendix A.I]Table 9): (3) Task- Invarince:: The identifica- tion of dominant FCs is largely task-agnostic. The saliency maps in Figure 12 derived from tasks such QA and summarization exhibit remarkable con-
Table Compound CA scores under varying number of selected FCs (F) and KV cache bud- gets (K). Each head has 64 FCs total.
K
256
512
768
1024
2048
Tdomn
Random
2.0
6.4
25.5
$1.1
Stream
34.4
26.8
24.4
26.5
30.7
53.9
SnapKV
37.9
40.9
41.9
45.4
49.5
66.6
F (1/8)
43.0
49.4
54.3
58.8
62.6
76.1
F
46.4
52.1
56.6
61.1
64.8
77.5
F.
49.7
54.7
58.9
63.4
66.8
79.0
F=
52.4
56.9
60.9
65.2
68.5
80.2
F (1/4)
55.3
59.7
62.8
66.9
70.1
81.4
sistency. Quantitatively, the overlap of dominant FCs across different calibration datasets consistently exceeds 70% in all tested models, as reported in Table IO This indicates that the functional roles of FCs are intrinsic to the underlying mechanics of RoPE, rather than being task adaptations.
Reconstructing Functionality from Ieom- The analysis above supports that the functionality of a full attention head can be reconstructed using only its most dominant F components Z To Lh.i |0 f < <d/2, F). Therefore, we measure the collective efficacy of this subset T.h.Tame I. and present the results in Table For comparison, we
using compound CA score, CA benchmark against token-eviction methods, which serve to emphasize the capability of predicting token importance. Our method demonstrates remarkable efficiency: with just 1/8 of the components selected under tight budget 64, Iuom achieves an accuracy of 43%, surpassing the strong baseline SnapKV (Li et al., 2024) by an average of 10.3% across all budget levels. We also present the predictive distribution of dominant FCs across tokens grouped by attention score quintiles in Table ability of dominant FCs to effectively capture both the relative ranking and true impact of context tokens.
K
IT This analysis further substantiates
4 METHOD
Grounded in the functional sparsity of FCs, our training-free framework FASA employs a two- stage, coarse-to-fine strategy to circumvent the prohibitive cost of full self-attention. First, the
4

---

Published as a conference paper at ICLR 2026
of calibration
TIP
FAC
Input
seq len
head 1
fc map
G
selaction
Q
prediction
selection
Vegts
dominunt RCx
dominast
head
Figure 2: Method Overview of FASA, First, the TIP stage leverages only dominant FCs to efficiently estimate token importance and select critical subset of tokens. Then, the FAC stage performs full-dimensional attention exclusively on this reduced subset to generate the next token. See discussion about design Appendix [D.2
Token Importance Predictor (TIP) stage utilizes computationally frugal proxy, defined by pre-calibrated set of dominant FCs, Taoom to efficiently identify a small subset of contextually salient tokens. Subsequently, the Focused Attention Computation (FAC) stage performs a full-fidelity attention computation exclusively on this salient subset, preserving high generation fidelity while drastically mitigating the computational and memory overhead of standard attention.
4.1 TOKEN IMPORTANCE PREDICTOR (TIP)
The TIP stage operates on the principle that dominant frequencies efficient proxy for token importance, where the dominant indices Idom are identified via one-time offline calibration.
Offline Calibration: Identifying Idom. The objective of the offline calibration is to identify
small, head-specific set of dominant frequencies, I for each attention head h1, We formulate this process as a search problem over frequency indices. Given a small calibration dataset and a target size Nrip. our goal is find the subset of FCs of cardinality Nzip that maximizes the expected average of CA scores. The objective is defined as:
Il.b dom
Σ C LiET
arginax IC .0.......///// w/t.d////-----111111
(5)
This calibration is highly efficient, one-time offline process because the resulting Ioom is empirically found be task-agnostic and can be robustly identified from minimal number of samples. Its associated computational cost is negligible. The detailed algorithm is provided in Algorithm
Online Prediction: Importance Scoring via Frequency Subspace Aggregation. During the online prediction phase at a given decoding step f. we leverage the pre-calibrated set of dominant frequencies, Z/A to efficiently estimate token importance in training-free manner. Conceptually, the full attention score for a query q, and keys k,tt can be decomposed into a sum of contributions from
all d/2 frequency components: a Kı+)) =i=0 h.. (q:, K Instead of performing this computationally expensive summation, our method constructs an importance score vector s.. by exclusively aggregating the contributions from the pre-identified dominant frequencies, i..., S EiETIA h. (q,, K1:). This formulation strategically bypasses computation for non- dominant frequencies. Finally, based these scores, identify the of top-N most important token indices, Te, for the subsequent FAC stage: Ti = TopK-I(S!., Naa)).
4.2 FOCUSED ATTENTION COMPUTATION (FAC)
Following the identification of the contextually important token set by the TIP module, this stage executes an attention computation on Te, enabling the model to concentrate its computational resources on the most salient parts of the context. Specifically, for the current query vector qi at decoding step f, instead of using the full key and value matrices (K1st, V1+) from the entire past context, we first gather the keys and values corresponding to the indices in T;:
K Gather(K, Tr), VT. Gather Ta)
(6)
5

---

Published as a conference paper at ICLR 2026
where the Gather(-) operation selects the rows from the original matrices specified by the index set Ti. The attention scores for each head (1, h) are then computed using only these selected keys. The final output vector for the head is subsequently produced by weighting the selected value vectors:
L.b FAC
Softmax (q.KT."VVa)
O
FAC VT.
(7)
Critically, the original absolute positions of the tokens in Ti are preserved. This directly maintains the integrity of their position embeddings and the vital spatial information they encode, preventing the performance degradation associated with positional distortion. In essence, the FAC stage functions as a high-fidelity computational filter, restricting full-precision attention to the most salient tokens to achieve compelling balance between computational efficiency and predictive accuracy.
4.3 Two IMPLEMENTATION OF FASA
We introduce two specialized, hardware-aware variants of FASA that offer a trade-off between memory and speed: (I). FASA-M (Memory-Optimizedl minimizes its GPU memory footprint by strategically offloading the value cache and non-dominant key components to CPU memory, making it ideal for TRAM-constraingg this approach can be effectively paired with prefetching techniques. (2) FASA-C (Computation- Optimized) prioritizes inference speed by retaining the full cache on-GPU but accessing only a sparse subset of key states, drastically reducing memory I/O for significant acceleration. (See Appendix D.] for details and memory analysis of FASA-M).
enviromments. To mitigate the latency from CPU-GPU data transfer,
4.4 EFFICIENCY ANALYSIS OF FASA
Computational Analysis. At the generation step i,, the complexity of computing q: Kr is Ottd) and the com- plexity of multiplying the value states with attention scores
20
Decode
10
Prefill latency ratio (%6
15
Prefill
is (ttd) per head. For FASA, (1) the complexity of the TIP stage is ((2.NNup) (each FC takes up 2 dimensions),
Prefill ratio
Latency
0.1
0.1
0.2
0.4
since this stage operates in low-dimensional subspaces, and (2) the FAC stage performs attention on a reduced set of Nac tokens, leading to a complexity of ONN Additionally, the detection dominant frequencies Idom offline. one-time, and applicable for various tasks and the burdens from this part could be neglected. Assuming the complexity of selecting the top-k tokens is small, the overall complexity of FASA is (2tNNrsp tip + (2)..) The theoretical speedup decoding stage Equation g
10
8.3
15
20
20.4
1024
2048
8192 16384 32768
Sequence Length (# Tokens)
Figure 3: Decoding latency dominates total latency in auto-regressive generation.
21d
d. d_ if NN <
Speedup 2t.Niip + 2N
Nupp / + Njuc/t" Speedup RJ
(8)
Nisp
Memory Movement Reduction. The auto-regressive decoding stage is notoriously memory-bound, as requiring loading the entire KV cache, creating significant latency bottleneck. This confirmed in Figure 3 where decoding constitutes 90% of the total latency at a 32K context. FASA, directly mitigates this bottleneck by drastically reducing memory traffic. At a decoding step t, standard attention loads 2tm bytes from the KV cache (with m as the byte size per state vector) while FASA accesses only ((NNeip/ m) bytes (only keys) for the TIP and 2. bytes for the FAC. The fraction that FASA must load is therefore: (26mNrip/ee + 2N 2. 2tm Nrip/d + Nfac/ RJ Neip/ ((NNge fac < l), which alleviates the memory-bound constraint of long-context decoding.
EXPERIMENTS
5.1 EXPERIMENTAL SETTING
Baselines and Models. To comprehensively evaluate FASA's performance, benchmark it against into two groups of robust baselines: (1) State-of-the-art methods: We compare against leading token eviction methods in efficient KV cache management, including Stream (Xiao et al.) 2024).

---

Published as a conference paper at ICLR 2026
Table 2: Performance of FASA on diverse models on Long VI benchmarks. For baselines, we retain
constant token budget (256) and 25% FCs for FASA. 'FKV and Oracle are full and look-ahead upper bounds.
Single-Due QA
Multi-Doc QA
Summarize
Summarize
Synthetic
Code
Method
NOA
Qaep
MF-en
Hqja
2Wiki
Musi
Gork
Quum
Mult
Tree
Tya
Pent
Pre
LEc
RB.P
AVG.
Jama3.2.0 2-3
FFV
26.0
40.7
50.4
32.2
29.6
15.1
33,5
22.9
25.3
71.5
88.9
3.5
87.8
52.0
54,2
42.2
Oracle!
26.6
41.2
49.8
31.9
29.9
162
32.6
22.2
25.0
715
89.
3.5
88.0
53.7
54,4
42
02
Quest
8.7
19.5
23.6
12.9
15.9
6.5
23.
18.
25.1
34.5
52.9
6.5
38.3
53.7
43.6
25.5
162
Stream
13.2
19.7
23.6
18.1
22.7
7.8
18.2
17.9
17.9
49.0
83.7
3.5
85.7
49.3
45.9
318.818888
SnapKV V
23.5
28.9
45.6
17.7
22.9
11.8
21.7
20.9
21.1
61.0
88.5
3.5
88.0
50.7
48.6
37.0.52
FASA
25.6
38.9
49.9
29.7
31.2
14.8
28.0
24.2
26.1
71.5
89.2
3.6
86.9
53.2
50.5
41.5.07
2wen2 5.78
FKV
24.2
43.5
52.1
55.9
46.9
28.6
31.8
23.1
23.9
71.5
89.3
7.5
92.0
60.2
66.5
47.8
Oracle
24.4
43.0
52.3
57.8
46.9
30
31.6
23.9
24.1
72.5
89.7
8.0
100.0
60.5
65.3
48.7"o9
Quest
9.1
24.5
30.4
24.7
24.1
8.8
26.8
19.9
24.4
41.
66.7
4.4
77.6
46.5
42.0
314-164
Stream
18.1
24.2
26.5
41
36.4
17.3
18.4
18.3
15.4
45.0
82.9
8.5
24.0
49.6
52.2 3199159
Snup V
26.6
36.0
50.8
55.6
43.8
26.5
21.9
21.9
19.3
58.0
86.2
8.0
98.5
35.6
60.6
42.6.52
FASA
28.3
43.8
51.9
57.4
46.0
30.1
31.2
22.8
24.3
72.0
89.4
8.0
99.5
60.3
64.0
47.9 01
V
29.1
41.6
52.9
49.4
39.5
29.1
34.8
25.7
27.8
76.0
88.6
5.5
98.0
58.4
59.7
47.4
Intall
Oracle!
31.0
40.2
52.4
50.3
39.4
28.8
34.0
25.74
27.2
76.0
89.4
5.0
98.0
59.3
61.0
47.9
Quest
15.7
30.7
41.0
37.4
27.1
119
29.3
21.3
26.6
57.0
80.7
5.0
85.5
56.9
53.0
38
Stream
11.8
15.3
20.9
32.1
27.1
10.6
20.2
17.3
20.1
44.5
69.0
1.6
3.2
56.5
49.8
26.7 $20.7
SnapKV
25.5
32.6
53.7
48.4
37.3
25.9
22.7
23.6
23.1
62.5
89.4
6.5
94.5
57.3
57.0
44.0
FASA
29.9
42.3
53.7
51.1
39.1
28.7
34.0
24.8
28.2
76.0
89.4
5.0
98.0
57.8
58.0
47.8
04
Lann3 1-BB
FKV1
30.0
45.3
55.6
55.8
43.7
30.2
35.1
25.4
27.0
72.5
91.7
7.1
99.5
63.0
56.3
48.7
Oracle!
30.3
44.5
55.0
54.9
44.6
32.0
34.8
25.1
26.9
72.5
91.5
7.0
99.5
63.3
57.4
78.7.00
Quest
13.7
33.1
38.4
35.8
32.2
12.8
26.5
20.9
26.7
38.0
65.6
388
95.0
52.5
45.7
35.4-133
Stream
21.9
23.4
31.8
45.1
36.7
24.3
20.0
21.0
19.3
45.5
87.9
6.9
99.5
59.4
49.
38.8
Snapk V
27.5
34.5
51.6
52.3
44.3
28.3
23.9
24.0
22.7
62.5
90.9
7.5
99.5
60.1
52.6
45.0.37
FASA
29.3
43.7
54.1
54.8
43.9
30.8
33.5
24.7
27.0
72.0
91.1
7.5
99.5
61.8
52.7
48.2.
FKV1
28.7
46.2
53.8
65.2
64.5
43.6
43.5
23.3
22.7
80.5
89.5
100.0
32.3
37.5 50.3
w2.5.
Quest
Oraclet
28.5
14.5
31.9
46.3
54.3
39.1
64.3
38.8
63.6
36.6
44.7
16.2
31.5
16.2
20.1
22.9
25.2
22.7
43.5
81.0
88.4
72.7
10.0
100.0
88.8
33.6
35.0
34.0
39.7
34.9.154
49.4.09
Stream
19.6
26.9
46.5
48
29.6
17.8
18.4
15.0
46.5
82.5
12.5
72.1
28.7
31.2 35.3.150
SnapKV
26.3
40.5
51.2
63.2
62.2
43.3
22.5
22.0
18.3
63.5
87.5
11.5
100.0
30.4
36.0
45.9
FASA
27.2
45.5
54.5
64.4
63.9
44.5
30.4
22.8
21.9
80.0
87.5
15.5
100.0
30.5
36.1
49.2
SnapKV Li et al. 2024), RKV (Cai et al. 2025a), Quest (Tang et al. 2024). H2O (Zhang et al. b:23;; (2) Upper bounds: two theoretical bounds, FKV, which represents standard inference with the complete, uncompressed KV cache, serving as the absolute performance ceiling due to no information loss, and Oracle, more pragmatic upper bound for eviction-based methods, assuming ideal knowledge to retain only the most critical tokens based on full-head scores. Our experiments span a variety of cutting-edge architectures and model sizes, specifically Llama (Touvron et al. 2023).
Mistral (Jiang et al. 2023), and Qwen (Bai et al. 2023).
Evaluation Benchmarks. To rigorously assess the capabilities of FASA across diverse long-context scenarios (Liu et al. (2025b). we conduct comprehensive evaluations spanning three paradigms: (1)
Long-context understanding We use diverse, real-world tasks from LongBench (Bai et al. 2024)
to assess the ability to identify critical information within lengthy contexts.
(2) Long-Sequence
Modeling:
We measure perplexity on PG-19 (Rae et al. 2019), WikiText (Merity et al. 2017), and
C4 (Raffel et al. (219)) corpus to evaluate generative fidelity over long dependencies. (3) Long-CoT
Reasoning:
To test performance in long-generation scenarios, we evaluate on complex mathematical
reasoning tasks from MATH500 (Hendrycks et al. 202T) and AIME24 (MAA 2024) on R1-LLMs.
5.2 PERFORMANCE COMPARISON ON LONG-CONTEXT TASKS.
240
20.0
20
5- Wickest
Qwe C4
11.
15
123
04
24
24
14
Figure 4: Perplexity results of FASA in comparison with FKV. Oracle, Stream, and Quest on Wikitext (top). PG19 (middle), and C4 corpus (bottom). Token sparsity indicates the retained ratio of tokens.
7

---

Published as a conference paper at ICLR 2026
Table 3: Performance and output length of FASA compared to baseline models on the MATH500 and AIME24 Nepp = 16. AIME24 results are reported as pass@1 1, based on 16 responses per question. PREF and DEC denote the prefill and decoding lengths, respectively. 'FKV and Oracle are full and look-ahead upper bounds.
MATH500
AIME24
Methods
Fixed Budget
Len Stats
Fixed Budget
Len Stats
300 500
700
1000
PREF
DEC TOTAL.| 500 1000 1500 2000 2500
PREP
DEC TOTAL.
DeepSeck-RR AL1-Disill-Llammlee
FKV1
72.4
72.4
2977
3104
43.9
43,9
13231
13392
Orucle
H20
70.4
6.8
72.6
33.0
53,87
24.2
42.8
71.8
127
8244
3195
8370
332]
30.0
0.7
36.7
47
37.3
39.3
14.0
20.0
36.0
161
21099
15638
21260
15799
Stream
9.6
24.6
40,4
47.4
3520
3647
0.0
3.3
8.6
10.7
15.3
10191
10352
Snapk V
21.6
32.6
46.8
54.6
7047
7174
4.0
8.0
16.0
23.3
29.1
17359
17520
RKY
24.
39.4
$7.0
7005
6.7
14.0
23.3
22916
23077
FASA
62.2
68.8
69.4
71.8
3171
3298
20.6
34.4
40.2
35.8
38.0
17166
17327
DeepSeck- -ii---R---DD-OO-OOO Sollll-wwe--1111
FKVt
92.4
92.4
2784
2914
66.6
66.6
11039
11204
Orucle
H20
92.2
29.6
50.2
92.4
92.4
62.8
92.2
77.0
127
2985
3413
3112
3540
67.9
5.3
20.5
66.7
67.3
37
70.7
67.3
52.7
165
11546
9519
11711
9684
Stream
27.8
44.0
57.8
64.
2801
2928
2.0
4.0
22
29.3
8468
8633
Snapk
34.
55.8
69.4
79.4
3586
3713
10.0
23
40.0
46.0
52.7
11922
12083
RKV
57.8
74.0
86.4
3865
3992
20.7
30.0
46
62.0
16274
16439
FASA
90.2
91.2
3139
3266
54.0
63.3
11553
11709
DeepSeck- Baicck-R--Dssilll DDistil-Qwenn3211 Qwen-32B
FKV1
92.6
92.6
2717
2846
72.8
72.8
10461
10626
Oracle!
H20
47.2
92.4
50.0
68.3
91.2
74.4
127
2886
3841
3968
3013
0.7
16.7
38
70.0
76
45
69
55.0
156
10904
11545
11710
11069
Stream
43.6
57.6
65.6
73.4
2773
2900
0.7
23
24.7
10732
10897
Snapk V
49.6
66.0
74.8
80.8
3704
3831
10.0
23.1
40.0
46
52.7
13650
13815
RKV
75.0
72.2
83.6
4229
4356
18078
18243
FASA
86.4
91.2
2887
3014
60.7
62.0
66.3
70.0
73.2
11735
11891
122 PAGK
MAG 10
FASK 12
PSA TC 92
Qwen2.5-14B
-an--n---- 1M, TREC
DeepSeek R1 Ai-DDistll---e Qwea-14B, MATH
3880
90
80
Pefermane
60
70
60
500
Token Budget ATG
Token
BAC
Figure 5: Evaluation of FASA on TREC (left) and MATH (right) datasets. The plots show the synergistic
effects under varying numbers of selected FCs and different token budgets.
FASA achieves near-lossless performance under various budgets. FASA consistently outperforms all baselines across various budgets (Appendix C.I and (6. preserving contextual integrity even under extreme compression (Table 2 In stark contrast, existing token-eviction methods suffer catastrophic performance degradation; for instance, Quest's accuracy plummets by 13.4% on NarrativeQA, underscoring their inability to retain critical information. Remarkably, under extreme budgets, FASA occasionally surpasses the FKV baseline (e.g., Mistral-7B). We attribute this phenomenon to the mitigation of attentional distraction from irrelevant tokens. This hypothesis is corroborated by the Oracle baseline, which also outperforms FKV sometimes, thereby validating our frequency-chunk- based framework's 's efficacy in precisely identifying semantically pivotal regions.
FASA models complex long- term dependencies. We sim- ulate token-by-token decod- ing process wherein the evic- tion strategy iteratively ap- plied before token prediction. The fixed-rule approach of Stream (Xiao et al.) 2024), which relies on "attention sinks," severely compromises its ability to capture long-range dependencies, leading to a drastic increase in perplexity as shown in Figure 4 Similarly, Quest's ss coarse, page-level granularity prevents it from adaptively retaining critical, non- contiguous tokens. In contrast, FASA's 's fine-grained, query- dependent mechanism accurately iden- tifies salient tokens, achieving performance comparable to FKV, even under aggressive compression.
Oradle
Steam
Snapity
PF 41 10wan2 5 525))
HotpotOA (Qwand 5 3231
Cooffecort 12war23 3281
Dureacer 10ee2.55 110)
Zass
accurayy
0.20
030
6.15
Figure 6: FASA under various token budgets (Neip = 16).
FASA excels at long-CoT reasoning, The chain of thought in long-form reasoning is a fragile thread, requiring the preservation of dynamically shifting "thought traces", thread that prominent baselines
8

---

Published as a conference paper at ICLR 2026
consistently sever. As shown in Table 3 their static compression heuristics, blind to the evolving importance of tokens, lead to a precipitous drop in performance. On RI-Llama, SnapKV's accuracy collapses to 21.6, stark contrast to the FKV's s 72.4, demonstrating fundamental failure to sustain the very logical dependencies required for reasoning. Conversely, FASA operates with surgical precision. It surpasses not only standard baselines but also R-KV, highly specialized method for CoT compression. It achieves an impressive 86.4% accuracy on a scant 10% context budget, narrowly trailing the 92.6% FKV upper bound. This feat cements its status as superior framework, one that can navigate the intricate web of complex reasoning without severing the essential threads of logic.
5.3
IN-DEPTH ANALYSIS
Table 4: Compatibility of FASA.
Table 5: Ablation on K.
Table 6: Ablation of offline calibration.
Budget 256
512
Qasp-
1024
2048
128
256 512 1024 2048
Token Budget
AVG.
Offline
2Wiki Musi Hap Qusp.
S-Doc QA
M-Doc Nqu
44.0
Base
43.7
30.2
55.8
45.3
55.6
29.9
+Pyky 14.4 7 44.5po 45.8 45.8
128
42.5
44.9 45.7 45.6 44.5
Nqa
44.5
31.6
55.0
44.
55.8
29.2
618
63.4
Lee
64.
6488
256
512
41.9
42.643.7
43.5
44.0
44.7 45.3 44.1
45.3 43.9
Musi
Qasp-
43.5
43.
43.0
31.0
30.8
55
54
448
44.0
54.4
54.6
54.6
29.6
29.1
02:2
63.0 G1. 64.9ro.
1024
42.2
44.2
44.3
44.7 45,0 44.1
.014
0011
.007
Effect on Generation Length. A neglected aspect of compression methods is the impact on output length. Some compression methods, like H2O, induce generative verbosity, imposing an overlooked computational burden (TableB). Conversely, others, such as Stream, prematurely terminate generation, which truncates valid reasoning and degrade performance. In contrast, FASA maintains output lengths nearly identical to the FKV while preserving high performance, demonstrating a superior balance.
Compatiblility of FASA. By design, FASA is orthogonal and synergistic with other KV cache optimization paradigms. We demonstrate this by integrating it with Py (Cai et al. (025b). which allocates varied budgets across layers. While PyramidKY V determines how many tokens keep per layer, FASA decides which tokens are most critical. As shown in Table this complementary pairing yields consistent performance gains, confirming FASA's 's high compatibility and modularity.
Efficiency Analysis. We assess the efficiency of our two FASA variants. FASA-M's memory savings are particu- larly pronounced in long sequences, as the KV cache's
Memory and Time Comparison (Nep = 16)
PASACC Marrery
FSV Numrory
MSV Time
MSS.C. Time
5.14
Momory (GB)
PASA M. Mumory
MSSAN
Time
footprint grows to dominate and dwarf the static memory costs of model parameters and activations. While its CPU- GPU data transfer introduces a slight latency overhead, this can be effectively mitigated by prefetching techniques that asynchronously load the required KV pairs in advance, FASA-C, implemented with Triton (based on Ribar al.)
2.04
3.02
1k
24
4%
56
15k
324
64k
Saquence Length
22024). ) delivers substantial inference acceleration. The Figure Memory vs. latency (Neip 16).
speedup effect intensifies with longer sequences, achieving up to a 2.56x with Neip 16 under 64K.
5.4 ABLATION STUDIES
Robustness to Calibration Window K. Our method exhibits remarkable robustness the calibration window size, A. Performance is largely insensitive to IC, with smaller KC values often yielding slightly superior results (Table (5). This suggests that due to the inherent sparsity of attention, even a small calibration window provides a sufficiently robust signal to identify the dominant FCs.
Trade-off between Nop and N/ac- The hyperparameters Nup (token selection precision) and Na (retention budget) govern a trade-off between the fidelity of token identification and the volume of retained context. As depicted in Figure 5 optimal performance can be achieved either with high-precise selection (large Neip) and a small budget, or more lenient selection (small Nesp) compensated by a larger one. Empirically, on the TREC dataset, we found that using just 10 dominamt FCs (15.6% of dimensions) with N Nfac 500 is sufficient to match the FKV's performance.
Impact of Offline Calibrated Data. As shown in Table our method exhibits remarkable robustness to the choice of calibration data. The minimal performance variation across different calibration datasets, as quantified by a low Coefficient of Variation (CV), confirms that our FC detection mechanism is stable and not reliant on specific calibration source.
9

---

Published as a conference paper at ICLR 2026
6 EXTENDING FASA TO NON-ROPE MODELS
In this section, we investigate the generalizability of FASA to non-RoPE architectures. The feasibility of this extension hinges whether functional sparsity is universal emergent property induced by alternative PE schemes. We first ascertain the presence of functional sparsity in ALiBi and Partial-RoPP followed by an empirical evaluation of FASA 's performance within these frameworks.
Functional Sparsity in ALiBi and Partial-RoPE, (MLA) We extend our analysis to two widely used PE variants: Attention with Linear Biases (ALiBi) and Partial-RoPP (in Multi-head Latent Attention (MLA)). While ALiBi encodes relative distances using head-specific linear biases added to the attention logits, MLA adopts decoupled approach where RoPE is applied only to a specific partition of the head dimensions. This strategy allows us to explore sparsity behaviors that go beyond the standard full-RoPE configurations. This strategy enables to examine sparsity behaviors beyond the typical full-RoPE configurations. As illustrated in Figures 8 and 9 both ALiBi and Partial-RoPE induce functional sparsity at the head dimension level. For ALiBI, heads indexed from 19 to 31 exhibit a regular pattern of functional sparsity, while the remaining dimensions in other heads show high CA scores (approximately 0.7). Therefore, FASA compatible with ALiBi.
Beichuan-1338 138-Chat Layer
Baichuan-138-- 133--hhtt
Hoad Index
Figure 8: CA Scores Heatmaps of Baichuan- 1ataan--3B-Gae 13B-Chat (ALiBi models).
DeepSennk A/--itt---tttttttte Layer 20
Mand
Hend
Iddo 0-12 C2 1121
merapetdion m-1277 111
Figure 9: CA Scores Heatmaps of DeepSeek-V2-Lite- (Partial-RoPE models).
FASA Evaluation on Other PEs As demonstrated in Tables and 8 FASA exhibits exceptional generalizability across diverse position encoding architectures. achieving this without incurring any significant performance trade-offs. The results consistently match or surpass those of FKV, establishing FASA as robust, high-performance method with broad applicability beyond RoPE.
Table 7: Performance on Partial-RoPP Models.
Table Performance on ALiBi Models.
Qasper 2Wiki Multi Passage Re Lee Samsum
Qasper Lsht Dureader Trec Repobench
FKV
33.18
19.83 47.27
49.00
63,40
34.04
FKV
9.11
24.25
23.18
23.00
17.30
FASA 33.46 20.25 46.50
48.50
62.49
32.53
FASA
7.80
21.25
21.70
21.50
16.46
7 CONCLUSION
In this work, we addressed the memory footprint and bandwidth introduced by the KV cache in L.Mss. Firstly, we cover an intriguing phenomenon: the functional sparsity of FCs. A subset of dominant FCs could show high contextual awareness. Based on this discovery, we introduce FASA, a coarse-to-fine two-stage freamwork. The first stage utilizes the dominant FCs to perform dynamic, query-aware token selection without costly training. Then, the second stage perform focused and precise attention computation this reduced subset. Our experiments indicate that FASA attains performance nearly on par with full KV even under constrained budgets. The memory- and speed-optimized variants of FASA offers practical and effective solution for efficient long-context inference.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their insightful comments and suggestions. We also thank the members of the Machine Learning Group at AMAP for their valuable feedback and support throughout the development of this work.
ETHICS STATEMENT
Our research focused on enhancing the computational efficiency of Large Language Model (LLM) inference by optimizing KV cache management. The primary positive impact of our work, FASA, to make large-scale models more accessible, affordable, and environmentally sustainable. By significantly reducing memory and computational overhead, our method can enable researchers and institutions with limited resources to develop and deploy powerful long-contex, models, thereby fostering broader innovation and democratization in the field of AL.
We acknowledge the dual-use nature of efficiency-enhanciin technologies. While our goal is positive, lowering the barrier to running large models could inadvertently make it easier for malicious actors to deploy them for harmful purposes, such as generating misinformation or spam at scale. is important to note, however, that our work is foundational and does not create new capabilities for generating harmful content; merely optimizes the performance of existing models.
conducted on publicly available benchmarks (LongBench, MATH, ALME) and open- pre-trained models. We did not use any private, sensitive, or user- generated data. We recognize that the foundation models used in evaluation may reflect and perpetuate societal biases present in their vast training corpora. Our method operates orthogonally to the challenge of model-level bias and does not address it directly, but we encourage users be mindful of the inherent limitations of the models they deploy with our technique.
All experiments
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide a detailed account of all models, datasets, experimental setups, and evaluation protocols, all of which are publicly available. An overview of the experiments is provided in Section [.1] with more comprehensive details described across several appendices. Specifically, the configuran configurations for all baselines and the detailed hyperparameters for FASA are presented in Appendix B.1 The descriptions of all benchmarks and their corresponding evaluation protocols are detailed in Appendix [.2 and Appendix B.3 respectively. Furthermore, the implementation and design choices for FASA are explained in Appendix B.4] Finally, the specific algorithms for FASA-M and other core functions provided in Appendix D.I and Appendix D.3
REFERENCES
Eigen attention: Attention in low-rank space for kv cache compression, 2024. URL https
//arxiv.org/abbb. /200 05646
Joshua Ainslie, James Lee- LeeTph Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023. URL
https: http::://cpenreviie wttp://ewweeeeeettteeeee www.rrmmwmm..hmm//o/oooow
net forum?
Yash Akhauri, Ahmed AbouElhamayed, Yifei Gao, Chi-Chih Chang, Nilesh Jain, and Mohamed Abdelfattah. Tokenbutler: Token importance is predictable, 2025. URL httts:://aaxiiii org/
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arx66:2309.16609, 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 31 3113-3137, 2024.

---

Published as a conference paper at ICLR 2026
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličkovié. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025. URL https://ppeneeeieer
net//foumm
Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, and Alexey Tumanov. Rocketkv: Accelerating long-comtext Ilm inference via two-stage kv cache compression, 2025.
http:///wwwwxw////////////////// URL /ttK:ttpss///axx.oor////////////
14051
Zefan Cai, Wen Xiao, Hanshi Sun, Cheng Luo, Yikai Zhang, Ke Wan, Yucheng Li, Yeyang Zhou, Li- Wen Chang, Jiuxiang Gu, al. R-kv: Redundancy-awand kv cache compression for training-free reasoning models acceleration. arXiv preprint arXiv:2505. 2025a.
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, 2025b. URL htttt::t///aaaaeeee wwwivv.iivvv.///ii// org/ abs/ 1.406.02063
Chi-Chih Chang, Wei-Cheng Lin, Chien- Yu Lin, Chong- Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. Palu: KV-cache compression with low-rank projection. In The Thirteenth International Conference on Learning Representations, 2025. URL
net/ wow.mmidddaas------....
Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, and Chao Huang. Seplim: Accelerate large language models by compressing one segment into one separator, 2025a.
abs 2412. 12094
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish. Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. URL httpt:tts//aaate
erg/ abs aaaaabb/2111 03374
Silin Chen, Shaoxin Lin, Xiaodong Gu, Yuling Shi, Heng Lian, Longfei Yun, Dong Chen, Weiguo Sun, Lin Cao, and Qianxiang Wang. Swe-exp: Experience-driven software issue resolution. arXiv preprint 1.XXi::2077233666 2025b.
Yanqi Dai, Yuxiang Ji. Xiao Zhang, Yong Wang, Xiangxiang Chu, and Zhiwu Lu. Harder is better: Boosting mathematical reasoning via difficulty- grpo and multi-aspect question reformulation. arXiv preprint 2rX1v22601.20614, 2026.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL httpss://axiiv.orggg 2205. 14135
Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. Get more with less: Synthesizing recurrence with kv cache compression for efficient Ilm inference, 2024.
URL https: org/ abs/240s. 2202 09398
Yixiong Fang, Tianran Sun, Yuling Shi, and Xiaodong Gu. Attentionrag: Attention-guided context
pruning in retrieval-auggmente generation. arXiv preprint arXiv:2503. 2025.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL httpt:t////..t....t...ttt http::://ppezzviieeee www.e//ooowweeeeeeon id=uNrFpDPMyo
12

---

Published as a conference paper at ICLR 2026
Tanya Goyal and Greg Durrett. Evaluating factuality in generation with dependency- entailment, 2020. BRLLttpps: www.xvv.org/ab///1...0000
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fisth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021. URL Setttp:://operreviiee net/ forum Iod 2mQsCe
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Monishwaran Maheswaran, Sebastian Zhao, June Paik, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Accelerating long context length Ilm inference, 2025a. URL
2411. 09688
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length I1m inference with kv cache quantization, 2025b. URL BtLhttts::// /wtpv//aavvvv//aa///////aaaaaaa. org/ org/abs/////eoaaoo..e..aae 22/2240111 180799
Dongsheng Jiang. Yuchen Liu, Songlin Liu, Jin'e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong, From clip to dino: Visual encoders shout in multi-modal large language 2023.
models. arXiv preprint urXiv:2310.080
Haoyang LI, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole HU, Wei Dong, Li Qing. and Lei Chen. A survey on large language model acceleration based on KV cache management. Trunsuctions on Machine Learning Research, 2025. ISSN 2835-8856. URL https:://ppenrese
net www.ooooummm..ee2m2.a
Renda Li, Hailang Huang, Fei Wei, Feng Xiong. Yong Wang, and Xiangxiang Chu. Adacurl: Adaptive curriculum reinforcement learning with invalid sample mitigation and historical revisiting, arXiv preprint St.i::2511.09447. 2025.
Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947-2005
2024.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, al. Deepseck-v2: strong, economical, and efficient mixture-of- experts language model. arXiv preprint urXiv:2405.04434 2024a.
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: KV cache compression in depth dimension for large language models. In The Thiny-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL httttsts///peeereet Let/Eoruumidd finrzuummiddsppe DQUMT
Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chen- gruidong Zhang, Bailu Ding, Kai Zhang, Chen Chen, Fan Yang. Yuqing Yang, and Lili Qiu. Retrievalattentioo:: Accelerating long-context I1m inference via vector retrieval, 2024c. URL
https:://axiiv wttp/////////////ooo 2009.1//222.21111
Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. Clusterkv: Manipulating Ilm kv cache in semantic space for recallable compression, 2025a. URL httpttts////xxii..rrrt
abs/ / 2212.412..00 03213
Runze Liu, Jiakang Wang, Yuling Shi, Zhihui Xie, Chenxin An, Kaiyan Zhang, Jian Zhao, Xiaodong Gu, Lei Lin, Wenping Hu, al. Attention compass: Efficient exploration for process- supervised rl in reasoning models. arXiv preprint DrXiv:2509.2662. 2025b.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Seissorhands: Exploiting the persistence of importance hypothesis for Ilm kv cache compression at test time, 2023.
arb/
2305. 17118
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint 20l.22402.02700. 2024d.
13

---

Published as a conference paper at ICLR 2026
MAA. American
invitational
mathematics examination
aime.
In Febru-
American Invitational
Mathematics
Examination
AIME
2024,
ary
https://maa.cra mat competitions/ ameri can- inviiiiiiiinnaaai mat.emaa ics -wwe-esseeaaecee nat.i on-oon-nn- me
2024.
URL
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture
models. In International Conference on Learning Representations, 2017. URL https://
Alexander Peysakhovich and Adam Lerer. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427, 2023.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling, 2019, URL http::://aaxvvvoo////bb/
1911. 0550
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Luka Ribar, Ivan Chelombiev, Luke Hudlass Galass,, Charlie Blake, Carlo Lusehi, and Douglas OTT. Sparq attention: handwwdtth-effiiien LLM inference. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL httpss:/
net/ for
Yuling Shi, Yichun Qian, Hongyu Zhang, Beijun Shen, and Xiaodong Gu. Longcodezip: Compress long context for code language models. arXiv preprint arXiv:25/0.0044: 2025.
Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. Loki: Low-rank keys for efficient sparse attention. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https:://ppeereeeeeeee
net. Eorum wtt.oorruummmdrrraeeaeeaa
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer; Enhanced transformer with rotary position embedding, 2023. URL htRpsttps:/ httttt:s://: http://aaxivvvv.i org argg/b/////0a 09864
Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. Shadowkv: Kv cache in shadows for high-throughput long-context Ilm inference, 2025. URL
http://wwwawiaagaaa...ooo/ 2tt.://///aa//...2222222 21465
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In First Conference on Language Modeling, 2024. URL httptttt////pereeiiiiii net Fttt//orrmmmi==7aaa11131
Jiaming Tang. Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context IIm inference. International Conference on Machine Learning, pp. 47901-47911. PMLR, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
Zhongwei Wan, Xinjian Wu. Yu Zhang, Yi Xin, Chaofan Tao, Zhihong Zhu, Xin Wang, Siqi Luo, Jing Xiong, Longyue Wang, and Mi Zhang. Sitext(D)_ [2) (O)S: Dynamic discriminative operations for efficient long- cong-context inference of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL net/ htte:/oormmiidHzBBBB
Ao Wang, Hui Chen, Jiaxin Li, Jianchao Tan, Kefeng Zhang, Xunliang Cai, Zijia Lin, Jungong Han, and Guiguang Ding. Prefixkv: Adaptive prefix kv cache is what vision instruction-followiiii models need for efficient generation, 2025a. ht..sttps: rxxiv. wwwpxiiioooraaaaoe abs /212. 03409
14

---

Published as a conference paper at ICLR 2026
Yifei Wang, Yuheng Chen, Wanting Wen, Yu Sheng, Linjing Li, and Daniel Dajun Zeng. Unveiling factual recall behaviors of large language models through knowledge neurons. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference
Empirical Methods in Natural Language Processing, pp. 7408. Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18653/66/vvvvv 11..116665111222.emmll w22024emmmlpmmainn..... URL
https://aclante
ma mamin... 420/
Yifei Wang, Yu Sheng, Linjing Li, and Daniel Dajun Zeng. Uncertainty unveiled:
Can exposure to more in-context examples mitigate uncertainty for large language models? In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.). Findings of the Association for Computational Linguistics: ACL 2025, pp. 20659 20678, Vienna, Austria, July 2025b. Association doi: 10. 10118863/1122255 http:////2222....nnn-geeeee findings-acl. f findi weddings-aaae acl. 1
for Computational Linguistics. ISBN 979 0777988-9119 -91---55-777--5-555
1062. htttstss::////// http//alaantttt wwtp:hhoooolo........oo/
Yifei Wang. Feng Xiong, Yong Wang, Linjing Li, Xiangxiang Chu, and Daniel Dajun Zeng. PO- SITION BIAS MITIGATES POSITION BIAS: Mitigate position bias through inter-position knowledge distillation. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 1495- 1512, Suzhou, China, November 2025c. Association for Com- doi: wwg. emw.ppaammmmpmmmmaammm 787
putational Linguistics. ISBN 979-8-89176-332-6
URL http:3//1122225.mm--mmomo
Yuhang Wang, Yuling Shi, Mo Yang, Rongrui Zhang, Shilin He, Heng Lian, Yuting Chen, Siyu Ye, Kai Cai, and Xiaodong Gu. Swe-pruner: Self-adaptive context pruning for coding agents, arXiv preprint 7rXY:22601.16746. 2026.
Zheng Wang, Boxiao Jin, Yuming Chang, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive KV cache merging for LLMs on long-context tasks, 2025d. URL
net wtt.:/toooooonddd05510....
Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, Xipeng Qiu, and Dahua Lin. VideoroPE: What makes for good video rotary position embedding? In Forty- International Conference on Machine Learning, 2025. BRLLtttss///ppeeeriiie wwt.://ww.e..eooeeeeeeeeeeeeee htt.://www....eeeee.oooooooooooo
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ar natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 1201666551000022mmmmmmeeeeeeo URL https: //s:://ack /ta/aalaaaa waatoonnoooooooo logyoolgy..oo org 2020 emn w 67
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference Learning
Representations, 2024. URL htthhtpp/:/// /ttphttts:://///eeeeeee/eo
Feng Xiong, Hongling Xu, Yifei Wang, Runxi Cheng, Yong Wang, and Xiangxiang Chu. Hs-star: Hierarchical sampling for taught reasoners via difficulty estimation and budget reallocation. arXiv preprint 25055 /9866, 2025.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412. 2024. URL
httpp::/dol/ wwog/100000 10. 48550/ arxiv. 2412. 15115
Qingyue Yang. Jie Wang, Xing Li, Zhihai Wang, Chen Chen, Lei Chen, Xianzhi Yu, Wulong Liu, Jianye Hao, Mingxuan Yuan, and Bin Li. Attentionpredictor: Temporal pattern matters for efficient Ilm inference, 2025. URL https:: ://a:///aiiiii wttp://wwww...r/////////222222. 04077
15

---

Published as a conference paper at ICLR 2026
Rongzhi Zhang, Kuan Wang, Liyuan Liu, Shuohang Wang. Hao Cheng, Chao Zhang, and yelong shen. LoRC: Low-rank compression for LLMs KV cache with progressive compression strategy, 2025. URLL/tttpss//ppeeeeeeee... http:/ewwww.etteteeeeeeeeeeeeeee wet.:eeeoorrmmaanc.nCAAAAAAAAAA
Zhenyu Zhang, Ying Sheng. Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy- hitter oracle for efficient generative inference of large language models. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023. URLhttps: / wwwnreevew..eee net/ forum? 2
16

---

Published as a conference paper at ICLR 2026
A
INVESTIGATION RESULTS OF DOMINANT FREQUENCY CHUNKS
A.1 FURTHER GENERALIZATION ON MODEL SCALES AND ARCHITECHTURES
Qwen2.5-14B-Instuuct Layer
Qwen2.5 5-12..5-445 1-4i:------n-sttt---l- Instruct-ru Layer
dim
high dim
bow dlen
hish dim
0.5
Index
23
02
22.
Frequency chunk index
Frequency chunk index
Figure 10: Functional sparsity is maintained Qwen2.5 series models (Yang al. 2024). Heatmaps visualize the Mean Contextual Agreement (C5k= for each Frequency Chunk (FC, x-axis) across all attention heads (y-axis) representative layer. We compare the standard Qwen2.5-14B-Instrus model (left) with its Long-context variant, Qasper dataset. The remarkable similarity between the two heatmaps demonstrates that the functional sparsity of FCs is robust property, consistently maintained even after long- onngcootext fine-tuning
(right), both calibrated on the
Lia Llama-3.2-38 3----32-----nttucc- Layer 22
Twwe2.5-32B-Inntree Instruct Layer 37
dim
high dim
0.35
0.4
Head Ind
0.3
0.20
02
01
15.
0.05
C an Frequency weg.araninnnooonii...ioo.
Frequency
Figure 11: Functional sparsity persists across model scales. Heatmaps show the Mean Contextual Agreement (=5x=)5)) for increasing scale (3B and 32B). The remarkable stability the dominant FC patterns (bright vertical columns) across these scales demonstrates that functional sparsity is fundamental and scalable characteristic RoPE.
Conclusions: Our cross- arsstterchtttll (Figure 10p and cross-scale (Figure TT] analysis reveals striking finding: the functional sparsity of FCs is universal and stable property. This powerful evidence suggests that the observed functional hierarchy not an emergent artifact of specific model's training dynamics or size, but rather an intrinsic characteristic deeply embedded within RoPE mechanism itself. The roles of different frequencies appear be fundamental and pre- determined, providing robust and predictable foundation for developing model-agnostic efficiency optimizations.
A.2 LASK-INVARIANCE PROPERTY OF FUNCTIONAL SPARSITY
We find that the saliency of dominant FCs is largely task-agnostic. This property is evidenced by the strong alignment between saliency maps generated for distinct downstream tasks, as shown in Figure I2 Despite the functional differences between question answering (left) and summarization (right), the resulting importance rankings are highly consistent. This indicates that these FCs perform fundamental role inherent to the model's architecture, rather than adapted for specific task.
A.3 MORE ANALYSIS RESULTS
Functional Sparsity across Layers. While the principle of functional sparsity is universal, the specific set of dominant FCs is far from static in Figure 13] instead, it exhibits high degree of
17

---

Published as a conference paper at ICLR 2026
Mistral-7B Layer
Mistral-7B Layer 10
dims
high law
high dim
Head Index
4.3
0.3
4.2
0.
61
Frequency chunk index
Frequency chunk index
(a) Qasper
(b) GovReport
Figure 12: Heatmaps of agreement score (CA, K = 256) across attention heads for the Qasper (Left)
and GovReport (Right) from LongBench-V VI (Bai al, (2024)) Mistral-7BB-Instret--000
R
Figure 13: Heatmaps of agreement score (CA, K 256) across different layers.
specialization across both model depth and individual attention heads. This dynamic behavior reveals sophisticated division of labor within the transformer architecture.
A.4
QUANTITATIVE EVIDENCE ON SPARSITY UNIVERSALITY & TASK- INVARIANCE
Table 9: The ratio of dominant FCs and non-dominant FCs.
Type FC
Dominant FCs (%)
Non-Dom FCs (%)
Model
CA scores 0.4
CA score < 0.15
Llama-3 1.2-3-3.2-3B
0.54
89.6
Meta-Llama-3.1-88
0.68
89.6
Mistral-7B-v0.3
0.68
92.7
Qwen2.5-7B
0.17
95.5
Qwen2.5-14p
0.27
94.7
Owen2.5-14BBB
0.65
90.5
Swen2.5-33BBIInse
0.52
91.2
RI-Distill-Llama-
0.79
89.5
EI-Distill-Qwent
0.76
90.2
RI-Distill- D-iistlll-Qeen-22B
0.67
90.9
18

---

Published as a conference paper at ICLR 2026
Table 10: Cross-task overlap matrix of dominant FCs (%). Each sub-table shows the percentage of intersection between dominant FCs identified on "row" dataset and "column" dataset.
Model
Overlap Dom-FCs
Qasper
Gov Report
Musique
Narrativeqa
2Wikimqa
Avg.
Qasper
100.00
75.90
82.30
70.50
83.20
82.38
Gov. Repore
75.90
100.00
82.10
70.80
81.90
82.14
Llama-3.2-BB
Musique
82.30
82.10
100.00
73.60
96.50
86.90
Narrativeqa
70.50
70.80
73.60
100.00
73.10
77.60
2Wikimqa
83.20
81.90
96.50
73.10
100.00
86.94
Qasper
100.00
71.10
77.10
67.30
77.00
78.50
Gov_report
71.10
100.00
79.40
65,50
78.90
78.98
Mistral-7B
Musique
77.10
79.40
100,00
67.80
97.90
84,44
Narrativeqa
67.30
65.50
67.80
100.00
67.30
73.58
2Wikimqa
77.00
78.90
97.90
67.30
100.00
84,22
Qasper
100.00
70.60
80.90
68.70
81.30
80.30
Gov_Report
70.60
100.00
79,40
68,20
78.70
79.38
Qwen2.5-7B
Musique
80.90
79.40
100.00
71.70
96.60
85.72
Narrativeqa
68.70
68.20
71.70
100.00
71.10
75.94
2Wikimqa
81.30
78.70
96.60
71.10
100.00
85.54
Qasper
100.00
84.30
71.80
Gov_ Report
69.20
75.00
67.60
74.80
77.32
Qwen2.5-14B
Musique
84,30
75.00
100.00
74.30
98.40
86.40
Narrativeqa
71.80
67.60
74.30
100.00
73.90
77.52
2Wikimqa
84.50
74.80
98.40
73.90
100.00
86.32
Table 11: Predictive distribution of dominant FCs across different attention score ranges.
Prediction accuracy across varying attention scale ranges
Model
Type of FCs
Top 20%
Top 20-40%
Top 40-60%
Top 60-80%
Top 80-100%
Dom
82.4*
79.1
72.1
59.2
44.9
Laama3.2-3B-Instrul
Non-dom
4.6
5.3
5.3
5.4
5.4
Dom
81.1
80.7
72.5
56.4
Eistral-7--Insruett-oo
Non-dom
3,6
4.2
4.9
4.5
Dom
81.9
82.4
76.9
49.3
Non-dom
5.5
Dom
74.3
66.4
56.6
44.9
34.7
Qwen2.5.1dB-Instruet
Non-dom
4.1
4.6
4.5
4.9
4.9
B EXPERIMENTS DETAILS
B. EXPERIMENT CONFIGURATIONS
Baseline Configurations. As FASA is designed optimize the decode phase, we forgo any KV cache optimizations during prefilling for all methods under evaluation. This experimental design isolates the performance impact of decode-stage acceleration, ensuring that our comparisons are direct and fair. For all baselines, we adopted configurations that are either standard in their original papers or represent fair and strong setup for comparison.
Oracle: serves oracle bascline to demonstrate the upper- bound performance of Top-k sparse attention. This method operates under the ideal assumption that the k most important KV tokens for each query be identified perfectly and at computational cost. Consequently, given token budget directly corresponds to this optimal Top-k set.
Stream (Xiao et al. 2024): This method is based on the "attention sink" phenomenon, preserving fixed number of initial tokens and a sliding window of recent tokens. Following its standard setup, we set the initial "start_ size" to 8 and the "recent "ize" to "budget 8".
SnapKV (Li et al. 2024): SnapKV estimates token importance based on accumulated attention scores within a observation window during prefilling. We adopted its "maxpool" strategy with a window size of 32 and kernel size of As its original design performs one-time filtering, is
19

---

Published as a conference paper at ICLR 2026
not directly suited for long- generation tasks. We therefore adapted it, following the methodology in (Cai et al., 2025a). by re-applying the filtering mechanism every n generated tokens.
Quest (Tang et al. 2024): Quest organizes the KV cache into pages and retrieves them based on coarse-grained query-page similarity. We set the page size to 16, value reported near-optimal, to balance the trade-off between retrieval granularity and overhead.
RKV (Cai et al., 2025a): RKV is state-of-the-a method for reasoning tasks that also employs a retrieval mechanism. We set its core hyperparameter X, which balances between recent and important tokens, to 0.1 as recommended for optimal performance.
FASA Configurations. Our configuration for FASA designed for both effectiveness and practical efficiency. Unless otherwise specified, the following setup was used across all experiments.
Dominant FC Identification: A core principle of FASA is that the set of dominant FCs is a univer- sal, task-agnostic property of the model architecture itself. Consequently, these indices (Idom) can be determined via highly efficient, one-time offline calibration. For our LongBench experiments, this calibration was performed on just a single data sample from the Qasper dataset. We found this minimal setup to be remarkably robust, as the generated response provides sufficient signal identify the dominant FCs. The universality of these calibrated indices is empirically validated by FASA's strong performance across diverse tasks, from summarization to code completion. For Long-Cof Tong reasoning, a similar single-instance calibration was performed on a question from the MATH500 dataset.
Hyperparameter Settings: For architectural simplicity and maximize computational parallelism, we employ uniform configuration across all heads and layers. The number of dominant FCs retain, denoted Nup- was consistently set to 16. This choice represents a balance between preserving sufficient contextual information and maximizing computational.
Task Configurations:
We configured the maximum sequence length to 32k for the AIME24 bench- mark, reflecting its higher reasoning complexity, and to 16k for MATH500. For the LongBench benchmark, we set the maximum prompt length to 127.5k for Llama3/0wen2.55 series models and 31.5k for Mistral-7B-Instreeen Instrucct-v..2.
B.2 BENCHMARK DETAILS
LongBench (Bai et al. 2024) is a comprehensive, multi-task benchmark designed to evaluate the long-context understanding capabilities of Large Language (Wang et al. 2024 2025b). It comprises a diverse set of tasks, including single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. In experiments, we report the average performance across all relevant tasks to provide a holistic measure of a model's ability to process and reason over extended contexts, with sequence lengths ranging from 4K to over 100K tokens.
MATH500 (Hendrycks et al., 2021) is a challenging benchmark for evaluating mathematical reasoning. It consists of 12,500 problems sourced from high school math competitions, spanning subjects like Algebra. Geometry, Number Theory, and Precalculus. Each problem is accompanied by a step-by-step solution, making it highly suitable for assessing CoT reasoning capabilities. We utilize the MATH500 subset for our long-Co'T generation experiments, where models must produce detailed reasoning chains to arrive the final answer.
AIME (MAA 2024) represents a significant step-up in reasoning complexity compared to the MATH dataset. It consists of problems from the AIME competition, which are known for their non-routine, multi-step solutions requiring deep mathematical insight and creativity (Li et al. 2025 Dai et al. 2026). These problems serve as a stress test for a model's 's most advanced reasoning and long chain generation abilities (Xiong et al.] 2025). Following standard practice, we evaluate performance using the pass@1 @k metric, specifically reporting pass@) based on 16 generated responses per question.
C4 (Raffel et al. 2019) is massive, general-domaie English text dataset derived from the Common Crawl web scrape. The "clean" version is created by applying a series of heuristics to filter out boilerplate content, code, and offensive language, resulting in high-quality, natural language corpus.
PG19 Rae et al., 2019) is a long-form text dataset derived from books in the Project Gutenberg library. It is specifically curated for evaluating long-range sequence modeling. Fach example in the
20

---

Published as a conference paper at ICLR 2026
dataset is full book text, making it an ideal benchmark for assessing model's ability handle and maintain coherence over very long dependencies, often exceeding the context windows of LLMs.
WikiText(Merity et al. 2017 is large-scale language modeling corpus sourced from high-quality "Good" and "Featured" articles on Wikipedia. Unlike raw web text, WikiText is well-formatted, grammatically correct, and retains its original punctuation and case. It is split into training, validation, and test sets at the article level.
B.3 EVALUATION PROTOCOLS
To provide a comprchensive and rigorous assessment of model performance, we employ a set of standard metrics tailored to each evaluation paradigm.
Long- Cont-CC Understanding (LongRench). For the diverse tasks within the LongBench (Bai et al. 2024), we follow its official evaluation protocol. Specifically, we use:
fl score for question-answering tasks.
rouge_scoe sccreescceeeee for summarization tasks.
aade__sim_$ee scoree for code completion tasks.
The final reported score for LongBench is the average performance across all constituent tasks.
Long-Sequence Modeling, To evaluate model's ability to maintain generative fidelity over long dependencies, we use perplexity (PPL). Perplexity measures how well a probability model predicts sample. For sequence of tokens W (wi, W2,..... ... Wx). PPL is defined as the exponential of the average negative log-likelihood in Equation | lower PPL indicates a better model, as it signifies higher confidence and accuracy in predicting the next token.
PPL((W) exp
Σ log P(w,|wes)
Long CoT Reasoning. For complex mathematical reasoning tasks such as MATH500 and AIME2024, evaluate the model's performance in long-generation setting. This paradigm is distinct from conventional long-context understanding tasks. Instead of processing a long static input, the model must maintain logical coherence and track thought traces auto-regressive generation process to produce the correct final answer. Performance is reported as pass@I (Dai et al. 2026).
extended,
For MATH500, we report pass @ I, where a single generation is sampled for each problem. For AIME2024, which features more challenging problems, we also report pass@) but the result is determined by checking if at least one correct answer exists within A: = 16 independent generations for each question. This sampling strategy standard for estimating performance on complex reasoning benchmarks.
B.4 IMPLEMENT DETAILS
Implementation Details Our implementation of FASA is built upon the HuggingFace Transformers library (Wolf al. 2020). We employ a non-invasive monkey patching approach integrate our logic. Specifically, we intercept the forward pass of the FlashAttention2 class within the model's modeling-py file. The core of our method resides in two components. First, leveraging the universal nature of dominant FCs, their pre-computed indices are stored in globally accessible dictionary, shared across all layers and heads. Second, the Token Importance Prediction (TIP) logic, which performs the critical token selection, is encapsulated within our tore_mmoule___e saalll__wttt___aeeeeeeee withh_paddig function. key advantage of our design is simplicity and minimal intrusion. The integration requires inserting just single line of code, the token selection logic, into the original attention function, making FASA easy to deploy and adapt. This minimal intrusion makes FASA highly portable and easy adapt. The corresponding pseudocode is provided in Figure 14
21

---

Published as a conference paper at ICLR 2026
bsz, q_len, hidden hhidde__sttttsseeeeeete states.size()
cos, sin = pposition_enee position_eebeediings _embeddings query states, key states apply rotary pos emblquery states, key states, cos, sin)
#token selection in TIP
if query states, shape|2] 1: for deocoding stage
key, wty.sttates,vllue value states
module with hath_aainnnguerrrssttteee key states, value states,seff.layer idx, idx.budgetreccrdds)
query_statts query serry_stttss.tarrne statess.ransposee1,
key states = key states.transpose(1, 2)
value states value trassstttaasspssee.. 2)
attn can__ouput=== laauut=__la___aaaa lit__aattetio_ooovee
query_ key states,
value states, attention_maan masiio,
lelen,
dropout=dropout rate, sliding window-getattr(sel, "sliding_winen window", None),
use top left mask=self. flassssssff__ffhh______e__ flash_atthhtt Seaas_attn_uses__eeee teeestttp_lftttt_ mask, $__cauaalssef.is_caa
attn_ hott_ouututt-- attn rutput..eshape((s: q_len, 1).cooniguous() attn output self.o self.o_proj(atthy output) return attn_outpurt aattn_weights jast_key_value
The FASA Pipeline: An Efficient, www.AttennionoCooaatii Approach. The algorithm details our two-stage process. key design feature is that the FAC stage seamlessly integrates with the standard FlashAttention API, leveraging its performance while enabling sparse computation.
Figure 14:
C ADDITIONAL EXPERIMENTAL RESULTS
C.1 PERFORMANCE ANALYSIS ON DIFFERENT BUDGETS
SnapKV
MultiFieldQA, zh (Qwen2.5 5-7)
2WIkIMQA (Qwen2.5-78)
GovReport (Qwen2.5-78)
Dureader (Qwen2.5-78)
Accuracy (96
80
30
30
50
25
20
20
20
30
1386
512
1024
312
1024
312
1024
2848
917
1024
2049
Figure 15: FASA on Qwen2.5-7B Instructt under various token budgets (Nip 16).
FKV
Oracle
Quest
SnapKV
FASA
Qasper (Meta-8B)
MultiFleldQA.eem en (Meta-88)
HotpotQA (Meta-8B)
45
Accuracy (%)
54
440
52
50
60
30
1286
512
1024
2048
1
512
1024
2048
1386
512
1024
2048
Figure 16: FASA on Meta-3, 1--amma 8B Instruct under various token budgets (Nup 16).
Comparison with Low-Rank Methods A closely related work to FASA is SparQ (Ribar et al. 2024), which also performs a form of dimension selection. SparQ operates on the heuristic that high-magnitude dimensions in query vector are the most indicative of importance, and thus selects corresponding key dimensions as proxy for token prediction. However, as our experiments in Figure 17 demonstrate, this heuristic proves be poor substitute for true contextual awareness.
22

---

Published as a conference paper at ICLR 2026
Under a constrained budget of 256 tokens, SparQ's performance collapses, indicating its inability to reliably identify critical tokens based solely on query magnitudes. Furthermore, from an efficiency standpoint, SparQ incurs significant overhead it must re-evaluate high-magnitude dimensions for every new query. In stark contrast, FASA leverages a one-time, offline calibration, making its per-token inference cost substantially lower.
Performance Comparison with Spart
55
45
Performanc
25
20
20
SparQ
FASA
15.
Token Budget
Figure 17: Comparision with SparQ on LongBench.
D DISCUSSION ON FASA
D.1 VARIANTS OF FASA
The memory-optimizze variant, FASA-M, is specifically engi- neered for scenarios with constrained GPU memory, such as consumer- grade hardware. As detailed in Algorithm its core strategy is to minimize the on-GPU memory footprint by strategically keeping only the most essential data on the GPU.
FASA-M (lemory-Optimizze
Specifically, only the dominant parts of the Key cache (Cyem)) ) which are required for the initial token importance prediction, are retained in GPU memory. The non-dominant parts the Key cache (Cey (Coondom) and the entire Value cache (Crat) are offloaded to and managed in the much larger CPU memory. During the Focused Attention Computation (FAC) stage, once the critical token indices (Ti) are identified, only the small, required subsets of the non-dominant key and value caches are transferred from the CPU to the GPU for the final attention calculation. This jjust-in-time' data transfer ensures that the GPU memory is primarily occupied by the most critical components, leading to substantial memory savings.
Memory Footprint Analysis The GPU memory footprint of the KV cache FASA-M can be formulated as follows. Let L be the total sequence length, b the token budget, d the model's hidden dimension, and Naayer the number of layers. Let doom be the dimension of the dominant FCs and drondom be the dimension of the non-dominant FCs (d ddom + dnondom). ). The memory occupied by the KV cache on the GPU is:
MeMGGUU Nioyers
L deom + b dnondom
+ b x d
x bytes_per_pantte cersspee_____e per_aarrmmmaapp-
(10)
Dominant Keys Non-dominaat Keys
Values
Compared full KV cache, which occupies Niagers X L 2d x bytess_er__aae_ per_parmm, FASA-M significantly reduces the memory burden, especially when the non-dominant and value components constitute large portion of the cache. For instance, if ddom 25% of and the budget is 10% of
L, the memory savings can be substantial, approaching an 8x reduction in typical configurations.
D.2 DESIGN CHOICES
On the Role of FC-Scores: A Proxy for Ranking, Not a Substitute for Attention. A crucial design principle we validated that our FC-based scores (s/ are not calibrated to function as direct attention weights. Although they provide remarkably accurate relative ranking of token
23

---

Published as a conference paper at ICLR 2026
importance, their direct substitution for attention probabilities leads to catastrophic performance degradation. This reveals their fundamental role as a selector a mechanism to identify salient tokens rather than approximator of the final attention distribution.
On the Indivisibility of Frequency Chunks. We investigated whether individual dimensions could serve selection units, and the answer is definitive A pipeline based on selecting "dominant dimensions" suffers a catastrophic performance degradation. This empirically validates that the Frequency Chunk (FC) an indivisible functional unit for this process. This principle is not coincidental but is a direct corollary of RoPE's core mechanism, which encodes position by applying rotations coupled pairs of dimensions. Disrupting these pairs severs the positional encoding, leading to model failure.
In summary, these two findings underscore two core design principles of FASA. First, an efficient proxy for token importance does not necessarily serve as valid substitute for attention weights. Second, any optimization for RoPE-basse models must respect the inherent coupling of dimension pairs, treating the Frequency Chunk indivisible functional unit.
D.3 ALGORITHM ON FASA
See the algorithm of offline calibration in Algorithm see the algorithm of FASA-M in Algorithm
Algorithm 1:
Offline Calibration for Dominant FCs
Input: A calibration dataset Q2 number of dominant FCs to select k. Output: The set of dominant FC indices, Idom.
// Stage
Collect Contextual Agreement (CA) scores
Initialize an empty map M to store CA scores for each (1, h, i) triplet foreach example in do foreach token generation step do foreach layer do foreach head h do
Compute full attention scores (qr, K1:t) foreach FC index do
Compute single-FC scores a (qt K1:t)
Calculate the CA score CA L.h.
using Eq.
Store CA L, in [[]]]]]
end
end end end end
// Stage 2: Select Domi nant. FCs
Initialize an empty map M for mean CA scores foreach (I, h. i) in M do
[ + Mean [(]])]
end
Idom < TTop Indices M, k) // Select top-k indices based on CA return Idom
E LLM USAGE
During the preparation of this manuseript, utilized the Al-based language model ChatGPT, developed by OpenAl. Its use was strictly limited to language refinement, including grammar correction, stylistic enhancement, and rephrasing for clarity. All scientific concepts, experimental
24

---

Published as a conference paper at ICLR 2026
Algorithm 2: Inference with FASA-M (Memory-Opimizzed Variant)
Current query qr: Current key ky; Current value v Dominant FC indices Idom
Input:
Token budget
Past KV cache: Okey (GPU), Cpendom (CPU), Coat (CPU)
Output: Next hidden state h++1
Updated KV cache: key key Chandom Cual
// Stage 1: Token Importance Prediction (TIP)
// Split key by dominant FCs
kdom .k. knondom f- (
11 Select corresponding query dimensions
qr dom + Select(q;;Idon
Kdom + UpdateCache (ee k(
Approximate scores using dominant parts
S. + qt dom [
// Identify fy indices of most salient tokens
T. f TopK-Indices(S..b))
// Stage 2: Focused Attention Computation (FAC)
1/ Select dominant key parts on GPU
K Kdom t SclectTokens(<<tr) Tr)
Update non-dominant cache on CPU
Mondom <- UpdateCache keg knondom)
key
Knonndom + LoadFronCPU (CRandom)
la
// Select non-dominant key parts CPU Knondom SelectTokens (K Kpondom, Tτ)
KT,
// Update value cache CPU
Geat < UpdateCachee(Caat Vt).
Vi - LoadFromCPU(Cuua)
// Select values on CPU
VT, SelectTokens(Vi... Tr)
Offload requi red non non don-iiminant keys GPU
K T: TransferTooGPP((XXqoooo
Rooddom
/ Offload requi red values to GPU
VT, TransferToGP((VT]
// Reconstruct full keys for selected tokens
KT. + Combine (K KKnondom .Idom)
// Compute full attention on the subset
Oiaec + Softmax(q.K3,//15)
h + Wolars VT,)
return h and updated caches
designs, data analyses, and conclusions presented herein are the original work of the authors and were conceived and executed without any substantive contribution from the language model.
25


Reviews:
[{"review_id": "gYh4l9OzMt", "reviewer": "Reviewer_YxYn", "summary": "The authors propose to use functional sparsity in RoPE at frequency-chunk level, basically a small per-head subset of FCs that act as a ‘filter’ for the head on what matters most. They use their importance predictor to score tokens, and run full attn on only selected positions. With this they are near full-KV accuracy on longbench with just 256 tokens.", "questions": "Comments on non-RoPE schemes would be appreciated. \nPage-based methods cannot use this efficiently, perhaps an ablation on page level sparsity could be useful to the reader, but not critical.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- novel idea (and observation) to use functional sparsity of RoPE \n- speedup demonstrated at long context, and can work with other KV compression schemes as well\n- robust across datasets\n- because they do not re-index token positions, original absolute positions of tokens are preserved", "weaknesses": "- Not applicable to non-RoPE variants, further discussion on that would help.\n- The idea is impressive, but it took quite some effort to understand. I think a huge amount of math can be simplified in the paper, maybe shifted to appendix for ‘more details’.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ITGeDufLyo", "reviewer": "Reviewer_pDZu", "summary": "This paper proposes a novel, training-free, query-aware token elimination framework called FASA. Its core contribution stems from a novel discovery of RoPE: functional sparsity exists at the frequency-chunk (FC) level. Based on this insight, FASA employs a two-stage framework:\n1. Token Importance Predictor (TIP): Utilizes a one-time offline-calibrated set of dominant FCs to efficiently estimate attention scores (aggregating only contributions from these FCs) and identify a critical subset of tokens.\n2. Focused Attention Computation (FAC): Performs full-dimensional, accurate attention computation only on the significant subset of tokens selected in the TIP stage.\n\nFurthermore, this paper provides two hardware-aware implementation variants: FASA-M (memory-optimized, offloading non-dominant K and V caches to the CPU) and FASA-C (computationally optimized, performing sparse access on the GPU). \n\nOn LongBench-V1,FASA reaches nearly 100% of full-KV performance when only keeping 256 tokens, and achieves 2.56× speedup using just 18.9% of the cache on AIME24.", "questions": "1. Regarding generalization beyond RoPE: Have you considered how to extend this insight into \"functional sparsity\" to models using other positional encoding schemes, or partially rope models like MLA?\n2. Why was N_{tip}=16 chosen as the primary configuration in the paper? Figure 6 seems to suggest that a smaller N_{tip} is also good enough in many cases.\n3. Regarding the efficiency analysis of FASA-M, does the comparison in Figure 7 already include prefetching operations? Could you also provide a comparison of the benefits of prefetching techniques?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The discovery of \"functional sparsity at the frequency block level\" in RoPE provides a new, theoretically grounded perspective for understanding attention mechanisms and designing sparse attention models, rather than relying solely on heuristics.\n2. The core hypotheses were sufficiently verified; in addition to verifying the sparsity of the dominant FCs, the universality and task-invariance of the CA index were also verified.\n3. Strong practicality: The paper considers different hardware constraints and proposes two variants, FASA-M (memory-constrained) and FASA-C (computation-constrained), which significantly enhances the practical application value of the method.", "weaknesses": "1. Strong dependence on RoPE: The entire methodology and core findings are built upon the analysis of RoPE. This severely limits the method's generalizability. It remains unclear whether or how FASA can be generalized to models using other positional encodings or even those without explicit positional encoding.\n2. Table 6 shows the robustness to the data. Only the TREC and MATH datasets are provided here, which is relatively limited in variety. Furthermore, the data size is not analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iiZkbExMZg", "reviewer": "Reviewer_VxaW", "summary": "This paper proposes a new method called FASA to achieve query-aware token eviction by dynamically predicting token importance. The authors state that they found only a small subset of frequency chunks (FCs) contributes significantly to the model's contextual awareness, thus allowing FASA to predict and use a critical set of tokens for focused attention computation. The authors conducted several experiments and concluded that FASA performs better than other attention sparsity algorithms.", "questions": "See Weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. Their insights about RoPE are interesting.\n2. According to their experiments, FASA appears to be effective.", "weaknesses": "- The authors mention that their insights about frequency chunks relate to RoPE and cite previous works. However, since this aspect is critical to FASA, I believe they should include their theoretical analysis directly in the paper. It is difficult for readers to be convinced by the current content alone.\n- Regarding the experiments conducted to support their insights about sparsity in frequency chunks, I think more demonstration is needed. The figure in the main text simply compares two heatmaps of FCs between different models and layers, which raises concerns about selective presentation. Although Figure 11 shows comparisons of different heatmaps across layers of the same model, and considering FASA's effectiveness, I am inclined to believe there might be solid evidence. However, this evidence must be clearly presented in future versions.\n- The authors claim to have empirically proven that the selection of FCs is task-agnostic and insensitive to calibration data, which is important for the method's practicality. However, the results are only briefly explained in Table 6, which is insufficient to convince readers.\n- Regarding the experiments on FASA's effectiveness, a potential issue lies in the choice of dominant FCs. In the Appendix, the authors explain that they used different FCs for LongBench experiments and Long-CoT reasoning. This seems contradictory to their claim about FC selection.\n\n- The font size in the figures is too small and difficult to read. It should at least match the main text's font size.\n- Some important experimental details are included in the Appendix, which can confuse readers and require frequent cross-referencing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zbh28D20z8", "reviewer": "Reviewer_UWTV", "summary": "This paper proposes FASA (Frequency-Aware Sparse Attention), a training-free, query-aware framework for token eviction by dynamically predicting token importance in LLMs. The authors identify functional sparsity in frequency chunks (FCs) within the RoPE positional encoding, observing that only a small subset (“dominant FCs”) contributes significantly to contextual awareness. Leveraging this insight, FASA consists of two stages:\n1. Token Importance Prediction (TIP): Uses dominant FCs to predict salient tokens.\n2. Focused Attention Computation (FAC): Performs full attention only on these tokens.\nTwo variants are proposed—FASA-M (memory optimized) and FASA-C (computation optimized)—achieving up to 2.56× speedup while maintaining near full-KV accuracy across LongBench-V1, MATH500, and AIME24.", "questions": "1. In Figure 10, it would strengthen the universality claim to evaluate across a broader range of popular models.\n2. I would appreciate a more intuitive explanation of dominant frequency chunks (FCs). The Contextual Agreement (CA) score measures alignment in token ranking between the full-head and single-FC attention scores. A high CA score indicates that the FC captures a similar ranking of token importance. However, since CA reflects order rather than magnitude, it is possible that some non-dominant FCs might exhibit smaller agreement but larger attention scale. Showing the distribution of CA scores for selected dominant FCs could help demonstrate that they accurately capture both ranking and influence on token importance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The discovery of functional sparsity at the frequency-chunk level in RoPE is both novel and conceptually insightful. The paper is clearly written and well-structured.\n2. FASA’s training-free, model-agnostic, and task-independent design enables straightforward integration with existing LLMs, greatly enhancing its practical deployability.\n3. The method demonstrates strong performance", "weaknesses": "1. Although FASA reduces computation by selective token attention, it still retains the full KV cache, meaning memory consumption remains a potential bottleneck for extremely long contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)