ProxyAttn: Guided Sparse Attention via Representative Heads
Authors: Yixuan Wang
Huang He
Siqi Bao
Hua Wu
Haifeng Wang
Qingfu Zhu
Wanxiang Che
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient LLM
Sparse Attention
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The quadratic complexity of attention mechanisms limits the efficiency of Large Language Models (LLMs) on long-text tasks. Recently, methods that dynamically estimate block importance have enabled efficient block sparse attention, leading to significant acceleration in long-text pre-filling of LLMs. However, their block-level coarse-grained estimation inevitably leads to performance degradation at high sparsity ratios. In this work, we propose ProxyAttn, a training-free sparse attention algorithm that achieves token-level estimation by compressing the dimension of attention heads. Based on our observation of the similarity among multiple attention heads in long texts, we use the attention scores of pooled representative heads to approximate the scores for all heads. To account for the varying sparsity among heads, we also propose a block-aware dynamic budget estimation method. By combining the scores from a set of representative heads with a multi-head dynamic budget, we can achieve a more fine-grained block attention evaluation at a low computational cost. Experiments on a variety of mainstream models and extensive benchmarks confirm the underlying similarity among attention heads in long texts. Leveraging a token-level fine-grained estimation, the proposed method achieves substantial gains in performance and efficiency compared to existing methods. More precisely, ProxyAttn can achieve up to 10.3x attention acceleration and 2.4x prefilling acceleration without significant performance loss. Our code is available at https://github.com/wyxstriker/ProxyAttn.

Published as a conference paper at ICLR 2026
PROXYATTN: GUIDED SPARSE ATTENTION VIA REP- RESENTATIVE HEADS
Yixuan :ang Huang He', Siqi Bao?, Hua Wu', Haifeng Wang",
Qingfu Zhu', Wanxiang Chelit
Haarbn Institute of Technology, China Baidu Inc., Beijing, China {yi lyixuanwang, qfzhu, (arr@ir.hitteee (hehuang, baosigi, wu_hua, fanghaiffeg)@baiddu
ABSTRACT
The quadratic complexity of attention mechanisms limits the efficiency of Large Language Models (LLMs) on long-text tasks. Recently, methods that dynam- ically estimate block importance have enabled efficient block sparse attention, leading to significant acceleration in long-text pre-filling of LLMs. However, their coarse-grained estimation inevitably leads to performance degradation high sparsity rates. this work, we propose PrexyAttn, a training-free sparse attention algorithm that achieves more precise block estimation by compress- ing the dimension of attention heads. Based on our observation of the similar- ity among multiple attention heads, we use the scores of pooled representative heads to approximate the scores for all heads. To account for the varying sparsity among heads, we also propose a block-aware dynamic budget estimation method. By combining the scores from representative proxy heads with multi-head dy- namic budgets, we achieve more fine-grained block importance evaluation low computational cost. Experiments extensive benchmarks confirm the underlying similarity among attention heads. Leveraging fine-grained estimation, the proposed method achieves substantial gains in performance and efficiency compared to existing methods. More pre- cisely, ProxyAttn can achieve up to 10.3x attention acceleration and 2.4x prefill- ing acceleration without significant performance loss. Our code is available at bttps://githur. http://www....o.oosssseee///eraa
variety of mainstream models and
INTRODUCTION
Large Language Models (LLMs) have shown outstanding performance in long context tasks
LLu
ct al.
2025a b) and widely applied in fields such code generation (Jimenez al. 2023 Hui
ct al. based [.LMs face efficiency limitations when processing long texts with millions of tokens. The
2024 and mathematical reasoning (Chen et al. 2025 Guo et al. 2025). However, transformer-
quadratic complexity of the attention mechanism poses significant challenge to the inference of
models. Recently, numerous studies (Brauwers & Frasincar] 2021 Zhang et al., 2025) have worked on refining the attention module to increase the training and inference efficiency.
In particular, sparse attention methods mitigate the latency issues of the attention mechanism by skipping certain computations. By leveraging the inherent sparsity (Liu et al. 2022; Deng et al.) 2024) of attention score distributions, these methods can reduce computational costs while preserv- ing model performance. Some static methods (Beltagy et al. 2020 /aheer et al.) 2020) use heuristic fixed templates to achieve sparse computation, making it difficult maintain model performance with diverse attention patterns (Likhosherstov et al. (2021). To achieve more accurate estimation, many efforts have focused on exploring various low-cost method to assess the importance of tokens, which in turn enables the dynamic sparse patterns. The performance of hardware-fries friendly sparse
Work done during internship at Baidu, Inc. Corresponding author.

---

Published as a conference paper at ICLR 2026
Lisma3.1-a0 1-B-itt-ffl
Owenz: 5 78-Insstnuct11
METUETTO
Baler
Xirteiion
Prexyalin
74.3
TTPT Speedup on 128K
TIPT Speedup on 125K
Figure Performance and speedup results of different sparse attention methods on RULER.
attention, which typically handled at the block level (Guo et al. 2024), heavily dependent on the efficiency and accuracy of its importance estimation. MInference (Jiang et al.) 2024) leverages multiple pre-defined templates and an online computed index to guide the construction of block sparse patterns. Furthermore, FlexPrefill (Lai et al.) 2025) proposes an input-aware approach that Sa dynamically determines the sparse patterns and sparsity rates.
While existing methods have proven effeective in long-context scenarios, their performance high sparsity rates is still limited by their coarse-grained importance estimation. To evaluate the importance of each block, these methods typically compress along the sequence dimension to ap- proximate their attention scores. This pparse-grained compression can cause a few high-scoring positions to be overlooked, which in turn impacts the final performance. A token-level dot-product calculation would solve this issue but is not feasible, as its time complexity scales identically to full attention.
In this paper. we achieve fine- line-grained and efficient trade-off in block importance estimation by exploring compression of the attention head dimension. We begin by analyzing the performance of multi-head attention on long sequences, where we observed two primary characteristics of how the heads behave: (1) The overall attention trends of different heads are consistent. primary differences between heads lie in their sparsity. Based these findings, we propose ProxyAtin, a training-free sparse attention algorithm that approximates attention scores using a few representative heads. Specifically, we leverage the similarity among attention heads to approximate the full attention scores using the complete scores from a few pooled heads. By incorporating max pooling, proposed method enables more accurate block importance estimation while ensuring efficiency. Furthermore, given the significant differences in head sparsity, we propose an online block-aware budget estimation method. By integrating the unified importance score with head-level budget, we can achieve diverse sparse attention patterns across different heads.
(2) The
In order to evaluate our proposed method, we conduct series of extensive experiments. We use widely-adopted LLMs that are specifically designed for long-context tasks like Llama3.1-8B. 7--..----BIInstctttte
(Yang al., (025). along with variety of mainstream benchmarks such as RULER (Hsich et al. 22024) InfiniteBench (Zhang et al.) 2024), and LongBench-v2 -2 (Bai et al. 2024a b). Experimental results show that ProxyAttn signifi- cantly outperforms other methods on both synthetic and worl-world datasets. As illustrated in Figure [ the proposed ProxyAttn achieves Pareto front for performance and efficiency compared to other coarse-grained estimation methods. Our contributions can be summarized as follows:
Instruct (Grattafiori et al., (024) and
We introduce ProxyAttn, training-free sparse attention estimation method. It leverages the attention scores from small number of proxy heads to efficiently estimate the scores for all heads, achieving more accurate importance measures.
Considering the diverse sparsity among heads, we propose a block-aware budget allocation method. Coupled with the unified importance score, our method can provide different head sparsity budgets online, which in turn yields diverse sparse attention masks.
Extensive experiments demonstrate that the proposed ProxyAttn outperforms other coarse- grained sparse attention methods across various model architectures and benchmarks. Our approach yields a maximum 10.3x speedup for attention computation without significant per- formance loss.
2

---

Published as a conference paper at ICLR 2026
Layer.
Appreieen Byahhard
Spera hasc
2.002
B1001
22003
Lay
Layor
baad
60435
c0099
e000)
350 2650
(a) The tokens that attention heads focus exhibit consistency long context.
(b) A shared token ranking can simultancously accommodate the scores from different heads.
(c) Attention heads vary in their sparsity. Both sparse and dense heads still share common focus on certain tokens.
Figure 2: Observational study on Llama3, 1otmaa.1-BB-Ian 1-8B-lnstruct using 8K-token data from the Needle in Haystack (NIAH) Single-1 task of the RULER (Hsieh et al. (2024) dataset.
2
OBSERVATION
2.1 MOTIVATION
Recent work (Jiang et al. 2024 Lai et al. 2025 Gao et al. 2024) on the importance of different attention blocks typically employs heuristic "vertical and slash" patterns pooling methods along the sequence dimension for approximation. These methods either rely on specific heuristic template assumptions or opt for a graseee estimation, rendering the importance measures of blocks inaccurate (Xu et al. (2025). Given the inherent sparsity of attention distributions, such coarse- grained pooling can easily overlook a few high-scoring positions.
A viable approach for both accuracy and efficiency in token-level dot-product computations is to compress along the head num dimension instead of the sequence dimension. This allows few key heads to serve as proxies for estimating the importance of all heads. However, the feasibility of this approach hinges on the assumption that attention heads exhibit a certain degree consistency in their scores. To validate this hypothesis, conduct some observational studies on the behaviors of multiple heads in a long-context setting.
2.2 CHARACTERISTICS OF MULTIPLE ATTENTION HRADS
To investigate the similarities and differences among different heads, we conduct experiments on the Llama model using long-come context data containing 8K tokens. As shown in Figure 2 by observing the performance of different heads, we can derive two key findings:
Attention heads exhibit consistency in token focus. We first calculate the overlap of tokens that different heads primarily focus on. As shown in Figure 2a the score at position (i, j) represents the cumulative attention score at head obtained by the top 1024 tokens from head i. It can be observed that while the very dense first layer has inherently low recall, the attention overlap scores between different heads in other layers are very high. This suggests that in long texts, multiple attention heads, particularly in deeper layers, focus on large intersection of tokens.
To further assess this similarity, consider whether it could be captured by shared metric, with a natural candidate being average pooling across the heads. As depicted in Figure 2b] we sort the tokens by their average score across all heads and then analyze the discrepancy between the cumulative probability curve based on shared ranking and the actual score curve for specific heads. The consistency between the two curves serves as the basis for our ProxyAttn algorithm, which leverages the scores from few heads to approximate the attention of all heads.
Attention heads exhibit variability in sparsity. While the diversity among attention heads is a
well-established observation (Jiang et al. 2024). we find that it does not contradict findings
3

---

Published as a conference paper at ICLR 2026
Hoad H-2
Seq
Seg
Bind
Poelig
Sag
Hend
Prou
Skg
Fodita
Machoeng
Teck
Prass
Head
Carrent cuarse-graieed block with sequence dimension compression.
Progy-Ail:: fine-grainel celinstin with head num dimendon compression.
Figure 3: Illustration of ProxyAttn. By compressing the head dimension, ProxyAttn can obtain token-level importance scores, leading more accurate block importance estimation. A proxy head able to obtain diverse masks by leveraging the online budget estimations from different heads.
of consistency. Consistent with the preceding setup, we utilize the same shared token ranking to examine the attention scores across various heads. Figure 2c illustrates that the primary variation between heads is in their sparsity rather than the tokens they focus on. This observation corroborates the findings of Xiao et al. (2024); Fuu et al. (2024). As seen in the figure, given the same ranking, all heads assign high attention to the leading tokens. The main distinction is that some heads are very sparse and only attend to the initial portion, while others are dense and maintain significant scores for later tokens. This motivates us to realize that while leveraging shared scores, flexible budget must be allocated to different heads to ensure performance.
PROXYATTN
Building upon above observations of attention heads, propose the ProxyAttn algorithm to efficiently achieve accurate, fine-grad grained importance estimation for block sparse attention. A schematic comparison with existing coarse-grained estimation methods is shown in Figure 3 Lever- aging the consistency of token focus among heads, we first employ an intra-group score sharing method ($3.1) for importance estimation. To account for the varying sparsity of heads, we then dynamically assign a unique budget ($ 3.2) to each head to generate diverse sparse masks.
3.1 UNIFTED SCORE ESTIMATION
In order obtain the token-level dot-product scores for each block in the attention map, compression must be applied along dimensions other than the sequence length to ensure efficiency. Given the consistency of token attention across heads on long contexts, we propose perform compression along the head num dimension. Specifically, we partition the attention heads into distinct groups. Within each group, a designated head is used to compute attention, which then serves as a proxy to guide the attention scores for the entire group. The entire process can be formulated as:
(QIKOT )
A' maxpool
softmax
E G,
(1)
where Q9, K E Nxd, and A, E X For all heads belonging to group g. a single token- level score will be shared. Combined with max-pooling, it enables fine-grained block importance estimation at the cost of computing only |G| proxy heads.
For the representative head queries Q" and keys K heuristically apply the same average pooling as for the sequence. This can also be seen as a form of compression along the head number dimension and can be formalized as:
Q"
Σo K
Σ
K
(2)
G
SEG
IEG
It should be noted that for models using Group-Query Attention (GQA) Ainslie et al. 2023), the group granularity will be aligned with the keys, ensuring that all queries belonging to the same group of keys also within that same group.
4

---

Published as a conference paper at ICLR 2026
Algorithm Block-aware Budget Estimation for Head
Input:
Query states Q', key states K', cumulative probability threshold y
Output: Estimated budget by
A < softmax Qise K.T vddk
D Compute approximate attention scores
a' < avgpool(A))
Aggregate into block-level scores >> Normalize and sort block-level scores
a' < sort(a') /sumnaa)
b; + ming
a a
227 /|al
D Determine budget ratio for selected head
return b;
To further reduce the computational cost of the estimation operation, also dropped certain qk pairs by stride, keeping only the first token each stride window. Driven by the local similarity inherent in attention mechanisms (Wu al. 2024). this operation reduces the computational cost without significant loss of accuracy (see Appendix B.2 for a detailed analysis). For a model with heads, the theoretical computational cost of using 9 representative proxy heads for estimation is expected to be n-strid?? of full multi-head attention. Leveraging head sharing, the proposed method reduces the computational cost for 7B model roughly one percent of full attention under con- ventional settings. This allows for fine-grained estimation without compromising computational efficiency.
3.2 DYNAMIC BUDGET ALLOCATION
After obtaining importance scores for different blocks, a common practice is to further obtain sparse block mask (Gao et al. 2024 Xu et al. 2025). often by using methods like Top-K selection. However, directly applying this approach to proposed method would result in identical masks for all heads in the same group, thereby impacting the maximum achievable sparsity rate. Given the find- ings from $2.2 the attention focuses different heads are consistent, with the primary difference being in their sparsity. We consider online approach to approximately evaluate the sparsity of individual heads, which in turn guides the selection of the budget.
Inspired by the work of Jiing et al. (2024), we propose use the queries from the last block to estimate the sparsity of different heads. The budget required for a given head is approximated the minimum budget for the query of the final block to achieve cumulative probability of y. Furthermore, given the significant discrepancy between token-level computation and the actual block budget due to attention sparsity, we employ average pooling ensure that the budget estimation is performed at the block level. [oporittm[[prreeeee the overall approach to budget estimation. After obtaining the budget b; for each head, we select the top b, blocks with the highest unified scores to form the sparse attention masks. The final sparse attention mask S MMN for head can be formalized as:
=f1: if E TopK(Af/m) K b,NV = [0. otherwise
S S
Where M, N is the block-level sequence length, and A is the previously estimated block-level im- portance of head i. As shown in F the proposed ProxyAttn achieves diverse sparse attention masks by combining shared head scores and the dynamic budget allocation method. By leveraging the efficient block sparse attention kernel (Guo et al., (024), we can substantially accelerate the attention operation for long contexts while maintaining comparable model performance.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Evaluation datasets. To thoroughly evaluate the performance of proposed method for long con- texts, we utilize two types of benchmarks: synthetic and real-world datasets. For synthetic bench- marks, we adopt the RULER (Hsich et al. 2024), which comprises a variety of synthetic tasks with token lengths ranging from 4K to 128K. This provides direct way to evaluate the long-context
5

---

Published as a conference paper at ICLR 2026
Table 1: Experimental results on RULER. Bolded values represent the best result under the same setting, while underlined values indicate the second best. * denotes methods that require additional training. Sparsity calculated by weighting the ratio of different lengths by their token count.
Method
Sparsity
4k
8k
16k
32k
64k
128k
Avg.
w.Ag..
Liama3. 1-8B-Instrwet
FtteAnttntion
0.00
96.17
94.21
93.72
89.56
86.13
76.95
89.46
86.49
Minfereniee
0.70
96.23
54.22
95.92
88.82
84.64
70.74
85.09
84.25
FeePPrtilll
0.72
96.03
94.07
94.35
92.05
85.21
75.12
89.47
86.28
XAllention
0.69
96.19
94.27
93.99
91.04
85,15
72.64
88.88
85.35
ScarAttention
0.77
96.20
94.44
93.80
90.01
84,99
74.34
88 96
85.60
Arin (7=00900)
0.80
95.76
94.32
93.52
91.05
86.14
75.75
89.42
86.31
ProxyAttn (y-0.99
0.69
96.07
94.57
94.01
91.55
86.63
78.25
90.18
87.43
Qwen2.5-777 Instruct- IM
Futemintttion
94.62
84.58
78.90
87.92
Minference
0.63
94.41
51.43
89..18
87.36
82.73
76.60
86,99
84.20
FlexPrefill
0.69
94.09
90.97
89.27
87.59
81.85
76.24
86.67
83.85
XAttention
0.61
92.92
88.22
84.45
84.09
79.99
73.74
83.90
81.02
ProxyAtin (y=0.90)
0.74
94.57
91.34
89.19
86.55
83.76
76.80
87.04
84.32
ProxyAttn (yy00.95)
83.57
capabilities of models and the performance trade-off introduced by sparse attention. Besides, We use InnchiteBnnch (Zhang et al. (2024) and LongBench-v2 (Bai et al., [202aa b) as benchmarks for real-world tasks. Both include diverse long-text tasks, such QA and summarization, providing more comprehensive evaluation in practical applications.
Selected baselines. To fairly evaluate the proposed ProxyAltn, compare against the follow- ing mainstream block sparse attention methods as strong baselines: (1) MInference (Jiang et al., 2024) enables dynamic sparse attention by defining three unique patterns and estimating the optimal indices for each pattern at inference time. (2) FlexPrefill (Lai et al. 2025) leverages input statistics enable context-aware sparse pattern determination and Top-P block selection, making the appli- cation of sparse attention more adaptable. (3) XAttention (Xu et al. 2025) mitigates the neglect of important tokens after pooling by more effectively capturing the Vertical-Slash Šlash pattern through the computation of the anti-diagonal. (4) SeerAttention (Gao et al. 2024) employs trainable MLP to process the latent information from the pooled queries and keys, which allows to capture block importance more effectively than direct pooling.
For the hyperparameter settings of the baseline methods, we strictly follow the optimal parameters reported in their original papers. Specifically, MInference pre-configures patterns and budgets for each head. To ensure fair sparsity. we configure FlexPrefill with a of 0.95 and a minimum budget of 2048. While XAttention employs head budgets determined via offline search, for models like Qwen where these budgets are not provided, we directly set a threshold of 0.95 to ensure the method's performance, SeerAttention implements dynamic block sparsity based threshold, which we set to 5e-4 as original paper.
Implementation details. Consistent with previous studies Jiang et al. (2024): Lai al. 2025 we choose to evaluate sparse attention methods exclusively during the pre-filling stage, which is the primary computational bottleneck, while using full attention for the decoding stage. All performance and speed experiments are conducted on an NVIDIA EVIDIA-H800-AT H800-80GB platform. We fuse dot-product and max- saxx-oooling operators to reduce latency during the block estimation phase, following the kernel design of (Gao et al. 2024). For the subsequent block sparse attention stage, we leverage the efficient implementation proposed by Guo al. (024)..
For the hyperparameter selection of our proposed ProxyAttn, we adopt two settings with values of 0.95 and 0.90, respectively, while maintaining a stride of and a minimum budget of 2048 tokens. See Appendix B for more hyperparameter experiments. When choosing the number of proxy heads for estimation, we selected and 4 for the LLaMA and Qwen models, considering the similarity of their attention heads. This indicates that a calculation using just one head accurately represent the behavior of all 32 heads the LLaMA3.1 1-BB model. will discuss this further in Section 5.1

---

Published as a conference paper at ICLR 2026
Table Main Results on InfiniteBench and LongBench-v2. LongBench results are reported without Chain-of-Thought Thought (CoT), with the "w. CoT" results shown in parentheses. The overall score is obtained by averaging the final scores from the two benchmarks.
InfiniteBench
Lonchench-v2 (w. CoT)
Methods
Overall
En
7h
Code
Math
Retri.
Short
Medium
Long
Lloma3. J. 88 Instrucr
Attention
32,09
13,65
23.35
33.43
84,84
33.90 (36.10)
26.50(30.70)
25.00 (27.80)
35.38
Minference
29.00
12.57
26.40
35,71
73.59
35.00 (38.90)
23.00
30,60 (26.90)
34.78
30.75
80.77
(36.70)
(25.10)
(28.70)
XAttention
31.45
13.28
22.84 84
32.57
80.03
35,00 (35.00)
26.00 (31.20)
26,90 (27.80)
34.89
ProxyAttn
33.21
13.35
25.13
32.00
81.49
34.40 (41.10)
27.40 (27.90)
29.60 (29.60)
36.06
Qwen2.5 1022255-8--nnteeee 788-attcc
FullAnention
30.57
15.65
31.98
41.14
77.87
39.40 (38,90)
28.40
34.30 (33.30)
38,22
Minference
30.72
13.15
30.30
47.71
78.53
40.60 (43.90)
(3..0
22.70(25.60)
37.91
FlexPrefill
31.42
15.66
30.96
42.29
75.07
(35.60)
30.20 (34.90)
27.80 (31.50)
37.39
XAttcation
29.63
15.56
32.74
43.71
70.40
41.70 (42.20)
27.40 (33.50)
32.40 (30.60)
37.26
ProxyAlin
(40.00)
(36.70)
(32.40)
10.3x
10
Rashhtttention
Minference
FlexPrefill
XAttention
ProxyAttn (Ours)
Attention Speadup
2.0c
10:00.ce..0...
4K
8K
16K
32K
64K
128K
256K
Figure 4: Kernel-level speedup of existing sparse attention methods with varying input lengths.
4.2 ACCURACY RESULTS
Results on synthetic tasks. Following the settings of Hsieh et al. (004). we also report weighted average scores (wAvg.), which assigns linear weights the results based on actual token lengths. The results for synthesis tasks with various length are presented in Table [ which shows that the proposed method has significant advantages over existing methods in terms of both sparsity and performance, By sharing scores between attention heads, ProxyAttn achieves excellent results on models with varying GQA settings, including Ilama3. and Qwen2.5. This highlights the strong generalization capabilities of the proposed method.
Additionally, we evaluate the performance of the proposed method various sparsity rates by ad- justing When Attn maintains a sparsity ratesimilar to other methods, it achieves the best average performance on RULER, even surpassing the performance of full attention on the Llama model. As the sparsity further increases, the proposed method can still achieve comparable exper- imental results while also reducing inference latency. contrast to the performance degradation XAttention faces when transferred to Qwen, ProxyAttn can be quickly migrated to other models without hyperparameter searching by using online budget estimation.
Results on real-world tasks. In addition to synthetic tasks, we also evaluate the performance of existing methods on real-world tasks using benchmarks such as InfiniteBench Zhang al.) (2024) and LongBench-v2 Baii et al.] (2024b). As shown in Table 2 despite the lower discriminabilie lity of real-world tasks compared to synthetic ones, the proposed ProxyAttn still achieve the best average results and even surpassed the performance of full attention. This demonstrates the potential of sparse attention methods for handling long documents. Full experimental results for the two datasets provided in Appendix C
4.3 EFFICIENCY RESULTS
Attention speedup. To intuitively demonstrate the efficiency of our proposed method, we first evaluate its attention acceleration ratio compared to other methods at kernel level. Considering that

---

Published as a conference paper at ICLR 2026
00
RULLE Score
RULER:
26:3
Rmopleth
1ul Alon (Lumel
Atunttan Soat: tugget
Preeplers (Dwer
74
(0wer
FullArtaation
Miderran dedeeronc
Plaitrofiil
XAstantion
ProsyAttnggn ProsyAttnigan
Dysamic bucgrt
The Number of Proxy Heads
Sparsity
(a)
(c)
Figure 5: Experimental analysis of proposed method. (a) The performance of different numbers of proxy attention heads across various models. (b) Latency for estimating block importance with 1 128K inputs using different methods. The latency of all methods less than 10% of the Full Attention. (c) Performance degradation with increasing sparsity rate for different budget allocation methods.
the inductive bias of sparse attention is not effective with random vectors, we cache the queries, keys, and values with true distribution from the RULER dataset for speed measurement. The final speedup is obtained by averaging multiple runs across different layers. We use the official implementation of FlashAttention (Dao al. 2022 Dao 2024) as our baseline for testing, and the experimental results are shown in Figure
As shown in the figure, sparse attention begins to show its speedup effect when the input text exceeds 32K tokens. Thanks to its fine-grained and low-cost block importance estimation, ProxyAttn can achieve a lower sparsity (see Appendix [) while maintaining performance, thereby leading to a more significant specdup. It is worth noting that the proposed method can achieve 10.3x attention acceleration with 256K tokens, which would significantly improve the pre-filling speed of the model.
End-to-end prefilling speedup. The pre-fill latency of LLMs is also influenced by the MLP mod- ules, which diminishes the acceleration effect gained from optimizing the attention operation alone. As shown Figure we report the performance and Time First Token (TTFT) speedup of our proposed method under the RULER 128K setting. By adjusting the settings, the proposed Proxy- Attn achieves Pareto frontier of performance and efficiency with fine-graines block importance estimation. With a minimal loss of model performance, an end-to-end speedup of up to 2.4x is achieved.
ANALYSIS
5.1
ATTENTION HEAD SIMILARITIES ACROSS MODELS
The effectiveness of Proxy Attn is grounded in the existence of similarities among different heads. To compare how this phenomenon varies across different models, we conduct experiments on RULER 128K with varying number of proxy heads. Figure[5 shows the results where test models with different GQA configurations by varying the number of proxy heads from 1 to the number key heads.
The multiple attention heads of the Llama model exhibit highly consistent performance. Conse- quently, single proxy head can effectively represent all 32 heads, achieving performance that sur- passes the baseline. Furthermore, for the Qwen model, approximately 4 proxy heads are typically required achieve satisfactory performance. We hypothesize that this may be related its very high GQA grouping ratio (seven queries per key). Although increasing the number of proxy heads raises the latency of importance estimation, the overall latency not significant due to the use of strided dropout on the sequence dimension.
5.2 COMPARISON OF ESTIMATION LATENCY
Beyond achieving a higher sparsity rate, the inherent latency of the block estimation algorithm must also be factored into the overall algorithmic performance. To this end, we evaluate the overhead of existing sparse attention methods at the block estimation stage. As shown Figure [bb] all methods achieve estimation latency of less than 10% of that of full attention. In particular,
8

---

Published as a conference paper at ICLR 2026
Layer
Layer 24
Layer'1
Layyrr 24
Layer 26
Layer 24
(a) Conventionally computed at- tention scores.
(b) Attention scores after remov-
(c) Normalized scores after re-
ing sink tokens (n=4).
moving sink tokens (n=4).
Figure 6: Illustration of token overlap across different heads.
the proposed ProxyAttn reduces computational costs by compressing the attention head dimension, thereby allowing it to account for more fine-grained, token-level dot-products while maintaining latency comparable to other methods. Furthermore, due to the use of dropout with a small stride, increasing the number of proxy heads does not significantly increase the estimation latency, which further enhances the generalizability of the method.
5.3 THE EFFECT OF DYNAMIC BUDGET ALLOCATION
As discussed in $2.2 the sparsity among attention heads is highly variable. Consequently, if a static Top-K allocation method is employed, it can lead excessive information loss in some dense heads, which in turn impacts the final performance. To validate our hypothesis, we compare the performance degradation of static and dynamic methods as the sparsity rate increased, using 128K- token inputs. As shown in Figure [cl_ the proposed method significantly outperforms the static baseline, exhibiting notable performance degradation only when the sparsity exceeds 90%. This highlights the importance of independently estimating the budget for each head.
5.4 MODEL SCALING EXPERIMENTS
Given that larger models typically have more at- tention heads, we scale our experiments to 70B model to validate the effectiveness of our method with greater number of attention heads. To ensure a fair comparison, we also conducte experiments on several baselines that compatible with the 70B model, while maintaining the same configurations as primary experiments. As shown in Table 3 even with the number of attention heads increased to 64, we can still achieve effective block importance estimation using single proxy head. While main- taining optimal performance, we can achieve a 2.08x end-to-end prefilling speedup for Llama3. 70B on 128K sequences.
Table 3: Ruler results on Llama3.1 1-B0BB Instruct with 64 attention heads.
Method
RULER
TTFT/s
FullAttention
65.03
91.84 (1.00x)
Minference
60.33
882.9 (1.1Tx)
XAttention
58.59
60.57 (1.52x)
ProxyAttn 62.23
44.23 (2.08x)
5.5 EFFECT OF SINK TOKENS
Recent studies (Xiao et al. 2023) have found that sink tokens, typically located the beginning of sequence, often receive a large amount of attention score. To eliminate the influence of sink tokens, we conducte a revised observational experiment based on $2.2 removing attention scores to the sink tokens. For given token, let its attention scores be A. with shape (head num, len). The overlap scores at (i, 1) in the figure are computed as follows:
token index (Api() k 1024).indice.
(4)
Score(i, sum(AAj, token_index]]

---

Published as a conference paper at ICLR 2026
The experimental results are shown in Figure 6 It can be observed that as the layer depth increases, the attention scores received by sink tokens grow progressively larger. After re- re-normalizing the at- tention, there is still substantial overlap across different heads. This indicates that the improvements provided by the proxy head are not merely a result of shared sink tokens, but also reflect its capacity to act proxy the global level.
6 RELATED WORK
Multi-head attention. While offering powerful modeling capabilities, Multi-Head Attention
(MHA) (Vaswani et al. (017) inherently suffers from efficiency issues. Many recent studies (Zheng
et al., (024)) aim to refine the attention mechanism from the perspective of multiple heads. Multi-
Query Attention (MQA) (Shazeer) 2019) and Grouped-Query Attention (GQA) (Ainslic al. 2023)
significantly reduce the memory footprint of the KV cache by having multiple queries share single
set of keys and values. Multi-head Latent Attention (MLA) (Liu et al.) 2024) implicitly groups via
low-rank projection, achieving further reduction in KV Cache memory consumption.
In addition, other work investigates the differential performance across various heads. MoA (uu et al. 2024) achieves efficient attention computation by assigning diverse, sparse attention patterns to each head every layer. AuoAttention (Xiao et al., 2024) utilizes an optimization method to effectively identify sliding-window heads and subsequently performs the corresponding sparse com- putation. Beyond the sparsity differences discussed above, the findings of this paper also indicate potential consistency across multiple heads, which can be leveraged for designing subsequent efficient attention architectures.
Sparse attention. Exploiting the inherent sparsity of the attention mechanism, MInference (Jang jct al. (024) and FlexPrefill (Lai et al., (2025) achieve efficient block-sparse attention by comput- ing dynamic sparse mask. SeerAttention (Gao al. 2024) introduces trainable MLP to tract information from different blocks, leading to a more accurate estimation. XAttention extends the pooling- based methods by incorporating more fine-gra fine-grained anti- miti-diagonal scoring mechanism, which leads to a better capture of the heuristic "vertical and slash" patterns. Similarly inspired by inter-head similarity, SharePrefill (Peng et al.) 2025)) performs offline clustering of attention heads. By fully sharing sparse patterns among similar heads, it achieves more accurate block attention estimation.
However, directly applying sparse attention to LLMs still incurs an unavoidable performance degra- dation. Some approaches (Team et al.) 2025) attempt to introduce optimization objectives during training to induce native sparsity of LLMs. NAS (Yuan et al.) (2025) achieves a native sparse at- tention that surpasses full attention by integrating three modes: compression, selection, and sliding window. Referencing the principles of MoE Mfxture-offEExperre MOBA (Lu et al. (0255 achieves a mix of block sparse attention without introducing any additional parameters.
CONCLUSION
In this paper, we propose ProxyAttn, training-free method for sparse attention. Considering the similarity among multiple attention heads, we leverage proxy heads to efficiently estimate the atten- tion scores for all heads. When integrated with dynamic budget allocation, ProxyAttn outperforms existing approaches across multiple datasets and models. We will further investigate the utility of the proxy head specifically within the decoding phase of LLMs.
ACKNOWLEDGEMENTS
We gratefully acknowledge the support of the National Natural Science Foundation of China (NSFC) via grant 62236004 and 62476073.
REFERENCES
Joshua Ainslie, James Lee-Thorp Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check-
10

---

Published as a conference paper at ICLR 2026
points. arXiv preprint arXiv:23005113245 2023.
Yushi Bai, Xin Lv, Jiajie Zhang. Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongRench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 3119-3137, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/ v1/2024.acl-long. 172. URL BeLLhtts:////
20az//2224422 acl-long.172
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang. Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong. Jie Tang, and Juanzi Li. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks, arXiv preprint arXiv:24/2.15204 2024b.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.0513 2020.
Gianni Brauwers and Flavius Frasincar. A general survey on attention mechanisms in deep learning. IEEE transactions knowledge and data engineering. 35(4):3279-332 -2999, 2021.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng. Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.0955 2025.
Tri Dao. FlashAttention 2: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICLR), 2024.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with 10-awareness. In Advances in Neural Information Process- ing Systems (NeurIPS), 2022.
Yichuan Deng, Zhao Song, and Chiwun Yang. Attention is naturally sparse with gaussian distributed input. CoRR, 2024.
Tianyu Fu, Haofeng Huang. Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406. 14909, 2024.
Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi. Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, et al. Secrattention: Learning intrinsic sparse attention in your llms. arXiv preprint arXiv:2410. 13276, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2113 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms 2025.
via reinforcement learning. arXiv preprint arXiv:2501.
Junxian Guo, Haotian Tang, Shang Yang, Zhekai Zhang. Zhijian Liu, and Song Han. Block Sparse Attention. httpps://gitiie com mit han lab/Block Spacckk--ppreeeeee Attention 2024.
Cheng-Ping Hsich, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.066
2024.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. /2 2186, 2024.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yer Yew Lin, et al. Minference I.0: Accelerating pre-filling for cong-context llms via dynamic sparse attention. Advances in Neural Information Processing Systems, 37:52481-52515 2024.
11

---

Published as a conference paper at ICLR 2026
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. arXiv preprint arXiv:2502.20766, 2025.
Valerii Likhosherstov, Krzysztof Choromanski, and Adrian Weller. On the expressive power of self-attention matrices, arXiv preprint 1rXiv:2106.03764 2021.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model. arXiv preprint arXiv:2405.04434 2024.
Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang. Jiebin Zhang, et al. A comprehensive survey on long context lan- guage modeling. arXiv preprint arXiv:2503. 17407, 2025a.
Liu Liu, Zheng Qu, Zhaodong Chen, Fengbin Tu, Yufei Ding, and Yuan Xie. Dynamic sparse attention for scalable transformer acceleration. IEEE Transactions Computers, 11612):3165- 3178, 2022.
Yijun Liu, Jinzheng Yu, Yang Xu, Zhongyang Li, and Qingfu Zhu. A survey on transformer context extension: Approaches and evaluation. arXiv preprint arXiv:2503. 13299, 2025b.
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, al. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502. 1311:/990//388 /3189, 2025.
Dan Peng, Zhihui Fu, Zewen Ye, Zhuoran Song, and Jun Wang. Accelerating prefilling for long- context Ilms via sparse pattern sharing. arXiv preprint 25Xi::2505.1957 2025.
Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:191.00210,, 2019.
MiniCPM Team, Chaojun Xiao, Yuxuan Li, Xu Han, Yuzhuo Bai, Cai, Haotian Chen, Wentong Chen, Xin Cong, Ganqu Cui, et al. Minicpm4: Ultra-effiieent Ilms on end devices. arXiv preprint arXiv:2506.07900 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Wei Wu, Zhuoshi Pan, Chao Wang. Liyi Chen, Yunchu Bai, Tianfu Wang, Kun Fu, Zheng Wang, and Hui Xiong. Tokenselect: Efficient long-context inference and length extrapolation for lims via dynamic token-level kv cache selection. arXiv preprint arXiv:2411.02888 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. arXiv preprint arXiv:2410.10819, 2024.
Ruyi Xu, Guangxuan Xiao, Haofeng Huang, Junxian Guo, and Song Han. Xattention: Block sparse attention with antidiagonal scoring. arXiv preprint arXiv:2503.16422. 2025.
An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, et al. Qwen2. 5-1m technical report. arXiv preprint arXiv:2501. 15383, 2025.
Jingyang Yuan, Huazuo Gao, Damai Dai. Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware- allggee and natively trainable sparse attention. arXiv preprint arXiv:2502.1 1089, 2025.
12

---

Published as a conference paper at ICLR 2026
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-11299 2020.
Jintao Zhang, Rundong Su, Chunyu Liu, Jia Wei, Ziteng Wang, Haoxu Wang, Pengle Zhang, Huiqiang Jiang, Haofeng Huang. Chendong Xiang, Haocheng Xi, Shuo Yang, Xingyang Li, Yuezhou Hu, Tianyu Fu, Tianchen Zhao, Yicheng Zhang, Boqun Cao, Youhe Jiang, Chang Chen, Kai Jiang, Huayu Chen, Min Zhao, Xiaoming Xu, Yi Wu, Fan Bao, Jun Zhu, and Jianfei Chen. A survey of efficient attention methods: Hardware-efficient sparse, compact, and linear attention. 2025.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang. Zhiyuan Liu, et al. Infinite bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Compulational Lingnistics (Volume 1: Long Papers), pp. 15262-15277, 2024.
Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Mingchuan Yang. Tang, Feiyu Xiong, and Zhiyu Li. Attention heads of large language models: A survey, arXiv preprint arXiv:2409.03752, 2024.
A
AI TOOLS
In this manuscript, AI tools such ChatGPT were used solely for refining the text and did not contribute to the content or ideas presented.
B HYPER-PARAMETERS
We present the hyperparameter search experiments related to ProxyAttn in this section. Ultimately, we chose a stride of 4 and a minimum budget of for LLaMA and 2048 for Qwen for our main experiments.
B.1 EFFECT OF GAMMA AND MIN BUDGET
Table 4: The hyperparameter search experiments concerning sparsity on RULER Benchmark.
7
min budget
sparsity
4k
8k
16k
32kc
64k
128k
Llama3. 1-88B-Instran
0.95
0.69
96.07
94.57
94.01
78.25
90.18
0.95
1024
0.69
95.75
94.04
93.65
90.66
86.05
76.92
89.51
86.63 63
0.95
2048
0.67
95.96
94.17
93.60
90.37
85.96
75.94
89.33
86.29
0.90
1024
0.80
95.82
94.23
93.31
90.18
84.72
73.63
83.65
85.25
@wen2.5
Instruct-IAN
0.95
0.63
93.91
91.26
90.16
86.89
83.46
76.99
87.11
84.46
77.51
0.95
2048
0.61
94.61
92.07
89.75
86.68
83.57
77.09
87.30
84.53
0.90
0.76
93.59
90.27
58.34
86.19
82.10
75.84
86.06
83.31
0.90
1024
0.76
94.07
90.62
89.32
86.48
82.65
74.71
86.31
83.37
0.90
2048
0.74
94.57
91.34
89.19
86.55
83.76
76.80
87.04
84.32
B.2 EFFECT OF STRIDE
As shown in Table 5 a slight increase in the stride does not significantly impair performance due to the local similarity of attention, but substantially reduces the estimation overhead.
13

---

Published as a conference paper at ICLR 2026
Table The impact of varying strides on the proposed method.
stride
min budget
4k
8k
16k
32k
64k
128k
Avg
Llama3. 1 1-8B---B---e Instruct
Full Autention
95.63
92.27
91.60
87.66
84.85
76.17
88.03
T
95.43
92.116
90.94
89.25
84,98
74.44
87.87
0.95
95.32
92.13
91.46
89.29
85.24
75.56
88.17
4
95.16
92.89
91.84
89.69
84.64
76.27
88.42
Qwen2.5 5-BB 17BB--nstuct--- IM
Full Attention
93.35
89.94
87.81
87.56
85.33
76.89
86.81
93.67
89.35
88.23
87.006
85,41
76.24
86.66
2
0.95
2048
93.67
89.88
88.54
87.66
85.13
75.10
86.66
4
93.57
90.48
88.30
86.55
84.21
74.97
86.35
C
DETAILED RESULTS ON REAL-WORLD DATASETS
Due to space limitations, the main text reports the average performance across two datasets. The complete experimental results are presented in this section.
Table Main results on InfiniteBench across various models.
En
Zh
Code
Math Find
Retrieval
Methods
Avg.
Sum
QA
MC
Dia
QA
Debug
PassKey
Number
KV
Llama3. J IBs-tsst
FullAnention
27.12
14.37
66.38
20.50
13.65
23.35
33.43
100.00
99.32
55.20
45.33
Minterence
25.31
17.30
60.70
13.55
T2.37
26.40
35.7T
100.00
47.97
22.80
4T.113
FlexPrefill
26.54
14.15
66.81
15.50
13.69
23.60
32.29
100.00
99.32
43.00
43.49
XAtienton
26.87
14.00
69.43
15.50
13.28
22.84
32.57
100.00
99.49
40.60
43.46
ProxyAtt
26.78
14.85
67.69
23.50
13.35
25.13
32.00
100.00
99.66
44.80
44.78
@won2.5-75 2.m.2.5778. Instruc--M
FullAttention
23.48
16.86
65.94
16.00
15.65
31.98
41.14
100.00
100.00
33.60
44.47
Xtinference
24.37"
13.87
61.25
13.00
T5.T6"
33.20
447.T
100.00
T00.06
35.60"
44.99
FlexPrefill
15.50
XAtiention
22.64
15.99
66.38
13.50
15.56
32.74
43.71
100.00
100.00
11.20
42.17
ProxyAtt
23.16
15.26
65.07
15.50
15.36
32.99
41.71
100.00
100.00
32.80
44.19
Table 7: Main results LongBench-v2 across various models.
Short
Medium
Long
Overall
Method
w.o. Cot
w. Col
w.o. Cot
w. Col
w.b. Cot
w. Col
w.o. Col
w. Cot
Llama3. 1-88- Kö-Instruct
PullAttention
33.90
36.10 10
26.50
30.70
25.00
27,80
28.80
32.00
Minference
35.00
38.90
26.00
31.20
30.60
26.90
30.20
33,00
FlexPrefill
30.60
36.70
27.90
25.10
25.90
28.70
28.40
30.00
XAttention
35.00
35.00
26.00
31.20
26.90
27.80
29.40
31.80
ProxyAttn
@wen2.5 7B Instruct JM
FullAttentonn
39.40
38.90
28.40
36.30
34.30
33.30
33.60
36.60
Minference
40.60
43.90
28.70
29.00
36.80
FlexPrefill
41.70
35.60
30.20
34,90
27.80
31.50
33.80
34.40
XAllention
41.70
42.20
27.40
33.50
32.40
30.60
33.60
36.00
ProxyAttn
43.90
40.00
28.40
36.70
27.80
32.40
33.80
37.00
D SPARSITY RATES
The sparsity rates achieved by different methods using the optimal configuration on RULER. A higher sparsity rate leads greater acceleration effect.
14

---

Published as a conference paper at ICLR 2026
Table 8: Sparsity rates achieved by different sparse attention methods across varying input lengths.
Model
4K
8K
16K
32K
64K
128K
Llama3. 1.....1.0 1e-B----Bsiiii Instruct
Minference
6.15
11.62
22.60
34.13
50.44
72.24
FlexPrefill
19.76
43.62
57.82
65.15
71.86
75.10
XAttention
29.10
39.81
49,91
58.96
68.42
73.20
ProxyAttn
49.63
63.21
73.31
78.27
83.19
83.86
Qwen2.5-77B
1..---B--nssructt IM
Minference
25.99
32.02
39.54
50.35
61.82
72.44
FlexPrefill
19.39
45.02
59.26
65.50
68.39
74.35
XAttention
34.11
44.47
51.54
56.99
60.55
65.32
Att
13.48
42.43
60.78
70.23
74.12
79.52
E OBSERVATIONS IN BOTH SYNTHETIC AND REAL-WORLD TASKS
To verify the domain-specifii differences of the phenomena observed in Section 2.2 we conducted observational experiments both synthetic and real-world datasets. As shown Figures and we observe the cumulative attention score curves for input tokens ranked by both the proxy head and the original heads across different datasets. For synthetic data, we use RULER dataset samples consistent with $2.2 while real-world data are sampled from LongBench v2 (Bai et al. (0224b). can be observed that the proxy head exhibits relatively consistent behavior across the two different domains, validating the generality of the proposed method.
Appornet
(a) Layer 0
(b) Layer
(c) Layer 15
(d) Layer 23
Figure 7: Cumulative attention curves across different layers of Llama3.1 I-8B-Instruct synthetic data from RULER.
(a) Layer O.
(b) Layer
(c) Layer 15.
(d) Layer 23.
Figure 8: Cumulative attention curves across different layers of Llama3.1 1--1--8----stte Instruct real-world data from LongBench v2.
15


Reviews:
[{"review_id": "38fCjyFUPX", "reviewer": "Reviewer_wqai", "summary": "The paper introduces ProxyAttn, a training-free sparse attention method designed to speed up long-context Transformers by exploiting redundancy across attention heads. Instead of pruning tokens directly, the authors group similar heads into “proxy heads” and use their aggregated (max-pooled) attention scores to estimate token importance. Each real head then receives its own Top-K allocation based on its intrinsic sparsity. This approach effectively compresses along the head dimension rather than the sequence dimension, which is both novel and impactful. Experiments on large-scale LLMs (LLaMA 3.1, Qwen 2.5) show up to a 10× speedup with almost no drop in accuracy.", "questions": "see weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The method can plug into existing Transformer architectures without retraining or parameter tuning, making it practical for deployment.\n2.Using proxy heads to estimate token-level importance leads to higher overall sparsity while preserving key contextual information—often matching or slightly outperforming full-attention baselines on long-context benchmarks.\n3.The reported efficiency gains are impressive: up to a 10.3× reduction in kernel-level attention time and a 2.4× speedup in total prefilling latency, with accuracy losses staying under one percentage point—an excellent trade-off for latency-critical use cases.", "weaknesses": "1.The method is only evaluated during the prefill stage; it’s unclear whether similar benefits (or issues) would arise during autoregressive decoding, especially for long generations.\n2.Important hyperparameters—like the sparsity threshold and minimum Top-K—are tuned via grid search. The lack of an adaptive or principled selection rule could make it harder to apply the method out-of-the-box to new models or domains.\n3.While computational savings are detailed, the paper doesn’t discuss the memory overhead from storing proxy key–value tensors, which could be a concern on GPUs with limited memory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a9DWtlowPB", "reviewer": "Reviewer_N8YJ", "summary": "The paper proposes ProxyAttn, a training-free sparse attention algorithm for long-context prefill. The key idea is to compress along the head dimension: group attention heads and use a small number of representative “proxy” heads to compute near token-level scores, then max-pool and aggregate them into fine-grained block importance estimates. To accommodate that different heads prefer different sparsity levels, the method adds a block-aware per-head dynamic budget based on a lightweight cumulative-probability criterion. This yields fine-grained, block-sparse masks at low estimation cost, delivering attention speedups and 2.4× prefill acceleration with no significant performance loss.", "questions": "1. The method hinges on cross-head token-focus similarity; corner cases with highly specialized heads may degrade the shared ranking quality. Do authors observe tasks or head types where the shared ranking under-represents important blocks (e.g., rare long-range dependencies)?\n2. How sensitive is performance to the group size and the choice of the representative head? Could learned or data-driven method further help beyond simple averaging (e.g., fine-tune somes head as proxyheads)? \n3. Could the proxy-heads be reused for decoding with query-aware sparse attention?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Novel sparse attention design. Instead of compressing along the sequence dimension (tends to be coarse), the paper compresses along the head dimension to obtain token-level importance at the cost of only a few proxy heads, which is conceptually clean and practically effective.  \n2. No retraining or model changes; works as a drop-in prefill accelerator and ports across model families.\n3. Strong speedups while maintaining full-attention accuracy on long context benchmarks.\n4.  The approach is validated by the observation that heads share token focus on long context, justifying proxy-head sharing.", "weaknesses": "I don't see any substantive weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BoGSXCDnR6", "reviewer": "Reviewer_C449", "summary": "This paper proposes ProxyAttn, an attention approximation method designed to accelerate the pre-filling phase.\n\nThe method is based on the key observation that attention patterns across different heads within the same layer are highly similar, differing primarily in their sparsity. Therefore, ProxyAttn employs a pooling head to generate shared attention scores for top-K block retrieval.\n\nFurthermore, the authors propose using the query from each head's final block to estimate its specific sparsity budget, which enables a more effective resource allocation.", "questions": "1. Regarding line 142: \"the score at position $(i,j)$ represents the cumulative attention score at head $j$ obtained by the top 1024 tokens from head $i$.\"This statement is unclear to me. Could you please provide the formula to explain this metric?\n\n2. What is the TTFT of your method on the Qwen2.5-1.5B, 3B, and 7B models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1.  The paper's motivation is well-articulated. Through multi-faceted comparisons and visualizations, it substantiates the hypothesis that differences between attention heads lie not in their *patterns*, but rather in their *magnitude* and *sparsity*. This insight validates the use of a pooling head for important block selection as a reasonable approach.\n\n2.  The experiments are thorough, conducted across the Ruler, LongBench-v2, and InfiniteBench benchmarks. The evaluation covers two mainstream models, Qwen and LLaMA, lending strong reliability to the results.", "weaknesses": "1.  The paper critiques prior methods for coarse-grained block retrieval. However, its own method still employs a stride, resulting in block-wise retrieval and top-k computations. This merely mitigates the coarse-granularity issue rather than fundamentally solving it—a true solution is likely infeasible due to prohibitive retrieval overhead.\n\n2.  The method offers limited speedup, achieving only ~2x on a 70B model. The baseline selection seems insufficient, as current mainstream models often use 7x or 8x GQA ratios. In such high-ratio scenarios, FlashAttention's computational speed increases significantly, representing a more practical setting (however, not optimal for your end-to-end TTFT comparison). Moreover, with the rise of local + global attention models (e.g., Minimax, Llama 4), the significance of pre-filling acceleration is diminishing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MzwjA13ehN", "reviewer": "Reviewer_kQrf", "summary": "The paper proposes ProxyAttn, a method for improving block selection in block-sparse attention. Instead of reducing along the sequence dimension to estimate high-attention regions, ProxyAttn computes a subset of attention heads and uses them as proxies to estimate which blocks to compute for the remaining heads. Experiments shows improved performance and efficiency when using the proposed sparse attention method.", "questions": "1. In the attention head consistency oracle experiment (Figure 2a), how much of the overlap is attributable to attention sinks? What happens if the influence of attention sinks is removed?\n2. What does the term “average score” in Line 149 refer to? Is it the mean *attention score* across heads or something else?\n3. Does the proxy head mechanism perform consistently across both retrieval and question-answering scenarios?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an interesting insight into the similarity of attention trends across different heads.\n2. The design choices of the proxy attention mask and dynamic budget allocation are well-motivated by corresponding empirical observations and oracle analyses, enhancing the interpretability of the proposed method.\n3. The proposed approach outperforms other sparse attention baselines on retrieval tasks (RULER benchmark), matching or even surpassing the dense attention baseline.", "weaknesses": "1. **Inconsistent performance on long-context understanding tasks:** Although the proposed method performs well on retrieval tasks, its advantage is less obvious on more realistic long-context benchmarks. This raises questions about whether the observed attention pattern similarity generalizes beyond retrieval settings, since real-world tasks often involve richer semantics and more diverse attention behaviors.\n\n2. **Definition clarity:** Some core concepts, such as *attention score*, are ambiguously defined. For instance, does it refer to the $(i, j)$ entry of the attention matrix, or the aggregated attention value across columns?\n\n3. **Figure clarity:** Several figures are not clearly explained. For example, the meaning of the x-axis in Figure 2(c) is unclear. How does it support the claim that the primary variation between heads lies not in the tokens they attend to?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)