XAttention: Block Sparse Attention with Antidiagonal Scoring
Authors: Ruyi Xu
Guangxuan Xiao
Haofeng Huang
Junxian Guo
Song Han
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Efficiency
Primary area: deep_learning->large_language_models
TL;DR: We introduce XAttention, a plug-and-play method that uses antidiagonal sums to efficiently identify important parts of the attention matrix, achieving up to 13.5x speedup on long-context tasks with comparable accuracy to full attention.

Abstract: Long-Context Transformer Models (LCTMs) are vital for real-world applications but suffer high computational costs due to attention's quadratic complexity. Block-sparse attention mitigates this by focusing computation on critical regions, yet existing methods struggle with balancing accuracy and efficiency due to costly block importance measurements.
In this paper, we introduce XAttention, a plug-and-play framework that dramatically accelerates long-context inference in Transformers models using sparse attention. XAttention's key innovation is the insight that the sum of antidiagonal values (i.e., from the lower-left to upper-right) in the attention matrix provides a powerful proxy for block importance. This allows for precise identification and pruning of non-essential blocks, resulting in high sparsity and dramatically accelerated inference.
Across comprehensive evaluations on demanding long-context benchmarks—including RULER and LongBench for language, VideoMME for video understanding, and VBench for video generation—XAttention achieves accuracy comparable to full attention while delivering substantial computational gains. We demonstrate up to 13.5x acceleration in attention computation. These results underscore XAttention's ability to unlock the practical potential of block sparse attention, paving the way for scalable and efficient deployment of LCTMs in real-world applications.

XAttention: Block Sparse Attention with Antidiagonal Scoring
Ruyi Xu 11 Guangxuan Xiao *2 Haofeng Huang' Junxian Guo Song Han 24
https://github.
xatttentinnnno
Abstract
multimodal capabilities. Long-Content Transformer Models (LCTMs) are emerging as essential tools in this evolution, particularly for tasks like video understanding (Lin et al., 2023; Wang et al., 2024) and video generation (Kong et al., 2025) that demand processing and generating exceptionally long sequences of information. These models hold the key to unlocking brilliant systems capable of interacting with the world in a human-like way, understanding and generat- ing not just text, but also visual information over extended periods. Imagine Al agents engaging in seamless, multi- modal, day-long interactions, or powerful world simulators generating hours of coherent video tasks that hinge on processing a tremendous number of tokens.
Cong-Context Transformer Models (LCTMs) are vital for real-world applications but suffer high computational costs due to attention's 's quadratic complexity. Block-sparse attention mitigates this by focusing computation on critical regions, yet existing methods struggle with balancing accu- racy and efficiency due to costly block impor- tance measurements. In this paper, we intro- duce XAttention, a plug-and-play framework that dramatically accelerates long-context inference in Transformers models using sparse attention. XAttention's key innovation is the insight that the sum of antidiagonal values (i.e., from the lower-left to upper-right) in the attention ma- trix provides powerful proxy for block impor- tance. This allows for precise identification and pruming of non-essential blocks, resulting in high sparsity and dramatically accelerated inference. Across comprehensive evaluations on demand- ing long-context benchmarks-iincluiin RULER and LongBench for language, VideoMME for video understanding, and VBench for video gener- ation XAttention achieves accuracy compara- ble to full attention while delivering substantial computational gains. We demonstrate up to 13.5 acceleration in attention computation. These re- sults underscore XAttention's 's ability unlock the practical potential of block sparse attention, paving the way for scalable and efficient deploy- ment of LCTMs in real-world applications.
However, realizing this vision requires overcoming signif- icant challenge: the computational burden of the attention mechanism (Vaswani et al., 2017). While crucial for captur- ing relationships within sequences, attention's 's cost scales quadratically with sequence length. This quadratic scaling creates substantial bottleneck during the pre-filling stage, hindering the practical deployment of LCTMs for complex. real-world applications.
In the pursuit of more efficient Transformers, block-sparse attention (Zaheer et al., 2020; Guo et al., 2024) has emerged as a promising avenue. The core idea is appealing: instead of computing attention between token pairs, focus resources on the most crucial regions of the attention map, creating "blocks" of relevant information. This selective computation promises to drastically reduce computational burden while preserving the model's ability to capture essential long- range dependencies.
Yet, existing block-sparse methods have struggled to deliver on their full potential, often grappling with a trade-off be- tween accuracy and efficiency. This stems from the lack of lightweight yet effective mechanisms for identifying and prioritizing truly important attention blocks. The overhead involved in determining block importance can negate the gains achieved through sparsity, rendering these methods impractical for real-world deployment.
1. Introduction
The transformative impact of Large Language Models (LLMs) (Dubey al., 2024; OpenAI, 2023) is expanding beyond natural language processing, steering in new era of
"Equal contribution 'Tsinghua University Massachusetts Insti- tute of Technology *SJTU "NVIDIA. Correspondence to: Guangx- uan Xiao <xgx @ mid.. Song Han <songhan @ mit.edu>.
This leads us to a question: Can we design a block-sparse attention mechanism that dramatically accelerates long- context Transformers without compromising accuracy, truly unlocking their potential for real-world applications?
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
(8)
Block Selection
Antidiagonal Scoring
Block Sparse Attention
Vertical-Slash Pattern
Zoom-in (a)
Zoom-in (b)
Figure 1. Illustration of XAttention: XAttention optimizes at- tention through a three-step process: (Left) Strided Antidiagonal Scoring: Each block (8x8 in this example) is scored by summing values along its strided antidiagonals (stride =4) 4), with red lines generally indicating higher summed values and blue lines lower values. (Middle) Block Selection: High-scoring blocks are se- lected hased these evaluations. (Right) Block Sparse Attention: Attention is computed only on the selected blocks (red blocks the right), achieving substantial computational savings. This example uses sequence length of 24.
Figure 2. XAttention's 's antidiagonal pattern intersects both vertical and slash patterns within block, enabling efficient detection of these patterns and guiding effective sparse attention computation.
2.1. Importance Prediction
The inherent sparsity of attention maps necessitates a robust strategy for predicting the importance of attention blocks. While methods like MInference (Jiang et al., 2024) and FlexPrefill (Lai et al., 2025) utilize combination of pooling and "vertical slash detection," our ablation study reveals that relying solely on average sum pooling yields inaccurate predictions. Pooling methods are particularly ineffective when only a few significant vertical or slash patterns exist within block, failing to capture these crucial indicators of importance.
We answer this question by introducing XAttention, novel and-lay framework designed to significantly improve the efficiency of block-sparse attention in long-context Transformers. XAttention is based on the key insight that the sum of antidiagonal values within the attention matrix can serve as a powerful, yet computationally efficient, indicator of block importance. Unlike existing methods that primar- ily rely on computationally intensive and lossy solutions like token pooling identify important blocks, XAtten- tion leverages this simple score to offer a potentially more streamlined and direct approach for rapidly and accurately identifying critical attention blocks.
MInference and FlexPrefill attempt to overcome this limi- tation by analyzing the last segment of the input query to identify important "vertical and slash indices." However, this approach faces two key challenges: firstly, important attention patterns may not persist in the final query segment; secondly, the search algorithm itself introduces substantial computational overhead (demonstrated in Figure 6).
This antidiagonal scoring algorithm allows XAttention to aggressively find and prune non-essential computations, achieving substantial sparsity without sacrificing accuracy. We extensively evaluate XAttention challenging long- context benchmarks, including RULER and LongBench for natural language processing, VideoMME for video under- standing, and VBench for video generation. Across these benchmarks, XAttention achieves accuracy comparable to full attention while delivering substantial computational gains, demonstrating up to 13.5x acceleration in atten- tion computation during pre-fillin prelfinging These results under- score XAttention's 's ability to unlock the practical potential of block-sparse attention, paving the way for scalable and efficient deployment of long-context Transformers in de- manding applications, especially in the expanding field of multimodal AI.
Fundamentally, an effective block importance prediction method should automatically and robustly identify signifi- cant patterns, including crucial vertical and slash patterns. To achieve this, we propose the antidiagonal selection method. Within each block of size B, we select elements along the antidiagonal using a stride S (visualized in Fig- ure 1). The sum of these selected elements serves as a proxy for the overall importance of the corresponding attention block.
The effectiveness of this method can be understood from two perspectives: (1) Information Preservation: This se- lection strategy ensures that information from all tokens is considered, as each token contributes to at least one antidiag- onal sum. (2) Pattern Detection: As illustrated in Figure the antidiagonal intersects every possible vertical and slash pattern within a block. XAttention's 's antidiagonal pattern intersects both vertical and slash patterns within a block, enabling efficient detection of these patterns and guiding effective sparse attention computation. This ensures that no crucial patterns are missed during the importance estimation process.
2. Method
In this section, we introduce our method, XAttention. The XAttention algorithm comprises three primary components: (1) importance prediction of attention map blocks, (2) selec- tion of important attention blocks, and (3) prediction of the minimum threshold for attention heads.
2

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
2.2. Threshold Block selection
2.3. Minimum Threshold Prediction
Based on the antidiagonal scoring pattern, we propose the following sparse attention block selection algorithm. Let S denote the stride, and let B be the size of the sparse attention blocks. The process begins with antidiagonal sum- mation, where we select elements along the antidiagonal within each S' x S block of the attention map and compute the sum of these elements for each antidiagonal. Subse- quently, we perform softmax normalization by applying the softmax function to these antidiagonal sums, yielding probability distribution over the antidiagonals. Finally, for block selection, the find blocks function is employed to identify the minimal set of blocks whose cumulative sum of antidiagonal probabilities exceeds predefined threshold T, Formally, this can expressed as:
We propose a dynamic programming approach to determine the optimal threshold for each attention head. Previous research indicates that different attention heads exhibit vary- ing sparsity levels and importance. Thus, it is beneficial to dynamically adjust thresholds for individual heads to optimize the balance between accuracy and computational efficiency.
Problem Formulation: Consider a model with H attention heads. We define dynamic programming table D [1k]] [m]. where h E {,,2, ..., Iif represents the /-th head, and m E {1, ments made. [[]][m]] stores the best performance achiev- able when exactly m threshold adjustments have been made across the first heads.
. .MM denotes the number of threshold adjust-
Dynamic Programming: Our objective is to find the opti- mal threshold for each head such that their joint contribution maximizes accuracy while minimizing computation. The recurrence relation for the DP table is:
find.blocks(4,r) arg min B
B
Σ BEB (t.j)
Σ
2
where A is the attention map, B is set of blocks, and [B] represents the number of blocks in the set. This process effectively determines the most important blocks in the at- tention map based on the antidiagonal scoring pattern and the specified threshold.
D[h][m] (ax 1][m], P(h, m))
where P(h, m) represents the performance of the model when the h-th head's threshold adjusted for the m-th time. This corresponds the model's performance after reducing the threshold of the h-th head by one step relative to the state Díh 1][m 1] in the optimization process.
Algorithm| Block Selection
We adjust the threshold for each head by reducing it by 10% at each step:
Require: Query matrix Q RLxdd Key matrix K E Rixdd block size B, stride S, head dimension dn. t
4. (m) = Ln (m - 1) x 0.9
Ensure: Sparse mask M
This ensures gradual reduction in computation while pre- serving each head's contribution to accuracy.
1: NB < [./B] (Number of blocks)
2: for = 0 to NB - 1 do
3: Qaiice < QbbB (b + :| (Extract Q block)
Note that this dynamic threshold prediction method can further optimize XAttention's sparsity but is not mandatory component. We present detailed results in the ablation study.
4: Qreshaped +
5: for i 5 1 down to 0 do
6: Qreshnpet -appnndddpppped(((( Sicefe (Qiice
5,:]) (Reshape along
antidiagonals with stride S}
3. Experiments
7: end for
8: Kreshuped +
This section presents our empirical investigation into the effecttiveness of XAttention. We first detail the implemen- tation specifics, followed by evaluation results on text and video understanding, well as video generation bench- marks, against strong baselines. We then test the accelera- tion performance of XAttention. Finally, we provide analyt- ical ablation studies to further understand the behavior of XAttention.
9: for 5
boeshaped-appedde Kii S,:)) (Reshape along an-
tidiagonals with stride S)
11: end for
K Ven.
12: Aapprox Softmax
(Approximate
attention scores}
13:
M₁ + find_blocks( blocks(Aapproo,, T) {Find blocks based
on threshold}
3.1. Experimental Setup
14: end for
We evaluate XAttention across three distinct do- mains. For natural language tasks, we employ Llama-3.1 8B-Instruct (Dubey et al., 2024). In the video understanding
Models
15: M
concatenate(MMp My,..., M MNB-1)
(Concatenate block masks)
3

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
augmented text prompts from VBench (Huang et al., 2024) to generate videos. We then compare the videos gener- ated by our proposed method, XAttention, against those produced by full attention baseline, evaluating the effec- tiveness of our approach in generating high-quality video content.
Table 1. Accuracy comparison of different methods and sequence lengths on RULER with Llama- 3. 1..1-8--nnstrre Instruct. XAttention is configured with Stride S = 8 and S 16 with Precisely Predicted Minimum Threshold.
Input Len
4k
8k
16k
32k
64k
128k
Avg.
Full
96.74 94.03 92.02 84.17 81.32 76.89 87.52
FlexPrefill
95.99 93.67 92.73 88.14 14 81.14 74.67 87.72
3.2. Accuracy Results
MInference 96.54 94.06 91.37 85.79 83.03 54.12 84.15
SecrAttn
95.32 92. 92.20 88.05 83.30 72.37 87.23
RULER On the RULER benchmark (Hsieh et al., 2024), we apply the dynamic programming method described in Section 3.3 for Minimum Threshold Prediction, utilizing strides of S 8 and S 16 with maximum adjustment number of M 1000. This yielded a set of minimum thresholds with an average of 0.8, further enhancing the computational efficiency of our sparse attention mechanism.
Xattn S=8 96.83 94.07 93.17 90.75 84.08 72.31 88.47
Xattn S=16 96.11 93.95 93.56 90.64
83.12
71.11 88.08
domain, we utilize 1o-n2--V--B-IIntruue (Wang et al., 2024). Finally, for video generation, we use the Hunyuan- Video model (Kong et al., 2025). To optimize the trade-off between computational efficiency and accuracy on natural language tasks, we apply our precise threshold prediction method to the 1-tiaa...----IIttttttt model.
Table 1 compares the accuracy of X XAttention against strong baselines on the Llama-3, 1.ama-3.1-BB-mma -88-ttutttttt model across vari- ous sequence lengths on RULER. Notably, both MInference and SecrAttention experience significant performance degra- dation as context length increases. In contrast, XAttention, configured with S = 8 and S = 16 and employing our precisely predicted minimum thresholds, not only surpasses the optimal sparse attention baseline, FlexPrefill, but also outperforms full attention at several sequence lengths. Ad- ditionally, we evaluate the same tasks on the above three models. The results shown in appendix are consistent across models. This demonstrates the robustness of XAttention in handling very long contexts.
Baselines We compare XAttention against several strong baselines. Our primary baseline for dense attention is FlashAttention (Dao, 2023), implemented within the Flash- Infer (Ye et al., 2024) framework. We also compare against MInference (Jiang et al., 2024), FlexPrefill (Lai et al., 2025), and Attennttn (Gao et al., 2024), strictly adhering to their public implementations. For SeerAttention, we incor- porate pretraining on the Gare weights. For MInference, we utilize their official configuration, where all attention heads adopt the "Vertical-Slare -lassh" sparsity pattern. For FlexPrefill, we set the hyperparameters 7 0.95 and 7 = 0.1, which, according to the original paper, resulted in the highest accu- racy among the provided parameter sets.
Table 2 presents the performance of XAtten- tion compared to strong baselines on the real-world tasks within the LongBench benchmark, using the Llama-3.1 1-8B- Instruct model. Maintaining the same configuration used for the RULER evaluation, we evaluate XAttention along- side MInference and FlexPrefill. XAttention achieves the highest average score across all tasks, demonstrating its ef- fectiveness in practical scenarios. Notably, the performance of X XAttention on individual tasks remains close to that of full attention, indicating that our method preserves accuracy while improving efficiency.
LongBench
Datasets We evaluate our model on diverse set of tasks spanning natural language understanding, video understand- ing, and video generation. For natural language tasks, we employ the RULER (Hsich et al., 2024) dataset, a synthetic benchmark specifically designed to assess long-context abil- ities in LLMs. RULER allows for customizable sequence lengths and task complexities, extending the traditional aandle-in-a-haystare test while introducing novel task cat- egories like multi-hop tracing and aggregation. We also evaluate on real-world long-contex: tasks from LongBench (Bai et al., 2023) to test performance in practical scenarios.
Video Understanding We apply Stride S = 16 and threshold 7 0.9 parameters on the Qwen VL-22-B model. As shown in Table 3, among the three sparse attention meth- ods, MInference and FlexPrefill fail to achieve optimal per- formance on Long video tasks. XAttention achieves the best average score among all sparse attention methods and even outperforms FlashAttention on long videos, with a frame rate of 1 frame per second for up to I hour.
For video understanding, we utilize the Video-MME (Fu et al., 2024) dataset, the first comprehensive benchmark for evaluating multimodal large language models (MLLMs) video analysis. Video-MME comprises 900 videos totaling 254 hours, with durations ranging from 11 seconds to 1 hour, providing a robust testbed for assessing long video comprehension.
Video Generation We evaluate XAttention's 's performance in the video generation domain using the Hunyuan Video model on prompts from VBench (Huang et al., 2024). The
In the video generation domain, we leverage 946 GPT-
4

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
Table Comparison of different attention methods real-world LongRench tasks using the Llama-3.1-80 1-8B-Instruct model. XAttention,
contigured with stride 8 and Precisely Predicted Minimum Threshold, achieves the best average scores against all baselines.
Single-Dos Dale--Deoor QA
Multi-Doc QA
Summarization
Few-shot Learning
Code
NiiQA
Qasper
MF-en
HPQA
2WikiMQA
MuSiQue
GovReport
QMSum
VCSum
MultiNews
TREC
TrivinQA
SAMSum
LSHT
LCC
RB-P
Method
Avg.
Full
31.44 25.07 29.40
16.89
17.00
11.79
34.22
23.25
15.91
26.69
72.50
91.65
43.74
46.00 52.19 49.14 40.34
MInference 31.59 24.82 29.53
17.03
16.46
11,58
34.19
23.06
16.08
26.71
72.50
91.18
43.55
46.00
52.33
49.93
40.30
FlexPrefill 27.30 28.56 27.66
17.20
15.14
9.46
32.76
23.66
16.05
27.25
64.00
88.18
41.28
31.00
45.69
47.54 36.83
XAttention
28.99 26.14
29.92
17.40
16.70
11.80
34.41
23.26
16.00
27.04
72.00
91.65
43.86
47.00
52.67
50.84
40.60
Full Attention (buseline)
XAttention (twarmup steps O. = 0.95)
XAtiention (Awarmup steps 5, = 0.9)
XAttention (Awarmup steps 5, -0.95)
Prompt: "A joyful, fuzzy ponda sits cross-legged by a crackling compfire, strumming small acoustic guitar enthusiasm. panda's 's black and white fur controsts beautifully with the warm glow of the fire, casting flickering shadows on the surroundngg snow-covered ground. Behind the panda, majestic snow-capped mountains rise against twilight sky, their peaks tinged with the last light of the setting sun. The panda's eyes sparkle with delight it plays cheerful tune, the serene mountain landscape and the cory campfire creating magical, heartwarming scene."
Figure 3. Qualitative comparison of video generation results on the VBench benchmark using the first prompt in the VBench dataset. Rows show frames from videos generated using: (1) Full Attention (baseline), (2) XAttention with no warmup and (r 0.95). (3) XAttention with 5 warmup steps and (r = 0.9), and (4) XAttention with 5 warmup steps and (r = 0.95). XAttention with warmup achieves high visual fidelity to the full attention haseline.
Hunyuan Video Video model utilizes the Diffusion Transformer (DiT) architecture (Peebles & Xie, 2023), which employs non-causal attention. As existing baselines are not imple- mented for non-causal attention, we compare XAttention solely against the full attention baseline. Our evaluation considers both quantitative metrics (PSNR, SSIM, LPIPS) and qualitative visual comparisons. We replace all attention computations in the DiT backbone with XAttention, and measure performance against the full attention output using the same random seed and prompt, averaging the results across all 946 VBench prompts. The generated videos have a resolution of 720x 1280 pixels and 129 frames, with 50 denoising steps. We configure XAttention with a stride of and thresholds of T 0.9 and = 0.95.
slight layout shifts in the output video compared to the full attention baseline, resulting in lower quantitative scores. In- spired by research on diffusion models (Xiao et al., 2023c; Li et al., 2024) demonstrating that early denoising steps are critical for determining content layout, we introduce a "warmup" phase. During this phase, we utilize full at- tention for the first denoising steps, before switching to XAttention. Figure illustrates the qualitative impact of this warmup strategy.
Table 4 presents the quantitative results of applying XAtten- tion to the Hunyuan Video model. Both configurations, with thresholds of T = 0.90 and T = 0.95, achieve high fidelity compared to videos generated with full attention. Specif- ically. we observe a PSNR up to 23.5, SSIM up 0.822, and LPIPS down 0.155, indicating level of similarity that difficult for the human eye to discem. As expected,
Initially, applying XAttention from the very beginning of the denoising process in the HunyuanVide Video model led to
5

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
Table Comparison of different methods on QwenVT.-2-7 in the Video-MME video understanding task. XAtttntiin is configured with Stride 16 and Threshold r 0.9. XAttention outper- forms Full Attention long video tasks and achieves the best average performance among all sparse attention methods.
Table 5. Density on Different Context Lengths. Stride S achieves lower sparsity, and as context length increases, sparsity generally increases (lower density).
8
SeqLen
Stride 4
Stride 8
Stride 16
Short (%) Medium (%)
Long (%) Overall (%)
4k
51.73%
52.16%
55.38%
40.96%
43.77%
43.55%
subs
w/o
w/
w/o
w/
w/o
w/
w/o
w/
16k
27.43%
27.49%
28.91%
Full
72.1 78.1 63.9
69.4
55.1
60.2 63.7
69.2
32k
21.09%
20.97%
27.93%
64k
9.43%
10.98%
11.32%
MInference 71.7 77.6 62.3
67.9
55.2
59.8
63.1
68.4
128k
6.20%
6.89%
7.32%
FlexPrefill 71.4 77.4 62.6
68.3
53.8
57.3
62.6
67.7
XAttention 71.9 78.8 62.6
68.5
55.7
60.3
63.3
69.1
ing from 8k to 256k. We conduct these experiments with strides of S = 16 and S = 8, and a threshold of r = 0.9. On shorter contexts, where attention density tends to be higher, both MInference and FlexPrefill experience increased over- head due to more extensive pattern selection. In contrast, XAttention maintains its speedup advantage. Notably, for context length of 256k, X XAttention achieves a maximum pre- fill attention speedup of 13.5x and 9.8x with corresponding densities of 7.32% and 6.89%, respectively (see Table 5),
Table Quantitative results applying XAttention to the Hun- yuan Video model on the VBench benchmark, using 5-step full- attention warmup. Higher (r) yields better fidelity (higher PSNR, higher SSIM, lower LPIPS) at the cost of slightly reduced sparsity (higher density). Both (r) settings demonstrate high similarity to the full attention baseline.
XAttn r PSNR (1)
SSIM (1)
LPIPS (4)
Density (%, 4)
0.90
21.5
0.767
0.215
34.4
0.95
23.5
0.822
0.155
45.5
End End Speed Up We further evaluate the end-to-end prefill acceleration of XAttention on the RULER benchmark using Llama-3. 1-88--Bnssruue Figure 5 illustrates the trade- off results: The left plot shows how varying sparsity levels affect the proportion of accuracy retained, while the right plot visualizes the relationship between end-to-end speedup and model accuracy. XAttention consistently surpasses baseline methods across both stride = 8 and stride = 16 settings, demonstrating superior socciency-accuraan trade- offs.
trade-off exists: a higher threshold 7 yields better results but slightly lower sparsity. Nevertheless, both configurations achieve over 50% sparsity.
Figure 3 provides qualitative comparison of videos gen- erated by the baseline (full attention) and XAttention with different configurations using the first prompt in the VBench set. Without the full attention warmup, the generated video, while still high quality, exhibits minor layout differences compared to the baseline. However, with the 5-step full attention warmup, the video generated by XAttention be- comes remarkably similar to the one generated by full at- tention, preserving both high quality and intricate details. These results demonstrate XAttention's effectiveness in video generation models, promising and increasingly im- portant application area for LCTMs.
Table 6. XAttention End-to-End Speedup on Llama-3.1-88 Instruct on RULER.
RULER
8k
16k
32k
64k
128k
256k
XAttn S=8 2.59
3.04
3.96
4.38
4.67
4.93
XAttn S=16
2.89
3.60
4.40
4.82
4.94
5.12
3.3. Efficiency Results
We further analyze the efficiency of XAttention on tasks with varying context lengths, comparing it against FlashAt- tention, MInference, and FlexPrefill. We focus on the prefill stage and measure the attention speedup achieved by XAt- tention. We also break down the computation time into pattern selection and sparse attention components, contrast- ing it with other trainingless pattern selection methods.
Attention Time Breakdown Figure 6 demonstrates that XAttention's antidiagonal pattern, coupled with its effi- cient block selection algorithm, results in significantly faster pattern selection compared MInference and FlexPrefill, which rely vertical slash index search. Specifically, XAt- tention's 's pattern selection time is up to 24.9x and 5.9x faster, respectively. Furthermore, the accuracy of the antidiagonal pattern allows XAttention to achieve a lower attention den- sity, leading to substantial speedups in the sparse attention computation itself.
Attention Acceleration Figure 4 illustrates the prefill speedup of Attention across token sequence lengths rang-
6

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
15
13.5
Attention Speedup
MInference
ScerAttention
FlexPrefill
XAttn S-8
XAttn S-16
11.7
12
9.8
8,4
9
7.1
7.6
6.0
6
5.1
3,94.3
4.2
0.20.
1.71.5
0.40.61.0
2.22.5
0.80.8
2.2.3.2
1.11.0
2.52.4
3.1
0
8K
16K
32K
64K
128K
256K
Figure Speedup comparison of attention methods across context lengths, relative FlashInfer's implementation FlashAttention. XAttention consistently outperforms other sparse attention methods, achieving up to 13.5x speedup at 256K tokens.
Accuracy Pencestnge
100%
Xatin S-8 Xattn S-16 Flex Prefill Minference Seer Atto
Table 7. Comparison of different patterns. For the computa-
96.5%
Accunicy
87.5
tion, the antidiagonal achieves the lowest density and the highest
93%
89.5%
84.5
Stride
=
Stride $ = 16
10%
30%
60%
90%
2.0
4.3
Metric
32k
Avg. Density
32k
Avg. Density
Category Spursity
Fod-to-Frn Speed up
Random
82.53 82.48 27.57% 82.35 80.94 31.36%
Diagonal
76.47 81.06 24.47%
58.26 79.63 25.31%
Antidiagonal
90.75 88.47 20.97%
90.64 88.08 27.93%
Figure 5. Trade-off of XAttention LLaMA-3.1-8D evaluated on RULER benchmark.1 The left plot shows the curve between sparsity and the percentage of accuracy retained, while the right plot presents the distribution of end-to sodeto--d speedup versus model accuracy. XAttention consistently outperforms baselines at stride 8 and stride = 16 settings.
one token selection per row and column. Table shows that the antidiagonal pattern achieves the highest accuracy while maintaining the lowest density across tasks, confirming its superiority.
120
Attention
15.8
Pattern Search
Stride Sizes We explore the impact of different stride sizes, S. Larger strides lead sparser sampled attention maps and thus lower computational overhead. However, excessively large strides can compromise the accuracy of block selection. We compare strides of 4, 16, and 64 in Table 8. Our results indicate that when the stride is too long. fails to accurately detect the previously identified slash attention pattern. An overly sparse antidiagonal cannot effectively distinguish slash patterns entering blocks from different positions, leading performance degradation.
90
24.9x
Time (ms)
60
30
5.9x
20.8
0
Full
Minfer
Flex
XAttn S=8
XAttn S=16
Figure 6. Breakdown of prefill attention time. Xattention signif- icantly reduces pattern selection time while maintaining density, achieving substantial acceleration compared to existing methods.
Table 8. Comparison of different Strides. Excessively long strides fail to distinguish slash patterns with different lengths, leading to decreased accuracy.
3.4. Ablation Study
To further analyze the components of XAttention, we con- duct an ablation study, evaluating the effectiveness of the Antidiagonal Pattern, Threshold Block Selection, and Mini- mum Threshold Prediction.
Stride
S
S
== 16
S 61
Avg
88.89
88.47
88.08
81.21
Density
21.09%
20.97%
27.93%
39.88%
Antidiagonal Pattern We investigate the importance of the antidiagonal pattern by comparing it with random and diagonal patterns as guidance for predicting attention block sums. For the random pattern, we ensure that elements are selected within each S S block, maintaining at least
Top-K vs. Top-Ratio vs. Dynamic Sparsity We evaluate different block selection strategies: Top-K, Top-Ratio, and our Threshold Block Selection (Dynamic Sparsity). For a fair comparison, we set K = 8192 and Ratio = 27% for S and K 16384 and Ratio 31% for

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
16, targeting computational costs similar our Threshold Block Selection. Table 9 demonstrates that both Top-K and Top-Ratio struggle to handle diverse and dynamic input sequence lengths with comparable computation. In contrast, our threshold-based approach, which retains blocks with at least the threshold-level attention, achieves the optimal balance between computation and accuracy.
4.2. Sparse Attention
The attention mechanism at the heart of LLMs exhibits inherent sparsity, meaning many attention weights neg- ligible and can be pruned without significant performance degradation (Child et al., 2019a). This sparsity becomes more pronounced as context length increases, presenting opportunities for optimizing inference speed. However, the dynamic and input-dependent nature of this sparsity, which varies across different inputs, attention heads, and even lay- ers, poses a significant challenge for effective exploitation.
Table 9. Comparison of different selection algorithms.
Stride
S 4
S = 8
S = 16
Metric
Avg
Density
Avg
Density
Avg
Density
Methods like Sparse Transformer (Child et al., 2019b), LongFormer (Beltagy et al., 2020), BigBird (Zaheer et al., 2020) and Selective Attention (1 (Leviathan et al., 2024) re- duce complexity through local or block-based attention, but often require retraining, limiting practicality. H2O (Zhang et al., 2023) and TOVA (Oren et al., 2024) discard tokens based on query patterns. StreamingI.LM (Xiao et al., 2023b) retains initial and recent tokens for consistent latency and memory usage, enabling processing of sequences longer than the pretraining length. Quest (Tang et al., 2024) uses query-aware token criticality estimation to load only im- portant KV cache pages, accelerating long-context LLM decoding. Retrieval head-based methods (Wu al., 2024; Xiao et al., 2024b) accelerate model decoding by focusing compute crucial retrieval heads.
Top K
84.96
17.40%
84.13 19.92%
83.11 30.15%
Ratio
85.96
21.00%
85,42 21.00%
84.24 27.00%
Threshold
88,89
21.09%
88.47
20.97%
88.08 27.93%
Minimum Threshold Prediction Finally, we compare the performance of our Minimum Threshold Prediction method against a fixed threshold of r = 0.9 on the RULER benchmark (Hsieh et al., 2024). Using Minimum Thresh- old Prediction, we start with F 0.9 and set M 1000, allowing the dynamic programming (DP) algorithm ex- plore 1,000 optimal threshold combinations. This results in a set of more refined thresholds, with an average value of 0.8. Table 10 demonstrates that the dynamically predicted threshold achieves lower density and improved accuracy, showcasing the effectiveness of this method.
To accelerate the prefill stage, recent methods have em- ployed sparse attention patterns. Minference (Jiang et al., 2024) and FlexPrefill (Lai et al., 2025) both utilize pat- tern selection algorithms to achieve significant speedups during prefill. However, the overhead of these selection algorithms remains a bottleneck. SeerAttention (Gao et al., 2024) achieves high sparsity through pretraining and fine- tuning of gate parameters, improving efficiency while main- taining low perplexity. Yet, it requires a costly training pro- cess and exhibits limited performance on downstream tasks. Therefore, a training-free free approach with a minimal-overhead selection algorithm needed to address the increasingly long prefill times associated with growing context lengths,
Table 10. Minimum Threshold Prediction yields improvements in both accuracy and sparsity, translating to faster inference.
Stride
S ==: 4
S 8
S 16
Metric
Avg
Density
Avg
Density
Avg
Density
0.9
87.51
23.06%
84.96
26.13%
85.83
28.36%
Minimum
88.89
21.09%
88.47
20.97% %
88.08 27.93%
4.3. LLM Inference Acceleration
4. Related Work
Numerous techniques have been developed to accelerate LLM inference. System-level solutions focus on optimiz- ing the original attention computation to better leverage hardware features. Notable examples include FlashAtten- tion (Dao et al., 2022; Dao, 2023), which optimizes memory access patterns for faster attention computation, and RingAt- tention (Liu et al., 2023), which distributes the attention computation across multiple devices. Other sestem-level approaches include FlashDecoding (Hong et al., 2024) and PagedAttention (Kwon et al., 2023), which focus on op- timizing the computation process and KV cache manage- ment, respectively. APE (Yang et al., 2025) aligns parallel
4.1. Long-Context Large Language Models
Progress in engineering and algorithms has extended the context length capabilities of Large Language Models (LLMs). Two primary approaches are: (1) compiling large datasets of long texts for continuous pretraining or fine- tuning (Peng et al., 2023; Chen et al., 2023), and (2) lever- aging external memory or -augmeted techniques to enhance long-range context processing (Burtsev et al., 2021; Xiao et al., 2024a; Wu et al., 2024). These advance- ments enable LLMs to handle increasingly complex tasks requiring reasoning over extended sequences.
8

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
and sequential attention for faster contex--augmented gen- eration. Model compression techniques, such as quantiza- tion, are also widely employed to reduce model size and memory footprint, leading to faster inference. Examples include SmoothQuant (Xiao et al., 2023a), AWQ (Lin et al., 2024), and QServe (Lin* et al., 2024), which quantize model weights and/or activations to lower bit-widths, thereby re- ducing memory bandwidth requirements and accelerating computation.
the efficiency of machine learning, particularly in the do- main of natural language and video processing, potential societal consequences warrant consideration.
The increased efficiency afforded by XAttention could en- able the deployment of LCTMs in resource-constrained environments, broadening access to advanced AI technolo- gies. This could have positive implications for fields like education, healtheare, and accessibility, where LCTMs can be used for tasks such personalized tutoring, medical diagnosis support, and real-time language translation.
4.4. Recent Works
However, the increased accessibility and efficiency of LCTMs also raise potential concerns. These include the potential for misuse in generating misleading or harmful content, exacerbating existing biases in training data, and the potential impact on employment in certain sectors. Further- more, the ability to process and understand longer contexts could raise privacy concerns if not handled responsibly.
Recently, several outstanding works have focused on ad- vancing sparse attention. Sparse Video Gen (Xi al., 2025) accelcrates video generation models by leveraging spatial and temporal heads while preserving generation quality. NSA (Yuan et al., 2025) introduces a natively trainable sparse attention mechanism for efficient long- long-context mod- eling. MoBA (Lu et al., 2025) addresses the quadratie com- plexity of traditional attention mechanisms without relying on strongly biased structures such sink or window at- tention by adopting a Mixture of Experts approach. Fast Video Generation (Zhang et al., 2025) reduces computation demands through Sliding Tile Attention, which employs localized spatial-temporee windows instead of full attention computation. Our work aligns with these efforts to democ- ratize AI by reducing computational costs and enabling efficient deployment.
We believe that the benefits of more efficient LCTMs out- weigh the potential risks, particularly when developed and deployed responsibly. We encourage researchers and practi- tioners consider these ethical implications and develop safeguards against potential misuse. Further research into the societal impact of efficient LCTMs is crucial for ensur- ing their beneficial deployment. We hope that X XAttention contributes to a future where powerful Al technologies are both accessible and used responsibly for the betterment of society.
5. Conclusion
ACKNOWLEDGMENT
We present XAttention, a novel plug-and-play framework for accelerating long-context inference in Transformer mod- els. By leveraging the insight that antidiagonal sums in the attention matrix serve as a robust proxy for block im- portance, XAttention efficiently identifies and prunes non- essential blocks, achieving substantial computational sav- ings without sacrificing accuracy. Our evaluations on chal- lenging long-context benchmarks in natural language un- derstanding (RULER, LongBench), video understanding (VideoMME), and video generation (VBench) demonstrate that XAttention achieves up to 13.5x speedup in attention computation while maintaining performance comparable to full attention. These results highlight XAttention's 's ability to unlock the practical potential of block sparse attention, paving the way for efficient and scalable deployment of Long-Contex. Transformer Models in real-world applica- tions.
We thank MIT-IBM Watson AI Lab, MIT and Amazon Sci- ence Hub, MIT AI Hardware Program, National Science Foundation, Hyundai, and Samsung for supporting this re- search. We thank NVIDIA for donating the DGX server.
References
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang. J., Huang, Z., Du, Z., Liu, X., Zeng. A., Hou, L., Dong, Y., Tang, J., and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. /4508, 2023.
Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer, 2020. arXiv:2004.05150.
Burtsev, M. S., Kuratov, Y., Peganov, A., and Sapunov, G. V.
Memory transformer, 2021. URL attps::/arxixs org/ httg/ab/22006.110 11527.
Impact Statement
This paper introduces XAttention, a novel approach for ac- celerating inference in Long-Context Transformer Models (LCTMs). While the primary goal of this work is to advance
Chen, S., Wong, S., Chen, L.,, and Tian, Y. Extending context window of large language models via positional interpolation, 2023, arXiv: 2306.15595.

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
Child, R., Gray, S., Radford, A., and Sutskever, I. Generat- ing long sequences with sparse transformers. 2019a.
Gonguet, V., Do, V., Vogeti, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Tan, X. E., Xie, X., Jia, X., Wang. X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, 7. D., Yan, Z., Chen, Z., Papakipos, Z., Singh, A., Grattafiori, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A.. Vaughan, A., Baevski, A.. Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Franco, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Wang. C.,, Kim, C.,, Zhou, C., Hu, C.,, Chu, C.-.., Cai, C., Tindal, C.,, Feichtenhofer, C., Civin, D., Beaty, D., Kreymer, D., Li, D., Wyatt, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang. D., Le. D., Holland, D.,, Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F. Kreuk, F., Tian, F., Ozgenel, F., Caggioni. F., Guzmán, F. Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpem,, G., Thattai, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Shojanazeri, H., Zou, H., Wang. H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Damlaj, L. Molybog, I., Tufanov, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Asher, J., Gaya, J.-B., Marcus, J., Tang. J., Chan, J.. Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin. J., Yang. J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang. J., Wu, K., U, K. H., Saxena, K., Prasad, K., Khandelwal, K., Zand, K., Matosich, K., Veeraragha- van, K., Michelena, K., Li, K., Huang, K., Chawla, K., Lakhotia, K., Huang, K., Chen, L., Garg, L., A, Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, I.,, Wehrstedt, I.., Khabsa, M., Avalani, M., Bhatt, M., Tsimpoukelli, M.. Mankus, M., Hasson, Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Ke- neally, M., Seltzer, M. L.., Valko, M., Restrepo, M., Patel, M., Vyatskov, M.. Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Laptev, N. P., Dong, N., Zhang, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj. P., Liang, Q. Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li, R., Hogan, R., Battey, R., Wang, R., Maheswari, R., Howes, R., Rinott, R., Bondu,
Child, R., Gray, S., Radford, A., and Sutskever, I. Gen- erating long sequences with sparse transformers, 2019b. URL https:///aaaa
wttgg///////////////////// 10509.
Dao, T. FlashAttention-2: Faster attention with better paral- lelism and work partitioning, 2023.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAt- tention: Fast and memory-efficient exact attention with awarennssss, 2022. urXiv:2205.14135.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B.. Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Al- lonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., SGarcia-llano D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Zhang, F., Syn- naeve, G., Lee, G. Anderson, G. I.., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I. Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J. Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J.. Hong, J., Lee, J., Fu, J., Chi, J.. Huang, J., Liu, J.. Wang, J., Yu, J., Bitton, J.. Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K. Malik, K., Chiu, K., Bhalla, K., Rantala- L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin L., Madaan, L., Malo, L., Blecher, L.. Landzaat, L.,, de Oliveira, L.., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov. N., Bogoychev, N., Chatterji, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q. Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R. Raileanu, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabas- appa. S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang. S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Her- man, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T.. Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V.,
10

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyighe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Kohler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, S., Mangla, V., Albiero, V., Jonescu, V., Poe- naru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wang, X., Wu, X., Wang. X., Xia, X., Wu, X., Gao, X., Chen, Y., Hu, Y., Jia, Y., Qi. Y., Li, Y.. Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Hao, Y., Qian, Y., He, Y., Rait, Z., Vito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., and Zhao, Z. The Ilama 3 herd of models, 2024. URL https: / /arxiv..v.v.//////a org/abs/////eeooo..rg...///..arr
Kong, W. Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang. J., Wu, K., Lin, Q., Yuan, J., Long, Y., Wang, A., Wang, A., Li, C.,, Huang. D., Yang, F., Tan, H., Wang. H., Song, J.. Bai, J., Wu, J., Xue, J., Wang, J., Wang, K., Liu, M., Li, P., Li, S., Wang, W. Yu, W., Deng, X., Li, Y., Chen, Y., Cui, Y., Peng, Y., Yu, Z., He, Z., Xu, Z., Zhou, Z., Xu, Z., Tao, Y., Lu, Q.. Liu, S., Zhou, D., Wang, H.. Yang, Y.. Wang. D., Liu, Y., Jiang, J., and Zhong, C. Hunyuanvideo: A systematic framework for large video generative models, 2025, URL https: /
Kwon, W., Li, Z.,, Zhuang, S., Sheng. Y., Zheng, L., Yu, C. H., Gonzalez, J. E.,, Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention, 2023.
Lai, X., Lu, J., Luo, Y., Ma, Y., and Zhou, X. Flexpre- fill: A context-aware sparse attention mechanism for efficient long-sequence inference. In The Thirteenth International Conference on Learning Representations, https://openreviie /ttpn//eeeeeeeeeeeeeeeeeeeeeeeee net/ /ortm?
abs/ 2407 21783.
Fu, C.,, Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075. 2024.
2025, URL https:
Leviathan, Y., Kalman, M., and Matias, Y. Selective attention improves transformer, 2024. URL https:
/ //arxiv. wwwpxivvorg/aaeo. wttp://////a......./////// 022003230.......
Gao, Y., Zeng, Z., Du D., Cao, S., So, H. K.-H., Cao, T., Yang, F. and Yang, M. Seerattention: Learning intrinsic sparse attention in your llms. arXiv preprint arXiv:2410. 12.2410..37766 13276, 2024.
Li, M., Cai, T., Cao, J., Zhang, Q. Cai, H., Bai, J., Jia, Y., Liu, M.-Y., Li, K., and Han, S. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
Guo, J., Tang, H., Yang, S., Zhang, Z., Liu, Z., and Han, S. Block Sparse Attention. https: https:// /gtthtb.thhhggggg com/ mit httpa/abb//lloo---a---ee-----e-- 2024.
Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, I. Video-Ilava: Learning united visual representation by alignment before projection, 2023.
Hong, K., Dai, G., Xu, J., Mao, Li, X., Liu, J., Chen, K., Dong. Y., and Wang. Y. Flashdecoding++: Faster large language model inference on gpus, 2024.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-ane aware weight quantization for Ilm compres- sion and acceleration, 2024.
Hsieh, C.PP. Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. Ruler: What's the real context size of your long-context language models? arXiv preprint ar wxXv:2240.006644 2024.
Lin Y., Tang*, H., Yang*, S., Zhang, Z., Xiao, G., Gan, C., and Han, S. Qserve: W4a8kv4 quantization and system co-design for efficient Ilm serving. arXiv preprint arXiv:2405.04553 2024.
Huang, Z., He, Y., Yu, J., Zhang. F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., Wang, Y., Chen, X., Wang, L., Lin, D., Qiao, Y., and Liu, Z. VBench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context, 2023.
Lu, E., Jiang, Z., Liu, J., Du, Y., Jiang, T. Hong, C., Liu, S., He, W., Yuan, E., Wang, Y., Huang, Z., Yuan, H., Xu, S., Xu, X., Lai, G., Chen, Y., Zheng, H., Yan, J., Su, J., Wu, Y., Zhang, N. Y., Yang, Z., Zhou, X., Zhang, M., and Qiu, J. Moba: Mixture of block at- tention for long-context Ilms, 2025. URL https: //axxiv. wwwpdii..rrgbbb/////ee/
Jiang, H. Li, Y., Zhang, C., Wu, Q. Luo, X., Ahn, S., Han, Z., Abdi, A. H., D., Lin, C.-Y., Yang, Y., and Qiu, L. Minference 1.0: Accelerating pre-filling for long- context Ilms via dynamic sparse attention. arXiv preprint wwXiv:2407.02490 2024.

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
OpenAl. Gpt-4 technical report, 2023.
Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y., and Han, S. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. arXiv, 2024b.
Oren, M., Hassid, M., Yarden, N., Adi, Y., and Schwartz, R. Transformers are multi-state mns, 2024. URL https: //arxiv. org.orgg/aao
06104.
Yang, X., Chen, T., and Chen, B. Ape: Faster and longer context-augmented generation via adaptive parallel en- coding. In ICLR 2025, 2025.
Peebles, W. and Xie, S. Scalable diffusion models with trans- formers, 2023. URL https: pettps://arxe http:///wwwwww/o///ae////e org//bs/ abs/ 2212. 2212.09748.
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models, 2023.
Ye, Z., Lai, R., Lu, R., Lin, C.-Y., Zheng, S., Chen, L., Chen, T., and Ceze, L. Cascade in- ference: Memory bandwidth efficient shared pre- fix batch decoding. https://llahinnfer: 2024 / /01///1//0/ htt/////ccccaaeee http://aa-iiiffeeeneeeeeeeeeeeee
Tang, J., Zhao, Y., Zhu, K., Xiao. G., Kasikei, B., and Han, S. Quest: Query-aware sparsity for efficient long-context Ilm inference, 2024.
Jan 01/08/ wwwcaadeinneeeeeeeceee html. Accessed on 2024-02-01.
2024. URL
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, I.., Zhang, 7., Xie, Z., Wei, Y. X., Wang, L., Xiao, Z., Wang, Y., Ruan, C., Zhang, M., Liang, W., and Zeng, W. Native sparse attention: Hardware-aligned and natively trainable sparse attention, 2025. URL https: / wwwpxii..org/aa..... 2502.11083
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang. J., Ge, W., Fan, Y., Dang. K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint (tti::2409.119900 2024.
Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Al- berti, C., Ontañón, S., Pham, P., Ravula, A.. Wang, Q. Yang, L.,, and Ahmed, A. Big bird: Transformers for longer sequences. In Larochelle, H., Ranzato, M., Had- sell, R.,. Balcan, M.-F., and Lin, H.-T. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020. Curran Associates, Inc., 2020.
Wu, W. Wang, Y. Xiao, G., Peng, H., and Fu, Y. Re- trieval head mechanistically explains long-context factu- ality, 2024.
Xi, H., Yang, S., Zhao, Y., Xu, C., Mi, Lin, Y., Cai, H., Zhang. J., Li, D., Chen, J., Stoica, I., Keutzer, K., and Han, S. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity, 2025, URL https: htaps://arxivy: /arxiv.vvvv. org/ ht///bb//2555 2502 01776.
Zhang, P., Chen, Y., Su, R., Ding, H., Stoica, I, Liu, Z., and Zhang. H. Fast video generation with sliding tile attention, 2025. URL https: www.aiiii...oraaaeoeeoaeoeeeeeee 2502.04500
Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., and Sun, M. Infllm: Training-free long- context extrapolation for ilms with efficient context memory, 2024a. URL https:: /ttp///avvvv//////a//ee///.aa..a org/orgggoo.o... abs/ 2402. 04617.
Zhang. Z., Sheng, Y., Zhou, T., Chen, T., Zheng, I.., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H20: Heavy-hitter oracle for efficient generative inference of large language models, 2023.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and efficient quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023a.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv, 2023b.
Xiao, G., Yin, T., Freeman, W. T., Durand, F,, and Han, S. Fastcomposer: Tuning-free free multi-subject image gen- eration with localized attention, 2023c. URL ht ttsp //a Varxivv orgg abs aba//b///3555 2305. 10431.
12

---

XAttention: Block Sparse Attention with Antidiagonal Scoring
Table 11. Quantitative results of applying XAttention the Wam2.1 model on the VBench benchmark, using 5-step full-attention
warmup. Higher (7) yields better fidelity (higher PSNR, higher SSIM, lower LPIPS) at the cost of slightly reduced sparsity (higher density). Both (7) settings demonstrate high similarity the attention baseline.
XAtt r PSNR (t) SSIM (t) LPIPS (L) Density (%, +)
0.90
21.2 22.7
0,745
0.231
39.2
0.95
0.819
0.129
33.6
A. Video Generation Results on Wan 2.1
We also evaluated XAttention on the Wan2.1 model. The resulting accuracy and sparsity metrics are shown in Table 11.
B. Various Language Model results on Ruler
Table 12. Generalization of XAttention across various model architectures, including Mistral Nemo 12B, Phi-3.5 Mini 3.8B, and Qwen2.5
7B. The results confirm XAttention's sustained effectiveness.
Model
Method
Average (4k-128k) Delta
Full
67.97 /-
MInference
64.49/ -3.48
Mistral Nemo 12B
FlexPretill
64.61 / -3.36
XAttn S=4
67.92 / -0,05
XAttn S=16
67.47 / -0.50
Full MInference
84,68 /-
81.89 -2.79
Phi 3.5 Mini 3.8B
FlexPrefill
82.83/-11.8
XAttn S=4
84.86 / +0.18
XAttn S=16
83.82 -0.86
Full
77.84 /-
MInference
74.02 / -3.82
Qwen2.5 7B
FlexPrefill
75.10 / -2.74
XAttn S=4
77.75 / -0.09
XAttn S=16
77.21 / -0.63
13


Reviews:
[{"review_id": "YzpORjZOYp", "reviewer": "Reviewer_5Q7x", "summary": "This paper introduces XAttention, a novel block-sparse attention mechanism leveraging an \"antidiagonal scoring\" method to efficiently approximate standard transformer attention. XAttention aims to accelerate inference in Long-Context Transformer Models (LCTMs) by using an antidiagonal scoring strategy to identify and prune less important blocks within the attention matrix. The authors argue that summing antidiagonal values within blocks effectively captures critical attention regions, allowing for computational savings without substantial accuracy loss. Empirical evaluations on multiple benchmarks (RULER, LongBench, VideoMME, VBench) demonstrate that XAttention achieves competitive accuracy compared to full attention methods while substantially reducing computational overhead, showing speedups up to 13.5×.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Antidiagonal Selection: Performance and Insights\n\nAntidiagonal selection offers significant advantages over other patterns:\n\n- Retains **all token information** while simultaneously detecting both **vertical and slash patterns** critical in LLM prefill\n- Diagonal patterns **miss** slash patterns with probability $\\frac{\\text{stride} - 1}{\\text{stride}}$\n- Other patterns (horizontal lines, pooling methods) lose token-level information\n- Empirically validated superiority: outperforms diagonal patterns (Table 6) and pooling-based methods used by FlexPrefill\n\n---\n\n### 2. Novel Contribution\n\nWe respectfully disagree with the assessment of limited contributions. While block-sparse attention exists, **effectively implementing it without accuracy/efficiency degradation remains challenging**:\n\n- FlexPrefill and MInference: Suffer from **high precomputation costs** and rely on last block for pattern detection, preventing **chunk prefill** and adaptation to **non-text tasks** like video\n- SeerAttention: Requires **additional parameter training**, limiting generalizability. Also it has poor empirical performance.\n- Our work: First to identify antidiagonal pattern as an effective block importance proxy, creating a **simple yet powerful method** that significantly improves the efficiency-accuracy trade-off\n\n---\n\n### 3. FlashAttention Comparison\n\nAs shown in the Figure 4 of our submitted paper, XAttention achieves up to **13.5× speedup** over FlashInfer FlashAttention (one of the fastest implementations):\n\n| Prefill Time (ms)         | 8k   | 16k  | 32k  | 64k   | 128k   | 256k   |\n| ------------------------- | ---- | ---- | ---- | ----- | ------ | ------ |\n| FlashInfer                | 5.0  | 19.3 | 77.2 | 314.5 | 1269.5 | 5192.1 |\n| FlashAttention (Official) | 4.9  | 19.1 | 76.8 | 312.2 | 1265.8 | 5186.5 |\n| XAttn Stride = 4                | 3.3  | 8.2  | 24.4 | 63.2  | 181.4  | 543.6  |\n| XAttn Stride = 16               | 2.6  | 5.2  | 13.4 | 38.3  | 134.0  | 383.6  |\n\n\n### 4. References and Formatting\n\nThank you for noting these issues. We will address them in the revision.\n\n---\n\n### 5. Appendices\n\nThe main paper effectively covers our contributions. We will include an appendix showing additional visual samples, theoretical analysis, and experimental results in the next revision.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no theoretical claims in this work.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The key contributions of this paper show that the sum of antidiagonal selected elements serves as a proxy for the overall importance of the corresponding attention block, which is related to block-wise sparse attention.", "essential_references_not_discussed": "The following essential reference about block-wise attention is not discussed in this paper:\n\n[1] Blockwise Self-Attention for Long Document Understanding, ACL 2020", "other_strengths_and_weaknesses": "Strengths:\n\n1. The observation that antidiagonal values within the attention matrix can serve as a powerful indicator of block importance makes sense.\n\n2. Clearly motivated method with an efficient antidiagonal scoring strategy.\n\n3. Extensive and solid experiments across diverse benchmarks (text and video domains), providing good empirical support.\n\n4. Demonstrates substantial computational efficiency with competitive accuracy relative to dense attention methods.\n\nWeaknesses:\n\n1. While the paper presents some refinements in block sparse attention methods compared to existing approaches such as [1], the improvements appear to be incremental. It would be valuable for the authors to elaborate on how antidiagonal selections offer significant performance.\n\n2. While the use of the antidiagonal as an importance indicator for the block is an interesting idea, it appears to be the primary novel contribution of the work. It appears that the improvements made to the methods are somewhat limited.\n\n3. Lack of deeper analyses and insights to justify why antidiagonal scoring outperforms other potential scoring methods compared to the other pooling methods. It would be beneficial if the authors could provide additional theoretical insights to further substantiate and highlight the advantages of this approach compared to existing methods.\n\n4. Lack of a speed comparison between Xattention and FlashAttention.\n\n\n[1] Blockwise Self-Attention for Long Document Understanding, ACL 2020", "other_comments_or_suggestions": "The spacing between the captions for tables and figures and the surrounding text appears insufficient, which could affect both readability and the overall presentation. It is recommended that additional spacing be introduced to enhance the visual clarity of the document. For example:\n\n1. The caption for Figure 2 (lines 66–68)\n2. The caption for the table (lines 165–169)\n3. Table 3 (lines 279–280)\n4. The caption in Table 4 (lines 294–295)", "questions_for_authors": "Could you please explain why the appendices have not been included?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "378RUFMPom", "reviewer": "Reviewer_aVq1", "summary": "- This paper proposes an efficient attention model. \n- It finds out that the antidiagonal values in the attention matrix provides a powerful proxy for block importance\n- Unlike existing methods that primarily rely on computationally intensive and lossy solutions\nlike token pooling to identify important blocks, Xattention directly use attention scores which are more efficient.\n- The proposed module is tested on several benchmarks including language process, video understanding and video generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Algorithm Clarification\n\nXAttention precomputes attention within the **antidiagonal pattern** and uses these scores to guide block-sparse attention selection. Our method:\n\n- Requires no fine-tuning\n- Achieves **lowest overhead** among prefill acceleration approaches (Figure 5)\n- Delivers up to **13.5× speedup** over FlashAttention (Figure 4)\n\n---\n\n### 2. Complexity Analysis\n\n**Precomputation complexity:**\n$$\\frac{2n^2d}{\\text{stride}} \\quad (\\text{approximate attention}) \\ + \\ \\frac{3n^2}{\\text{stride}^2} \\quad (\\text{approximate softmax})$$\n\n**Full attention complexity:**\n $$4n^2d + 3n^2 + 4nd^2$$\n\nWith **stride = 4**, precomputation cost is < **1/8** of full compute\nWith **stride = 16**, precomputation cost is < **1/32** of full compute.\n\n**Figure 5** shows a breakdown of precompute time that closely aligns with this theoretical result. (While the actual latency closely aligns with theoretical predictions, it may be slightly higher due to factors such as vector reordering, memory overhead, and the implementation details of the Triton kernel.)\n\n---\n\n### 3. Threshold DP Design Choice\n\nAs described in Section 2.3, this is an **optional component** for optimizing sparsity by assigning different thresholds per head. XAttention also works effectively with global thresholds on LLMs, VLMs, and video generation.\n\n---\n\n### 4. Antidiagonal Pattern Effectiveness\n\nThe antidiagonal pattern:\n\n- Retains all token information\n- Simultaneously detects vertical and slash patterns with $1/\\text{stride}$ probability\n- Prevents information loss compared to horizontal lines or pooling methods\n- Empirically outperforms both diagonal patterns (Table 6) and pooling methods (e.g., FlexPrefill)\n\n---\n\n### 5. Two-Level Loop Structure\n\nIn Algorithm 1:\n\n- Outer loop: blocks (`blocknum`)\n- Inner loop: stride slices (`stride`)\n\nDespite appearing as nested loops, the **actual complexity is O(n)**, not O(n²), where n represents sequence length.\n\n---\n\n### 6. Softmax Approximation\n\nLine 120 explains why we normalize attention scores - to create a probability distribution for threshold selection. The Softmax complexity here is $\\left(\\frac{\\text{stride}}{\\text{blocksize}}\\right)^2$ of full attention (e.g., $\\frac{1}{1024}$ with stride = 4).\n\nTop-k could replace Softmax, but Table 8 demonstrates this performs worse than our threshold method.\n\n---\n\n### 7. Implementation Details\n\nAs noted in line 20 of the abstract, our method is:\n\n- **Plug-and-play**\n- Requires **no finetuning**\n- **Specifically designed for prefill stage**\n\nWe used both Block Sparse Attention (https://github.com/mit-han-lab/Block-Sparse-Attention) CUDA kernel and Triton kernel for implementation.\n\n\n---\n\n### 8. Figure Presentation\n\nThank you for these suggestions. We will improve figure clarity in the revision.", "claims_and_evidence": "- Not exactly, \n- Although extensive experiments are provided. The algorithm itself is confusing, the implementation details, calculation equations, theoretical computational complexity are not provided to help understand this method.\n- The motivation and design choice of threshold prediction and dynamic programming are confusing.\n- The method achieves both notable performance and efficiency improvements compared to previous methods, but the underlying reason remains unclear.", "methods_and_evaluation_criteria": "Yes, three different tasks (language process, video understanding and video generation) on several benchmarks are adopted for evaluation.\nEfficiency comparison of the proposed attention module is also provided.", "theoretical_claims": "Yes, the  Algorithm 1 is checked.\n- The Algorithm 1 involves two levels of loops to select blocks, what's its computational efficiency and why it is more efficient?\n- Why the approximate softmax attention is calculated inside the loops.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "- Some implementation details of experiments are not provided. For example, does XAttention needs finetuning or it can directly replace standard attention during inference? In Language tasks, does XAttention both works in prefill and decoding stages?\n- The figures shown in the paper are confusing, which seem not align with the main algorithm and demand extensive effort to understand.", "other_comments_or_suggestions": "I think the paper writing needs extensive improvement. Especially the algorithm, implementation details, design choice analysis and computational complexity comparison.", "questions_for_authors": "Does this method use any cuda or triton implementation? Ot it uses any existing efficient attention components or code?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gSZO2FtJ9k", "reviewer": "Reviewer_F2rz", "summary": "This paper introduces XAttention, a plug-and-play framework that accelerates long-context inference in Transformer models through block sparse attention. The key innovation is using the sum of antidiagonal values in the attention matrix as a proxy for block importance, allowing for identification and pruning of non-essential blocks. This approach achieves considerable speedups (up to 13.5x) while maintaining comparable accuracy to full attention across language, video understanding, and video generation tasks. The method doesn't require retraining and shows promising results compared to other block-sparse approaches like MInference, FlexPrefill, and SeerAttention.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Model Generalizability\n\nWe tested XAttention across diverse architectures with consistent results:\n\n**LLMs Accuracy (RULER)**\n\n| Model             | Method      | Average (4k-128k) | Performance Delta |\n| ----------------- | ----------- | ----------------- | ----------------- |\n| Mistral Nemo 12B  | Full        | 67.97             | –                 |\n|                   | MInference  | 64.49             | -3.48             |\n|                   | FlexPrefill | 64.61             | -3.36             |\n|                   | XAttn S=4   | **67.92**         | **-0.05**         |\n|                   | XAttn S=16  | 67.47             | -0.50             |\n| Phi 3.5 Mini 3.8B | Full        | 84.68             | –                 |\n|                   | MInference  | 81.89             | -2.79             |\n|                   | FlexPrefill | 82.83             | -1.85             |\n|                   | XAttn S=4   | **84.86**         | **+0.18**         |\n|                   | XAttn S=16  | 83.82             | -0.86             |\n| Qwen2.5 7B        | Full        | 77.84             | –                 |\n|                   | MInference  | 74.02             | -3.82             |\n|                   | FlexPrefill | 75.10             | -2.74             |\n|                   | XAttn S=4   | **77.75**         | **-0.09**         |\n|                   | XAttn S=16  | 77.21             | -0.63             |\n\n**LLMs Speed-up**\n\n| Model        | Method     | 8k       | 16k      | 32k      | 64k      | 128k      | 256k      |\n| ------------ | ---------- | -------- | -------- | -------- | -------- | --------- | --------- |\n| Mistral Nemo | XAttn S=16 | **1.7×** | **2.6×** | **4.7×** | **8.3×** | **9.9×**  | **10.9×** |\n| Phi 3.5      | XAttn S=16 | 1.4×     | **2.3×** | **4.7×** | **8.1×** | **11.0×** | **11.9×** |\n| Qwen2.5 7B   | XAttn S=16 | **1.8×** | **2.7×** | **4.9×** | **8.5×** | **12.4×** | **13.9×** |\n\n**Video Generation (Wan 2.1 14B)**\n\n| Threshold | PSNR (↑) | SSIM (↑) | LPIPS (↓) | Density (%) | Speed-up |\n| --------- | -------- | -------- | --------- | ----------- | -------- |\n| 0.90      | 21.20    | 0.739    | 0.212     | 35.8        | 2.5×     |\n| 0.95      | 22.67    | 0.794    | 0.167     | 51.2        | 1.8×     |\n\n---\n\n### 2. Hyperparameter Sensitivity  \n\nThreshold = 0.9 demonstrates strong **generalizability** across models:\n\n**Same threshold (0.9) on different models:**\n\n| Model         | Llama  | Mistral Nemo | Phi3.5-mini | Qwen2.5 |\n| ------------- | ------ | ------------ | ----------- | ------- |\n| Sparsity      | 23.06% | 24.93%       | 29.32%      | 21.15%  |\n| Performance Δ | -0.01  | -0.05        | +0.18       | -0.09   |\n\n**Trade-off curve with LLama 3.1 8B (stride=4):**\n\n| Threshold | 0.1   | 0.7   | 0.8    | 0.9    | 0.95   | 1.0     |\n| --------- | ----- | ----- | ------ | ------ | ------ | ------- |\n| Sparsity  | 4.31% | 5.62% | 10.35% | 23.06% | 49.88% | 100.00% |\n| Accuracy  | 41.34 | 73.96 | 84.39  | 87.51  | 87.64  | 87.52   |\n\n---\n\n### 3. Video Generation Quality\n\nThese quantitative metrics are well-acknowledged methods used in efficient visual content generation works to compare the difference between original generated content and the efficient generated content, such as used in Distrifusion (Li et al., https://arxiv.org/pdf/2402.19481) and Sparse VideoGen (Xi et al., https://arxiv.org/pdf/2502.01776). They are extremely strict compared to content level scores since they require pixel-level exactness, and the scores XAttention achieved is a level which very similar to the original generation. Beyond quantitative metrics, we've included qualitative samples in the Figure 3 comparing between XAttention-generated videos and full attention videos, confirming they are virtually indistinguishable.", "claims_and_evidence": "+The claims made in this submission are generally supported by convincing evidence. The main claim of achieving significant speedups while maintaining accuracy is well-supported by experimental results across multiple benchmarks. \n\n-However, the evidence is somewhat limited in model diversity, with primary evaluations on Llama-3.1-8B-Instruct for language tasks, Qwen2-VL-7B-Instruct for video understanding, and HunyuanVideo for video generation. The efficacy of the approach across model architectures beyond these specific models is not fully established, which weakens the generality claim somewhat. It would test more models, e.g., Mistral Nemo 12B Instruct, Phi 3.5 Mini 3.8B Instruct, Qwen2.5 7B Instruct, and so on.", "methods_and_evaluation_criteria": "+The paper uses established benchmarks (RULER, LongBench, VideoMME, and VBench) that directly test the challenges of long-context understanding, which align well with the goal of the work. The comparison against strong baselines (FlashAttention, MInference, FlexPrefill, SeerAttention) provides good context for understanding the gains.\n\n+The ablation studies are valuable in examining the contributions of different components (antidiagonal pattern, threshold block selection, minimum threshold prediction), though more exploration of hyperparameter sensitivity would strengthen the evaluation.", "theoretical_claims": "The paper does not have theory.", "experimental_designs_or_analyses": "+The experimental design is generally sound. The authors evaluate on diverse tasks (language, video understanding, video generation) with varying sequence lengths (4k to 256k tokens), providing a comprehensive picture of the method's capabilities.\n\n-One issue is the relatively simple setup for video generation experiments. While the comparison against full attention is done using the same random seed and prompts, only PSNR, SSIM, and LPIPS metrics are reported without detailed analysis of generation quality or more nuanced evaluation. More qualitative analysis would be beneficial.", "relation_to_broader_scientific_literature": "The authors connect their approach to previous work on sparse attention (Sparse Transformer, LongFormer, BigBird, etc.) and more recent work on attention optimizations like FlashAttention and inference acceleration methods. The paper clearly identifies its novelty compared to related approaches like MInference and FlexPrefill, highlighting that those methods incur significant computational overhead for pattern selection, which XAttention addresses through its antidiagonal scoring technique.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n- The antidiagonal scoring idea is novel and interesting - it provides a simple yet effective heuristic for identifying important attention blocks without requiring expensive computation. \n- The speedup achieved (up to 13.5x for 256k context) is impressive. Figure 4 clearly shows how XAttention consistently outperforms other sparse attention methods across different sequence lengths.\n- The method is training-free and can be applied as a drop-in replacement, making it immediately useful for practitioners without requiring costly retraining or fine-tuning. The dynamic threshold prediction approach shows thoughtful consideration of the varying sparsity patterns across different attention heads.\n\nWeaknesses:\n- Limited model diversity - the evaluation focuses primarily on Llama-3.1-8B-Instruct for language tasks, with limited exploration across model families or sizes. This raises questions about how well the approach generalizes across different model architectures and scales.\n- The video generation experiments, while novel, feel preliminary. More detailed analysis beyond basic metrics would strengthen these results.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "ctNe6vJvvS", "reviewer": "Reviewer_ZhZQ", "summary": "This paper introduces a plug-and-play attention sparsity method with minimal additional computational overhead. The proposed approach uses the sum of antidiagonal values in the attention matrix as a proxy to determine block importance, enabling block selection to reduce the computational density of attention mechanisms. Extensive experiments demonstrate that the method significantly enhances attention sparsity and accelerates attention computation, all while maintaining the model's performance.\n\n## update after rebuttal\nWhile the rebuttal has addressed my main concerns, I agree with reviewer 5Q7x that the technical mechanism appears incremental. So I keep my score (3).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### 1. Antidiagonal Selection: Performance and Insights\n\nAntidiagonal selection offers significant advantages over other patterns:\n\n- Retains **all token information** while simultaneously detecting both **vertical and slash patterns** critical in LLM prefill\n- Diagonal patterns **miss** slash patterns with probability $\\frac{\\text{stride} - 1}{\\text{stride}}$\n- Other patterns (horizontal lines, pooling methods) lose token-level information\n- Empirically validated superiority: outperforms diagonal patterns (Table 6) and pooling-based methods used by FlexPrefill\n\n### 2. Precomputation Process\n\nPrecomputation occurs before computing the full attention for $Q\\times K^T$. Based on block selection results, we determine which blocks require full attention computation. The process is detailed in Algorithm 1.\n\n### 3. Minimum Threshold Prediction\n\nMinimum Threshold Prediction is an **optional algorithm** which finds the optimal threshold for each head, further optimizing efficiency and accuracy. This process:\n\n- Is conducted **offline** (not during prefill)\n- Does not increase runtime computational complexity\n- Profiles configurations in advance, determining minimum thresholds per head\n\n### 4. Baseline Selection Justification\n\nOur selected baselines represent SOTA methods for sparse prefill attention. Previous approaches:\n\n- Have high precomputation complexity and overhead (Figure 5)\n- Limited applicability (FlexPrefill/MInference don't support video generation or chunk prefill)\n- Require additional fine-tuning (SeerAttention)\n\n\n### 5. Stride Selection Strategy\n\nStride is indeed critical for balancing efficiency and accuracy:\n\n**Different stride values (tested on RULER with Llama3.1 8B):**\n\n| Stride   | 64    | 32    | 16    | 8     | 4     | 2     | 1     |\n| -------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Overhead | 1.97% | 3.12% | 4.81% | 7.54% | 14.9% | 28.9% | 57.4% |\n| Accuracy | 81.21 | 84.23 | 88.04 | 88.42 | 88.38 | 88.39 | 88.27 |\n\nThese results show stride = 8 maintains accuracy comparable to stride = 4 with less overhead, and performance remains effective for stride ≤ 16.\n\nAccording to our complexity analysis:\n\n- Precomputation: $\\frac{2n^2d}{\\text{stride}} + \\frac{3n^2}{\\text{stride}^2}$\n- Full attention: $4n^2d + 4n^2 + 6nd^2$\n\nWith stride = 4, precomputation cost is < 1/8 of full compute\n With stride = 8, precomputation cost is < 1/16 of full compute\n\n(As shown in Figure 5, while the actual latency closely aligns with theoretical predictions, it may be slightly higher due to factors such as vector reordering, memory overhead, and the implementation details of the Triton kernel.)\n\n### 6. Predefined Sparsity Implementation\n\nFixed sparsity can be achieved by setting a block limit k and selecting the top-k highest-scoring blocks. This approach is equivalent to the Top-k strategy (Table 8).\n\nHowever, we believe fixed sparsity across all inputs is suboptimal since information density varies between requests. XAttention's **dynamic sparsity determination** enables better generalization and accuracy across diverse scenarios.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "There are several issues:\n1. The proposed method relies on the antidiagonal value-based block importance estimation, and the paper claims that the antidiagonal selection method is effective due to its advantages in Information Preservation and Pattern Detection. However, these advantages are not unique to antidiagonal values, as the main diagonal can also preserve information and detect patterns. Further theoretical or empirical evidence is required to clarify the rationale and necessity behind the antidiagonal selection strategy.\n2. The method relies on a precomputed attention matrix to predict block importance, but the paper does not clearly explain when and how this step is performed. \n3. How the Minimum Threshold Prediction works is not that clear. The paper should explicitly explain the definition of a time of threshold adjustment and when the threshold is adjusted, especially during the prefilling process.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "There are several issues:\n1. There is a lack of sufficient justifications for why the selected baselines are the most appropriate for evaluating the proposed method.\n2. The experiments only test a narrow range of stride values, and the paper does not provide a clear strategy for determining stride. This may limit the generalizability of the method, as stride selection could significantly impact performance.", "relation_to_broader_scientific_literature": "Different from existing works that rely on pooling methods, the paper proposed an antidiagonal selection strategy to achieve attention sparsity, which introduces less additional computational overhead.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper addresses an important problem and shows promise, but the motivation and justification for the proposed method are currently unclear.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. What are the unique motivations for using antidiagonal values to estimate block importance?\n2. If the sparsity level is predefined, how can the proposed method achieve a predefined sparsity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents XAttention, a plug-and-play method for accelerating long-context inference in Transformers via a novel antidiagonal block scoring strategy. The core idea is to use the sum of antidiagonal values in attention blocks as a proxy for block importance, enabling efficient block-sparse attention without sacrificing performance. The authors demonstrate strong empirical gains with competitive accuracy across a diverse set of benchmarks, including language modeling, video understanding, and video generation.

The reviewers generally applauded for the novelty of the antidiagonal scoring mechanism and its simplicity, and are satisfied with the strong empirical support with evaluation across multiple domains (RULER, LongBench, VideoMME, VBench) and architectures (LLaMA-3, Qwen2.5, Phi-3.5, Mistral Nemo), where the method consistently matches or exceeds baseline accuracy while delivering substantial speedups. While there were some initial minor concerns, most of those are cleared through the process of discussion and reviewers unanimously suggested acceptance. Hence I recommend acceptance.
Decision: Accept (poster)