DuSA: Fast and Accurate Dual-Stage Sparse Attention Mechanism Accelerating Both Training and Inference
Authors: Chong Wu
Jiawang Cao
Renjie Xu
Zhuoheng Ran
Maolin Che
Wenbo Zhu
Hong Yan
Venue: NeurIPS 2025 poster ()
Keywords: Efficient Attention Mechanism
Sparse Attention Mechanism
Transformer
Primary area: deep_learning
TL;DR: DuSA, a novel dual-stage sparse attention mechanism, further advances the basic single-stage sparse attention mechanisms to match or even outperform the vanilla scaled-dot product attention mechanism in different tasks.

Abstract: This paper proposes the Dual-Stage Sparse Attention (DuSA) mechanism for attention acceleration of transformers. In the first stage, DuSA performs intrablock sparse attention to aggregate local inductive biases. In the second stage, DuSA performs interblock sparse attention to obtain long-range dependencies. Both stages have low computational complexity and can be further accelerated by memory acceleration attention mechanisms directly, which makes DuSA faster than some extremely fast attention mechanisms. The dual-stage sparse attention design provides a lower error in approximating vanilla scaled-dot product attention than the basic single-stage sparse attention mechanisms and further advances the basic sparse attention mechanisms to match or even outperform vanilla scaled-dot product attention. Even in some plug and play situations, DuSA can still maintain low performance loss. DuSA can be used in both training and inference acceleration. DuSA achieves leading performance in different benchmarks: long range arena, image classification, semantic segmentation, object detection, text to video generation, and long context understanding, and accelerates models of different sizes.

DuSA: Fast and Accurate Dual-Stage Sparse Attention Mechanism Accelerating Both Training and Inference
Chong Wu"
Jiawang Cao" Opus AI Research gavin cao@opus pro
Renjie Xu" Dept of Electrical Eng City University Hong Kong harryxu950610@gaai com
Dept of Electrical Eng City University Hong Kong chongwu2-c@my city..eee.. eettyeeduuu. edu. hk imroxaswc@gmail. com
Zhuoheng Ran
Maolin Chel
Dept of Electrical Eng City University of Hong Kong zhuoheran2-cem cityu. edu. hk
School of Mathematics and Statistics
Guizhou University
@incml@outlook com
Wenbo Zhu
Hong Yan
Opus AI Research vito. zhu@opus, pro
Dept of Electrical Eng City University of Hong Kong h. yan edu, hk
Abstract
This paper proposes the Dual-Stage Sparse Attention (DuSA) mechanism for tention acceleration of transformers. In the first stage, DuSA performs intrablock sparse attention to aggregate local inductive biases. In the second stage, DuSA performs interblock sparse attention to obtain long-range dependencies. Both stages have low computational complexity and be further accelerated by mem- ory acceleration attention mechanisms directly, which makes DuSA faster than some extremely fast attention mechanisms. The dual-stage sparse attention design provides lower eтror in approximating vanilla scaled-dot product attention than the basic single-stage sparse attention mechanisms and further advances the ba- sic sparse attention mechanisms to match or even outperform vanilla scaled-do product attention. Even in some plug and play situations, DuSA can still maintain low performance loss. DuSA can be used in both training and inference acceler- ation. DuSA achieves leading performance in different benchmarks: long range arena, image classification, semantic segmentation, object detection, text to video generation, and long context understanding, and accelerates models different sizes.
Introduction
The vanilla scaled-dot product attention (VSA) mechanism [ is a core technique in transformers due to its ability to capture and model long-range dependencies 2 However, the softmax operation after the scaled-dot product becomes the main bottleneck that makes the computational complexity of VSA the square of the sequence length and significantly limits efficiency of VSA
Equal contribution Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To improve efficiency, various attention acceleration methods have been proposed. Among them, FlashAttention 4 is milestone in attention acceleration that provides exact method of attention acceleration by optimizing memory input/output (I/O) operations. Its successors 5 further improve memory efficiency. addition these memory acceleration attention mechanisms, other methods focus on optimizing the computation bound by using the kernel trick to find a linear approximation 19 or utilizing the sparsity of attention matrices to perform local attention operations [18] 20 30
3
These methods are usually called computation accelcration attention mechanisms. Memory acceleration attention mechanisms can usually provide an exact attention operation to replace the vanilla scaled-dot product attention mechanism directly. However, they usually hardware-aware and their computational complexity is still quadratic with respect to the sequence length [18]. Computation acceleration attention mechanisms mechanisms, and some of them [8] attention mechanisms. However, most of them have inferior performance compared to VSA and cannot directly replace VSA without training or tuning.
usually faster than memory acceleration attention 29 30 can be further accelerated by memory acceleration
Stage
0- 2014
softraa
&.B
2014
sofonex
Mi
Stage
Pege
Figure 1: The overview of DuSA. Boxes with grey color denote zeros. Figs. 2 and examples for obtaining A2.
shows two
Intrablock
Interblock attention (Strategy
attention
A. JLPVICE
Fc B2
M: g0
Figure 2: DuSA performs intrablock sparse attention within each block to obtain local inductive biases in Stage DuSA performs interblock sparse attention to obtain global information in Stage Here the blockify strategy is Strategy in Remark [ The different colors in attention matrices (4, and A₂ A') denote the different blocks. The color changes between and H₃ and between H; and He denote the information aggregation process.
Figure 3: DuSA using Strategy 2 in interblock sparse attention aggregate more information.
Ag = A" A" = A' and (l + 1) can be obtained using the rolling indices introduced in Remark 2
This paper proposes Dual-Stace Stage Sparse Attention (DuSA), novel attention mechanism, for attention acceleration of transformers. This work builds on the basic single-stage sparse attention mechanisms 25 and advances the field further through the novel accurate dual-stage sparse attention mechanism design as shown in Fig. [ DuSA performs intrablock sparse attention the first stage and interblock sparse attention in the second stage, which makes DuSA capable of obtaining intrablock information (local inductive biases) and interblock information (long-range dependencies) at the same time and is significantly different from commonly used sparse attention mechanisms 223.25 27-30). Both

---

stages have low computational complexity and can be further accelerated by memory acceleration
attention mechanisms directly. In summary, this paper has the following contributions.
A novel and effective dual-stage sparse attention (DuSA) mechanism is proposed to signifi- cantly accelerate training and inference processes.
(ii) A mathematical relationship between DuSA and VSA is given: 1. The theoretical upper bound using DuSA to approximate VSA derived. 2. The numerical results demonstrate that the dual-stage design has a lower approximation error than the single-stage design, despite the higher theoretical upper bound.
(iii) DuSA can approximate VSA with low error: DuSA can maintain 97%+ performance using 75%-94% sparsity ratios by replacing VSA in vision transformers (VīTs) directly without training, and surpasses VSA after training.
(iv) DuSA surpasses state-of-the-an ahe methods in terms of better performance and faster speed in different tasks.
Related work
Efficient attention mechanisms have been extensively studied to mitigate the quadratic complexity of attention. These methods can be classified into two categories based on their optimization focus: (I) memory acceleration methods and (2) computation acceleration methods.
2.1 Memory acceleration methods
Memory acceleration methods are usually hardware-aware and focus on minimizing memory I/O access to reduce complexity for some certain types of GPU architectures. FlashAttention 4 the milestone for memory acceleration in attention computation. FlashAttention-2 (FA2) (5) is major update version of FlashAttention, which further improves efficiency. FlashAttention-3 is a version that supports new features or characteristics of the new Hopper GPU architectures. Inspired by FlashAttention, some variants FlashAttention have been proposed. FlashSigmoid [31] implements memory efficient sigmoid attention mechanism based on FA2. Native sparse attention (NSA) [29] and SpargeAttn (3) implement memory efficient selective sparse attention based on FlashAttention. Different NSA, Attn is designed only for inference acceleration, which cannot be used to accelerate the training process.
2.2 Computation acceleration methods
Computation acceleration methods can be classified into two types: (1) the linear kernelized attention mechanism; (2) the sparse/local attention mechanism according to whether not the sparse property of the attention matrices utilized.
2.2.1 Linear kernelized attention mechanisms
Linear kernelized attention mechanisms use the kernel trick to approximate the softmax normalization of the vanilla scaled-dot product of the query matrix and the key matrix into linear product of two independent feature maps. Powerful feature map design becomes the key problem in this kind of efficient attention mechanism. Rectified linear unit (RcLU) 33 based feature maps are commonly used to approximate softmax normalization of the vanilla scaled-dot product of the query matrix and the key matrix, such as the ReLU-based feature map dividing by sequence length [9. the cubic ReLU-based feature map 77]. the convolution-enhant
ReL--based feature map (3), and the
exponential linear unit (ELU) there are some other types of feature maps. such as the cos-based nonlinear feature map 14] and the independent boftmax-based feature map [8
24 based feature map [0]. In addition ReLU-based feature maps,
15 S The attention matrix is approximately low-rank 17]. therefore, another kind of feature map design is based on low-rank decomposition methods, such as the Nyström approximation 2 decomposition (SVD) (36,, and matrix decomposition
the CUR decomposition [17], the singular value
377

---

2.2.2 Sparse/local attention mechanisms
Sparse/local attention mechanisms are based on the sparse property of attention matrices and usually use blockify methods, such as sliding windows, to partition the original sequences into smaller blocks [18]. Attention computation is limited within these blocks. Computational complexity can be reduced to almost sublinear complexity with an increasing number of blocks. Blockify methods based on sliding windows were first introduced by Longformer [24] and Swin 20 Due to the local nature of the blocks, global information easily lost. Different compensation strategies have been proposed, and most existing methods are based on the combination of different sparse attention mechanisms. sliding windows in two parallel attention heads and combines the attention results of two heads to reduce global information loss. Some methods use several fixed-step or randomly selected tokens to introduce long-range dependencies [25 27). Reformer (28) proposes
CSWin [21] introduces two different cross-
serill--seesiiiveasssiiiee (LSH) algorithm for selecting tokens to perform sparse attention, while the explicit sparse transformer [23] proposes top-k selective attention. NSA 29 combines compressed attention, top-k selective attention, and sliding window-based attention to propose a hierarchical sparse attention mechanism to obtain different levels of information. Unlike these hierarchical sparse attention mechanisms, ELFATT [8] proposes hybrid head architecture which combines simple sparse blockify attention with global kernelized linear attention in two parallel heads obtain global information. XAttention (38) uses the sum of antidiagonal values of the attention matrix to select blocks for further sparse attention computation within these blocks.
This work is significantly different from the above sparse attention mechanisms. DuSA introduces novel dual-stage sparse attention mechanism design as shown in Fig. [ The first stage (intrablock sparse attention) is used to obtain intrablock information (local inductive biases) and the second stage (interblock sparse attention) is used to aggregate all blocks according to their similarities (attention scores) obtain interblock information (long-range dependencies). Hence, DuSA can approximate VSA with a low error. Both stages have low computational complexity and are compatible with mem- ory efficient methods, which can be further accelerated. As enhanced alternative for basic sparse attention, DuSA can further improve and accelerate some advanced sparse attention mechanisms.
Methods
3.1 Vanilla scaled-dot product attention
For an input embedding matrix H E Rm after three linear transformations, we can obtain the following,
Q = HWQ, = HWK, V= HWv,
(1)
where Wo Rexe, WK E Re and Wy E Rexe are three scaling transformation matrices, Q E Rmxe E Rmxe, and V E Rm are the query matrix, key matrix, and value matrix, respectively. Vanilla scaled-dot product attention (VSA) obtains the attention matrix A E mm and the updated embedding matrix H as follows,
QK VC
A = softmax
H AV.
Since m >> c, the softmax operation for the scaled-dot product of Q and K makes VSA quadratic computational complexity with respect m and becomes the main bottleneck of efficiency.
3.2 Dual-stage sparse attention
Dual-Stage Sparse Attention (DuSA) can be expressed as a two-stage sparse attention process as Figs. 2 and show. In the first stage, DuSA performs local attention (intrablock sparse attention) within each block as follows,
QK
A = softmax
21
H1 + A.V,
Vc
where @ denotes the Hadamard product (39 and Z (m/b) 29 U()) with @ denoting the Kronecker product (39, b being the block size, (m//)) E IR (mm/)))((a/ (m/bb) being a matrix of which diagonal elements are and all the other elements are 00, and U(b) E bxb being the all-ones matrix.

---

Before considering the second stage of DuSA, we need to consider the mathematical expressions for blockify Strategies and 2 in Figs. 2annaandd and 3
Remark 1. Let J = [j.j b ......... - (m/6- 1)6} with j = 1,2,.... ........, bb. Hence, for blockify Strategy
the matrix Z2 E mmm is defined as [2] J, + U (m//b) with j = 1, 1,2, .......... b, and the other
elements of Z2 are -o.. Therefore, one has
OKK Vc
A' softmax
② Z2
H' + A'H.
(4)
Remark 2. We also introduce several notations (rolling indices) follows:
{o) 1 (1)
1 { = {JI, JJ1J2::: 2................. Joh.
(u( (2))
(2) ( = (J2, JIb.
(1T)) (b) 1
(b) 1 (Jb, 11,..... Jbb
6-1
For each k, the matrix A is defined A J A(J , with 1,2,...,b, and (J], Jp)) 0 (m/b) with 0 (m/b) E [R(m/bbx(m//)) being the all-zeros matrix and j + J....NN.. Note that
A = A'. Therefore, for blockify Strategy 2, we have
A"
A
(5)
k=1
The upper bounds of Eqs. 5 are discussed in Section Al in Appendix. Based on Remarks the second stage of DuSA performs interblock sparse attention as follows,
and
A2 = A",
H - H" = A"H = A2(1
(6)
or
A A',
H2 + H' A'H = A2(AVV).
(7)
4 Experiments and results
4.1 Experiment settings
DuSA is evaluated in image classification (ImageNet-IK 400,, long range arena (LRA
4 text to
video generation (Open-Sora 2.0 [42 prompt sets semantic segmentation (ADE20K 43 object detection (MS COCO 2017 44) , and long context understanding (LongBench 45 tasks. DuSAO
denotes DuSA (without using Strategies and 2), DuSAI denotes DuSA (Strategy 1), and DuSA2
denotes DuSA (Strategy 2). If not specified, all results of DuSA were obtained using DuSA2. For image classification, semantic segmentation, and object detection tasks, we compared DuSA with Agent [16), cross-shaped sliding window attention (CSW) (21. EFFATT [15), ELFATT [18], FA2
(5). FLatten 77 sliding window attention (SW) 120] and VSA M. The VIT backbones used for image classification, semantic segmentation, and object detection tasks are: CSWin-T/B 7 2] and Swin-T/B 20). The original backbones using CSW or SW are named with the suffix "LOCAL":
CSWin-T/B L/OLCALL and Swin-T/B-LLC T/B-LOCAL. The naming scheme for backbones using other attention mechanisms to replace CSW or SW is to add the corresponding attention mechanism name as suffix after the backbone name, for example, CSWin-T-DuSA. We followed the training protocol of
21 46 to adopt the same training settings and data augmentation methods to train all methods. Some advanced non-Vo ViT architectures: ConvNeXt-T [47 and VMamba-T 46 are selected for comparison. The experiments were conducted using 8 NVIDIA vGPU (32GB) GPUs. For LRA, we compared DuSA with FA2, Linformer [22], Nyströmformer [3], Primal (36), Reformer [28], and VSA. We followed the training protocol and the settings of [8] 36) and performed the experiments using 1
18
For given matrix B E _m and two index sets (f...., is} (1,2..... m) with 1 S < i, S m and
j.} C (1,2....,n) with < f1
S n. we let BLL, denote the
submatrix E RR of with entries (bi hei,eJ.
5

---

NVIDIA A1 (40GB) GPU. We followed the settings of [30] to rum text to video generation using DuSA. The backbone used is Cog CogVideoX22B [48]. We compared DuSA with FA2 and SpargeAttm
30
Complexity analysis, the ablation study about different block sizes, and more experiments
(plug and play examples, the use of DuSA vanilla ViT architectures [49], memory consumption
patterns, and the comparison of DuSA with FlashAttention of FlashInfer [50] and XAttention [38] in accelerating the prefilling stage of Llama-3. -ns-tust [51] on LongBench) are available in Appendix. The source code of DuSA is available in Supplementary Material For ImageNet-IK
ADE20K and MS COCO 2017, all methods including DuSA that are compatible with FA2, are accelerated by FA2. For text to video generation, DuSA is further accelerated by SpargeAttn.
Table 1: The test accuracy comparison of different methods on LRA. The best values are in bold.
Ditaser (sequence length)
Linformer
Nystrümfoemer
Prinal
Refeemer
VSA
DuSA
ListOps (2K)
37.3
37.2
37.3
19.1
37.1
38,1
Text (4K)
55,9
65.5
61.2
64.9
63.0
65.3
79.4
79.6
77.8
79.4
Image (1K)
37.8
41.6
43.0
43.3
38.2
42.7
Pathtinder (1K)
67.5
70,9
68.3
09.4
74.2
73.3
Average accuracy (%)
55.6
99.0
57,5
55.1
58.8
60.2
Table 2: The comparison of rumning time (s) per 1K training steps and peak memory consumption of different methods LRA using 1 NVIDIA H20 (96GB) GPU.
(sequence longth)
Nystromfommer
DuSA
(5) Memory (Ci8)
ListOps
0.4
19.6
11.1
28.3 0.6
21.2 0.5
34,9
71,2
16.8
0.5
Text (4K)
0:9
69.3
263.2
Retrieval (4K)
126.3
1.4
69.1
4.1
96.0
2.4
66.3
1.9
137.8
3.4
523.0
17.2
66.1
2.2
Image (1K)
48.9
0.8
45.8
2.2
69.7
1.4
49.8
1.0
82.9
2.9
109.2
4.5
44.8
1.2
Patirinder
4.2 Long range arena
DuSA compared with state-of-the thet--rf- attention mechanisms in LRA Table It can be seen that DuSA achieves state-of-ther performance in LRA. DuSA significantly outperforms VSA in long hierarchically structured data modeling (ListOps (2K)), text classification (Text (4K)), image classification on sequences of pixels (Image (1K)), and document retrieval (Retrieval (4K)) tasks. Only in long-range spatial dependency learning (Pathfinder (I))), DuSA slightly inferior to VSA, but it is significantly better than other efficient attention mechanisms in this task. Furthermore, DuSA faster than all comparison methods and offers 2.4-8.6x speedups over VSA and 1.1 x speedups over FA2 and achieves the third lowest memory consumption shown in Table
Speda 33
Speadep 1:26.
acuusay 26
sodeing Say 1
300
ELEATT
SLAATT
Plake
2000
Spoodap 2.5% accusssy (33%
Snoadap
100
aocuray
Spectip
500
500
Speclap Jc0uccy 5.8%
Spoodap
(a) Backbone: CSWin- T/B
(b) Backbone: Swin-T/B
Figure 4: Accuracy-efficiency (inference throughput) curves of different methods on ImageNet- I...
4.3 Image classification performance
Fig. prresents comprehensive performance comparison of various methods in the image classi-
fication task (ImageNet-IK3 IK). The experimental results demonstrate that DuSA achieves superior

---

classification accuracy compared to all methods in different model sizes and input resolutions. In addition, DuSA faster than all comparison methods and offers 2.2-4.3x speedups over VSA and 1.2-1.8 speedups over FA2 using CSWin/Swin backbones. More detailed results can be found in Table AT in Appendix. Fig. shows the visual comparison of class activation maps (CAMs) of DuSA, ELFATT, and VSA. Compared to ELFATT and VSA, CAMs of DuSA (Strategy 2) are more accurate than theirs. DuSA2 focuses on the groudtruth objects more accurately than DuSAO/DuSAL.
Black footed forret
ELFATT
DuSA0
Cabbage burterfly
Arabian camel
ELFATT
VSA
DuSA0
DrSAI
DuSA2
Barmcoute
FLTT
VSA
DuSA0
DUSAI
DaSA2
Kite
ELFATT
VSA
Du5A0
DeSAI
DuSA2
Skunk
ELFATT
VSA
DuSA0
DeSAI
DuSA2
Figure 5: Class activation maps (CAMs) comparison of CSWin-T-DuSA, CSWin-T-ELFATI and CSWin-T-VSA using Score-CAM (52) Note: CAMs obtained by DuSA2 and DuSAI focus on the groudtruth objects of the dual-stage spase attention design. In most cases, CAMs DuSA2 cover more area of the groudtruth objects and less area of the background than those of DuSAI. Compared ELFATT and VSA, CAMs of DuSA2 are even more accurate than theirs.
accurately than those of DuSAO which further demonstrates the effectiveness
4.4 Semantic segmentation performance
Table 3 presents comprehensive performance comparison of various methods on ADE20K. DuSA significantly outperforms all comparison methods terms of mean class accuracy (mAce) and mean intersection over union (mloU). As for inference speed, DuSA matches the speed of some extremely

---

Table 3: ADE20K semantic segmentation comparison. Note: Number of floating point operations
(FLOPs) are based on an input size of 512 x 2048. # is parameter numbers.
UpenNet
using 160K fine-tuning tenrioos Inference throughhpt (FP5) messured NVIDIA H2O with butch size of and mixed precision.
Method
mAc:
mioU
PPS (ingsh)
PLOPS
CSWas I 1Swin-T
Agent
60.8158,
48.5146.7 146,7
171.4
50M 666
929 319.64G 1957, 5OG
EFFATT
606155.3
1467
331.39
30M 660M
930.36G 1939.35G
1943.944
FLatten
61,41570
19.3144.8
271
51M 661M
930. 193 19A1662G
LOCAL
61.1155.6
49.6 144.5
28
50M 660
928 68C 1945 B6G
VSA
61.1159.33
147.
((12)
(FA2)
2458775G/92866CC (PAZ) 12873/944/9777
61.415988
Otbers
Com-eeXi-
58.3
46.1
VManba-T
59.3
47.9
fast methods: EFFATT and ELFATT, and offers 5.3-7.4x speedups over VSA and 2.3-2.6x speedups over FA2. Both backbones using DuSA significantly outperform ConyNeXt-T and VMamba-T. Swin-T-DuSA is as fast as Con and faster than VMamba-T in semantic segmentation.
4.5
Object detection performance
Table compares the object detection performance of various methods on MS COCO 2017. DuSA matches the state-of-the-art performance of ELFATT and significantly outperforms other attention mechanisms with higher box/mask average precision (AP*/APm) both 'Ix' fine-tuning training schedule and '3x' multiscale (MS) fine-tuning training schedule. DuSA offers 5.6-8.3 speedups over VSA and 2.2-2.9x speedups over FA2. CSWin-T-DuSA significantly outperforms VMamba-T in object detection with higher box/mask average precision and faster speed. Swin-T-DuSA also significantly outperforms ConvNeXt-T in terms of both performance and speed.
Table 4: MS COCO 2017 object detection comparison. Note: FLOPs are based on an input size of 1280 800. FPS was measured on NVIDIA H20 with batch size of 1 and mixed precision.
Method
AP
AP
Apm
APED
APR
FPS (ingsss)
GELORS
CSWin T
Ageni
16.8
68.9170.8
51.3
42.3143.9
65.91 67.9
45.3147.3
40M
254.5
EFFATT
46.1 148.5
68.3170.0
50.5153.2
41.9143.4
65.51 67.3
45.3146.9
36
40M
255.20
FLFATT
47.0149.4
69.2170.9
51.4154.4
42.6144.
66.4 68.0
45.91
33
40M
254.40
255.05
LOCAL
46.5149.33
68.5170.8
51.0154.3
42.1
65.61
45.3147.5
40M
253.57
VSA
47.0148.8
69.1
15.55
42.6143.6
66.1 67.4
45.91451
7/18 (FA2)
40M
1712.76/53.56
(FF22
DESA
46.9149.4
69.2
31.7154.4
42.7 44.1
66.4 67.8
45.91
39
40M
253.58
Swis-T
Agent
44.6147.3
67.5169.5
151.9
14.7142..7
64.4 66.4
43.4146.2
48M
278.42
EFAIT
44.7147.6
67.01994
64.01
44.4
48M
261.95
45.31
260.43
Flanen
44.116.5
67.3168.5
48.5 150.8
40.2 142.1
63.81 65.4
43.0145.1
41
40M
266.43
LOCAL
42.7146.0
66.2168.
46.8 150.3
39,31 41.6
62.2 65.1
42.2 42.2144.9
45
48M
207.01
VSA
45.4148.0
67.9170.0
152.7
143.3
65.01
44.8
6/17
2106.75/20046
DESA
460148.4
tiny
ComNeXt
262.29
VMamba
47.1148.9
69.5 170.6
$2.0 13366
42.7 14377
66.31 67.7
46.01 45.8
35
30M
271.16
Table 5: The comparison of text to video generation using Cog VideoXX22 -2B (sequence length: 17K) as the backbone. Note: The speed metric, TOPS (tera operations per second) (30), is the attention kernel speed. The speed was obtained using NVIDIA vGPU (32GB) with BF16 precision.
CLIPSIM
CLIPTT
VQA-a
VQAit
Fow
0.9971
$1,6921
48.3386
SpangeAltn (spuniiy: 0.46)
198
02015
0.9958
$1.5676
48:2785
5.7611
DuSA (sparsity: 0,50)
258
0.2644
0.9966
51,6099
48.3379
6,0482

---

(a) FA2 (CLIPSIM: 0.2161, CLIP-T: 0.9983, VQA-a: 53.4576, VQA-t: 48.9721, Flow-score: 12.3805)
(b) SpargeAttn (CLIPSIM: 0.2109, CLIP-T: 0.9984, VQA-a: 52.9862, VQA-t: 48.3282, Flow-score: 10.4248)
(c) DuSA (CLIPSIM: 0.2108, CLIP-T: 0.9986, VQA-a: 53.5017, VQA-L: 48.7888. Flow-score: 8.7635) Figure 6: DuSA achieves noninferior video generation quality compared to VSA using FA2 for acceleration and outperforms SpargeAttn.
PONDS
PONDS
(a) FA2 (CLIPSIM: 0. 0.1955, CLIP-T: 0.9992, VQA-a: 54.6491, VQA-t: 49.1951, Flow-score: 6.6526)
(b) SpargeAttn (CLIPSIM: 0.2044, CLIP-T: 0.9991, VQA-a: 53.2634, VQA-L: 48.8170, Flow-score: 6. 6.144
(c) DuSA (CLIPSIM: 0.2065, CLIP-T. 0.9982, VQA-a: 53.5948, VQA-E: 49.1732, Flow-score: 6.7921)
Figure 7: DuSA achieves noninferior video generation quality compared to VSA using FA2 for acceleration and outperforms SpargeAttn.

---

4.6
Text to video generation performance
Table 5 compares DuSA, FA2, and SpargeAttn in text to video generation (Open-Sora 2.0 prompt sets) using CogVideoX-2D 2B as the backbone. We followed [30] to use CLIPSIM [55], CLIP-T [55] VQA-a (56), VQA-t [56], and Flow-score [57 to evaluate text-video alignment, aesthetic and technical quality of the video, and temporal consistency. Using a similar sparsity ratio, DuSA obtains better text-video alignment (CIIPSIM), higher video aesthetic and technical quality (VQA-a and VQA-t), and better temporal consistency (Flow-score) than SpargeAttn and its performance is almost at the same level compared VSA using FA2 for acceleration. DuSA faster than FA2 and StargeAttn using 6F16 precision. DuSA uses 98s, while FA2 uses 147s and SpargeAtttn uses 106s on NVIDIA vGPU (32GB) with BF16 precision for end-to-end generation. Figs. 6 and show the visual comparison of DuSA, SpargeAttn, and VSA (FA2) in text to video generation. DuSA achieves better video generation quality than SpargeAttn and noninferior quality compared to VSA.
Table 6: Effect of different blockify strategies or stage execution orders of DuSA on ImageNet- I Note: FPS was measured on NVIDIA H20 using batch size of 512 and mixed precision.
Method
Resolution
Accuracy (%)
FPS (imgsis)
FLOPs
CSWin-T
DuSAO
224
82.9
3197
20M
4.09G
DuSAI (1": Intrablock;; 2nd Interblock)
224
83.0
2909
20M
4,09G
DuSAI (1": Interhlock; 2d1, Intrablock)
224
82.9
2880
20M
4.09G
Du5A2 (1": Intrablnck; 2d1, Interblock)
224
83.2
2908
20M
4.09G
DuSA2 (1*: Interblock; 2dd, Intrablock)
224
83.0
2866
20M
4,09G
Swin-T
DuSAO
224
30M
DuSA1 (1": Intrablock;; 2dd, Interblock)
224
82.6
3136
30M
4,63G
DuSAI (1": Interhlock; 2d1, Intrablock)
224
82.6
3095
30M
4.63G
DuSA2 (1": Intrablock; 2dd. Interblock)
2212
82.7
3154
30M
4,63G
DaSA2 (1":
Interblock; 2n1, Intrablock)
224
82.6
3062
30M
4.63G
4.7 Ablation studies about different blockify strategies and different stage execution orders
Table shows the comparison of DuSA using different blockify strategies on ImageNet-1K IK. Without using any blockify strategies, the speed is the fastest; however, its performance is also the lowest. Strategy 2 (DuSA2) obtains the highest accuracy because includes more token information for each token in token mixing, Fig. 3 shows. And its speed is almost the same that of Strategy 1. Additionally, as illustrated in the visual comparison of Fig. 5] CAMs obtained using Strategies 1 and 2 focus more accurately on the ground-truth objects than those obtained without using Strategies 1 and 2, further demonstrating the effectiveness of the dual-stage sparse attention design. In most cases, CAMs of Strategy 2 cover more area of the groudtruth objects and less area of the background than those of Strategy 1. Table also shows the comparison of DuSA using different execution orders of two stages on ImageNet- IK Due to the change in execution order, the speed is slightly different due to the different reshaping operations and convolutions for different value matrices used for position encoding. Performing intrablock sparse attention first usually outperforms performing interblock sparse attention first in terms of both speed and accuracy.
5 Conclusions
In this paper, we propose DuSA for attention acceleration of transformers. DuSA further advances single-stage sparse attention mechanisms through the novel dual-stage sparse attention mechanism design. Both stages are designed have low computational complexity and can be further accelerated by memory acceleration methods. The dual-stage sparse attention design provides a lower error in approximating vanilla scaled-dot product attention than the basic single-stage sparse attention mechanisms. DuSA can still maintain low performance loss in some plug and play situations. DuSA can be used in both training and inference acceleration. After training, DuSA can even outperform VSA. DuSA achieves tfte-oof-the-ars performance in different benchmarks: long range arena, image classification, semantic segmentation, object detection, text to video generation, and long context understanding, and accelerates models of different sizes.
10

---

Acknowledgements
This work is supported by the Hong Kong Innovation and Technology Commission (InnoHK Project CIMDA), the Institute of Digital Medicine, City University of Hong Kong (Projects 9229503 and 961 9610460), the National Natural Science Foundation China (No. 12561095), and the Special Posts of Guizhou University (No. [2025]06).
References
[1] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems, Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017, pp. 5998-6008.
[2] J. Han, L. Zeng, L. Du X. Ye, W. Ding, and J. Feng. "Modify self-attention via skeleton decomposition for effective point cloud transformer." Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 1, 2022, pp. 808-816.
[3] Y. Xiong. Z. Zeng, R. Chakraborty, M. Tan, G. Fung. Y. Li, and V. Singh, "Nyströmformer: Nyström- based algorithm for approximating self-attention," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 16, 2021, pp. 14 138-14 148.
[4] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, "FlashAttention: Fast and memory-effictee exact attention with IO-awareness," Advances Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 16 349-16359.
[5] T. Dao, "FlashAttention-22: Faster attention with better parallelism and work partitioning." " in International Conference on Learning Representations, 2024. [Online]. Available: https://openrvview.et/ffoooo 2x-22m2
[6] Shah, G. Bikshandi, Y. Zhang. V. Thakkar, P. Ramani, and T. Dao, "FlashAttention-3: Fast and accurate attention with asynchrony and low-precision," in Advances in Neural Information Processing Systems, A. Globerson, Mackey, D. Belgrave, A. Fan, Paquet, J. Tomczak, and C. Z.aang Eds., vol. 37. Curran Associates, Inc., 2024, pp. 68 658-68 685.
[7] D. Han, X. Pan, Y. Han, S. Song, and G. Huang. "FLatten transformer: Vision transformer using focused linear attention," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 5961-59711
[8] K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Q. Davis, A. Mohiuddin, E.. Kaiser, D. B. Belanger, L. Colwell, and A. Weller, "Rethinking attention with performers," in International Conference on Learning Representations, 2021. [Online]. Available:
[9] M. Wortsman, J. Lee, J. Gilmer, and S. Kornblith, "Replacing softmax with ReLU in vision transformers," arXiv preprint arXiv:2309.08555 2023.
[10] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, "Transformers are RNNs: Fast autoregressive transformers with linear attention," in Proceedings of the 37th International Conference on Machine Learning. ser. Proccedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119, PMLR, 2020, pp. 5156-5165. [Online]. Available: https:/ppoceeddgg..lrrr
[11] D. Bolya, C.-Y. Fu, X. Dai, P. Zhang, and J. Hoffman, "Hydra attention: Efficient attention with many heads," in Computer Vision ECCV 2022 Workshops, L. Karlinsky, T. Michaeli, and K. Nishino, Eds. Cham: Springer Nature Switzerland, 2023, pp. 49.
[12] Lu, J. Yao, J. Zhang, X. Zhu, H. Xu, W. Gao, C. Xu, T. Xiang, and L. Zhang. "SOFT: Softmax- free transformer with linear complexity." in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 21 297-21 309.
[13] H. Peng, N. Pappas, D. Yogatama, R. Schwartz, N. Smith, and L. Kong, "Random feature attention," " International Conference on Learning Representations, 2021. [Online]. Available: http:://appmeiiivaaaa//uuurrrlld

---

[14] Z. Qin, W. Sun, H. Deng, D. Li, Y. Wei, B. Lv, J. Yan, L. Kong, and Y. Zhong, "cosFormer: Rethinking softmax attention," in International Conference on Learning Representations.
2022. [Online]. Available:
[15] Z. Shen, M. Zhang, H. Zhao, S. Yi, and H. Li, "Efficient attention: Attention with linear complexities." in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021, pp. 3531-3539.
[16] D. Han, T. Ye, Y. Han, Z. Xia, S. Pan, P. Wan, S. Song, and G. Huang, "Agent attention: On the integration of softmax and linear attention," in Computer Vision - ECCV 2024, A. Leonardis, E. Ricei, S. Roth, O. Russakovsky, T. Sattler, and G. Varol, Eds. Cham: Springer Nature
Switzerland, 2025, pp. 124-140.
[17] C. Wu, M. Che, and H. Yan, "The CUR decomposition of self-attention matrices in vision transformers,, JEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. [Online]. Available: https://doii.rr/100.1110 h0tP://000000/PAM2200222..2.2
[18] C. Wu, M. Che, R. Xu, Z. Ran, and H. Yan, "ELFATT: Efficient linear fast attention for vision transformers," in Proceedings of the 33rd ACM International Conference on Multimedia, ser. MM '25. New York, NY, USA: Association for Computing Machinery, 2025, pp. 9140-9149.
[Online]. Available: https://doi.or//Iii https://oooomm/1111155//77771110 (4.014445//74007.35542222
[19] Z. Ran, Z. Ye. C. Wu, R. C. C. Cheung, and H. Yan, "FastViT: Real-time linear attention accelerator for dense predictions of vision transformer (ViT)," " in 2025 IEEE International Symposium on Circuits and Systems (ISCAS), 2025. [Online]. Available: htpp://ddoi.rg//100 100.1119999 TOATISCAAS55722222... 11046624
[20] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in Proceedingy of the IEEF/CVF
International Conference on Computer Vision (ICCV), 2021, pp. 10012-10022.
[21] X. Dong, J. Bao, D. Chen, W. Zhang, N. Yu, L. Yuan, D. Chen, and B. Guo, "CSWin transformer: general vision transformer backbone with shaped windows," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 12 124 12 134.
[22] S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, "Linformer: Self-attention with linear complexity," arXiv preprint 20Xiv22006.04768. 2020.
[23] G. Zhao, J. Lin, Z. Zhang, X. Ren, Q. Su, and X. Sun, "Explicit sparse transformer: Concentrated attention through explicit selection," " CoRR, vol. abs/1912.11637, 2019. [Online].
Available:
[24] I. Beltagy, M. E. Peters, and A. Cohan, "Longformer: The long-document vol. abs/2004.05150, 2020. [Online]. Available: lntp:://axi..orr/aa///0000111113
traaecdocuumeettttasttneeeee transformer." CoRR,
[25] R. Child, S. Gray, A. Radford, and I. Sutskever, "Generating long sequences with sparse transformers." CORR, vol. abs/1904.10509, 2019. [Online]. Available:
https:/aariivorg/abb//o.. 10509
[26] Y. Tay, D. Bahri, L. Yang, D. Metzler, and D.-C. Juan, "Sparse Sinkhorn attention," in Proceedings of the 37th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 2020, pp. 9438-9447. [Online]. Available: https://pooeeddigs.ll...esss
19/
[27] M. Zaheer, G. Guruganesh, K. A. Dubey. J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang. and A. Ahmed, "Big Bird: Transformers for longer sequences," in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 17 283-17297, 297.
[28] N. Kitaev, E. Kaiser, and A. Levskaya, "Reformer: The efficient transformer," in International Conference on Learning Representations, 2020. [Online]. Available: https://ppeneeviwwneeffooom wwwiiiorrnnnnnnndddee KKKKkkgN
[29] Yuan, H. Gao, D. Dai, J. Luo, L. Zhao, Z. Zhang, Z. Xie, Y. Wei, L. Wang, Z. Xiao, Y. Wang. C. Ruan, M. Zhang, W. Liang, and W. Zeng, "Native sparse attention: Hardware-aligned and natively trainable sparse attention," in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, Eds. Association for Computational Linguistics, 2025, pp- 23 23077-22 097.. [Online], Available: http:///ccaathooooyyy//222..... 1126/
12

---

[30] J. Zhang, C. Xiang, H. Huang, J. Wei, H. Xi. J. Zhu, and J. Chen, "SpargeAtteettion: Accurate and training-free sparse attention accelerating any model inference," in Proceedings of the 42md International Conference
Machine Learning, ser. Proceedings of Machine Learning Research, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj. K. Wagstaff, and J. Zhu, Eds., vol. 267. PMLR, 2025, pp. 76 76397-76413, [Online]. Available:
https///proceedings.llcc 2twpsmmssss/222272222222
[31] Ramapuram, Danieli, E. G. Dhekane, F. Weers, D. Busbridge, Ablin, T. Likhomanenko,
J. Digani, Z. Gu, A. Shidani, and R. Webb, "Theory, analysis, and best practices for sigmoid self-attention," " in International Conference on Learning Representations, 2025. [Online]. httpe//wwww.eeeoommmdddddahhaaa
Available:
[32] V. Nair and G. E. Hinton, "Rectified linear units improve restricted Boltzmann machines," " in Proceedings of the 27th International Conference on Machine Learning, ser. ICML' 10, J. Fürnkranz and T. Joachims, Eds. Madison, WI, USA: Omnipress, 2010, pp. 807-814.
[33] H. Cai, J. Li, M. Hu, C. Gan, and S. Han, "EfficientViT: Lightweight multi-scale attention for high-resolution dense prediction," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 17 173--11733 313.
[34] D.-A Clevert, T. Unterthiner, and S. Hochreiter, "Fast and accurate deep network learning by exponential linear units (ELUs)," arXiv preprini urXiv:/511.000 1.07289, 2015.
[35] H. Kang, M.-H. Yang, and J. Ryu, "Interactive multi-head self-attention with linear complexity," arXiv preprint arXiv:2402.1 1 17507, 2024.
[36] Y. Chen, Q. Tao, F. Tonin, and J. Suykens, "Primal-Attention: Self-attention through asymmetric kernel SVD in primal representation," in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 65 088-65 101.
[37] Z. Geng, M.-H. Guo, H. Chen, X. Li, K. Wei, and Z. Lin, "Is attention better than matrix decomposition?" in International Conference on Learning Representations, 2021, [Online]. Available: https://ppenreviiw.ee///oo Iiimmidd===vPvosssssss
[38] R. Xu, G. Xiao, H. Huang. J. Guo, and S. Han, "XAttention: Block sparse attention with antidiagonal scoring," in Proceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, A. Singh, M. Fazel, D. Hsu, S. Laceste-Julien, F. Berkenkamp, T. Maharaj. K. Wagstaff, and J. Zhu, Eds., vol. 267. PMLR, 2025. pp. 69 819-69 69 831. [Online]. Available: https://proceedings mir. wttp://eeeeiinnnnmi.rrer/eeeeeee pres/v267//////2agg htmll
[39] R. A. Horn and C. R. Johnson, Matrix Analysis, 2nd ed. Cambridge University Press, 2012.
[40] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, "ImageNet large scale visual recognition challenge," International Journal of Computer Vision, vol. 1 15, no. 3, pp. 211-252, 2015.
[41] Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler, "Long range arena: benchmark for efficient transformers." in International Conference on Learning Representations, 2021. [Online]. Available: https///peaneviee..ee/ffoooo wwwp:e/wwww.....o.ooooeeaewwaaee
[42] X. Peng, Z. Zheng, C. Shen, T. Young, X. Guo, B. Wang, H. Xu, H. Liu, M. Jiang, W. Li et al., "Open-Sora 2.0: Training a commercial-level video generation model in $200K," arXiv preprint arXiv:2503.0966 096422 2025.
[43] B. Zhou, H. Zhao, X. Puig, S. FiFiller, A. Barriuso, and A. Torralba, "Scene parsing through ADE20K dataset," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA. USA: IEEE Computer Society, 2017, pp. 5122-5130.
[44] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft COCO: Common objects in context," in Computer Vision - ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014,
pp. 740-755.
[45] Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li, "LongBench: A bilingual, multitask benchmark for long context understanding." " in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), L.-W. Ku, A. Martins, and V. Srikumar,
13

---

Eds. Association for Computational Linguistics, 2024, pp. 3119-3137. [Online]. Available: www.e//aanhoooooooo/rA.aaaaaaaoa
[46] Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang. Q. Ye, Jiao, and Y. Liu, "VMamba: Visual state space model," in Advances in Neural Information Processing Systems, A. Globerson, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, Eds., 37. Curran Associates, Inc., 2024, pp. 033--1033063
[47] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, "A ConyNet for the 2020s," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 11 976-11 986.
[48] Z. Yang, J. Teng, W. Zheng. M. Ding, S. Huang, Xu, Y. Yang, W. Hong. X. Zhang, G. Feng, D. Yin, Y. Zhang. W. Wang. Y. Cheng, B. Xu, X. Gu, Y. Dong, and J. Tang, "CogVideoX: Text- to-video diffuusion models with an expert transformer," " in International Conference on Learning Representations, 2025. [Online]. Available:
[49] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x 16 words: Transformers for image recognition at scale," in International Conference on Learning Representations, 2021. [Online]. Available:
[50] Z. Ye, R. Lai, B.-R. Lu, C.-Y. Lin, S. Zheng, L. Chen, Т. Chen, and L. Ceze, "Cascade inference: Memory bandwidth efficient shared prefix batch decoding," February 2024. [Online]. Available:
[51] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur,
A. Schelten, A. Vaughan et al., "The Llama herd models," arXiv preprint arXiv:2407.21783, 2024.
[52] H. Wang. Z. Wang. M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu, "Score-CAM: Score- weighted visual explanations for convolutional neural networks," " 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Los Alamitos,
CA, USA: IEEE Computer Society, 2020, pp. 111-119.
[53] Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun, "Unified perceptual parsing for scene understand- ing." in Computer Vision - ECCV 2018, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds. Cham: Springer International Publishing, 2018, pp. 432-448.
[54] K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask R-CNN," in 2017 IEEE International Conference on Computer Vision (ICCV). Los Alamitos, CA, USA: IEEE Computer Society, 2017, pp. 2980-2988.
[55] Y. Liu, X. Cun, X. Liu, X. Wang. Y. Zhang, H. Chen, Y. Liu, T. Zeng, R. Chan, and Y. Shan, "EvalCrafter: Benchmarking and evaluating large video generation models," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024, pp. 22 139-22214 149.
[56] H. Wu, E. Zhang, L. Liao, C. Chen, J. Hou, A. Wang. W. Sun, Q. Yan, and W. Lin, "Exploring video quality assessment on user generated contents from aesthetic and technical perspectives," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 20 144-20 154.
[57] T. Zhao, T. Fang, H. Huang, R. Wan, W. Soedarmadji, E. Liu, S. Li, Z. Lin, G. Dai, S. Yan, H. Yang, X. Ning, and Y. Wang, "ViDiT-Q: Efficient and accurate quantization of diffusion transformers for image and video generation," in International Conference on Learning Representations, 2025. [Online]. Available: http://pwwwwwaaaeee/o//omm/md/oe
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section A8 Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Each theoretical result in Section of the main text and Section A1 of Appendix is provided with complete (and correct) proot.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Subsection 4.1
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
See Supplementary Material
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuiu wwwtii/gguie///eeeeceoocc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Subsection
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The paper does not include such experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section and Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://tttt////rpp.....//o///ppa http://p.....llla/c//cccccieiino
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section A7
Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the existing assets used are properly cited or credited in the paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

A1 Upper bounds analysis
The following derivations omit the normalization term of the softmax operation for convenience.
To characterize the difference between H and H1 we set I = {(i 186--11, 1)b + (1- 1)6 +2 2..... ib) with ..,, m/b. Then the entries of A can also be defined A1 (T,, A M2) with
i i= 2, m//b and A (Ii, Iv) = Os with + i', where 0, Rg is the all-zeros matrix. Hence, we have
m//b
|| A
Σ
IA(T,
W-=Li
which implies that
m/bb
H1 H |A AVIA S
Σ
(A1)
For ||A' AAlle and ||A" - AAlle according to Remarks
and
we have
b
IA'' A S
[
(A2)
id==
6
AA AlE S
|
Σ
[A
id'=1iA11"
(A3)
IA
||F
Σ
(A ,f))) )|e..
3-1==157
Remark 3. Based the above descriptions and Eqs. can be discussed in following two cases.
6 and 77. the upper bound for H H
Case 6 corresponding to Eq. (6): an upper bound for H2 H| is deduced as
2 H| F ||A A V
AVI F AA"AV A V A V AV
AA V A, VII |A AVI
m/b
< ||A" Im mmx |Ai VII + Σ
IAT,,T))
m/b
AA" Im max |AV| + (IIA"
Im Imax
+ 1)
Σ
TT)re|) (A4)
Case corresponding Eq. given as
similar to Case 6 an upper bound for H2 Fis
H2 HIIF F SIA Im (maax|AV/
m//
+ CIA Im mmx + 1)
Σ 111="
[A(T,,
(A5)
where for any matrix B _gmx,, the symbol |B |max is defined as
B mmxx = max ([be)l i = 1, ..,,....7 m;; = 122.
We now estimate two terms IIA" Emllmax and I| Im Immax Remark based on Inequalitics
A2h and (3)) as follows:
LA" " Im max = I|A" "
+ A Im Imlx S AAA"
A max + I|A Imllmax S ||A" A
b
+ ||A - Im |l <
(A(J 1 )00)
NA(JS") M(C3S"I("))ll 1 - -m l
A 2=1
5.j - 1
(A6)
22

---

and
||
mm Ilmax
Σ
I ||A -mllmmax
(A7)
15'=1.773
Therefore, the upper bound for | and A66:
H in Case
6 is obtained by combining Inequalities
A4
m/h
H2 HIFF S aallAVil + 8
Σ 11=
(A8)
with
ΣΣ 2=2/=1
[A
E
|A( , Ile + ||A -Imllmax
==a+b 1,
13-=1377
and the upper bound for H H I in Case A7):
is obtained by combining Inequalities
A5 and
H2 H S d'IIAV 8
(A9)
with
a'
Σ
JJ
||A -m mmlmmax
a'=a++1. +1
0.7
0.7
4-Wiiwer ssay Stateies sol
Seuaeer
0.6
Relatve 2200
0.4
0.4
0.1
0.1
Staacer Whihe adey Staaeded
Epach
Epoch
(a) Backbone: CSWin-T
(b) Backbone: Swin-T
Figure A1: The comparison of relative error of DuSA using Strategy 1 (its relative approximation
||ا and the upper bound of the term H2 HH can be obtained using Inequality error is: [[
A9)). DuSA using Strategy 2 (its relative approximation error is: 111-111ix and the upper bound of [
the term |Hg H|r r can be obtained using Inequality (A8). and DuSA without using Strategies and 2 (its relative approximation error is: | |||l- and the upper bound of the term || H can be obtained using Inequality (AT) in approximating VSA during the training of C/Swn-T/Swin-I on ImageNet-1K: The approximation error shows a decreasing trend. Strategy shows a lower error curve the beginning training which consistent with the analysis of the relationship between the upper bound Inequality (A8) and the upper bound Inequality (A9). As the training process progresses, the actual approximation error of Strategy becomes error of Strategy which also consistent with more token information introduced by Strategy 2 in token mixing enhance performance. Even the backbone is untrained, DuSA using Strategy still provides significantly lower approximation error than DuSA without using Strategies and which further demonstrates that DuSA is more similar to VSA than simple sliding window based sparse attention.
smaller than the actual approximation
23

---

may be relatively rough. In FF in Cases O6 and (7) Fig. AT shows the comparison of the actual relative error of DuSA using blockify Strategy 1, DuSA using blockify Strategy 2, and DuSA without using blockify Strategies and 2 in approximating VSA. Blockify Strategy shows a lower error curve at the beginning of training which is consistent with the analysis of the relationship between the upper bound Inequality (A8) and the upper bound Inequality (A9 As the training process progresses, the actual approximation error of blockify Strategy 2 becomes even smaller than the actual approximation error of blockify Strategy which is also consistent with more token information introduced by blockify Strategy 2 in token mixing to enhance performance. Even the backbone is untrained, DuSA using blockify Strategy 1 or 2 still provides significantly lower approximation error than DuSA without using blockify Strategies I and 2 which further demonstrates that DuSA A is more similar to VSA than simple sliding window based sparse attention (DuSA without using blockify Strategies and 2).
Remark 4. For the term || H Fl, its upper bound (88) or (A9 the future, key issue is obtain a tighter upper bound for H2 =Ir
Table A1: ImageNet-Il 1K classification comparison. Note: FPS was measured on NVIDIA H20 using batch size of 512 for tiny models and 256/32 for base models with resolution of /242 and mixed precision.
Method
Resolution
Accuracy (96)
HPS timges)
FLOPS
CSWin
Agent
224 13842
84,7
994 276
73M
14.491442.55 42,.77
224 13842
10591
ELFATT
224213880
84,71
1187 355
75M
14.461 42, 442488
FLatten
224 13842
LOCAL
224 13542
85.5
10371 323
75M
14,39L1 42.28G
VSA
224 13842
(FA2))
(PA2)
73M
22.33G/14.39G (PA2)) 11.89/222.0
(FA2)
DoSA
224
1384 1
8 85.9
1203 356
73M
14.3961 42.28G
CSWin T
224
83.1
2425
20M
4.14G
EFRATT
2243
82.6
2526
20M
4,17G
BLFATT
224
83.1
2856
20M
4.13G
FLatien
224
83.1
2025
21M
4.16G
LOCAL
2242
20M
409G
VSA
22.
83.1
1305/2200 (FA2)
20M
7.60G44 (FA2)
D
224
83.2
20M
409G
Swin-B
Agent
224 1
1367
|5.48GI 4XG1166440
BEBAUT
224 1384
84.2185.3
15361 481
85M
15,33G1 45.04G
ELEXTT
224 1
91M
18.681111
FLatten
224 1384
83.8185.0
12261388
89M
13.4601 46.49G
LOCAL
224 1384
1
15.47G1 47.19G
VSA
84.21 85.3
74311325 (FA2)) 134/313 (FA2)
88M
21 5TCV15 1961 (FA2) 199.766344 64G (FA2)
224913842
85.5
91M
15.61G1
Swin-T
Agent
224
82.6
2847
29M
4.53G
LFBATT
2243
H2.1
28M
4.45G
ELFATT
22.22
82.7
3159
30M
4.67G
PLaticn
2243
82.1
2502
29M
4,50G
LOCAL
2242
81.4
2943
25M
4.51G
VSA
224
82.4
1269/2771
(FA2)
8.81 (FA2)
DoSA
224
82.7
3154
30M
4.63G
Others
ConvNeXI-T
224
82.1
3911
29M
4.47G
VMambo-1
224
82.5
30M
4.84G
A2 Complexity analysis
For VSA, the nature of the scaled-dot product attention calculation leads to the complexity of
O(m x c). For blockify Strategy 1, the complexity of DuSA is determined by Eqs. (3) and (4). In Eq. 3 the complexity of the computation of the attention matrix A₁ is O(m x x c) and the
complexity of the token mixing to obtain H' is also O(m x x c). Hence, the total complexity of
Eq. 33) is (mm x x c). Similarly, for the complexity of Eq. (4 its complexity is O((m² /b) x c).
c). and it can achieve the The total complexity of DuSA using blockify Strategy is O mxxb+mm +m?
24

---

minimum Om x vm x c) when b = mm. For blockify Strategy 2. compared to blockify Strategy 1, only has one more attention score aggregation process, as shown in Eq. { of which the complexity is O(m²/0) /b) & (( (b) x c). Hence, the total complexity of DuSA using blockify Strategy 2 is also O mxxb +m c) and its minimum is also O(m x vm X e) when b /m. If m > c, the complexity of DuSA O(m x ym), which is less than O O(m²) ) of VSA. Our method is designed for acceleration when m >> The larger m is than the more obvious the acceleration effect is.
b
Table A2: The effect of different block sizes at each level (L1-L4) the performance of DuSA and some plug and play (PP) examples using DuSA to replace VSA directly without training in the ImageNet- IK classification task. Note: FPS measured on 1 NVIDIA H20 using batch size of 512 and mixed precision. "PT" denotes the models pre- are-trained from scratch. CSWin-T-VSS here the same as CSWin-T-VSA in Table while Swin-T-VSA here uses the enhanced version from 18] which replaces the concatenation operations for partition/merging with convolutions to enhance performance, hence it is a little different from Swin-T-VSA in Table AT
L2 L3
1.4
Resolution
Accuracy (PT/PP)
FPS (imgsh)
FL.0%s
CSWin-T
49
49
196
49
224
K2.7/78.8
2989
20M
4.09G
224
83.2/82.0
30M
784
784
196
49
22412
83/2/8311
2860
20M
4.09G
3136
VSA (full sequence length
784
196
49
224
83.17
1303/2210 (FA2)
20M
7.60G14 (PA2)
Swaa-T
49
224
82.5773.1
3223
30M
4.63G
196
196
196
49
224
82.7/50.3
31544
30M
4.63G
224
82.8/83.4
30M
4.630
VSA (full longth
224
82.77
1188/2257 (FA2)
30M
9.06134 63G
A3 The effect of different block sizes and plug and play examples
Table A2 shows the effect of different block sizes at each level (L.-L44) on the performance of DuSA and some plug and play (PP) examples using DuSA to replace VSA directly without training in the ImageNet-IKK classification task. The sequence lengths of VSA in 4 levels of CSWin-T/Swin-T are 3136, 784, 196, 49). The sequence lengths of the last two levels are too short to affect ViT speed 16-18. Hence, we only modified the block sizes of the first two levels. As the block sizes increase, the inference speed is reduced and the accuracy is improved. Even using the 75% sparsity ratio of the first level, DuSA still offers a 1.3x speedup over FA2 Table A2 and outperforms VSA. Also, as Table A2 shows, when using the 75%-94% sparsity ratios of the first two levels, DuSA can almost maintain 97%+ accuracy in a plug and play application. The current version of DuSA is a static sparse attention mechanism, which cannot preserve the accuracy of VSA without loss in plug and play application. is designed for accelerating both the training and inference processes. After training, it can match or even outperform VSA. Inspired by XAttention, we have provided training-fiee free dynamic sparse attention version of DuSA for accelerating the prefilling stage of large language models (LLMs). The dynamic version of DuSA can match the performance of VSA in plug and play applications and provides a significant acceleration effect. Details are available in Section
A6
A4 The use of DuSA in simple and general architectures
Table shows the performance comparison of vanilla ViT architectures and the versions using DuSA to replace VSA. The sequence length m of the vanilla VIT architecture is relatively short (for the input resolution of /342 using a constant input patch size of 16, m = 197/577 with a constant embedding dimension 64), which limits the acceleration effect of DuSA and cannot completely show the superiority of DuSA for acceleration. Because DuSA designed for acceleration when
m ><
25

---

Table A3: The performance comparison of vanilla ViT architectures and the versions using DuSA to replace VSA. All models are trained from scratch on ImageNet-IK. IK Block size is 14/24 for (argge)))) models. Padding is used to make the sequence length divisible by b. FPS and the inference memory are obtained on 1 NVIDIA H20 using a batch size of 512/32 for the T/L. models.
Method
Resolution
(3)
FPS (imges)
FLOPs
Inference
VIT-LI6-VSA
3642
179222
307M
191.3IG/174850
3,7/2.9 (FA3)
BT-L/I--DUAA
384
77.2
230
307M
174,54G
3.0 (FAZ)
224
(FAZ)
VIT-//6--DSA
224
72,9
10718
6M
1,07G
2,0
A5 Memory consumption patterns of DuSA
Tables A4 and A5 show the memory comparison results. Compared to ELFATT and FA2, DuSA still has the overall lowest memory usage and its memory usage is significantly lower than that of VSA.
Table A4: Peak memory usage comparison ImageNet using NVIDIA H20 (96GB). The batch size used is 32 in both inference and training. "OOM" denotes out of memory (MEM).
Mehho
Resolution
Inference MEM (GB)
Trinning MEM (GB)
CSWin D ELFATT
384
2.5
CSWin B-VSA
384
62.3/2.4
00M/17.9 (FA2)
CSWin B DESA
384
233
21.0
Table A5: Peak memory usage comparison on ADE20K and MS COCO 2017 using NVIDIA H20 (96GB). The batch size used is for inference and 2 for training. "OOM" denotes out of memory.
Method
ADE20K
ADE20K training (EM
training (GB)
Swis-T
ELFATT
2.8
1.6
5.6
82
(FAZ)
(PA2)
DeSA
2.5
8.1
CSWin-T
2.7
VSA
33.2/2.7 (FA2)
18.2/1.4(FA2)
16.7/5.4 (PA2)
OOMX & (FA2)
DOSA
2.7
1.4
5.4
8.7
12
XAttention (stride=8, antidiagonal patterns)
10
DuSA (stride=8, antidiagonal pattems)
Speedup ratio (x)
DuSA (stride=16, antidiagonal palterns)
8
6.6
6
6.0
5.8
4
2,8
2.8
2
1.9
1/8
2//3
1.1
4K
8K
16K
32K
64K
128K
Sequence length
Figure A2: Speedup relative to FlashAttention of FlashInfer obtained by different methods across different sequence lengths using -.1-aa...------------- on NVIDIA H20.
A6 The application of DuSA in accelerating the prefilling stage of LLMs
Based on XAttention, we introduce DuSA (Strategy 2: Rolling indices- attention scores fusion) search antidiagonalddiagonall patterns attention matrices to filter important or unimportant blocks for further dynamic sparse attention performed within important blocks only. As Fig. A2 and Table
26

---

show, DuSA can use larger stride (larger sparsity) to achieve similar or higher performance than XAttention, further accelerating FlashAttention of FlashInfer. By introducing antidiagonal patterns of XAttention, the performance of DuSA can be further improved.
Table A6: The performance comparison of different attention methods LongBench using the
Llama-3.1 1amma-31--B--nstree 1-Stttnsttuutt model. Note: Higher scores mean better performance.
PhosggReens Ronsseenneelloon
Muliii MaaoFiccuag--a
MahiFsali SaiERaiiiaaa A--ah
PassgeeRttieal
NaratheQA
Qasper
HespoQQ
MusiQue
DuReadel
GovRopon
MulnNows
VCSUM
TREC
TihiQA
SAMSum
LSST
PaoapeCoum
Repollench
Ansrage
LOC
VSA (FushAAtemion of Flshlnfer)
31.44
25.07
20 40
61,68
16.80
17.00
34.22
23.25
6969
15.1
2,50
91.65
43.74
5.95
77,11
52.10
40. 14
XAnention (stide=8, antidiagonal pattems)
50
26.44
29.28
61.57
17.39
1631
34.60
23:24
27.08
16.11
11.50
90.97
44.33
46.50
5.23
74
53.23
50.94
(strides, andiiaguunal postems)
80
28.08
58.23
18.00
16.05
35.4
34,08
25.21
26.99 92
15.77
92.15
43.27
7.25
76.02
48.99
DoSA Ginde--, diegonal patterns)
31.34
28.78
61.65
1692
16.5
11.86
35.00
34.55 S4
23:43
27.17
16.21
73,00
90062
45.59
00.50
6.38
7506
55.24
90 16
(strdde116 antidiagoaal pattorms)
30.33
27.57
20 20.00
60.99
17.34
17,02
34,59
23.18
15.73
90.90
45.67
00.00
6.39
23.25
40.56
26
A7
Broader impacts
DuSA provides efficient attention mechanism accelerate training and inference of long-context transformers, and it is not bardware-aware which will benefit academia and startups to reduce deployment costs using rediuum-performant DuSA can be used in advanced hicrarchical sparse attention mechanisms to improve their performance and speed as novel alternative for basic sparse attention. Although the enhanced efficiency of long-context transformers may increase risks such as misuse in generating fake/harmful contents, the benefits brought by efficient long-context transformers outweigh the risks in general.
GPUs and unlock cross- cross-disciplinary breakthroughs.
A8 Limitations
As Remark 4 shows, for the term ||H
[. its upper bound (Inequality (A8) or (A9 provides
worst-case guarantee and may be relatively rough: In the current version, the upper bound (Inequality (8 (A9 for Strategy 1 or 2 is larger than the upper bound (Inequality (AIp) for simple block sparse attention. However, as shown in Fig. All the actual approximation eтror DuSA (Strategy 1) DuSA (Strategy 2) is usually much lower than that of DuSA without using Strategies I and the future, a key issue to obtain a tighter upper bound for Strategy or 2. In addition, the acceleration effect of DuSA may be limited when is not significantly larger than
27


Reviews:
[{"review_id": "pKT79KnDxU", "reviewer": "Reviewer_87kG", "summary": "This paper proposes the Dual-Stage Sparse Attention (DuSA). DuSA performs two stage sparse attention mechanism where in the first it performs intra block to learn local inductive biases and in second stage they learn interblock attention mechanism to learn long range dependencies. this two stage mechanism reduces the error in vanilla scaled-dot product attention compared to basic single-stage sparse mechanisms. The first stage  obtains  the local inductive biases and the second stage aggregates all blocks according to their similarities  to aggregate  long-range dependencies.", "questions": "Questions:\n- even though DuSA has less memory, DuSA has equal FLOPs or more compared to existing methods like ELFATT.\n- Can DuSA be extended auto-regressive based methods?", "limitations": "Limitations are discussed in the paper", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- DuSA has low error in approximating vanilla scaled-dot product attention\n- Both stages have low computational complexity and are compatible with memory efficient methods, and can be further accelerated.  DuSA can further improve and accelerate some advanced sparse attention mechansims\n- the paper clearly explained the math behind the approximation process of the two stage process in DuSA.\n- DuSA is compared with state-of-the-art attention mechanisms in LRA.  DuSA conducted experiments in image classification, semantic segmentation and object detection performance\n\nWeaknesses:\n- ELFATT performance is same or better than DuSA in tables 3 and 4. ELFATT has less number of FLOPs compared DuSA", "was_revised": true, "final_justification": "Thankyou Authors for providing rebuttal. After reading the reviews from other reviewers and authors responses I would like to maintain my initial rating.", "rebuttal": "**We would like to thank the reviewer for the constructive comments provided to help us improve the quality of the paper. We have addressed the issues raised in the comments point-by-point as follows:**\n\n---\n\n**Response to W1:** ELFATT [1] is a SOTA for ViT acceleration, which was just online several months before DuSA. The FLOPs of DuSA are lower than ELFATT. You may have confused ELFATT with EFFATT [2]. EFFATT is a kernelized linear method, and its theoretical complexity is indeed lower than that of DuSA. But its performance is significantly inferior to DuSA. Its real speed doesn’t show a significant superiority compared to DuSA. Because both stages of DuSA can be further sped up by memory-efficient methods.\n\n---\n\n**Response to Q1:** The number of floating point operations (FLOPs) doesn’t represent the real speed of the method, as pointed out by [1,3,4]. Even though some methods have lower FLOPs, their real speed could be slower. Hence, we provide the inference throughput of all methods, which will be more accurate to represent the real speed of each method.\n\n---\n\n**Response to Q2:** Yes, it can. XAttention [5] has shown that the diagonal pattern-based attention mechanisms can be used in accelerating the prefilling stage for recent auto-regressive transformer-based (decoder-only) LLMs. Based on XAttention, we introduce DuSA2 (Strategy 2: Rolling indices-based attention scores fusion) to search antidiagonal or diagonal patterns of attention matrices to filter important or unimportant blocks for further dynamic sparse attention only performed within important blocks. As **Table R1-1** shows, compared to XAttention, the performance of DuSA2 is closer to that of full attention. As **Tables R1-1** and **R1-2** show, DuSA2 can use a larger stride (larger sparsity) to achieve similar or higher performance than XAttention, further accelerating full attention. By introducing antidiagonal patterns, the performance of DuSA2 can be further improved.\n\n## Table R1-1. The performance comparison of different attention mechanisms on LongBench using the Llama-3.1-8B-Instruct model. Note: Higher scores mean better performance.\n|Method|Full (FlashInfer’s implementation of FlashAttention)|XAttention (stride=8)|DuSA2 (stride=8) antidiagonal patterns|DuSA2 (stride=8) diagonal patterns|DuSA2 (stride=16) antidiagonal patterns|\n|-|-|-|-|-|-|\n|NarrativeQA|31.44|30.48|30.81|31.34|30.21|\n|Qasper|25.07|26.04|24.28|25.96|**27.57**|\n|MultiFieldQA-en|**29.40**|29.28|28.88|28.78|29.00|\n|MultiFieldQA-zh|**61.68**|61.67|58.28|61.65|60.99|\n|HotpotQA|16.89|17.33|**18.09**|16.92|17.34|\n|2WikiMultihopQA|17.00|16.34|16.03|16.35|**17.02**|\n|MuSiQue|11.79|11.88|10.21|11.86|**12.04**|\n|DuReader|34.93|34.64|35.40|35.60|**36.13**|\n|GovReport|34.22|34.60|**34.68**|34.54|34.59|\n|QMSum|23.25|23.24|23.21|**23.43**|23.18|\n|MultiNews|26.69|27.08|26.92|**27.17**|26.96|\n|VCSUM|15.91|16.11|15.77|**16.21**|15.73|\n|TREC|72.50|71.50|72.50|**73.00**|71.50|\n|TriviaQA|91.65|90.97|**92.15**|90.62|90.90|\n|SAMSum|43.74|**44.13**|43.27|43.59|43.67|\n|LSHT|46.00|46.50|**47.00**|46.50|46.00|\n|PassageCount|5.95|5.23|**7.25**|6.38|6.39|\n|PassageRetrieval-en|**98.20**|88.68|96.84|89.26|97.76|\n|PassageRetrieval-zh|**77.11**|74.40|76.67|75.06|73.76|\n|LCC|52.19|53.23|52.28|53.24|**53.36**|\n|RepoBench-P|49.14|**50.94**|48.97|50.16|49.56|\n|Average|**41.18**|40.68|40.92|40.84|41.13|\n\n## Table R1-2. Speedup relative to FlashAttention (FlashInfer’s implementation) obtained by different attention mechanisms across different sequence lengths using Llama-3.1-8B-Instruct on a single H20 (96GB) GPU.\n|Sequence Length (K)|DuSA2 (stride=8) antidiagonal patterns Speedup|DuSA2 (stride=16) antidiagonal patterns Speedup|XAttention (stride=8) Speedup|\n|-|-|-|-|\n|4|1.02|1.12|**1.93**|\n|8|1.59|2.36|**2.83**|\n|16|2.25|**4.67**|4.05|\n|32|2.79|**6.63**|5.06|\n|64|3.36|**9.11**|6.00|\n|128|3.09|**11.20** |5.57|\n\n---\n\n**References:**\n\n[1] Wu, C., Che, M., Xu, R., Ran, Z., & Yan, H. (2025). ELFATT: Efficient linear fast attention for vision transformers. *arXiv preprint arXiv:2501.06098*.\n\n[2] Shen, Z., Zhang, M., Zhao, H., Yi, S., & Li, H. (2021). Efficient attention: Attention with linear complexities. In *Proceedings of the IEEE/CVF winter conference on applications of computer vision* (pp. 3531-3539).\n\n[3] Chen, J., Kao, S. H., He, H., Zhuo, W., Wen, S., Lee, C. H., & Chan, S. H. G. (2023). Run, don't walk: chasing higher FLOPS for faster neural networks. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition* (pp. 12021-12031).\n\n[4] Zhang, J., Xiang, C., Huang, H., Wei, J., Xi, H., Zhu, J., & Chen, J. (2025). Spargeattn: Accurate sparse attention accelerating any model inference. *arXiv preprint arXiv:2502.18137*.\n\n[5] Xu, R., Xiao, G., Huang, H., Guo, J., & Han, S. (2025). Xattention: Block sparse attention with antidiagonal scoring. *arXiv preprint arXiv:2503.16428*."}, {"review_id": "js5Aa5L7QA", "reviewer": "Reviewer_mVoD", "summary": "The authors introduce a new multi-stage sparse attention method to reduce the computational complexity compared to naïve full self attention, while keeping the approximation error to a minimum. The efficacy is validated across different vision and long-sequence experiments.", "questions": "**[Q1]:** The models used for DuSA are CSWin and Swin – both models that use convolutional down-sampling in their multi-stage architecture;   \n$\\textrightarrow$  I would be interested if the authors have tried to use their approach in a conventional ‘monolithic’ ViT? Since this keeps the resolution of the feature maps consistent throughout the architecture, I’d assume this provide even more opportunity to improve efficiency given that quadratic complexity of VSA?   \n$\\textrightarrow$  It could also provide ‘cleaner’ insights given the simple structure of the architecture.\n\n\n**[Q2]:** The models across all experiments seem to be around 50-60M parameters; Have the authors tried different scales of the architectures to validate that the attention works similarly across these, given that larger models can sometimes perform different internal processing steps than small models? How does it work for very small model? \n\n**[Q3]:** While I do appreciate thorough derivations of the proposed methods, I feel like the derivation of the upper bound somewhat stands ‘by itself’ in this paper. After the derivation, the theoretical results are not really used to further analyse or interpret the empirical findings; While there are some empirical findings placed in the appendix in terms of error, they don’t really back up the theoretical ones; Rather, the authors themselves comment that the bounds are probably not tight enough.   \n$\\textrightarrow$  Have the authors thought about better combining these theoretical results with some empirical validation regarding whether the lower approximation errors actually are reflected in the results on actual tasks?  \n$\\textrightarrow$  If not, I’d suggest shortening the bound derivation and using the space instead to provide further insights to the reader in terms of the method, e.g. by including some of the visualisations/analyses currently placed in the appendix – e.g. Fig A2, or the swapped execution of stages (Tab A4), or similar.\n\n\n**Further Comments / Corrections**:  \n- While Strategy 1 and 2 are introduced, it is often unclear which one has been used -- and why there are two in the first place; It might be worth providing further analysis in the main paper, including some visualisations that are currently in the appendix (Fig. A2)!\n- The authors use ‘hard-aware’ multiple times in their manuscript – I assume they mean “hardware-aware”?  \n- In Method 3.1 (VSA) l123: The statement “Since m >> c, …” is incorrect in two ways:   \n  1) The number of patches being much larger than the dimension is not a required condition for the attention-matrix to grow quadratically with m (it does this always, given the nature of its computation)\n  2) The statement itself is not necessarily true: Typical ViT-style VSA on ImageNet results in a sequence length of 196+1 tokens ((224/16) x (224/16)) +cls token; but the number of channels is typically of the same scale or larger (192 up to 1280)", "limitations": "Some limitations are discussed in the appendix of the paper, but these relate mainly to the tightness of the derived bound. I’d like to recommend the authors to think about some additional limitations inherent to their current approach in terms of applicability and/or other aspects that might be of interest to the reader/practitioner.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths \n**Originality & Significance:**  \n- Novel two-stage method that improves upon other sparse single-stage ones while being still competitive in terms of efficiency/throughput\n- Authors demonstrate the applicability via improvements across five different tasks\n\n**Quality:**  \n- Contributions are generally well-placed within the wider area of research\n- Theoretical derivations provided for the underlying concepts\n\n**Clarity:** \n- Contributions and underlying motivation clearly outlined in the intro \n- Explanation of key-concepts well-supported through a good mix of figures \n-\tThe paper is mostly well written and easy to read and follow\n\n## Weaknesses  \n- Clarity and insights provided to the reader could be improved -- see questions\n- Since the proposed DuSA aims to approximate/replace vanilla scaled dot-product attention, at least one ‘simpler’ architecture that exclusively uses this type of attention on a global scale (e.g. ViT) would provide valuable insights -- see questions\n- Somewhat limited experiments in terms of variation in model scale – see questions", "was_revised": true, "final_justification": "Taking the authors' rebuttal, my fellow reviewers' and my own concerns and questions into account, I think this paper is still a borderline case that leans slightly more towards 'accept'; although major restructuring of the paper should be performed by the authors (as recommended) in case this work gets accepted.", "rebuttal": "**We would like to thank the reviewer for the constructive comments provided to help us improve the quality of the paper. We have addressed the issues raised in the comments point-by-point as follows:**\n\n---\n\n**Response to Q1:** We have applied DuSA2 (DuSA using Strategy 2) to the simple vanilla ViT architectures (ViT-T/16 and ViT-L/16). **Table R2-1** shows the performance comparison of vanilla ViT architectures and the versions using DuSA2 to replace VSA. The sequence length $m$ of vanilla ViT architecture is relatively short; for the resolution of $224^2$, its maximum sequence length is only $196+1$, and its four levels have sequence lengths: $197$-$197$-$197$-$197$ with a constant embedding dimension $c=64$ for each head. Even for the resolution of $384^2$, its maximum sequence length is only $576+1$, and its four levels have sequence lengths: $577$-$577$-$577$-$577$ with a constant embedding dimension $c=64$. The maximum sequence length of Swin and CSWin is $m=3136$/$9216$ for the resolution of $224^2$/$384^2$ and is significantly larger than $197$/$577$. Their four levels have sequence lengths: $3136$-$784$-$196$-$49$/$9216$-$2304$-$576$-$144$ for the resolution of $224^2$/$384^2$ with a constant embedding dimension $c=32$ for each head. As the ablation study of **Table A2 of the original manuscript** shows, if the sequence length $m$ is not significantly larger than the head embedding dimension $c$, the acceleration effect will be limited, which cannot completely show the superiority of our method for acceleration. Although the acceleration effect of DuSA on the ViT series is limited due to the short sequence length, DuSA achieves a much better acceleration effect than FlashAttention-2 (FA2) on other monolithic transformers, such as CogvideoX-2B, which uses 30 attention blocks with a constant sequence length $m=17776$ and head embedding dimension $c=64$.\n\n## Table R2-1. The performance comparison of vanilla ViT architectures and the versions using DuSA2 to replace VSA. All models are trained from scratch on ImageNet-1K. Block size $b$ is 14 for the tiny model and 24 for the large model. Padding is used to make the sequence length divisible by the block size.\n|Backbone|Resolution|Top1 Acc|Param|GFLOPS|FPS (imgs/s)|Inference Memory (GB)|\n|-|-|-|-|-|-|-|\n|ViT-DuSA2-L/16|$384^2$|**77.2**|307M|**174.54**|230|3.0|\n|ViT-L/16|$384^2$|76.5|307M|191.21/174.85 (FA2)|179/**272 (FA2)**|3.7/**2.9 (FA2)**|\n|ViT-DuSA2-T/16|$224^2$|**72.9**|5.8M|**1.07**|10718|**2.0**|\n|ViT-T/16|$224^2$|72.6|5.8M|1.26/1.08 (FA2)|10644/**15350 (FA2)**|2.2/**2.0 (FA2)**|\n\n---\n\n**Response to Q2:** We have applied DuSA2 for small models like ViT-T/16, which only has 5.8M parameters, and large models like ViT-L/16, which has 307M parameters. Details are available in the **Response to Q1**. For larger models, please see the acceleration of using DuSA2 on CogvideoX-2B, as **Table 5 of the original manuscript** shows, and using DuSA2 on the acceleration of the prefilling stage of the Llama-3.1-8B-Instruct model, as **Tables R2-2** and **R2-3** show.\n\n## Table R2-2. The performance comparison of different attention methods on LongBench using the Llama-3.1-8B-Instruct model. Note: Higher scores mean better performance.\n|Method|Full (FlashInfer’s implementation of FlashAttention)|XAttention (stride=8) antidiagonal patterns|DuSA2 (stride=8) antidiagonal patterns|DuSA2 (stride=8) diagonal patterns|DuSA2 (stride=16) antidiagonal patterns|\n|-|-|-|-|-|-|\n|NarrativeQA|31.44|30.48|30.81|31.34|30.21|\n|Qasper|25.07|26.04|24.28|25.96|**27.57**|\n|MultiFieldQA-en|**29.40**|29.28|28.88|28.78|29.00|\n|MultiFieldQA-zh|**61.68**|61.67|58.28|61.65|60.99|\n|HotpotQA|16.89|17.33|**18.09**|16.92|17.34|\n|2WikiMultihopQA|17.00|16.34|16.03|16.35|**17.02**|\n|MuSiQue|11.79|11.88|10.21|11.86|**12.04**|\n|DuReader|34.93|34.64|35.40|35.60|**36.13**|\n|GovReport|34.22|34.60|**34.68**|34.54|34.59|\n|QMSum|23.25|23.24|23.21|**23.43**|23.18|\n|MultiNews|26.69|27.08|26.92|**27.17**|26.96|\n|VCSUM|15.91|16.11|15.77|**16.21**|15.73|\n|TREC|72.50|71.50|72.50|**73.00**|71.50|\n|TriviaQA|91.65|90.97|**92.15**|90.62|90.90|\n|SAMSum|43.74|**44.13**|43.27|43.59|43.67|\n|LSHT|46.00|46.50|**47.00**|46.50|46.00|\n|PassageCount|5.95|5.23|**7.25**|6.38|6.39|\n|PassageRetrieval-en|**98.20**|88.68|96.84|89.26|97.76|\n|PassageRetrieval-zh|**77.11**|74.40|76.67|75.06|73.76|\n|LCC|52.19|53.23|52.28|53.24|**53.36**|\n|RepoBench-P|49.14|**50.94**|48.97|50.16|49.56|\n|Average|**41.18**|40.68|40.92|40.84|41.13|\n\n## Table R2-3. Speedup relative to FlashAttention (FlashInfer’s implementation) obtained by different attention mechanisms across different sequence lengths using Llama-3.1-8B-Instruct on a single H20 (96GB) GPU.\n|Sequence Length (K)|DuSA2 (stride=8) antidiagonal patterns Speedup|DuSA2 (stride=16) antidiagonal patterns Speedup|XAttention (stride=8) Speedup|\n|-|-|-|-|\n|4|1.02|1.12|**1.93**|\n|8|1.59|2.36|**2.83**|\n|16|2.25|**4.67**|4.05|\n|32|2.79|**6.63**|5.06|\n|64|3.36|**9.11**|6.00|\n|128|3.09|**11.20** |5.57|\n\n---\n\n**Response to Q3:** We will revise the final version of the paper according to reviewer’s suggestion to shorten the bound derivation and use the space instead to provide further insights to the reader in terms of the method, e.g. by including some of the visualizations/analyses currently placed in the appendix – e.g. **Fig. A2**, or the swapped execution of stages (**Table A4**), or similar.\n\n---\n\n**Response to Further Comment 1:** We will clarify the use of Strategies 1 and 2, move more visualizations from the appendix to the main body, and shorten the bound derivation.\n\n---\n\n**Response to Further Comment 2:** We will revise all “hard-aware” to “hardware-aware”.\n\n---\n\n**Response to Further Comment 3:** Indeed, as the reviewer pointed out, the nature of the scaled dot product attention calculation leads to a complexity of $O(m^2\\times c)$. We will make the statement more rigorous. For vanilla ViT, its sequence length is $m=(196+1)$/$(576+1)$ for a resolution of $224^2$/$384^2$, and its embedding dimension is $c=64$ for each head for both $224^2$ and $384^2$ resolutions. Hence, it still satisfies $m > c$. And in the current stage, a long sequence length $m$ is preferred. Take Swin and CSWin for example, their embedding dimension is $c=32$ for each head for all four levels in both $224^2$ and $384^2$ resolutions. Their four levels have sequence lengths: $3136$-$784$-$196$-$49$ (for the input resolution of $224^2$)/$9216$-$2304$-$576$-$144$ (for the input resolution of $384^2$). CogvideoX-2B uses a sequence length $m=17776$ with $30$ heads. Each head has an embedding dimension $c=64$. Llama-3.1-8B-Instruct has $32$ heads. Each head has an embedding dimension $c=128$. Llama-3.1-8B-Instruct supports a maximum sequence length of 128K. $m$ is also significantly larger than $c$. Our method is designed for the acceleration for the case $m \\gg c$. The larger $m$ is than $c$, the more obvious the acceleration effect.\n\n**Response to Limitations:** We will add some additional limitations (for example, the acceleration effect of DuSA may be limited when $m$ is not significantly larger than $c$) inherent to our DuSA in terms of applicability or other aspects that might be of interest to the reader or practitioner.\n\n**Response to Paper Formatting Concerns:** We will revise the checklist and provide proper justifications as well as references to the sections where these questions are answered."}, {"review_id": "MvfwGBon4K", "reviewer": "Reviewer_iq85", "summary": "This paper proposes DuSA, which is a Dual-Stage Sparse Attention (DuSA) mechanism to accelerate the attention part of the vision transformers. A theoretical upper bound using DuSA to approximate VSA is derived.  Experiments demonstrated that the proposed DuSA has a lower approximation error than the single-stage design, despite the higher theoretical upper bound. DuSA can be used in both training and inference acceleration.", "questions": "1. Could the authors provide more explanation on larger theoretical upper bound but experimentally better results?\n2. Any insights of the proposed techniques on general transformer architectures? Or is there any thing unique making the proposed techniques especially effective for Vits?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Theoretical derivation and analysis of the proposed attention mechanism \n2. The plug and play nature of the technique at both training and inference \n3. Overall good results\n\n\nWeakness:\n\n1. More insights on the experimentally shown good results given the slightly weaker theoretical analysis would be better\n2. More analysis on memory consumption patterns during training would be more insightful\n3. The plug-and-play's accuracy drops might be a little too large", "was_revised": true, "final_justification": "1. The performance is good; the additional stats by the authors further verify the savings memory usage\n2. Still, all the tricks being put together seems to be a little too empirical and lacks enough analysis", "rebuttal": "**We would like to thank the reviewer for the constructive comments provided to help us improve the quality of the paper. We have addressed the issues raised in the comments point-by-point as follows:**\n\n---\n\n**Response to W1 and Q1:** The theoretical upper bound is used to provide the worst-case guarantee. A larger bound represents a larger confidence interval, and it may be larger than its real approximation error. Hence, as we mentioned in the limitation, a tighter bound is needed that can more accurately represent the real approximation error.\n\n---\n\n**Response to W2:** **Tables R3-1** and **R3-2** show the memory comparison results. Even compared to SOTAs (ELFATT and FlashAttention-2 (FA2)), DuSA2 (DuSA using Strategy 2) still has the lowest memory usage, and its memory usage is significantly lower than that of VSA.\n\n## Table R3-1. Peak memory usage comparison on ImageNet-1K using a single H20 (96GB) GPU. The batch size used is 32 for both inference and training. OOM denotes out of memory.\n|Methods|Resolution|Inference Memory (GB)|Training Memory (GB)|\n|-|-|-|-|\n|CSWin-B-DuSA2|$384^2$|**2.3**|21.0|\n|CSWin-B-ELFATT|$384^2$|2.5|18.8|\n|CSWin-B-VSA|$384^2$|62.3/2.4 (FA2)|OOM/**17.9 (FA2)**|\n\n## Table R3-2. Peak memory usage comparison on ADE20K and COCO using a single H20 (96GB) GPU. The batch size used is 1 for inference and 2 for training. OOM denotes out of memory.\n|Methods|ADE20K (Inference Memory GB)|COCO (Inference Memory GB)|ADE20K (Training Memory GB)|COCO (Training Memory GB)|\n|-|-|-|-|-|\n|Swin-T-DuSA2|**2.5**|**1.4**|**5.5**|**8.1**|\n|Swin-T- ELFATT|2.8|1.6|5.6|8.2|\n|Swin-T-VSA|49.2/2.8 (FA2)|71.9/1.5 (FA2)|20.1/**5.5 (FA2)**|OOM/8.4 (FA2)|\n|CSWin-T-DuSA2|**2.7**|**1.4**|**5.4**|**8.7**|\n|CSWin-T-ELFATT|**2.7**|**1.4**|5.5|8.8|\n|CSWin-T-VSA|33.2/**2.7 (FA2)**|48.2/**1.4 (FA2)**|16.7/**5.4 (FA2)**|OOM/8.8 (FA2)|\n\n---\n\n**Response to W3:** The current version of DuSA is a static sparse attention mechanism, which cannot preserve the accuracy of VSA without loss in a plug-and-play application. It is designed for accelerating both the training and inference processes. After training, it can match or even outperform VSA. And we have provided a dynamic sparse attention version, which is training-free and maintains higher performance. The dynamic version of DuSA can match the performance of VSA in a plug-and-play application and provides a significant acceleration effect. We introduce DuSA2 (Strategy 2: Rolling indices-based attention scores fusion) to search antidiagonal or diagonal patterns of attention matrices to filter important or unimportant blocks for further dynamic sparse attention only performed within important blocks. As the following **Table R3-3** shows, compared to XAttention [1], the performance of DuSA2 is closer to that of full attention. As **Tables R3-3** and **R3-4** show, DuSA2 can use a larger stride (larger sparsity) to achieve similar or higher performance than XAttention, further accelerating full attention. By introducing antidiagonal patterns of XAttention, the performance of DuSA2 can be further improved. \n\n## Table R3-3. The performance comparison of different attention mechanisms on LongBench using the Llama-3.1-8B-Instruct model. Note: Higher scores mean better performance.\n|Method|Full (FlashInfer’s implementation of FlashAttention)|XAttention (stride=8) antidiagonal patterns|DuSA2 (stride=8) antidiagonal patterns|DuSA2 (stride=8) diagonal patterns|DuSA2 (stride=16) antidiagonal patterns|\n|-|-|-|-|-|-|\n|NarrativeQA|31.44|30.48|30.81|31.34|30.21|\n|Qasper|25.07|26.04|24.28|25.96|**27.57**|\n|MultiFieldQA-en|**29.40**|29.28|28.88|28.78|29.00|\n|MultiFieldQA-zh|**61.68**|61.67|58.28|61.65|60.99|\n|HotpotQA|16.89|17.33|**18.09**|16.92|17.34|\n|2WikiMultihopQA|17.00|16.34|16.03|16.35|**17.02**|\n|MuSiQue|11.79|11.88|10.21|11.86|**12.04**|\n|DuReader|34.93|34.64|35.40|35.60|**36.13**|\n|GovReport|34.22|34.60|**34.68**|34.54|34.59|\n|QMSum|23.25|23.24|23.21|**23.43**|23.18|\n|MultiNews|26.69|27.08|26.92|**27.17**|26.96|\n|VCSUM|15.91|16.11|15.77|**16.21**|15.73|\n|TREC|72.50|71.50|72.50|**73.00**|71.50|\n|TriviaQA|91.65|90.97|**92.15**|90.62|90.90|\n|SAMSum|43.74|**44.13**|43.27|43.59|43.67|\n|LSHT|46.00|46.50|**47.00**|46.50|46.00|\n|PassageCount|5.95|5.23|**7.25**|6.38|6.39|\n|PassageRetrieval-en|**98.20**|88.68|96.84|89.26|97.76|\n|PassageRetrieval-zh|**77.11**|74.40|76.67|75.06|73.76|\n|LCC|52.19|53.23|52.28|53.24|**53.36**|\n|RepoBench-P|49.14|**50.94**|48.97|50.16|49.56|\n|Average|**41.18**|40.68|40.92|40.84|41.13|\n\n## Table R3-4. Speedup relative to FlashAttention (FlashInfer’s implementation) obtained by different attention mechanisms across different sequence lengths using Llama-3.1-8B-Instruct on a single H20 (96GB) GPU.\n|Sequence Length (K)|DuSA2 (stride=8) antidiagonal patterns Speedup|DuSA2 (stride=16) antidiagonal patterns Speedup|XAttention antidiagonal patterns (stride=8) Speedup|\n|-|-|-|-|\n|4|1.02|1.12|**1.93**|\n|8|1.59|2.36|**2.83**|\n|16|2.25|**4.67**|4.05|\n|32|2.79|**6.63**|5.06|\n|64|3.36|**9.11**|6.00|\n|128|3.09|**11.20** |5.57|\n\n---\n\n**Response to Q2:** We have applied DuSA2 (DuSA using Strategy 2) to the simple vanilla ViT architectures (ViT-T/16 and ViT-L/16). **Table R3-5** shows the performance comparison of vanilla ViT architectures and the versions using DuSA2 to replace VSA. The maximum sequence length of the vanilla ViT architecture is relatively short. For the resolution $224^2$, its maximum sequence length is only $196+1$, and its four levels have sequence lengths: $197$-$197$-$197$-$197$ with a constant embedding dimension $c=64$ for each head. Even for the resolution $384^2$, its maximum sequence length is only $576+1$, and its four levels have sequence lengths: $577$-$577$-$577$-$577$ with a constant embedding dimension $c=64$ for each head. While the maximum sequence length of Swin and CSWin is $3136$/$9216$ for the resolution of $224^2$/$384^2$, and their four levels have sequence lengths: $3136$-$784$-$196$-$49$/$9216$-$2304$-$576$-$144$ for the resolution of $224^2$/$384^2$ with a constant embedding dimension $c=32$. As the ablation study of **Table A2 of the original manuscript** shows, if the sequence length $m$ is not significantly larger than the head embedding dimension $c$, the acceleration effect will be limited, which cannot completely show the superiority of our method for acceleration. Although the acceleration effect of DuSA on the ViT series is limited due to the short sequence length, DuSA achieves a much better acceleration effect than FlashAttention-2 (FA2) on other general transformers, such as CogvideoX-2B, which uses 30 attention blocks with a constant sequence length $m=17776$ and head embedding dimension $c=64$.\n\n## Table R3-5. The performance comparison of vanilla ViT architectures and the versions using DuSA2 to replace VSA. All models are trained from scratch on ImageNet-1K. Block size $b$ is 14 for the tiny model and 24 for the large model. Padding is used to make the sequence length divisible by the block size.\n|Backbone|Resolution|Top1 Acc|Param|GFLOPS|FPS (imgs/s)|Inference Memory (GB)|\n|-|-|-|-|-|-|-|\n|ViT-DuSA2-L/16|$384^2$|**77.2**|307M|**174.54**|230|3.0|\n|ViT-L/16|$384^2$|76.5|307M|191.21/174.85 (FA2)|179/**272 (FA2)**|3.7/**2.9 (FA2)**|\n|ViT-DuSA2-T/16|$224^2$|**72.9**|5.8M|**1.07**|10718|**2.0**|\n|ViT-T/16|$224^2$|72.6|5.8M|1.26/1.08 (FA2)|10644/**15350 (FA2)**|2.2/**2.0 (FA2)**|\n\n---\n\n**References:**\n\n[1] Xu, R., Xiao, G., Huang, H., Guo, J., & Han, S. (2025). Xattention: Block sparse attention with antidiagonal scoring. *arXiv preprint arXiv:2503.16428*."}, {"review_id": "UyKXpobrUS", "reviewer": "Reviewer_Ah1n", "summary": "DuSA introduces a novel Dual-Stage Sparse Attention method to accelerate both the training and inference of vision transformers. The main idea is to split the attention computation into two stages. Both stages implement sparse attention: the first stage is designed to capture local inductive biases within individual blocks, while the second stage aggregates cross-block information. Since both stages can work with the existing attention acceleration techniques, and sparse attention has low computational complexity, DuSA achieves a significant speedup over the vanilla scaled-dot product attention (VSA). Experiments on different benchmarks demonstrate the effectiveness of the proposed DuSA method.", "questions": "- There's been a lot of recent works on sparse attention for LLMs. Can DuSA be applied to LLMs — for both prefill and decoding?\n- DuSA is similar to a recent paper XAttention [1], they both use block sparsity and diagonal sparse patterns. From the results of XAttention, antidiagonal patterns outperform diagonal ones, which is leveraged in DuSA. Could you compare DuSA and XAttention in more detail?\n\n[1] Ruyi Xu, et al. XAttention: Block Sparse Attention with Antidiagonal Scoring. arXiv:2503.16428.", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- Strengths\n\t- The proposed method is compatible with existing efficient attention implementations, allowing it to be easily integrated into the existing vision transformer models to get better performance. The experimental results also show the real speedup, which makes the method solid and practical.\n- Weakness\n\t- This paper spends a considerable amount of space analyzing the upper bound of the approximation error, but this bound is not used to guide the design of sparse attention. Additionally, as the author noted, this upper bound is relatively rough and does not seem helpful for sparse attention analysis and design right now (though that could change later). Maybe it’d be better to move this section to the appendix.\n\t- Many discussions and ablation studies are in the appendix. It'd be better to have them in the \"Experiments and Results\" session to help readers understand.\n\t- In the abstract, the paper claims that DuSA maintains low performance loss in plug-and-play scenarios. But this topic is only discussed in Section A2, with evaluations limited to ImageNet. The results show a significant drop in accuracy.", "was_revised": false, "final_justification": "", "rebuttal": "**We would like to thank the reviewer for the constructive comments provided to help us improve the quality of the paper. We have addressed the issues raised in the comments point-by-point as follows:**\n\n---\n\n**Response to W1 and W2:** We will move the section about upper bound analysis of the approximation error to the appendix and move more experimental discussions and ablation studies to the main body in the next version. \n\n---\n\n**Response to W3:** The current version of DuSA is a static sparse attention mechanism, which cannot preserve the accuracy of VSA without loss in a plug-and-play application. It is designed for accelerating both the training and inference processes. After training, it can match or even outperform VSA. Inspired by XAttention [1], which is an excellent training-free dynamic sparse attention SOTA for accelerating the prefilling stage of LLMs, we have provided a dynamic sparse attention version of DuSA. The dynamic version of DuSA can match the performance of VSA in plug-and-play applications and provides a significant acceleration effect. Details are available in **Response to Q1**.\n\n---\n\n**Response to Q1:** The answer is yes. DuSA can be applied to accelerate the prefilling stage. Based on XAttention [1], we introduce DuSA2 (Strategy 2: Rolling indices-based attention scores fusion) to search antidiagonal/diagonal patterns of attention matrices to filter important or unimportant blocks for further dynamic sparse attention performed within important blocks only. As the following **Table R4-1** shows, compared to XAttention, the performance of DuSA2 is closer to that of full attention. As **Tables R4-1** and **R4-2** show, DuSA2 can use a larger stride (larger sparsity) to achieve similar or higher performance than XAttention, further accelerating full attention. By introducing antidiagonal patterns of XAttention, the performance of DuSA2 can be further improved. \n\n## Table R4-1. The performance comparison of different attention mechanisms on LongBench using the Llama-3.1-8B-Instruct model. Note: Higher scores mean better performance.\n|Method|Full (FlashInfer’s implementation of FlashAttention)|XAttention (stride=8) antidiagonal patterns|DuSA2 (stride=8) antidiagonal patterns|DuSA2 (stride=8) diagonal patterns|DuSA2 (stride=16) antidiagonal patterns|\n|-|-|-|-|-|-|\n|NarrativeQA|31.44|30.48|30.81|31.34|30.21|\n|Qasper|25.07|26.04|24.28|25.96|**27.57**|\n|MultiFieldQA-en|**29.40**|29.28|28.88|28.78|29.00|\n|MultiFieldQA-zh|**61.68**|61.67|58.28|61.65|60.99|\n|HotpotQA|16.89|17.33|**18.09**|16.92|17.34|\n|2WikiMultihopQA|17.00|16.34|16.03|16.35|**17.02**|\n|MuSiQue|11.79|11.88|10.21|11.86|**12.04**|\n|DuReader|34.93|34.64|35.40|35.60|**36.13**|\n|GovReport|34.22|34.60|**34.68**|34.54|34.59|\n|QMSum|23.25|23.24|23.21|**23.43**|23.18|\n|MultiNews|26.69|27.08|26.92|**27.17**|26.96|\n|VCSUM|15.91|16.11|15.77|**16.21**|15.73|\n|TREC|72.50|71.50|72.50|**73.00**|71.50|\n|TriviaQA|91.65|90.97|**92.15**|90.62|90.90|\n|SAMSum|43.74|**44.13**|43.27|43.59|43.67|\n|LSHT|46.00|46.50|**47.00**|46.50|46.00|\n|PassageCount|5.95|5.23|**7.25**|6.38|6.39|\n|PassageRetrieval-en|**98.20**|88.68|96.84|89.26|97.76|\n|PassageRetrieval-zh|**77.11**|74.40|76.67|75.06|73.76|\n|LCC|52.19|53.23|52.28|53.24|**53.36**|\n|RepoBench-P|49.14|**50.94**|48.97|50.16|49.56|\n|Average|**41.18**|40.68|40.92|40.84|41.13|\n\n## Table R4-2. Speedup relative to FlashAttention (FlashInfer’s implementation) obtained by different attention mechanisms across different sequence lengths using Llama-3.1-8B-Instruct on a single H20 (96GB) GPU.\n|Sequence Length (K)|DuSA2 (stride=8) Speedup|DuSA2 (stride=16) Speedup|XAttention (stride=8) Speedup|\n|-|-|-|-|\n|4|1.02|1.12|**1.93**|\n|8|1.59|2.36|**2.83**|\n|16|2.25|**4.67**|4.05|\n|32|2.79|**6.63**|5.06|\n|64|3.36|**9.11**|6.00|\n|128|3.09|**11.20** |5.57|\n\n---\n\n**Response to Q2:** XAttention uses the sum of antidiagonal/diagonal values of the attention matrix as the metric for ranking block importance. While in stage 2, DuSA uses the diagonal values of the attention matrix for block-level mixture to introduce long-range dependencies from other diagonal blocks from stage 1. We take the version of XAttention that uses diagonal patterns for comparison.  For the input $\\textbf{Q}$ and $\\textbf{K}$ with a size of $(B, H, M, C)$, where $B$ is the batch size, $H$ is the number of heads, $M$ is the sequence length, and $C$ is the embedding dimension of each head, the version of XAttention using diagonal patterns reshapes the $\\textbf{Q}$ and $\\textbf{K}$ to a shape like $(B, H, M//stride, stride\\times C)$ and performs attention computation using softmax on the final two dimensions to obtain diagonal values of the attention matrices. While DuSA1 (DuSA using Strategy 1) reshapes the $\\textbf{Q}$ and $\\textbf{K}$ to a shape of $(B, H, M//stride, stride, C)$ and transposes the third and fourth dimensions to a shape like $(B, H, stride, M//stride, C)$. DuSA1 also performs attention computation using softmax on the final two dimensions to obtain diagonal values of the attention matrices (*For satisfying the input requirements of some memory-efficient methods, $(B, H, stride, M//stride, C)$ may be further reshaped to $(B, H\\times stride, M//stride, C)$. The softmax operation is still performed on the final two dimensions, hence they give the same results.*). The difference between DuSA1 and XAttention is that DuSA1 introduces a multi-stride computation to obtain diagonal values of the attention matrices. And DuSA2 (DuSA using Strategy 2) further sums the stride dimension to obtain $\\textbf{A}_2$ as **Figure 3 of the original manuscript** shows.\n\n---\n\n**References:**\n\n[1] Xu, R., Xiao, G., Huang, H., Guo, J., & Han, S. (2025). Xattention: Block sparse attention with antidiagonal scoring. *arXiv preprint arXiv:2503.16428*."}]

Author rebuttal:

Meta-review: This paper proposes DuSA, a dual-stage sparse attention mechanism that is empirically strong across tasks and shows practical speedups while remaining compatible with memory-efficient implementations. Reviewers highlight clear motivation and derivations and broad cross-task validation, with the method improving upon single-stage sparse baselines in accuracy/throughput trade-offs. The reviewers find the contribution is novel and practically relevant, with strong evidence across diverse benchmarks; remaining issues are editorial/clarificatory rather than technical. Thus, the AC decides to accept the submission.
Decision: Accept (poster)